Word-based Tokenization Approach Splitting the text by spaces Other delimiters such as punctuation can be used Advantages Easy to implement Disadvantages High risk of missing words; e.g., Let and Let’s will have two different types Languages like Chinese do not have space Huge vocabulary size (token type) ○Limit the number of words that can be added to the vocabulary Mi ll d d ill b id d t k