Tokenization is the task of chopping text into pieces, called tokens Token is an instance of a sequence of characters in some particular document (grouped as a useful semantic unit) Type is the class of all tokens containing the same character sequence Term is a (perhaps normalized) type that is included in the corpus dictionary Example: to sleep more to learn Token: to, sleep, more, to, learn Type: to, sleep, more, learn Term: sleep, more, learn (stop words removed) In the sentence, “In Kolkata I took my hat off. But I can’t put it back on.”, total number of word tokens and word types are: 1. 14, 13 2. 13, 14 3. 15, 14 4. 14, 15 Answer: 1. 14, 13. Solution:
Here, the word “I” is repeated two times so type count is one less than token count.