Sub-word Tokenization Approach Frequently used words should not be split into smaller subwords Rare words should be decomposed into meaningful subwords Uses a special symbol to indicate which word is the start of the token and which word is the completion of the start of the token Tokenization → “Token”, “##ization” State-of-the-art approaches for NLP and IR rely on this type Advantages Out-of-vocabulary word problem solved