Subword tokenization Three common algorithms: Byte-Pair Encoding (BPE) (Sennrich et al., 2016) Unigram language modeling tokenization (Kudo, 2018) WordPiece (Schuster and Nakajima, 2012) All have 2 parts: A token learner that takes a raw training corpus and induces a vocabulary (a set of tokens). A token segmenter that takes a raw test sentence and tokenizes it according to that vocabulary