Byte-Pair Encoding (BPE) Tokenization Uses Huffman encoding for tokenization (greedy algorithm) Training Steps: 1. Starts with splitting the input words into single characters (each of them corresponds to a symbol in the final vocabulary) * In practice we commonly add special end of word symbol “__” before space 2. Find the most frequent occurring pair of symbols from the current vocabulary 3. Add this to the vocabulary and size of vocabulary increases by one 4. Repeat steps (2) and (3) till the defined number of tokens are built or no new combination of symbols exist with required frequency