Byte Pair Encoding (BPE) token learner Let vocabulary be the set of all individual characters = {A, B, C, D,…, a, b, c, d….} Repeat: Choose the two symbols that are most frequently adjacent in the training corpus (say 'A', 'B') Add a new merged symbol 'AB' to the vocabulary Replace every adjacent 'A' 'B' in the corpus with 'AB'. Until k merges have been done.