BPE token segmenter algorithm On the test data, run each merge learned from the training data: Greedily In the order we learned them (test frequencies don't play a role) So: merge every e r to er , then merge er _ to er_ , etc. Result: Test set "n e w e r _" would be tokenized as a full word Test set "l o w e r _" would be two tokens: "low er_"