Issues in Tokenization Can't just blindly remove punctuation: m.p.h., Ph.D., AT&T, cap’n prices ($45.55 ) dates ( 01/02/06 ) URLs ( http://www.stanford.edu ) hashtags ( #nlproc ) email addresses ( someone@cs.colorado.edu ) Clitic: a word that doesn't stand on its own "are" in we're, French " je " in j'ai, " le " in l'honneur When should multiword expressions (MWE) be words? New York, rock ’n’ roll