Simple Tokenization in UNIX (Inspired by Ken Church’s UNIX for Poets.) Given a text file, output the word tokens and their frequencies tr -sc ’A-Za-z’ ’\n’ Change all non-alpha to newlines | sort | uniq Sort in alphabetical order Merge and count each type 1945 A 72 AARON 19 ABBESS 25 Aaron 5 ABBOT ... ... 6 Abate 1 Abates 5 Abbess 6 Abbey 3 Abbot .... …