Lynkco — Lynkco NLP Introduction Course Manual (EN)

Space-based tokenization A very simple way to tokenize For languages that use space characters between words Arabic, Cyrillic, Greek, Latin, etc., based writing systems Segment off a token between instances of spaces Unix tools for space-based tokenization The "tr" command Inspired by Ken Church's UNIX for Poets Given a text file, output the word tokens and their frequencies