Lynkco — Natural Language Processing Introduction

Sentence Segmentation !, ? mostly unambiguous but period “.” is very ambiguous Sentence boundary Abbreviations like Inc. or Dr. Numbers like .02% or 4.3 Common algorithm: Tokenize first: use rules or ML to classify a period as either (a) part of the word or (b) a sentence-boundary. An abbreviation dictionary can help Sentence segmentation can then often be done by rules based on this tokenization.