“Dr. Watson, Mr. Devi Priya”, said Stamford, introducing us. can be tokenized as in the following example, where each token is enclosed in single quotation marks: `"' `Dr.' `Watson' `,' `Mr.’ `Devi’ `Priya' `"' `,’ `said' `Stamford' `,' `introducing' `us' `.’ Purpose The primary goal is to segment text into meaningful units, which are often individual words, to facilitate further analysis such as text classification, sentiment analysis, machine translation, or information retrieval. Process It typically involves identifying word boundaries within a text, often based on whitespace, punctuation, or other defined delimiters.