Tokenization

Definition(s)

An operation that examines a document or block of text and breaks the text into words. Typically, a space is used to separate words, but special characters such as a hyphen, period, or quotation mark can also be used.

Print Friendly, PDF & Email