3.9k views
2 votes
What is the distinction between tokens and types. Give an example to illustrate the difference.

1 Answer

3 votes

Final answer:

Tokens are individual units of text whereas types refer to unique categories of tokens. For example, in the sentence 'The cat chased the dog,' there are five tokens and four types.

Step-by-step explanation:

Tokens and types are two concepts related to linguistics and text processing. In natural language processing, a token is an individual unit of text such as a word or a character. On the other hand, a type refers to a unique category of tokens that share similar attributes. For example, in the sentence 'The cat chased the dog,' the tokens are 'the,' 'cat,' 'chased,' 'the,' and 'dog' while the types are 'the,' 'cat,' 'chased,' and 'dog.' So, there are five tokens and four types in this sentence.

User Martze
by
7.2k points