Home  >   FAQs   >  What are tokens in Chat GPT?
FAQs in this section

What are tokens in Chat GPT?

Table of Contents
Tokens are chunks of text into which a given input or output is divided. Chat GPT uses Natural Language Processing and Machine Learning languages, where a token can be as short as one character or as long as one word.

For example, ChatGPT is Great! can be tokenized into the following tokens:
  1. Chat
  2. G
  3. PT
  4. is
  5. Great
  6. !
Each token corresponds to a meaningful unit in the text. Spaces are usually tokens themselves, and punctuation marks can be separate tokens or part of adjacent words.
Was this FAQ helpful?
0  out of  0  found this helpful.
Comments 0
Be the first to comment
© Copyright 2023 Vtiger. All rights reserved.