A Token Is a Piece of Split Text
A token is the smallest unit an LLM uses to read and write text. The model does not handle characters directly; it first converts text into a sequence of tokens and then processes them. Frequently used words fit into a single token, while long or unfamiliar words are split into several fragments.
How many characters make up one token depends on the language and the model. OpenAI and Google each give a rule of thumb of about 4 characters per token for English. According to Anthropic's documentation as of October 2026, models that use its newer tokenizer split the same text into about 30 percent more tokens than earlier models.
Estimating Japanese Text
Even when the content is simply translated, the number of tokens varies widely by language. A 2023 study comparing many tokenizers found differences of up to 15 times between languages. Applying the English rule of "4 characters per token" directly to Japanese text can throw an estimate far off.
The price notes on this site assume about 1 to 1.5 tokens per Japanese character, and treat one million tokens as the text of seven to ten Japanese paperback books. Both are rough figures, so check exact counts with the token counting tools that developers provide.
Prices and Length Are Set in Tokens
The token is the shared unit for both price and length. API costs are set by multiplying the number of tokens you have the model read (input) and write (output) by the price per million tokens for each. How much a model can read at once is also stated in tokens, so whether you can pass in a long document is judged by its token count too. Text that turns into more tokens for the same content costs more and leaves room for less at once.