Tokens, the Context Window and Why AI Forgets
You are forty messages into a chat, the AI has been brilliant, and then it suddenly ignores a rule you gave it at the start. Or you paste a long report and it answers as if it had read only half. People often decide the AI is being lazy or "getting tired". It is neither. It is running into two ordinary limits that every chat AI has: it reads in small pieces called tokens, and it can only hold a limited amount of text in view at once, which is called the context window. This tutorial explains both in plain words, shows why they make an AI seem to forget, and gives you a handful of habits that fix most of the problems. You will also run two small experiments so you can see the effect for yourself.
Difficulty: ★☆☆☆☆ (no technical background needed)
Tools needed: Any chat AI you already use (ChatGPT, Claude, Gemini or similar)
Last updated: November 2026
Overview
Imagine a very fast reader who works at a desk. Everything you say to them, and everything they say back, is written on sheets of paper that they lay on the desk. They can only answer using what is on the desk. The desk is large, much larger than it used to be, but it is not infinite. When it fills up, something has to go. And when you start a new conversation, they clear the desk entirely.
That picture is close to how a chat AI works. The sheets of paper are made of tokens, the desk is the context window, and "forgetting" is what happens when something is off the desk, or buried so deep in a heap that it hardly gets read.
None of this is secret, and none of it requires maths. But most people never hear it, so they meet it as a series of small mysteries. Why did it contradict itself? Why does it follow my instructions for ten minutes and then drift? Why does it lose a detail I gave it at the very start? Why is the same task fine in a fresh chat and worse in a long one?
This tutorial answers those questions in seven steps. First you learn what a token is, and look at one. Then how a model uses tokens to write an answer. Then the context window: what counts towards it, and what happens when it is full. Then the three real reasons an AI seems to forget. Then how saved "memory" features relate to all this. Then a set of habits that work in any tool. And finally what tokens mean for cost, speed and usage limits, without quoting a single price.
You will finish with a clearer mental model, a short handoff prompt you can reuse, and a feel for when to start a fresh chat.
Who This Is Useful For
What You Will Learn
What You Need
1. Any chat AI you can open in a browser or an app.
2. A long piece of text to experiment with, such as an article or a report you have permission to paste. A few pages is enough.
3. About thirty minutes, including the two experiments.
Paste any text and see how it is cut into tokens. It is one provider's tokenizer, so other tools will count slightly differently, but the idea is the same.
Information as of 14 November 2026. Context windows have grown a great deal in recent years and keep changing, and every tool and plan is different. This tutorial deliberately gives no window sizes, prices or limits. Check the official page of the tool you use for its current numbers, and use the principles here, which hold across tools.
Step 1 — What a token is
An AI does not read words the way you do. Before a model sees your text, a program chops it into small pieces called tokens. A token can be a whole short word, part of a longer word, a punctuation mark or a space.
A few examples show the idea. The word "cat" is probably one token. A longer word such as "unbelievable" might be cut into pieces such as "un", "believ" and "able". A number like "2026" might be one token or several. A rare name, or a word in a less common language, is often cut into more pieces.
Two things follow.
1. Tokens are the unit of everything. How much the model can hold, how much it can write, how fast it answers and how much usage you consume are all counted in tokens, not words.
2. The conversion is only a rule of thumb. In English, a commonly quoted rough figure is that a token is about three-quarters of a word, so a page of text is a few hundred tokens. Other languages can need more tokens for the same meaning, and the exact split depends on the tool. Chinese text, for instance, often takes more tokens per character than English takes per letter, and the count differs between tools.
Try it now. Open the tokenizer page, or any similar one, and paste a sentence. Look at how it is divided. Then paste the same idea in another language, or a long technical word, and see the difference. Two minutes spent here makes the rest of this tutorial much easier to picture.
Step 2 — How a model uses tokens to answer
The model does one thing, over and over. Given the tokens so far, it predicts which token is most likely to come next. It picks one, adds it to the text, and repeats, token by token, until the answer is finished. An answer of three paragraphs is hundreds of these small steps in a row.
Two consequences matter here.
It only works from what is in front of it. The prediction for the next token depends on all the earlier tokens the model can see: your instructions, your messages, its own earlier replies and anything you pasted. If something is not among those tokens, it cannot influence the answer, except through what the model learned during training.
It does not look things up, and it does not keep a diary. When you chat, the model is not searching a database of your past conversations, and it is not storing what you said in a place it consults later. Each new reply is produced by reading the whole conversation so far, from the beginning, again. The "memory" of a chat is nothing more than the text of the chat being fed back in.
This is a surprising fact for many people, so it is worth stating plainly. The AI does not remember your conversation. Your conversation is re-read every time. Remembering and re-reading feel identical from the outside, which is why the difference only shows up when re-reading goes wrong.
Step 3 — The context window
The context window is the maximum amount of text, counted in tokens, that the model can take in and work with at one time. Everything has to fit inside it together.
Many people think the window holds only what you type. It holds more.
| What fills the window | Examples |
|---|---|
| Instructions you or the app set | System instructions, custom instructions, a Project's saved instructions |
| Everything you have said | Every message in this chat so far |
| Everything the AI has said | Every reply in this chat so far, including long ones |
| Anything you attached or pasted | Documents, spreadsheets, web pages, images turned into tokens |
| Results from tools | Search results the AI fetched, files it opened |
| Room for the reply being written | The answer itself needs space too |
So a long chat is not just your messages. It is your messages, all of the AI's long replies, and every file, added up.
What happens when the window is full. Different tools handle it differently, and they do not always tell you. Some refuse and ask you to start a new chat. Some quietly drop the oldest messages. Some compress old parts into a summary and keep going, which saves space but loses detail. In all three cases, something you said earlier is no longer fully in view.
That is the first, simplest reason an AI forgets: it is literally no longer there.
Step 4 — The three reasons an AI seems to forget
"It forgot" can mean three different things, and the fix is different for each. When an AI misbehaves, ask which one you are facing.
Reason 1: It is a new chat. A fresh conversation starts with an empty desk. It does not contain your last chat unless a memory feature has put something back. If you said "as I told you yesterday", the AI has no yesterday. Fix: restate what matters, or start from a saved summary.
Reason 2: The window overflowed. In a very long chat, early material may have been dropped or compressed. Typical signs: it stops following a rule from the first message, it contradicts a decision from the early part, or it asks for something you already gave. Fix: restate the key rules, or start a new chat with a summary of where you got to.
Reason 3: It is in the window but poorly used. Even when everything fits, a model does not treat all parts of a long text equally. Research on long inputs has found that models can be less reliable at using information buried in the middle of a very long text than information near the start or the end. In practice, a rule given at the start of a very long chat may get less weight by message forty, and one detail in the middle of a pasted report can be missed while the opening and closing are used well. Fix: put the most important material at the beginning or the end, shorten what you paste, and ask for the specific detail directly.
There is also a fourth thing that looks like forgetting but is not: it never knew. A model's training has a cut-off date, so it may not know very recent events unless it can search. And it may simply not have learned a rare fact. If a detail was never in your chat and never in its training, no amount of "remembering" will produce it. That is closer to the next tutorial's topic, hallucination, where an AI fills the gap with something plausible.
Step 5 — Saved memory, Projects and how they relate
Many tools now offer ways to carry information between chats: saved memory, custom instructions, Projects or workspaces with attached files. They can look like magic, but they follow the same logic.
They put text on the desk for you. A saved preference such as "I prefer short answers" or a Project's instructions and files are not a separate brain. The tool takes that saved material and inserts it into the context at the start of a chat. It takes up room in the window like anything else, and what the model does with it is subject to the same limits.
Four practical points follow.
1. Saved material is not always used. It may be added in a shortened form, or only the parts the tool judges relevant. If something must be respected, also say it in the chat.
2. Wrong saved memory is persistent. A note that is out of date will be put on the desk every time, and the AI will treat it as true. Look at what is saved, and correct it.
3. Saved memory and privacy go together. Whatever you let a tool remember about you is stored by that tool. Check what it keeps and how to delete it, and do not store anything you would not want kept.
4. Features differ and change. Which tools have memory, how it works and how to switch it off differ by product, plan and date. Look in the settings of the tool you use, and treat any list as a snapshot.
A simple way to think about it: saved memory and Projects are a convenient way of pre-loading the desk. They do not make the desk bigger.
Step 6 — Habits that keep an AI on track
You cannot change how big the desk is, but you can use it well. These habits work in any tool.
1. One topic per chat. A chat that wanders through five subjects fills the window with material that is irrelevant to the sixth. When the topic changes, start a new chat.
2. Start fresh when it drifts. If the AI has begun to ignore rules, repeat itself or contradict earlier answers, do not struggle on. Ask for a summary, and start a new chat with it. This is the single most useful habit.
3. Use a handoff summary. Before you leave a long chat, ask for a structured summary you can paste into the next one.
Please write a handoff note for a new conversation. Include: (1) the goal we are working on, (2) decisions we have made and why, (3) rules and preferences I have set that must be kept, (4) what is finished, (5) what is still open, and (6) any facts or numbers I gave you that must not be changed. Keep it under 300 words.Read the summary and correct it before you paste it into the new chat. A summary is a compression, and a compression can drop or distort something.
4. Restate the rules that matter. In a long chat, repeat important constraints every so often, or when the task changes: "Reminder: keep it under 100 words, no jargon, and do not invent any numbers."
5. Put key material first or last. When you paste something long and your real question depends on a particular part, tell the AI which part, and place your question and the most important instruction at the end, close to where it will write.
6. Paste less. If you only need three pages of a fifty-page report, paste those three. A smaller, focused input is easier for the model to use well, and leaves more room for the answer.
7. Ask for specifics, then check. For a long document, ask a pointed question ("What does section 4 say about refunds? Quote the sentence.") rather than a vague one ("Summarise everything"). A quote can be checked against the original in seconds.
8. Break a big job into stages. Do the research in one chat, the outline in another and the draft in a third, passing a short summary between them. Each stage has a clean desk.
Step 7 — Tokens, cost, speed and usage
Because tokens are the unit of all the work, they are also how many tools measure usage, speed and cost. You do not need numbers to use this wisely.
If you ever use an AI through an API, token counts appear directly on the bill, and these same ideas apply with more force. For ordinary chat use, the rule of thumb is enough: keep chats focused, keep inputs trimmed and start fresh when it drifts.
Experiments to Run
Do these two. They take about fifteen minutes in total and make everything above concrete.
Experiment 1: the tokenizer. Paste three things into the tokenizer page: a short English sentence, the same idea in another language you know, and a long made-up word or a string of numbers. Write down how many tokens each took. Notice how the same meaning can cost a different number of tokens.
Experiment 2: the buried detail. Take a long text of a few pages. Add a made-up fact somewhere in the middle, for example "The meeting room code is 4417". Paste the text into a chat and ask: "What is the meeting room code?" Then add a second made-up fact right at the start and a third at the end, and ask for all three. Try again with a much longer text if you can. Note which facts are found, which are missed and whether the AI admits when it cannot find something. This is a small, safe way to see how attention behaves on long input.
Write down what you find. Your own result is worth more than any claim about a particular tool.
Worked Example: a long planning chat goes off track
Wen is planning a three-day workshop with an AI. In the first message he sets four rules: the audience is new managers, sessions are 45 minutes, no jargon, and the budget is small. For an hour the chat goes well.
By message thirty-five, the AI is suggesting a two-hour session with technical vocabulary and an expensive venue. Wen's first reaction is irritation. "It is ignoring me." Then he thinks it through. The chat is long, it includes several long replies and a pasted agenda. Either the early rules have been compressed or dropped, or they carry little weight so far back.
He applies the matching fix. He asks for a handoff note, reads it, and notices it has lost the 45-minute rule and the budget. He corrects those two lines, opens a new chat and pastes the corrected note. He adds a one-line reminder at the end of his first message: "Rules: new managers, 45-minute sessions, no jargon, small budget. Please check every suggestion against these."
The new chat stays on track. He keeps the topic narrow: day one only. When day one is done, he asks for a short summary, and starts a fresh chat for day two. The whole workshop takes four short chats instead of one long one, and each stays obedient.
Common Mistakes to Avoid
Three habits cause most of the frustration.
Mistake #1: Assuming the AI remembers the way a person does. People talk to a chat AI as if it holds a memory of them, then are surprised when a new chat starts blank or an early rule vanishes. Fix: remember that the conversation is re-read each time, restate what matters, and use a handoff summary when you move to a new chat.
Mistake #2: Keeping one giant chat going for everything. A long, wandering chat fills the window, buries early rules and makes answers drift. Fix: one topic per chat, start fresh when the AI drifts, and break big jobs into stages with a short summary passed between them.
Mistake #3: Pasting everything and trusting the answer. A whole report goes in, a confident summary comes out, and a detail from the middle quietly goes missing. Fix: paste only what you need, put your question and key instruction at the end, ask for quotes you can check, and verify any detail that matters against the original.
Going Further
Build a reusable starter block. Write a short block of your standing rules and context, such as role, audience, tone and limits, and paste it at the top of every new chat. It is a hand-made version of saved memory that you control completely.
Compare tools. Run the buried-detail experiment in two different tools, and write down which finds more. Redo it every few months, since the tools change.
Learn to read the signs. Keep a note of the symptoms that told you a window was full: repeated answers, ignored rules, vague references to "earlier". You will recognise them sooner next time.
What to learn next. The next tutorial is about hallucination: why an AI can sound completely sure and be wrong, which kinds of error to expect, and how to catch them. It builds directly on this one, because a missing detail is often exactly where an AI fills the gap with something plausible.
Key Takeaways
In this tutorial you learned:
Before you move on, a challenge. Run the two experiments, then take your longest recent chat and write a handoff note for it using the prompt above. Read the note carefully and mark anything it got wrong or dropped. That one exercise will teach you more about how an AI "remembers" than any explanation.