noun · AI technique
Prompt caching
also called prefix cachingcontext caching
The AI keeps its place in a book you keep handing back. If the first pages are unchanged, it skips rereading them.
Reusing the already-processed start of a prompt across requests when the beginning is identical, to cut latency and input cost.
The provider stores the processed state of an identical prefix of your input, and if the next request starts with exactly the same tokens that work is reused, so it responds faster and usually costs less. Change anything early in the prefix and the saving is lost.
Real-life example
Your assistant sends the same long instructions and a big file with every question. Without caching, each question pays to process all of it again. With caching on, the unchanged start is reused, and only your new question is processed fresh. A single changed line near the top ends the saving.
Playground
Send requests and watch how much of each one is reused. Then change something near the top.
Your app
No saved copy yet.
What gets sent
Press “Send a request” to see what is reused. The first request has nothing saved yet.
Illustrative: made-up sizes and speeds. Reused text is counted at 10% of normal and the saved copy lives 5 minutes. Real providers differ.
Where the analogy breaks
-
It only works on an exact match from the start. Putting a changing timestamp early in the prompt can quietly turn the saving off.
-
Caches are short-lived. If a while passes between requests, the stored work may be gone and you pay the full price again.
-
It does not make the context window bigger. Cached text still counts toward the limit; it is only cheaper and quicker to process.