Context Windows

The Idea

The context window is the most tokens a model can attend to in one call. Instructions, history, retrieved passages and the reply all share it, and anything outside it simply does not exist for the model.

Real-World Example

An air-traffic controller's rack of paper flight strips. A new strip goes on only when one comes off, and a flight with no strip on that rack is, for that controller, not in the picture.

The Code

history = [("system", 20), ("user", 300), ("assistant", 250),
           ("user", 180), ("assistant", 400), ("user", 120)]
budget = 700

kept, used = [], 0
for role, size in reversed(history):     # newest turns first
    if used + size > budget:
        break                            # everything older is dropped
    kept.append((role, size)); used += size

print(list(reversed(kept)), used)        # note: the system turn fell off

history = [("system", 50), ("user", 200), ("assistant", 300), ("user", 100)]
budget = 500

used = 0
for role, size in reversed(history):
  if used + size > budget:
      break
  used += size

print(used)