Intelligence Report

Nvidia team says summaries and search can replace giant AI context windows

Nvidia researchers report that periodic summaries and keyword lookups usually matched full-context results at a fraction of the cost.

INSIGHTEX Editorial Desk

2 min read

Nvidia team says summaries and search can replace giant AI context windowsIntelligence Report

What happened

A research group at Nvidia contends that large language models frequently have no need to use the full length of their context windows, The Decoder reported. The researchers describe long context as slow, costly and error-prone. Instead, they put forward an approach that compresses and retrieves information rather than keeping all of it in view at once.

In the team's tests, two cheaper methods generally performed on par with giving a model its entire context. One summarised material at regular intervals, and the other ran targeted keyword searches to pull up relevant passages. Both reached similar results at a small share of the cost. The group also suggests a new category of models designed to work from brief notes instead of extended context.

Context

A model's context window acts as its working memory, covering all the material it can consider while producing a response. That capacity has expanded quickly. GPT-4 debuted in March 2023 with 8,192 tokens, while GPT 5.6 Sol now handles roughly one million, around 120 times more and enough to fit Tolstoy's War and Peace.

NVIDIA profile

handles profile

SourceThe Decoder

The briefing

The week in tech, in one email.

The stories worth your time, every week. Unsubscribe anytime.