MIT memory system lets robots flag and repair mislabeled objects on the fly

What happened

Researchers at MIT have developed UQ-DAAAM, a memory system for robots that detects stored object descriptions likely to be wrong and corrects them, MIT News reported. Graduate students Harry Zhang and Nicolas Gorlo are co-lead authors of the paper, with associate professor Luca Carlone as senior author. The paper is headed to the Conference on Neural Information Processing Systems.

The system measures how confident a vision-language model is in the words of a caption, then checks that caption against descriptions of the same object captured from other angles. Both signals are merged into a single uncertainty score. Above a set threshold, a simple rule chooses extra frames to examine. The team proved mathematically that this rule can pick the best frames without rerunning the model, sharply cutting computation. A large language model then combines the selected captions, favoring newer and clearer ones.

According to the researchers, the refinement step resolved ambiguities for roughly 92 percent of objects while needing only two additional captions per object, and it raised description accuracy by 5 percent over the state of the art. It also handled questions about objects, where they were and when they were last seen more accurately than baseline methods. Carlone said that letting a robot recognize the limits of its own knowledge will be essential for deploying physical AI safely.

Context

The work extends DAAAM, an earlier system from the same group. DAAAM used a vision-language model to caption objects seen through a robot's onboard cameras and stored those captions in a 3D map for later queries. Partial, distant, blurry or dim views could lead it astray, such as mistaking a gray chimney for a squirrel. Support came partly from the U.S. Office of Naval Research and the U.S. Army Research Laboratory.