Drill-down: Interactive Retrieval of Complex Scenes using Natural Language Queries

News Release Summary

Researchers at the University of Virginia and IBM Research have developed a system called Drill-down that lets users find specific images by typing a series of natural language descriptions, each one narrowing the search further rather than trying to capture everything in a single query. The problem they were tackling is a familiar one: existing image search tools struggle when a user wants to locate a very particular photo of a complex scene containing multiple objects, because cramming an entire scene description into one sentence is both difficult and imprecise. Instead of forcing that single-shot approach, Drill-down lets users start broad — say, "a group of people posing in a park" — and progressively add more specific details across several turns, such as "there is a bride among them," with the system updating its results each time. The key technical contribution is a compact set of state vectors that store and organize the history of a user's queries, with each vector learning to track a distinct part of the scene rather than collapsing everything into one representation, which is how earlier dialogue-based retrieval systems worked. Crucially, the team found they could train the model without collecting expensive human-annotated search sessions, instead using existing image region captions from the Visual Genome dataset as a cheap substitute for real user queries. Tests on both simulated and real human users showed that Drill-down outperformed competing methods while actually using less memory and fewer parameters, and more than 80 percent of human testers successfully located their target image within five turns. The work suggests that breaking image search into a conversational back-and-forth is a practical path toward retrieving highly specific images in large, diverse collections.