Obj2Text: Generating Visually Descriptive Language from Object Layouts

News Release Summary

Researchers at the University of Virginia have built a system that can automatically write captions describing a scene using nothing more than a list of objects and their positions in an image, bypassing the need for raw pixel data. The system, called OBJ2TEXT, works by feeding object labels and their bounding box coordinates into one neural network that encodes the layout as a sequence, and then passing that encoded representation to a second neural network that generates a sentence word by word. Testing on the standard MS-COCO image dataset, the team found that object location and object count both meaningfully improved caption quality — removing either one caused measurable drops in performance — demonstrating that even a sequential encoding of spatial information carries real descriptive value. Perhaps more practically, when the researchers combined OBJ2TEXT with an object detector called YOLO and a conventional image-based captioning model, the hybrid system outperformed the image-only captioning baseline, pushing its CIDEr score from 0.863 to 0.950 on the MS-COCO benchmark; human evaluators also preferred the combined system's captions roughly 65 percent of the time when they all agreed. The work matters because it shows that structured, symbolic information about a scene — the kind produced by object detectors or used in graphic design and storyboarding — can complement or even partially substitute for pixel-level visual features in language generation, offering a cleaner way to study what image captioning models actually need to know about a scene.