Feedback-prop: Convolutional Neural Network Inference under Partial Evidence

News Release Summary

Researchers from the University of Virginia and CyberAgent have developed a technique that lets existing image-recognition neural networks make better predictions when some information about a photo is already known ahead of time. The method, called feedback-prop, addresses a gap between how computer vision systems are typically tested — using only visual input — and how they are often used in practice, where surrounding text, GPS data, user tags, or other contextual clues are frequently available. Rather than retraining a network to incorporate that extra information, the researchers found they could instead feed known labels back through a trained network during the inference step itself, adjusting the network's internal activations until predictions for the remaining unknown labels improved. They tested two variants of the approach — one that updates layers sequentially and one that injects small corrective variables at multiple layers simultaneously — across several tasks, including identifying objects in images when some labels are already known, predicting fine-grained scene categories when coarse categories are given, and generating image captions when object annotations are available. Across all tasks and multiple standard network architectures including VGG-16 and ResNet, adding partial evidence consistently improved accuracy, with relative gains ranging from roughly 10 to 13 percent depending on the task. Notably, the technique requires no changes to the original model's training and works with an arbitrary mix of known and unknown labels, making it broadly practical for real-world deployment scenarios where images rarely arrive without any accompanying context.