Deep Feature Aggregation and Image Re-ranking with Heat Diffusion for Image Retrieval
News Release Summary
Researchers at Xi'an Jiaotong University and the University of Virginia have developed a new image search system that borrows a concept from physics — heat diffusion — to make visual search engines more accurate and efficient. The core problem they tackled is that standard image retrieval systems can be thrown off by repetitive visual patterns, such as the rows of identical windows on a building's facade, which flood the system with redundant information and make it harder to identify what is actually distinctive about an image. To fix this, the team treated each local feature extracted from a convolutional neural network as a heat source, then used the mathematics of heat diffusion to measure how "bursty" or repetitive that feature is — features that spread heat widely through a network of similar neighbors are flagged as redundant, while isolated features that generate little heat transfer are treated as more distinctive. The system then assigns weights to features accordingly before combining them into a single compact image descriptor. The same heat-diffusion principle was also applied at the image level, where a query image acts as a heat source and the warmth it spreads to candidate database images is used to re-rank search results. Testing on standard benchmarks including the Oxford Buildings and Paris datasets, the approach outperformed competing methods, in some cases improving retrieval accuracy by more than five percentage points on large-scale datasets, while still running fast enough for practical use — all without requiring any additional labeled training data.
abstract
Image retrieval based on deep convolutional features has demonstrated state-of-the-art performance in popular benchmarks. In this paper, we present a unified solution to address deep convolutional feature aggregation and image re-ranking by simulating the dynamics of heat diffusion. A distinctive problem in image retrieval is that repetitive or \emph{bursty} features tend to dominate final image representations, resulting in representations less distinguishable. We show that by considering each deep feature as a heat source, our unsupervised aggregation method is able to avoid over-representation of \emph{bursty} features. We additionally provide a practical solution for the proposed aggregation method and further show the efficiency of our method in experimental evaluation. Inspired by the aforementioned deep feature aggregation method, we also propose a method to re-rank a number of top ranked images for a given query image by considering the query as the heat source. Finally, we extensively evaluate the proposed approach with pre-trained and fine-tuned deep networks on common public benchmarks and show superior performance compared to previous work.
details
citation
@article{pang2019deep,
title = {Deep Feature Aggregation and Image Re-ranking with Heat Diffusion for Image Retrieval},
author = {Pang, Shanmin and Ma, Jin and Xue, Jianru and Zhu, Jihua and Ordonez, Vicente},
year = {2019},
journal = {IEEE Transactions on Multimedia 2019 (Journal).},
url = {https://arxiv.org/abs/1805.08587},
}