International Conference on Emerging Techniques in Computational Intelligence (ICETCI) ·
Hard Negative-Aware Fine-Tuning for Multi-Modal Image Retrieval
Published · Conference · Multimodal retrieval
BLIP and OpenCLIP use hard negative examples to learn to retrieve similar fashion images.
Abstract
The notion of multi-modal image retrieval involves retrieving semantically relevant images with a textual query from a very large image gallery. However, most of the current methods based on single modality fail to address complex relationships that exist between visual and textual content. In this paper, we propose a framework to learn embeddings by contrasting image representations with textual query representations in order to perform effective multi-modal retrieval in the fashion domain. We investigate two latest state-of-the-art vision-language models, i.e., OpenCLIP and BLIP, on the FashionIQ dataset which is made up of fashion images with detailed human annotations in their captions. To improve the performance of our framework, hard negative samples are used in the contrastive loss function to increase the discriminative power of learning embeddings, and then progressive fine-tuning of image and text encoders is explored on different architectures. As our experimental results show, when tested on three fashion categories, viz. Dress, TopTee and Shirt, the recall@50 scores are 50.13% and 46.08% respectively for BLIP and OpenCLIP, outperforming other baselines such as TIRG, ComposeAE, and CIRPLANT.
Findings
- Reports Recall@50 of 50.13% for BLIP and 46.08% for OpenCLIP across the evaluated fashion categories.
- Combines hard-negative contrastive learning with progressive encoder fine-tuning.
Sources and citation
DOI 10.1109/ICETCI71772.2026.11710416
View BibTeX citation
@inproceedings{hard-negatives-2026,
title = {{Hard Negative-Aware Fine-Tuning for Multi-Modal Image Retrieval}},
author = {Appana Saketh Krishna Rao and BHVSP Subrahmanyam and Jyotsna Pandranki and Neeraj Choudhary and Alok Kumar Kamal},
booktitle = {International Conference on Emerging Techniques in Computational Intelligence (ICETCI)},
year = {2026},
pages = {1--5},
publisher = {IEEE},
doi = {10.1109/ICETCI71772.2026.11710416},
url = {https://ieeexplore.ieee.org/abstract/document/11710416/}
}