This repository is for CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval from CUHK-SenseTime Joint Lab (appear in ICCV 2019).
It is built on top of the VSE++ and SCAN in PyTorch.
We recommended the following dependencies.
import nltk
nltk.download()
> d punktDownload the dataset files and pre-trained models. We use the same pre-extracted features and splits as SCAN.
The splits are produced by Andrej Karpathy. The raw images can be downloaded from from their original sources here, here and here.
The precomputed image features of MS-COCO are from here. The precomputed image features of Flickr30K are extracted from the raw Flickr30K images using the bottom-up attention model from here.
The pre-extracted image features are from SCAN, produced by Kuang-Huei Lee. The data can be downloaded from:
wget https://scanproject.blob.core.windows.net/scan-data/data.zipWe refer to the path of extracted files for data.zip as ./data directory.
Run train.py in the directory of the corresponding config path:
Training the cross-attention model on Flickr30K dataset:
cd ./experiments/f30k_cross_attention
python python -u ../../train.py --config ./config_256.yamlTraining the full CAMP model on Flickr30K dataset:
cd ./experiments/f30k_gate_fusion
python python -u ../../train.py --config ./config_finetune.yamlWe initialize the network weights from the pretrained cross-attention model to train the full CAMP model. The weights for attention map are fixed for the first several epochs and then we finetune the whole network.
Changing the resume arguments in the coreesponding config file and running evaluation in the project root directory:
from test_modules import test_CAMP_model
#config_path = "./experiments/f30k_cross_attention/config_test.yaml"
test_CAMP_model(config_path)Pretrained model for Flickr30K could be downloaded here.
If you found this code useful, please cite the following paper:
@InProceedings{Wang_2019_ICCV,
author = {Wang, Zihao and Liu, Xihui and Li, Hongsheng and Sheng, Lu and Yan, Junjie and Wang, Xiaogang and Shao, Jing},
title = {CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval},
booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
month = {October},
year = {2019}
}
I have left CUHK and the email address is deprecated. Please directly open a new issue or contact [email protected] if you have further quetions. Thanks!