Please use this identifier to cite or link to this item:
Full metadata record
DC FieldValueLanguage
dc.contributor.authorWang, Haoen_US
dc.contributor.authorLin, Guoshengen_US
dc.contributor.authorHoi, Steven C. H.en_US
dc.contributor.authorMiao, Chunyanen_US
dc.identifier.citationWang, H., Lin, G., Hoi, S. C. H. & Miao, C. (2022). Paired cross-modal data augmentation for fine-grained image-to-text retrieval. 30th ACM International Conference on Multimedia (MM 2022), 5517-5526.
dc.description.abstractThis paper investigates an open research problem of generating text-image pairs to improve the training of fine-grained image-to-text cross-modal retrieval task, and proposes a novel framework for paired data augmentation by uncovering the hidden semantic information of StyleGAN2 model. Specifically, we first train a StyleGAN2 model on the given dataset. We then project the real images back to the latent space of StyleGAN2 to obtain the latent codes. To make the generated images manipulatable, we further introduce a latent space alignment module to learn the alignment between StyleGAN2 latent codes and the corresponding textual caption features. When we do online paired data augmentation, we first generate augmented text through random token replacement, then pass the augmented text into the latent space alignment module to output the latent codes, which are finally fed to StyleGAN2 to generate the augmented images. We evaluate the efficacy of our augmented data approach on two public cross-modal retrieval datasets, in which the promising experimental results demonstrate the augmented text-image pair data can be trained together with the original data to boost the image-to-text cross-modal retrieval performance.en_US
dc.description.sponsorshipAI Singaporeen_US
dc.description.sponsorshipMinistry of Education (MOE)en_US
dc.description.sponsorshipMinistry of Health (MOH)en_US
dc.description.sponsorshipNational Research Foundation (NRF)en_US
dc.rights© 2022 The owner/author(s). Publication rights licensed to ACM. All rights reserved. This paper was published in the Proceedings of 30th ACM International Conference on Multimedia (MM 2022) and is made available with permission of The owner/author(s).en_US
dc.subjectEngineering::Computer science and engineeringen_US
dc.titlePaired cross-modal data augmentation for fine-grained image-to-text retrievalen_US
dc.typeConference Paperen
dc.contributor.schoolSchool of Computer Science and Engineeringen_US
dc.contributor.conference30th ACM International Conference on Multimedia (MM 2022)en_US
dc.description.versionSubmitted/Accepted versionen_US
dc.subject.keywordsImage-to-Text Retrievalen_US
dc.subject.keywordsComputing Methodologiesen_US
dc.citation.conferencelocationLisbon, Portugalen_US
dc.description.acknowledgementThis research is supported, in part, by the National Research Foun- dation (NRF), Singapore under its AI Singapore Programme (AISG Award No: AISG-GC-2019-003) and under its NRF Investigator- ship Programme (NRFI Award No. NRF-NRFI05-2019-0002). This research is supported, in part, by the Singapore Ministry of Health under its National Innovation Challenge on Active and Confident Ageing (NIC Project No. MOH/NIC/HAIG03/2017). This research is also supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG-RP-2018-003), and the MOE AcRF Tier-1 research grant: RG95/20.en_US
item.fulltextWith Fulltext-
Appears in Collections:SCSE Conference Papers
Files in This Item:
File Description SizeFormat 
Full text.pdf1.51 MBAdobe PDFThumbnail

Citations 50

Updated on May 17, 2024

Page view(s)

Updated on May 18, 2024

Download(s) 50

Updated on May 18, 2024

Google ScholarTM




Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.