Please use this identifier to cite or link to this item: https://hdl.handle.net/10356/162546
Title: Cross-modal graph with meta concepts for video captioning
Authors: Wang, Hao
Lin, Guosheng
Hoi, Steven C. H.
Miao, Chunyan
Keywords: Engineering::Computer science and engineering
Issue Date: 2022
Source: Wang, H., Lin, G., Hoi, S. C. H. & Miao, C. (2022). Cross-modal graph with meta concepts for video captioning. IEEE Transactions On Image Processing, 31, 5150-5162. https://dx.doi.org/10.1109/TIP.2022.3192709
Project: AISG-GC-2019-003 
AISG-RP-2018-003 
NRF-NRFI05-2019-0002 
MOH/NIC/HAIG03/2017 
RG95/20 
Journal: IEEE Transactions on Image Processing 
Abstract: Video captioning targets interpreting the complex visual contents as text descriptions, which requires the model to fully understand video scenes including objects and their interactions. Prevailing methods adopt off-the-shelf object detection networks to give object proposals and use the attention mechanism to model the relations between objects. They often miss some undefined semantic concepts of the pretrained model and fail to identify exact predicate relationships between objects. In this paper, we investigate an open research task of generating text descriptions for the given videos, and propose Cross-Modal Graph (CMG) with meta concepts for video captioning. Specifically, to cover the useful semantic concepts in video captions, we weakly learn the corresponding visual regions for text descriptions, where the associated visual regions and textual words are named cross-modal meta concepts. We further build meta concept graphs dynamically with the learned cross-modal meta concepts. We also construct holistic video-level and local frame-level video graphs with the predicted predicates to model video sequence structures. We validate the efficacy of our proposed techniques with extensive experiments and achieve state-of-the-art results on two public datasets.
URI: https://hdl.handle.net/10356/162546
ISSN: 1057-7149
DOI: 10.1109/TIP.2022.3192709
Schools: School of Computer Science and Engineering 
Rights: © 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. The published version is available at: https://doi.org/10.1109/TIP.2022.3192709.
Fulltext Permission: open
Fulltext Availability: With Fulltext
Appears in Collections:SCSE Journal Articles

Files in This Item:
File Description SizeFormat 
Cross-Modal Graph with Meta Concepts for VideoCaptioning.pdf6.23 MBAdobe PDFThumbnail
View/Open

SCOPUSTM   
Citations 50

5
Updated on Apr 16, 2024

Web of ScienceTM
Citations 50

2
Updated on Oct 30, 2023

Page view(s)

118
Updated on Apr 18, 2024

Download(s) 50

39
Updated on Apr 18, 2024

Google ScholarTM

Check

Altmetric


Plumx

Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.