Please use this identifier to cite or link to this item:
https://hdl.handle.net/10356/162632
Title: | Learning to compose and reason with language tree structures for visual grounding | Authors: | Hong, Richang Liu, Daqing Mo, Xiaoyu He, Xiangnan Zhang, Hanwang |
Keywords: | Engineering::Electrical and electronic engineering Engineering::Computer science and engineering |
Issue Date: | 2019 | Source: | Hong, R., Liu, D., Mo, X., He, X. & Zhang, H. (2019). Learning to compose and reason with language tree structures for visual grounding. IEEE Transactions On Pattern Analysis and Machine Intelligence, 44(2), 684-696. https://dx.doi.org/10.1109/TPAMI.2019.2911066 | Journal: | IEEE Transactions on Pattern Analysis and Machine Intelligence | Abstract: | Grounding natural language in images, such as localizing "the black dog on the left of the tree", is one of the core problems in artificial intelligence, as it needs to comprehend the fine-grained language compositions. However, existing solutions merely rely on the association between the holistic language features and visual features, while neglect the nature of composite reasoning implied in the language. In this paper, we propose a natural language grounding model that can automatically compose a binary tree structure for parsing the language and then perform visual reasoning along the tree in a bottom-up fashion. We call our model RvG-Tree: Recursive Grounding Tree, which is inspired by the intuition that any language expression can be recursively decomposed into two constituent parts, and the grounding confidence score can be recursively accumulated by calculating their grounding scores returned by the two sub-trees.RvG-Tree can be trained end-to-end by using the Straight-Through Gumbel-Softmax estimator that allows the gradients from the continuous score functions passing through the discrete tree construction. Experiments on several benchmarks show that our model achieves the state-of-the-art performance with more explainable reasoning. | URI: | https://hdl.handle.net/10356/162632 | ISSN: | 0162-8828 | DOI: | 10.1109/TPAMI.2019.2911066 | Schools: | School of Computer Science and Engineering School of Electrical and Electronic Engineering |
Rights: | © 2019 IEEE. All rights reserved. | Fulltext Permission: | none | Fulltext Availability: | No Fulltext |
Appears in Collections: | EEE Journal Articles SCSE Journal Articles |
SCOPUSTM
Citations
10
42
Updated on Mar 23, 2024
Web of ScienceTM
Citations
10
26
Updated on Oct 28, 2023
Page view(s)
67
Updated on Mar 28, 2024
Google ScholarTM
Check
Altmetric
Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.