Please use this identifier to cite or link to this item:
Title: Multistage spatio-temporal networks for robust sketch recognition
Authors: Li, Hanhui
Jiang, Xudong
Guan, Boliang
Wang, Ruomei
Thalmann, Nadia Magnenat
Keywords: Engineering::Electrical and electronic engineering
Issue Date: 2022
Source: Li, H., Jiang, X., Guan, B., Wang, R. & Thalmann, N. M. (2022). Multistage spatio-temporal networks for robust sketch recognition. IEEE Transactions On Image Processing, 31, 2683-2694.
Journal: IEEE Transactions on Image Processing 
Abstract: Sketch recognition relies on two types of information, namely, spatial contexts like the local structures in images and temporal contexts like the orders of strokes. Existing methods usually adopt convolutional neural networks (CNNs) to model spatial contexts, and recurrent neural networks (RNNs) for temporal contexts. However, most of them combine spatial and temporal features with late fusion or single-stage transformation, which is prone to losing the informative details in sketches. To tackle this problem, we propose a novel framework that aims at the multi-stage interactions and refinements of spatial and temporal features. Specifically, given a sketch represented by a stroke array, we first generate a temporal-enriched image (TEI), which is a pseudo-color image retaining the temporal order of strokes, to overcome the difficulty of CNNs in leveraging temporal information. We then construct a dual-branch network, in which a CNN branch and a RNN branch are adopted to process the stroke array and the TEI respectively. In the early stages of our network, considering the limited ability of RNNs in capturing spatial structures, we utilize multiple enhancement modules to enhance the stroke features with the TEI features. While in the last stage of our network, we propose a spatio-temporal enhancement module that refines stroke features and TEI features in a joint feature space. Furthermore, a bidirectional temporal-compatible unit that adaptively merges features in opposite temporal orders, is proposed to help RNNs tackle abrupt strokes. Comprehensive experimental results on QuickDraw and TU-Berlin demonstrate that the proposed method is a robust and efficient solution for sketch recognition.
ISSN: 1057-7149
DOI: 10.1109/TIP.2022.3160240
Rights: © 2022 IEEE. All rights reserved.
Fulltext Permission: none
Fulltext Availability: No Fulltext
Appears in Collections:EEE Journal Articles

Page view(s)

Updated on Dec 3, 2022

Google ScholarTM




Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.