Visual localization at NTU campus

Abstract

Visual localization is a key problem in various computer vision applications such as augmented reality and autonomous driving. Major challenges for visual localization include varying weather conditions, dynamic foregrounds, and varying viewpoints as seen in environments with dynamic objects such as the Nanyang Technological University Campus. Some efficient methods to represent images for the Visual Place Recognition task like Fischer Vectors (FV), Scale-Invariant Feature Transform (SIFT), and Vector of Locally Aggregated Descriptors (VLAD) can handle some of these challenges. Although VLAD provides a rich and effective method for image storage and retrieval, it models a static function. NetVLAD modifies the same to create a trainable function, that minimizes the Euclidean distance between the query and the correct positive image and is used as baseline in this work. Soft assignment to clusters makes NetVLAD readily pluggable into Convolutional Neural Network architectures for end - to - end training. Instead of uniform pooling as in the case of NetVLAD, Attention Pyramid Pooling of Salient Visual Residuals (APPSVR) uses attention, generated based on semantic segmentation, to de-prioritize task irrelevant features. Three levels of attention in the form of local integration, global integration and parametric pooling handle the cases of task - irrelevant features, contextual information and weighting between clusters respectively. This paper aims to study the effect of semantic segmentation in visual localization; NetVLAD and APPVSR as potential solutions for visual localization in an indoor location like the Nanyang Technological University (NTU) Campus. Utilizing semantic information to generate attention has shown to be helpful with an increase in Recall@1 rates from 0.8381 to 0.8563.