Real-time visual SLAM for embedded systems

Abstract

vSLAM (Visual Simultaneous Localization And Mapping) is a key step that enables many autonomous robots to self-navigate and map a given space. Recently, there has been a shift to incorporate Deep Learning (DL) based methods into the vSLAM pipeline due to their high performance. An important part of vSLAM, known as feature extraction, is responsible for consistently extracting salient regions in the camera images over time, in order to feed downstream matching and tracking algorithms. DL based feature extractors face two main challenges: robustness of the features and latency. This thesis makes the following contributions. The first contribution focuses on quantization and power efficiency optimization of SuperPoint, a DL based feature extractor. It is shown that SuperPoint can be quantized to half precision and executed on a neural accelerator with up to 40% reduction in power consumption with almost no effect on SLAM tracking performance. The second contribution propose hardware-aware inference time optimizations of ViT-based feature extractors. The proposed methods achieve approximately 12.5% prune rate at negligible performance degradation. The pruning methods can be further coupled with hardware-aware latency/accuracy trade- off search algorithms, to exploit non-linearity characteristics that leads to 7% faster inference for a single pruned token. Next, the proposed image-adaptive learned pruning mechanism demonstrates up to 20% speedup by selectively forwarding less important tokens. It is also demonstrated that lightweight ViT backbones like EfficientFormer can achieve high performance on the feature extraction task.