Dynamic query routing for distributed edge chatbots on NVIDIA Jetson platforms

Abstract

In the current age of cloud-based computing and the widespread adoption of large language models (LLMs), concerns regarding data privacy, security and efficiency have become largely prominent. Many widely used LLM solutions rely on centralized cloud infrastructures, which is particularly concerning for industries handling sensitive information, such as healthcare, finance, and government services, where strict data protection regulations apply.

Deploying LLMs on local edge devices offers a compelling alternative, providing greater control over data while reducing reliance on external cloud services. However, running LLMs on edge devices presents challenges in terms of computational efficiency, power consumption, and latency. A potential solution lies in distributed LLM systems, where multiple edge devices work together to process queries based on their complexity, optimizing performance while maintaining security.

This project explores a distributed LLM-based chatbot deployed across multiple Nvidia Jetson devices. The system is designed to dynamically route queries based on their complexity, ensuring that high-performance devices handle resource-intensive tasks while lightweight devices manage simpler queries. The goal is to strike a balance between latency and energy efficiency while maintaining local inference capabilities that prioritise privacy and security. The system utilizes the Nvidia Jetson AGX Orin for high-performance tasks and the Jetson Nano Developer Kit for simpler tasks, operating in a client-server model where each device acts as an independent server. The queries are assigned to the respective devices for processing based on a complexity threshold.

Experiments confirmed expected theoretical outcomes—lower latency but higher power usage on the Orin, and the opposite on the Nano. However, further investigation and testing was conducted to determine the optimal threshold point for a balance in efficiency and performance for different use cases.