Benchmarking AI workflows: performance evaluation of DietCoke in visual question answering pipelines

Abstract

Recent advances in Large Language Models (LLMs) have led to the emergence of complex, agentic AI workflows that go beyond single-model inference. These workflows, such as those used in visual question answering (VQA), involve multiple interdependent steps including image understanding, external knowledge integration, and iterative reasoning. However, current benchmarking tools largely focus on isolated model performance and fail to capture the nuanced performance characteristics of such multi-stage AI systems. This gap makes it difficult to evaluate and optimize real-world AI deployments holistically. This project extends the Agentic Workflow Benchmark framework to support the end-to-end evaluation of complex LLM workflows. Specifically, the DietCoke VQA pipeline was re-implemented and modularized into a benchmarkable workflow, deployed with vLLM as inference backends. LAVIS’ Img2LLM was also integrated as a separate microservice to simulate a full VQA pipeline. Key findings show that vLLM achieves significantly higher throughput—up to 25× over the Hugging Face baseline—while maintaining comparable accuracy. However, Img2LLM introduces a bottleneck due to its sequential design and memory-bound QA generation, underscoring the need for further optimization in upstream components. These insights highlight the importance of benchmarking AI workflows as full systems rather than isolated models.