Machine translation of multilingual cybersecurity reports with large language models

Abstract

Researchers are exploring the integration of large language models (LLMs) across diverse cybersecurity applications, such as OpenAI’s GPT models and DeepSeek’s reasoning models due to their robust knowledge and linguistic capabilities. Generative LLMs such as GPT-4 and DeepSeekV3 have demonstrated the ability to generate coherent text in multiple languages while simultaneously possessing extensive knowledge in many different domains of security. Although LLMs exhibit impressive knowledge capabilities, their performance often falls short compared to industry experts when their performance is evaluated on domain-specific tasks. However, the multilingual capabilities show potential in their application for more domain focused machine translation. This project explores the use of GPT-4o and DeepSeek-V3 for translation of cybersecurity reports to extract cyber threat intelligence from multilingual sources. The project includes the creation of a few security domain-focused benchmark datasets of parallel sentences extracted from cybersecurity reports as well as the OPUS corpora. Existing translated cybersecurity reports were parsed for matching sentence-pairs in the source and target languages. The data from the OPUS Wikimedia corpus was filtered for sentences containing security related keywords, then manually filtered for quality sentences. GPT-4o and DeepSeekV3 were evaluated on the benchmark datasets using the standard machine translation evaluation metrics BLEU, chrF and TER. The results show that DeepSeek-V3 and GPT-4o can generate translations of reasonably high quality that surpass open-source translation models and DeepSeek yielded slightly better translations for Russian and Chinese, but GPT-4o is able to consistently outperform DeepSeekV3 in Indonesian translations.