TTP identification from unstructured text

Abstract

This project aims to study and compare the efficiency of different Large Language Models (LLMs) for the identification of Tactics, Threats, Procedures (TTPs). Accurate identification of TTPs from Cyber Threat Intelligence reports and documentation is vital for threat analysis, proactive defense, and understanding adversarial behavior. Traditional approaches for TTP extraction rely heavily on manual analysis, often requiring extensive expertise and resources.

In this study, we investigate both encoder-only models, specifically BERT variants, and decoder-only models such as GPT and LLAMA. Additionally, we evaluate prompt engineering techniques and Retrieval-Augmented Generation (RAG) to enhance TTP extraction. To mitigate limitations posed by sparsely available annotated cybersecurity datasets, we explore various data augmentation methods.

We performed comparative experiments, evaluating each model type and enhancement technique. Based on our results, encoder-only models perform much better than decoder-only models for TTP extraction tasks with big gains in accuracy and dependability.

Lastly, to make practical use and operational integration easier, we created a web-based application that includes our best-performing model along with Nautral Langauge Processing (NLP) preprocessing pipelines that allow the model to be improved over time.

Overall, our findings suggest that encoder-only models such as BERT with data augmentation and fine-tuning represent the most effective approach for reliably extracting TTPs from unstructured cybersecurity texts.