Event extraction for cybersecurity using large language models

Abstract

This project studies and compares the efficiency of different Large Language Models (LLMs) for the extraction of cybersecurity events. Cybersecurity event extraction is a critical task in Cyber Threat Intelligence, it is aimed at identifying and categorizing incidents such as data breaches, malware attacks, and vulnerabilities from unstructured text sources like news articles, threat reports, and social media.

Traditional methods for cybersecurity event extraction often rely on rule-based systems or supervised machine learning models, which require extensive labelled data and are limited in adaptability. The nature of cybersecurity is that it is ever changing. One method of acquiring Cyber Threat Intelligence is through Open-Source Intelligence, where articles across the web are sourced and analysed. As LLMs have a good understanding of semantics and context, it is possible to leverage on LLMs for Cybersecurity event extraction.

In this study, the focus will be on the conversational LLMs that many are familiar with, such as ChatGPT3.5, ChatGPT-4, LLAMA and Cohere. We investigate the efficacy of these conversational LLMs in extracting Cybersecurity events without further fine tuning but with the help of prompting techniques as well as Retrieval Augmented Generation.

The effectiveness of our approach is evaluated through experiments on the CASIE dataset, comparing the performance of the different LLMs over zero shot, prompting techniques and retrieval augmented generation. The results demonstrate that the current state of base LLMs is unable to fulfil the task of Cybersecurity Event Extraction.