Evaluating large language models for multilingual vulnerability detection at dual granularities

TL;DR

This is an empirical study comparing the performance of PLMs and LLMs for multilingual vulnerability detection at function-level and line-level.

Problem

Existing studies primarily focus on specific languages (e.g., C/C++) and function-level detection, leaving the strengths and weaknesses of PLMs and LLMs in multilingual and multi-granularity scenarios largely unexplored.

Approach

Using over 30,000 real-world vulnerability-fixing patches across seven programming languages, the study systematically assesses the performance of state-of-the-art PLMs and LLMs at both function-level and line-level. GPT-4o is enhanced through instruction tuning and few-shot prompting.

Results & Contribution

GPT-4o, enhanced with instruction tuning and few-shot prompting, significantly outperforms all other evaluated models, including CodeT5P. The LLM-based approach demonstrates superior capability in detecting unique multilingual vulnerabilities, particularly excelling in identifying the most dangerous and high-severity vulnerabilities, revealing substantial improvements over PLM approaches.