Vulnerability Evolution and the Promise of Automated Gatekeeping in Open-Source Software

Abstract

Open-source software (OSS) has become increasingly more popular across different domains, yet the security mechanisms governing it have not scaled with its growth. This paper presents a longitudinal study of 30,572 unique vulnerability reports spanning 10 programming languages and eight package managers from 2017 to late 2025. Our analysis exposes a fundamental imbalance: reported vulnerable packages are growing at 91.5% annually, nearly four times the 26% growth rate of total packages, while maintainer remediation capacity has remained roughly flat at 1,650 deployed fixes per year since 2022. This mismatch has resulted in a growing backlog of unresolved vulnerabilities. We also examine vulnerability distributions across ecosystems and find that only six CWE types account for over 50% of all reports across ecosystems, and that intentional supply chain attacks through embedded malicious code are heavily concentrated in NPM and PyPI, which together account for over 98% of all such incidents. This imbalance in maintainer capacity, combined with the concentration of risk in a small set of weakness patterns, motivates the exploration of large language models (LLMs) as automated defenses within CI/CD pipelines. We evaluate 10 state-of-the-art LLMs on 582 pairs of real-world vulnerable and patched package versions covering 1,434 advisories. Our results show that no single model resolves the trade-off between precision, recall, and latency. GPT-5.1 achieves the strongest balanced performance (62.8% F1), while reasoning-intensive models such as DeepSeek-R1 incur considerably higher inference costs (~50s), and high-recall models like LLaMA-2 generate excessive false positives. These findings support a hybrid defense pipeline that balances recall and precision, combining efficient first-pass screening with more precise downstream analysis for validation and classification. Such designs offer a practical path to narrowing the growing gap between vulnerability discovery and remediation.