LLM Hallucinated Security Reports A Nightmare For Open Source Projects

Python And pip Today, Maybe Your Repository Next

There are a lot of arguments about what LLMs are truly capable of, but one thing they are obviously good at is creating a large amount of content in next to no time.  The only limitation of the volume of output they can produce is the hardware they run on.  This has become obvious in things like AI generated SEO optimization, which invisibly fills product descriptions with immense amounts of keywords that may or may not apply to the product.  Regardless, search engines love that sort of thing and happily give higher weights to products with all that AI generated SEO garbage.  There is now a new way that LLMs are ruining people’s online experiences, LLM generated security reports are bombarding open source projects.

Recently a large volume of AI generated bug reports have been bombarding open source projects, and while the reports are not based in reality but are indeed LLM hallucinations, it is impossible to determine that until they are investigated.  It can take a bit of time to verify the reported security problem is indeed a load of nonsense and with the volume of reports increasing daily they can paralyze an open source project’s development while they are investigated.

To make matters worse, these reports are not necessarily malicious.  A person interested in trying out an open source project might ask their favourite LLM if the program is secure and not question the results they are provided.  Out of the kindness of their hearts they would then submit the bug report by copying and pasting the results provided by the LLM without bothering to read them.  This leads to the project developer having to spend time to prove that the data provided is crap hallucinated by an LLM, when they could have been working on real issues or improvements.

The reports could also be weaponized, if someone wanted to interfere with the development of a project.  A conscientious developer can’t just ignore bug reports submitted to their projects without the risk of missing a valid one.  If you are delving into open source and asking your favourite LLM to check projects for security issues, maybe just don’t do that!  Learn enough about the program to verify there is an issue, or leave it to those who can do that already.