Georg Zoeller on LinkedIn: This is the wall all tools with web access will run into. AI is even more…

Why this example is a big deal

In order to give generative AI systems agency to take actions or to make decisions, we need to be reasonably sure they are operating in our best interest.

Unfortunately, because these models are trained on uncurated data at a scale that’s impossible to (economically) curate and validate, even if we could make AI systems without hallucination and immunity to prompt injection (which we can not), they would still be vulnerable to training data being maliciously manipulated.

This creates a familiar pattern: In order to operationalize AI generated work, a human needs to perform meticulous security analysis (in this case, auditing every package used in the app).

Additionally, this problem is not limited to static training data. Via prompt injection, or existing compromised training data, models can

Currently, lofty valuations and tales of full automation, do not price in this reality correctly, creating yet another naive, insecure by design technology which will have to be patched with ropes and wires for years to come.

The Cybersecurity angle

Generative AI supercharges familiar cybersecurity issues. In this case, a supply chain attack (malicious package) being injected into code. Running that code would lead to systems compromise.

However, even if best security practices are applied, the problem goes much further. The right prompt injection in combination with “AI tool calling” (the ability for AI systems to invoke other tools) is another can of worms…