A car dealership’s chatbot once agreed, in writing, to sell a customer a vehicle for one dollar — because someone figured out how to manipulate what it was told to do. That’s not a hypothetical risk. It’s the kind of failure a demo won’t show you, and it’s a fair thing to ask about before you put a chatbot in front of real customers.
Two different problems, often confused
Hallucination is the bot confidently saying something false — a wrong price, a policy that doesn’t exist, an answer it made up because it didn’t actually know. Prompt injection is someone deliberately manipulating what the bot does, often by hiding instructions in a message it processes. They’re different failure modes, but injection often makes hallucination worse — a compromised bot doesn’t just guess wrong, it can be steered into saying whatever the attacker wants.
Why this isn’t a rare edge case
Independent research has found real, existing exposure: a meaningful share of sampled e-commerce chatbots were already vulnerable to third-party content injection, and several widely used chatbot plugins pass conversation history to the underlying AI model with no integrity checks at all — meaning nothing stops a forged message from being treated as legitimate input.
What actually prevents it
- Grounded in real data, not open-ended guessing — a bot that answers from your actual inventory, pricing, and policies is far less likely to invent an answer than one improvising from general knowledge.
- A defined scope — the bot should only be able to do the specific things it’s meant to do (check availability, answer a known question), not execute arbitrary requests phrased convincingly enough.
- Validation before anything consequential happens — a price change, a refund, a booking cancellation should never happen purely because a message asked for it.
- A real escalation path — anything the bot doesn’t have a confident, grounded answer for should hand off to a person instead of guessing.
Where this fits
This is why AI agent development work is built around your actual data and a defined scope from the start, not a generic model given open-ended instructions and hoped to behave. It’s the same principle behind how we handle escalation in AI agent for customer service work — the bot should know what it doesn’t know.
Ask us how a specific workflow would be protected before it goes live.