Large Language Learning Models or LLMs are very vulnerable to prompt injection attacks. Microsoft security researchers have found multiple ways to circumvent the Guardrails.
It makes sense why Prompt Injection was included in the OWASP LLM Top 10.