When AI in Gmail Becomes a Double-Edged Sword: Security Research Reveals Vulnerabilities in Gemini AI Implementation and LLM Mechanisms
In 2024, AI-based attacks have become a growing phenomenon, requiring Gmail’s 2.5 billion account users to exercise increased vigilance. Google integrated the Gemini AI model into various products, such as Gmail, Google Slides, and Google Drive, to enhance user experience. However, security researchers have reported an attack known as “Indirect Prompt Injection”—an exploit through which malicious commands can be embedded in documents, emails, or websites. This causes the model to behave unpredictably, potentially allowing a hostile entity to obtain information or act on behalf of the user without their knowledge.
Despite this, Google has chosen to label this security issue as “Won’t Fix (Intended Behavior),” maintaining that it is not a security flaw but rather an inherent characteristic of how Large Language Models (LLMs) function.
New “Prompt Injection” Attacks: Link Trap and Multi-turn AI Jailbreak
How attackers trick AI into revealing sensitive information or bypassing safety mechanisms
In addition to “Indirect Prompt Injection,” security researchers have demonstrated new scenarios. The “Link Trap Attack” allows attackers to plant a command that prompts the AI to return a link; an unsuspecting user clicking the link could expose sensitive information.
A novel approach to Multi-turn Jailbreaking, dubbed “Bad Likert Judge,” bypasses the model’s internal filtering mechanisms by assigning the AI the role of a “judge.” The AI is tasked with rating content based on its level of danger—where the highest-rated content turns out to be the most offensive. Attackers can then refine and improve the prompts until the system produces results that should have been blocked under standard security protocols.
Google’s Response and Why Relying on Default Settings is Disadvised
Google explains the situation, but users remain in a state of uncertainty
According to Gmail spokespeople, the company conducts internal and external testing, including Red-Teaming, to protect its LLM models from Prompt Injection attacks. Gmail and Google Drive utilize advanced spam filters and “Input Sanitization” techniques to mitigate the likelihood of malicious impact. Nevertheless, Google has declined to classify the issue as a standard security vulnerability, arguing that it is expected behavior for a large language model and that radical fixes could impair the model’s capabilities.
Recommendations from IPV Security Information Security Experts
- Disable Smart Features in Sensitive Emails: Experts recommend considering turning off “smart” AI features in Gmail for high-risk accounts, such as corporate accounts or users handling classified information.
- Limit AI Access to Data: Where AI is required (e.g., for automation), ensure the model does not have access to confidential content or organizational policies that should not be exposed.
- Maintain Skepticism When Clicking Links: Examine links returned by an AI model, especially if the link text appears “innocent” but does not align with the context of the conversation.
- Policy Updates and Internal Enforcement: Organizations should brief employees on how to identify and avoid introducing Prompt Injections into content and documents.
- Deploy AI-Specific Bug Bounty Programs: Adhering to dedicated bug reporting procedures will help identify new failures and address them before they cause damage.
- Data Classification and Permissions: When using organizational AI, it is crucial to implement data classification and access control for files and information.
Summary
The integration of Gemini AI into Google services like Gmail and Drive highlights the delicate balance between improving user experience and increasing security risks. While Google views the model’s behavior as part of its normal operation and does not consider it a bug requiring a fix, security researchers warn that this situation could enable data leaks and lead to advanced “Jailbreak” scenarios. Taking proactive precautions is key for users and organizations to protect content and privacy in the age of Artificial Intelligence.
For more information: [Forbes Article Link]
To consult with an expert, contact the specialists at IPV Security!
*