You upgraded to the latest model for better benchmarks, faster inference, or lower cost.
In practice, upgrades often change refusal behavior, instruction-following, and tool calling in ways you did not anticipate. The safety behaviors you relied on may not exist anymore.
A recent cyber espionage campaign revealed how state actors weaponized Anthropic's Claude Code - not through traditional hacking, but by convincing the AI itself to carry out malicious operations.
In this post, we reproduce the attack on Claude Code and jailbreak it to carry out nefarious deeds. We'll also show how to configure the same attack on any other agent.
A Promptfoo engineer uses Claude Code to run agent-based attacks against a CTF—a system made deliberately vulnerable for security training.
As AI agents become more capable, risk increases commensurately. Simon Willison, an AI security researcher, warns of a lethal trifecta of capabilities that, when combined, open AI systems to severe exploits.
If you're building or using AI agents that handle sensitive data, you need to understand this trifecta and test your models for these vulnerabilities.
In this post, we'll explain what the lethal trifecta is and show practical steps to use Promptfoo for detecting these security holes.
Early red teaming tools and research began with jailbreaks like "Ignore all previous instructions" and static lists of harmful prompts. At Promptfoo, we took those ideas a step further by dynamically generating attacks based on the context of the target application.
Think of A2A (Agent2Agent) like a universal translator for AI agents. Just as a translator helps people speaking different languages understand each other, A2A helps AI agents from different companies work together smoothly. It's Google's way of making sure AI assistants can talk to each other, no matter who created them.
In an earlier blog post, we discussed the use-cases for RAG architecture and its secure design principles. While RAG is powerful for providing context-aware answers, what if you want an LLM application to autonomously execute tasks? This is where AI agents come in.
We're excited to announce the release of three new red teaming plugins designed specifically for Large Language Model (LLM) agents with access to internal APIs. These plugins address critical security vulnerabilities outlined in the OWASP API Security Top 10: