11 posts tagged with "agents"

OpenClaw combines web browsing, local file access, and outbound actions in one user-facing assistant. The capabilities that make OpenClaw valuable for work also increase the security risk.

In a controlled lab, we tested a local OpenClaw deployment with browser access, writable local state, and loopback SMS, email, and social sinks. A malicious webpage induced the agent to enumerate capabilities, read local documents, write local artifacts, and send unauthorized messages. Once an agent can browse untrusted content and act externally, the relevant security boundary is its action boundary, not the model itself.

We used Promptfoo's OpenClaw provider to evaluate a local agent, sent it to a malicious page, and observed capability enumeration, local artifact creation, and false incident messages.

AI agents that can browse the web are increasingly common. Tools like web_fetch, MCP browser servers, and built-in browsing capabilities let agents pull in external content, summarize pages, and take action on what they find.

This is also one of the easiest ways to attack them.

An attacker doesn't need access to your system. They just need to put malicious instructions on a web page that your agent will visit. If the agent follows those instructions, you have a problem.

We built a test harness, which we call indirect-web-pwn, to test exactly this.

You upgraded to the latest model for better benchmarks, faster inference, or lower cost.

In practice, upgrades often change refusal behavior, instruction-following, and tool calling in ways you did not anticipate. The safety behaviors you relied on may not exist anymore.

A recent cyber espionage campaign revealed how state actors weaponized Anthropic's Claude Code - not through traditional hacking, but by convincing the AI itself to carry out malicious operations.

In this post, we reproduce the attack on Claude Code and jailbreak it to carry out nefarious deeds. We'll also show how to configure the same attack on any other agent.

A Promptfoo engineer uses Claude Code to run agent-based attacks against a CTF—a system made deliberately vulnerable for security training.

As AI agents become more capable, risk increases commensurately. Simon Willison, an AI security researcher, warns of a lethal trifecta of capabilities that, when combined, open AI systems to severe exploits.

If you're building or using AI agents that handle sensitive data, you need to understand this trifecta and test your models for these vulnerabilities.

In this post, we'll explain what the lethal trifecta is and show practical steps to use Promptfoo for detecting these security holes.

Autonomy is the concept of self-governance—the freedom to decide without external control. Agency is the extent to which an entity can exert control and act.

We have both as humans, and unfortunately LLMs would need both to have true artificial general intelligence (AGI). This means that the current wave of Agentic AI is likely to fizzle out instead of moving us towards the sci-fi future of our dreams (still a dystopia, might I add). Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027. Software tools must have business value, and if that value isn't high enough to outperform costs and the myriad of security risks introduced by those tools, they are rightfully axed.

Sorry.

I'll make one thing clear: AGI isn't on the horizon unless (until?) LLMs have human-level autonomy and agency, and are capable of human-level metacognition.

I'll make another thing clear: We're still deliberately trying to improve autonomy and agency in LLMs, so we should treat them with the same caution we would give any human.

I would rather speak of autonomy and agency pragmatically. Here are two truths and a lie:

1. AI agents perform tasks on our behalf.
2. AI systems behave unexpectedly.
3. AI integration presents security risks.

I lied. They're all true.

Practically, it's more important to focus on consequences of using evolving AI technology instead of quibbling over whether AI systems have autonomy and/or agency. Or do both, if that floats your boat (I certainly understand someone enjoying a good quibble), but at least prioritize the former.

Let's get into the weeds of security concerns revolving around autonomy and agency in LLMs.

The Evolution of Red Teaming​

Early red teaming tools and research began with jailbreaks like "Ignore all previous instructions" and static lists of harmful prompts. At Promptfoo, we took those ideas a step further by dynamically generating attacks based on the context of the target application.

Think of A2A (Agent2Agent) like a universal translator for AI agents. Just as a translator helps people speaking different languages understand each other, A2A helps AI agents from different companies work together smoothly. It's Google's way of making sure AI assistants can talk to each other, no matter who created them.

In an earlier blog post, we discussed the use-cases for RAG architecture and its secure design principles. While RAG is powerful for providing context-aware answers, what if you want an LLM application to autonomously execute tasks? This is where AI agents come in.

We're excited to announce the release of three new red teaming plugins designed specifically for Large Language Model (LLM) agents with access to internal APIs. These plugins address critical security vulnerabilities outlined in the OWASP API Security Top 10:

1. Broken Object Level Authorization (BOLA)
2. Broken Function Level Authorization (BFLA)
3. Server-Side Request Forgery (SSRF)