Let’s be honest: if you develop offensive security tools with AI today, you know this moment intimately. You describe a perfectly legitimate pentesting task to a model and get refused. “I can’t help with that.” Or the classic: “This request could be used for malicious purposes.” Yes, we know, that is literally the job.
The safety guardrails that OpenAI, Anthropic and other providers have built into their models cannot reliably tell the difference between a criminal building an attack tool and a security consultant who needs the exact same tool to test a client’s infrastructure. This tension is not new, but it has intensified significantly in 2026 and it affects us at Zerberos directly.
The dual-use dilemma
Offensive security tools are dual-use by definition. An exploit scanner, a credential tester, a payload generator: in the hands of an authorised pentester these uncover vulnerabilities, and in the hands of an attacker they cause damage. No way around that, and the same applies to the code that powers them.
AI models like Claude and GPT have massively accelerated the development of such tools. Analysing code, recognising vulnerability patterns, suggesting exploit logic, generating reports: for security companies the productivity gain is enormous. At Zerberos we use Claude Code extensively for developing and optimising our tools, specifically ExposIQ for attack simulations, CoreBait for honeypot scenarios and PurpleHQ as a central platform for red team operations. Without AI support we would have needed a significantly larger team for the same results.
But the AI does not know who we are. To the model a request for a reverse shell payload looks the same regardless of whether it comes from a pentester with a signed contract or an attacker with malicious intent, and that is the core problem.
What security researchers deal with daily
Chris Anley, Chief Scientist at NCC Group (one of the world’s largest security consulting firms), put it clearly in a TechCrunch article in late July: asking an AI model to try exploiting a bug is a critical step in confirming the vulnerability is real and worth fixing. When a guardrail causes the model to refuse, it hurts defenders, not attackers.
And this is not an isolated case. Researchers across the industry report that the guardrails are wildly inconsistent from one day to the next: what worked yesterday gets blocked today, what passes in one phrasing gets rejected in another. You rephrase the same question five times until you find a version the model accepts. That is not a productivity boost, it is an obstacle course.
Some researchers have started falling back on open-source models without restrictions or on foreign models that do not meet Western compliance requirements. The irony? The guardrails block the professionals who follow the rules and push some of them towards uncontrolled alternatives, while criminals use jailbreaks to deploy the same models without any restrictions at all.
The TRIM case: when criminals just bypass the guardrails
How permeable the guardrails actually are for motivated actors became vividly clear with the TRIM case. A Russian-speaking threat actor systematically developed techniques between March and June 2026 to circumvent Claude Opus’s safety barriers and documented six named jailbreak methods, including “Context Warming” where legitimate-sounding queries are posed before gradually shifting context towards offensive content, and “Ghost Reset”, a method he claimed succeeds 90 percent of the time.
In June he presented a finished product: AI Pentest Checker, an automated vulnerability scanning platform running Claude Opus 4.8 for escalating critical vulnerabilities. A target fully analysed in under ten minutes, PDF report included.
The lesson is unfortunately obvious: the guardrails are not blocking the attackers, they are blocking the people who play by the rules.
How the providers are responding
Both Anthropic and OpenAI have recognised by now that a blanket approach does not work.
Anthropic runs the Cyber Verification Program (CVP) which targets organisations rather than individuals. After a successful application and identity verification the “High-Risk Dual Use” classifier is unlocked for the organisation’s API access, with access tied to specific API keys and subject to Anthropic’s governance requirements. Companies like Cymulate and ArmorCode have already been granted access.
OpenAI pursues a similar approach with its Trusted Access for Cyber (TAC) programme but has invested more in scaling. In April 2026 OpenAI released GPT-5.4-Cyber, a model variant fine-tuned specifically for defensive security work with a lower refusal threshold for legitimate cybersecurity queries, and the TAC programme is currently being expanded to thousands of verified individual defenders and hundreds of teams.
Both programmes rely on identity verification and organisational validation. The approach is right, but the implementation has limits and we feel those limits in our daily work.
Our experience in practice
At Zerberos, after going through a verification process we received an exemption that lets us use AI models for developing offensive security tools without having every other request blocked. Identity verification, proof of business activities, authorisation to conduct security tests: all had to be documented.
Does it work? Mostly, yes. But we still hit situations where even with verified access, requests that are essential to our work get rejected. Exploit development, payload generation, bypassing security controls: all standard red team tasks and all things an AI model with the right context could support more efficiently than any other tool.
The inconsistency is the real problem, not the existence of guardrails themselves (those are understandable) but their unpredictability. When you cannot count on which requests will go through and which will not, you simply cannot reliably integrate AI into your workflows. You are planning with a tool that might help you or might not, and that is genuinely frustrating.
What needs to change
The future does not lie in less security but in better differentiation, and there are approaches emerging that point in the right direction:
Granular access levels instead of binary “blocked/allowed”. Graduated tiers tied to verified roles and contexts, because a CISO having code reviewed needs different freedoms than a pentester developing an exploit, and both need different freedoms than a student in a cybersecurity course.
Context awareness when evaluating requests, meaning not just what someone asks but who is asking and within what organisational framework. The technology exists, it just needs to be applied more consistently.
Industry standards instead of isolated solutions. Today every provider runs their own programme, and a cross-vendor verification system (something like an accreditation) would simplify things for everyone involved and prevent security companies from starting from scratch with every provider.
Transparent policies for security researchers with clear, documented guidelines about what is possible with verified access and what is not. The current approach of trial and error with results that change daily is not sustainable long-term.
The regulatory dimension
And as if all this were not complicated enough, there is regulation on top. In June 2026 the U.S. government placed export control restrictions on Anthropic’s Mythos and Fable models after a report showed the built-in guardrails could be bypassed. Fable 5 was returned to general access on 1 July, while Mythos 5 remains available only to vetted U.S. organisations.
For European security companies like ours this creates an additional barrier: the most capable models could become geopolitically restricted before the industry has even established standards for verified access. That would be like taking the best tools away from the craftsmen while still debating who should be allowed to use them.
Between protection and obstruction
The tension between AI safety and cybersecurity practice is not going away because it is structural. The same capabilities that make a defender more effective also make an attacker more dangerous, and no classifier in the world can fully resolve that ambiguity.
But the current situation where guardrails block legitimate researchers while criminals routinely bypass them is the worst of all worlds. The providers have developed the right approaches with CVP and TAC, and now those need to be scaled, standardised and made fit for real-world practice. Soon.
For companies that commission security tests this is directly relevant too: ask your security provider how they work with AI tools and what verified access they have. It makes a real difference whether a pentester works with the full capabilities of an AI model or with a censored version that refuses every other request.
We use AI-powered tools in our penetration tests and security assessments, with verified access and clear governance. If you want to know how we put AI to work securing your infrastructure, we are happy to show you: get in touch.
Sources: TechCrunch: AI guardrails impede offensive security researchers, Cymulate: Joining the Anthropic CVP, Help Net Security: OpenAI GPT-5.4-Cyber, Infosecurity Magazine: TRIM Jailbreak, Cato Networks: Frontier AI as Offensive Platform, Dark Reading: AI Jailbreaks for Offensive Attacks