Who checks the AI labs? The rise of independent evaluators
Outside groups are getting deeper access to check the work of AI labs. On Oct. 15, 2026, Anthropic said it had given six employees of METR, a nonprofit that evaluates AI models, "employee-like access" inside the company, with the right to publish what they find. The move came as METR, Britain's AI Security Institute and others published a wave of reviews of Anthropic and OpenAI.
What is an independent evaluator, in one sentence?
An independent evaluator is an outside organization that tests an AI company's models or safety claims and publishes its own conclusions. Anthropic's Oct. 15 post lists what it expects of one: expertise, no equity in or financial dependence on the company, enough access and freedom to publish.
How do the evaluators get access?
Anthropic describes three tiers. In the first, Anthropic decides what access to give, and findings are published with its approval. In the second, evaluators "can publish without Anthropic's approval," though Anthropic can still require the removal of some categories of information. The third adds employee-like access; METR is the first evaluator there.
The six METR staff have company hardware, office space and access to internal tools "roughly on par" with Anthropic's own risk teams, the post says. "Anthropic can’t veto publication," but it can require redactions in predefined categories, such as confidential intellectual property.
What have they found so far?
Recent independent reviews of AI labs, September to November 2026
| Evaluator | What it reviewed | Published | Access |
|---|---|---|---|
| METR | Claude Opus 5.5 before release (Anthropic) | Sept. 22, 2026 | API access over 10 business days, unpaid |
| METR | Misalignment section of Anthropic's August 2026 Risk Report | Oct. 20, 2026 | Unredacted report sections; Anthropic could veto |
| METR and Redwood Research | OpenAI's internal agent monitoring | Oct. 22, 2026 | Three weeks onsite, internal code; OpenAI could redact |
| METR | Automated R&D section of Anthropic's August 2026 Risk Report | Nov. 5, 2026 | Not described in this table's sources |
| UK AISI | More than 55 models before release | As of Oct. 12, 2026 | Pre-release testing |
In a review published Oct. 20, METR agreed that the risk of misalignment in high-stakes settings from Anthropic's models, meaning models working against what their developers intended, is low, but said this is "primarily due to models' limited capabilities, rather than safeguards and alignment training." It called Anthropic's risk report "far from adequate for confidently ruling out misalignment." That review was done before the new arrangement: Anthropic had the right to veto or redact it, and METR says no redactions were made.
In a separate exercise published Oct. 22, a METR employee and a Redwood Research employee spent three weeks inside OpenAI testing its system for monitoring its own AI agents. They estimated that roughly 75% to 94% of internal agent traffic from models at least as strong as GPT-5.5 was eligible for monitoring at the time, with substantial uncertainty, and reported a way agents could avoid having hundreds of actions monitored.
Earlier, METR's pre-release evaluation of Claude Opus 5.5, published Sept. 22, ran on API access over 10 business days under an unpaid agreement, and Anthropic reviewed and edited the summary before METR signed off.
What the arrangements don't say
The Oct. 15 Anthropic post does not say whether METR is paid for the embedded work. The OpenAI exercise was designed "not to provide robust public transparency," and OpenAI could redact anything it considered confidential. Anthropic also says it "can’t publicly share all the information relevant to our safety work."
Context: governments are testing too
Britain's AI Security Institute (AISI) says it has "independently evaluated more than 55 models before release," in an Oct. 12 post that lists "enabling independent auditors and evaluators" among its priorities. AISI adds that it "is not seeking to mandate requirements for frontier developers."
METR published another review of Anthropic's August 2026 Risk Report, on its automated research section, on Nov. 5, 2026.
FAQ
What is METR?
METR is a nonprofit that evaluates frontier AI models. Since October 2026, six of its staff have employee-like access inside Anthropic.
Can Anthropic block what METR publishes?
Under the arrangement announced Oct. 15, 2026, Anthropic can't veto publication but can require redactions in predefined categories, such as confidential intellectual property.
Does anyone independently test OpenAI?
Yes. METR and Redwood Research spent three weeks in 2026 testing OpenAI's internal agent monitoring and published a summary on Oct. 22, 2026.
How many models has the UK AI Security Institute tested?
More than 55 models before release, the institute said on Oct. 12, 2026.
Sources
- Anthropic, An update on independent evaluators (Oct. 15, 2026) · archived copy
- METR, Review of the misalignment section in Anthropic's Risk Report (Oct. 20, 2026) · archived copy
- METR, Red-Teaming OpenAI's Internal Agent Monitoring Systems (Oct. 22, 2026) · archived copy
- METR, Summary of METR's predeployment evaluation of Claude Opus 5.5 (Sept. 22, 2026) · archived copy
- UK AI Security Institute, Towards technical best practice for frontier AI development (Oct. 12, 2026) · archived copy
- METR, Review of the Risks from automated R&D section in Anthropic's Risk Report (Nov. 5, 2026) · archived copy
Cite this
Who checks the AI labs? The rise of independent evaluators, New Mind Museum, November 6, 2026. https://newmindmuseum.com/safety/2026/11/independent-ai-evaluators-metr-anthropic-openai-aisi/
New Mind Museum. (2026, November 6). Who checks the AI labs? The rise of independent evaluators. https://newmindmuseum.com/safety/2026/11/independent-ai-evaluators-metr-anthropic-openai-aisi/
“Who checks the AI labs? The rise of independent evaluators.” New Mind Museum, 6 Nov. 2026, https://newmindmuseum.com/safety/2026/11/independent-ai-evaluators-metr-anthropic-openai-aisi/.
@misc{new-mind-museum-2026-who-checks-the-ai-labs-the-rise-of-indep,
author = {{New Mind Museum}},
title = {Who checks the AI labs? The rise of independent evaluators},
year = {2026},
url = {https://newmindmuseum.com/safety/2026/11/independent-ai-evaluators-metr-anthropic-openai-aisi/}
}Free to quote and cite with a link to this page.
Keep reading
Sources
- Anthropic, An update on independent evaluators (Oct. 15, 2026)
- METR, Review of the misalignment section in Anthropic's Risk Report (Oct. 20, 2026)
- METR, Red-Teaming OpenAI's Internal Agent Monitoring Systems (Oct. 22, 2026)
- METR, Summary of METR's predeployment evaluation of Claude Opus 5.5 (Sept. 22, 2026)
- UK AI Security Institute, Towards technical best practice for frontier AI development (Oct. 12, 2026)
- METR, Review of the Risks from automated R&D section in Anthropic's Risk Report (Nov. 5, 2026)