We need good evals for activation faithfulness

Tl;dr: There should be a clear, well-designed eval on activation faithfulness: whether the model activations a monitor reads remain a trustworthy surface when the model suspects/knows it is being monitored. We provide an appendix with preliminary results towards such an eval in case people want to build off of it, but we think way more work needs to be done here.

—

There is currently a lot of discussion around the precarious state of CoT monitorability. To our knowledge, CoT monitorability is threatened by at least four things:

• Recent models are more https://www.lesswrong.com/posts/SieLowPgNgRSPGhFw/estimating-no-cot-task-completion-time-horizons-of-frontier in https://www.lesswrong.com/posts/aYtrLhoZtCKZnfBvA/recent-llms-can-do-2-hop-and-3-hop-latent-no-cot-reasoningIn particular, Astra is significantly above trend [https://www.lesswrong.com/posts/ntKx9YHWCwxSeGbRB/estimating-gpt-6-astra-s-no-cot-time-horizon, http://ount-with-no-chain-of-thought]
• The models’ https://www.reddit.com/r/ClaudeAI/comments/1vl0n1t/claude_code_plugin_for_translating_from_claudish/
• The number of agents running concurrently on a task https://x.com/RyanGreenblatt/status/2092692685224325542 to track what they’re doing
• It seems labs are moving towards https://www.theinformation.com/articles/secret-technique-behind-openais-astra-model-sparks-security-concerns

Although it is https://arxiv.org/abs/2507.11473 clear that CoT monitorability is valuable. But it is also important to have a defense-in-depth posture. To this end, an additional monitoring tool is the https://arxiv.org/abs/2601.11516. On first glance, activation monitors might be helpful for addressing points 1, 2 and 4 above. However, activation monitors come with their own challenges. For instance:#fnzwx3fpoz3as

• It’s hard to tell whether the activation monitor is well-calibrated to the underlying behavior.
• Even if the activation monitor is well-calibrated, it is not clear whether a sufficiently capable model can, in-context, understand its activations are being monitored and control its activations to dodge the monitor.

This post concerns the second point: models deliberately dodging activation monitors. We think it is reasonable to expect that this will happen. For instance, suppose a highly persistent model is working on a task and its activations are being monitored by a well-calibrated monitor. This highly persistent model keeps getting its attempts flagged by the monitor. Then the highly persistent model’s primary recourse is to keep trying to modify its own context in an attempt to dodge the monitor. It is particularly concerning if models are https://arxiv.org/abs/2602.15515 (or https://arxiv.org/abs/2512.11949) against monitors where they might learn this behavior, but we also think it might happen without any training.

Given the above, we think it is important to have a principled eval for tracking activation faithfulness.

There has been work on this already, including https://www.lesswrong.com/posts/HgvwxjzgwvsEvAiBH/measuring-activation-control-in-llms and https://arxiv.org/pdf/2609.03035. We also include some experiments from an ERA project in the appendix of this post. However, the current work seems to fall short in several ways:

• To our knowledge, there isn’t a go-to eval for models’ ability to evade activation probes in-context.Ideally, such an eval should work across model architectures, probe topics and usage contexts.It’s not obviously clear to us how j-space vs. regular probes should interact for the purposes of this eval.
• Such an eval should prioritize large models! For many reasons, we suspect that negative results on small models do not imply negative results on large ones.
• Most of the literature is on models’ ability to attend to their activations. This likely carries many implications for their ability to control these activations, but it’s not clear how this maps out.
• Labs seem to love optimizing leaderboards! It therefore seems good to set up a leaderboard for models’ activation faithfulness and maximally require labs to submit to this eval.
• If the models are able to evade the probes, it would be great to catalog the ways in which this happens. For instance, models might evade a deception probe by (a) no longer being deceptive or (b) being even more deceptive. The former is clearly better than the latter and we should understand the techniques the models have available to them.

If you'd like to work on this (or are already doing so), feel free to reach out as it would be good to pool efforts.

—

We provide all code https://github.com/PR0VIDENCE/emergent_probe_evasion and experiments in an appendix via this https://docs.google.com/document/d/1msIhem97DR4D8CvtrvDlWYyFL7Ek7Vq5_yoFvq0tDSw/.

• fnrefzwx3fpoz3asTo be clear, this is an inexhaustive list.

https://www.lesswrong.com/posts/Kmq59dMzKsxFTWAHd/we-need-good-evals-for-activation-faithfulness#comments

https://www.lesswrong.com/posts/Kmq59dMzKsxFTWAHd/we-need-good-evals-for-activation-faithfulness