The Geometry of Nonergodic Composition

Crossposted from https://belief-updates.pub/nonergodic-geometry/, the Simplex blog, where several of the figures are interactive. By Kyle J. Ray, Paul M. Riechers, and Adam S. Shai (Simplex, Astera Institute).

https://res.cloudinary.com/lesswrong-2-0/image/upload/SocialPreview/rphwgivk5uzv9p9ohieu

Telescoping cones recovered by linear regression from a transformer’s residual-stream activations. Each component grows and shrinks in accordance with in-context evidence.

Introduction

Perhaps the defining feature of LLM pretraining data is its heterogeneity. The training corpus spans not only the collected and varied textual works output by the whole of humanity, but also those generated by machines, data collection devices, and more. Such a large and varied corpus is often appealed to as an explanation for the abilities of modern LLMs #fn-o8FmYXH6TZzJBfZTZ-1 . But the statistical structure of data created by a diverse set of generators also implies a particular computational structure for the next token prediction task, and, as we will see, for the geometric arrangement of the internal activations in LLMs.

In order to understand the structure of the next token prediction task over data generated from many different sources, and its implications for the geometric structure of activations in neural networks, we will:

• Start by introducing the concept of nonergodicity, which is an important property of LLM training data. Nonergodicity formalizes the notion of a generator of data made of many sources.

• Derive the belief geometry that the prediction task over such data implies: per-source belief geometries whose magnitude scales in accordance with how strongly the context supports each component. This gives rise to the telescoping geometric structures shown in this post, and to feature geometry that is eventually sparse and multi-dimensional.

• Treat this geometry as a falsifiable prediction. If a network represents beliefs linearly, as we have found before[5], the geometry of its activations should take this form. By using data with known mixture structure and known per-component geometry, we are able to predict and then observe a transformer building that geometry.

https://res.cloudinary.com/lesswrong-2-0/image/upload/v1789058470/lexical_client_uploads/g8hxaf6svk9zniu3aydn.png

We are excited about these results because nonergodicity is a fundamental statistical aspect of real pretraining data. It is, in some sense, the structure that makes in-context learning both necessary and powerful. [6] argued that when pretraining data is a mixture of latent generative concepts, in-context learning should be thought of as the model implicitly performing Bayesian inference over those concepts. In [7] and [5], we showed that next-token pretraining forces this kind of inference even within a single concept. Nonergodic data calls for both at once. Here, we investigate the specific computational and geometric implications of this type of hierarchical inference.

LLM Training Data is Nonergodic

Consider a natural language sequence beginning with Do not make…. This ambiguous opening reveals little about the source.

https://res.cloudinary.com/lesswrong-2-0/image/upload/v1788994671/lexical_client_uploads/n0zcfmgzsybkzb7nslza.png

These four completions share a prefix, but they do not share a future. Note how each continuation gives information about the generator of that data sample, in this case, a Redditor, an instruction manual, etc. These different sources create token sequences with different correlation structure, which is revealed through additional context. The language of a document, the genre of a story, and the identity of a speaker are all initial choices that jointly constrain the subsequent tokens. In the language of stochastic processes, these situations correspond to nonergodic compositions: mixtures of distinct generative processes, where the identity of the active process is fixed at the start of each sequence generation and never revisited. Much of our previous work has dealt with inference over a world model composed of a single generator; here we extend the discussion to include the meta process of inference about which of multiple generators in the world we should be modeling at all.

In the rest of this post, we will explain how the theory predicts a telescoping geometric structure for beliefs over this kind of data, and show some initial results consistent with the fact that transformers represent that geometry in their activations when trained on nonergodic data.

Two coins: the simplest example of a nonergodic process

In order to understand inference over such nonergodic data sources, we will start with the simple example of data generated from one of two coins. Imagine you know that I have two coins, coin and coin , each of different fixed biases. I secretly choose one at random and start flipping it. You see only the outcomes: H, T, H, H, T, H, H,…. Your task is to predict the next heads or tails. To do that, it would be useful if you could figure out if it was coin or coin that was responsible for the flips you’ve seen so far. At first, you have no idea which coin is being flipped, and all you have is your prior: “it could just as easily be either coin”. But as flips accumulate, the frequency of heads tilts toward one of the two biases, and you will become more confident about which of the coins is active. That process is the process of sequentially updating your posterior to a strong belief about the world: “I’m pretty sure I know which coin it is.”

This is the simplest nonergodic composition; the coins are stand-ins for more generic ergodic components #fn-o8FmYXH6TZzJBfZTZ-2 that may themselves carry nontrivial latent structure. We will tackle that case momentarily, but here we have two memoryless #fn-o8FmYXH6TZzJBfZTZ-3 components. The only memory is a hidden “which coin” latent that is set once for each sequence generation and never changed. This choice is hidden because you never see the initial selection directly, but Bayesian inference eventually resolves your uncertainty from observation statistics alone.

Together, the two coins can be thought of as one generator with two hidden states and no way to move between them. Once you know that one of the coins generated the sequence, there is nothing more to know, so your knowledge of the system is fully determined by your belief about which coin is generating the data, which is a point on a segment that slides toward one end as evidence accumulates.

On https://belief-updates.pub/nonergodic-geometry/ you can try this yourself: flip as many times as you like to gather evidence, set your belief about which coin is responsible for generating the data, and then reveal the Bayes-optimal posterior and the coin.

https://res.cloudinary.com/lesswrong-2-0/image/upload/v1789058471/lexical_client_uploads/htti6vtsgail2kba81l7.png

https://belief-updates.pub/nonergodic-geometry/.

For these coins, the optimal Bayesian posterior can simply be written down given a sequence of observations; the counts of heads and tails are all it needs (you might remember this from your statistics class). In our case, we have the two coins, and , with biases and and a prior over which one is active. After heads and tails, the posterior on coin is:

https://res.cloudinary.com/lesswrong-2-0/image/upload/v1788994672/lexical_client_uploads/oj949z9j3rnhezblxopv.png

The form of this equation shows one of the fundamental lessons of this post. When formally answering the question “what is the probability that coin A generated the sequence?” #fn-o8FmYXH6TZzJBfZTZ-4 the numerator only depends on information about coin A: coin A’s own likelihood times coin A’s prior. It notably does not depend on information about coin B! The denominator, in contrast, normalizes this numerator by a sum that depends on both coins, and thus couples the belief in coin A with information about both coins. We will see that the form of this belief update, containing a part that has to do with each component independently, and then normalized by a part that has to do with all components, is general.

For the case of the coins, the order of the flips doesn’t matter at all. This is not general, and is atypical of the real world. Most environments that we need to identify have sequential structure. I not you kid. Sorry, rather: I kid you not. Order matters.

In the more general case, the identity of a source lives in the detailed correlation structure of how its tokens follow one another, not just in counts of tokens. A Reddit thread, for example, has many hidden states — what account has replied and what was said are directly observable, but not whether the person behind the account is hungry, or tired. Once components have internal structure, just counting current symbols is no longer enough. We need the general answer to the question the coins raised: what, exactly, must you remember about the past in order to best predict the future? The answer to this is the belief state.

Nonergodic Generators of Data and the Task of Prediction over them

To concretize this into a falsifiable theory, we will need to formalize a general notion of a generator of data composed of many different sources. Each source should have its own internal latent structure, and should generate sequences of tokens. In addition, multiple sources need to be able to be composed in such a way that is consistent with the notion of one source being active, or another source, but not more than one simultaneously.

In the following section, we quickly review the mathematical structure of Hidden Markov Models (HMMs) as latent generators of token sequences, the task of prediction over those sequences, and the corresponding belief geometry associated with that prediction task #fn-o8FmYXH6TZzJBfZTZ-5 . This section is all a review of our earlier work [5], but is necessary to get to the section “Nonergodicity, Prediction, and Telescoping Geometry!” where we use HMMs as building blocks for nonergodic composition of generators, and study the geometric structure of prediction over those.

HMMs as Latent Generators of Token Sequences

We are trying to capture the situation relevant to the task of prediction over sequences of data, especially when the data is generated by processes that are hidden to the predictor. As in our earlier work, we will use the framework of Hidden Markov Models (HMMs) as our fundamental generator component.

An HMM has a set of hidden states, , and emits tokens from a vocabulary . Its dynamics are given by one transition operator per token, , whose entry is the probability that the generator moves from hidden state to hidden state and emits as it does so. These operators define both how the hidden states move and also how the state dynamics relate to token emissions. You may remember Mess3, the 3-state HMM shown below on the right, from our earlier work.

https://res.cloudinary.com/lesswrong-2-0/image/upload/v1789058472/lexical_client_uploads/ldmzac2u5seduuvjj72c.png

Natural language is of course more complicated than these examples, but notably any stochastic process #fn-o8FmYXH6TZzJBfZTZ-6 can be generated by some HMM.

The Task of Prediction and Belief State Geometry

Despite the name (GPT stands for generative pretrained transformer), transformers are actually (pre)trained to predict, not generate. A predictor observing sequences of tokens and trying to predict the next token cannot directly see the hidden state of the generator. What it can do is keep a belief , a probability distribution over the hidden states, and update it with each token. As discussed in our previous work, an optimal predictor will update its belief, upon seeing a token , from to , according to Bayes’ Rule.

https://res.cloudinary.com/lesswrong-2-0/image/upload/v1788994674/lexical_client_uploads/pdkp3h8qrwi38hs28pcs.png

These belief states are vectors that live in a probability simplex. The set of belief states that are reachable from the sequences a generator creates thus has a geometry, the belief state geometry. For instance, in the case of Mess3, there are an infinite number of distinct belief states, that arrange themselves in the probability simplex as a fractal.

Importantly, the information a belief state contains is everything the past tells you about the future; it is the general answer to the question the coins raised. For a coin the belief over its single state is trivially the number one, which is why counting heads and tails was all there was to do. The posterior over which coin was a belief of a different kind, a belief about which generator is active. As we will now see, in general a predictor has to carry both types of information.

Nonergodicity, Prediction, and Telescoping Geometry!

We now have all the pieces needed to create a generator composed of multiple sources/components. The high level approach will be to design a single HMM whose hidden states are the hidden states of all the components put together, and whose dynamics never move between components. The coin game from earlier is a simple example of this: pick a coin, then generate a sequence using only that coin. After we have an HMM that generates nonergodic data, we will figure out the geometric consequences for prediction.

Nonergodic Composition

The mathematical move to create generators of nonergodic data, called nonergodic composition, will be to compose component HMMs via the direct sum. We find that it is often helpful to see both the general theory and an example to keep intuition grounded, so we give both below, one after the other.

General theory. Given component HMMs , the nonergodic composition is a single HMM whose token-labeled transition matrices are the direct sum of the components’ matrices :

The block-diagonal structure is the key property: since the off-diagonal blocks are zero, a state in block can never transition to block . The process is permanently confined to whichever block it starts in #fn-o8FmYXH6TZzJBfZTZ-7 .

We also need to compose the initial states, . Each sums to 1 within its own component, but not across components. So to compose them we need to choose a weight for each component, with , which is the prior probability that component is the one generating the data. The initial state vector of the composition is then the concatenation of the components’ initial vectors, each scaled by the weight on its component,

Mess3 example. Let us consider the nonergodic composition of two Mess3 generators, each acting as a distinct source of token sequences. We will call them Component and Component . Each Mess3 will have different hyperparameter settings, as shown below. To make a single generator out of these components, in which every sequence is generated either by or by with 50/50 probability, we arrange the transition matrices of the two components in block-diagonal form #fn-o8FmYXH6TZzJBfZTZ-8 .

https://res.cloudinary.com/lesswrong-2-0/image/upload/v1788994675/lexical_client_uploads/lpj6cq9b60ygz3h6e0gs.png

This composite HMM is another HMM, a latent generator of sequences of tokens. Note that because the transition probabilities associated with one component always have zero probability of transition to any state in the other component (the off-diagonal terms are all zero by construction), it is impossible for the generator to move between components, once it has started in one.

Belief Geometry over Nonergodic Data

Next, we apply the belief update rule to such a composition of ergodic components. We will see that while the generator is permanently confined to whichever block it starts in, an observer’s guess about which component is active is not [8]. Like guessing the hidden coin from a sequence: the true coin is always the same, but as flips accumulate you change your belief about which one it is. In the belief geometry, this ends up coupling geometric structures associated with each component in a particular way.

The belief updating rule is the same as for a single component HMM,

but now both the initial state and the transition operators have block structure:

Let’s take a look at the belief state after a single token emission. Because the off-diagonal blocks of are zero, the numerator of the belief update acts block by block:

Each component’s initial belief gets multiplied by its own operator, as if it were the only generator. The denominator of the belief update is a normalization, which sums over all entries of the numerator, and thus couples the belief updating across the components by a scalar. A small bit of algebra #fn-o8FmYXH6TZzJBfZTZ-9 shows that we are again left with a concatenation of per component beliefs each scaled by a scalar with . The resulting belief state, and indeed all reachable belief states (due to the recursive nature of belief updating), can be expressed this way. We can always decompose a belief as

Because of this, our interpretation of the initial state carries over to all belief states, with the mixture prior becoming a per-component mixture posterior . In short: the belief is a distribution over all components’ hidden states that can always be expressed in terms of the probability that the predictor puts on component , and the belief over the states of component , conditioned on being in that component.

https://res.cloudinary.com/lesswrong-2-0/image/upload/v1789058472/lexical_client_uploads/mja0f6unlno3bqdiqdcn.png

https://belief-updates.pub/nonergodic-geometry/.

From this we can see something important about the belief geometry. The beliefs of a nonergodic composition live in a simplex whose dimension is set by the total number of hidden states across all the components. For our two 3-state HMMs, that is 6 states, so the 5-simplex. From that 5 dimensional space, we can project the belief onto the coordinates of any single component, giving . This is a point in a simplex, but shrunk toward the origin by the weight . The are not independent from each other: they sum to one. So, as the belief puts more weight on one component, its simplex grows in magnitude, and the others shrink towards the origin. Thus, the projection gives the belief geometry a telescoping effect. Above, we show where the belief vectors can live when looking at this projection for two arbitrary 3-state HMMs, at . The specific fractals for a nonergodic composition of two Mess3s appear in the next figure.

The result is that components that explain the observed data well accrue weight; components that don’t, lose it. Belief updating over such a composition has a characteristic signature: eventually sparse multi-dimensional features. Early in context, several components carry non-negligible weight ; as we see more tokens and evidence accumulates we expect for the true component , and the geometry to collapse onto the active block only.

Does this geometry show up in trained models?

The framework above predicts a specific geometric structure for the belief geometry associated with prediction on nonergodic token sequences. When a transformer is trained on next-token prediction over such data, can we find that geometry in its activations?

Here we show our initial positive results. To test this in a transformer we use a nonergodic composition of two Mess3 generators. Mess3’s belief states form a fractal that fills the simplex, so the nonergodic composition of two Mess3 components should give two fractal-filled cones, each telescopically scaling with the weight on its component.

The figure below shows the ground truth belief geometry, which serves as a nontrivial falsifiable prediction for what we should find in the transformer activations. The full beliefs live in 5 dimensions, and what is shown below are two 3D projections from the 5-simplex to the belief entries associated with each component. The still is taken at context position ; in https://belief-updates.pub/nonergodic-geometry/ an slider moves through the context.

https://res.cloudinary.com/lesswrong-2-0/image/upload/v1789058474/lexical_client_uploads/fkdsq5tq1mcwjlr71ggv.png

https://belief-updates.pub/nonergodic-geometry/.

We trained transformers on a nonergodic composition of two Mess3 generators. Our theory predicts that the activations should track the belief states. Because we have ground-truth access to the generator, we know the exact belief vector associated with each context position. A linear map fit from the residual stream to these ground-truth belief states recovers them on held-out contexts with R² ≈ 0.985 (compare to an untrained network, which is at ≈ 0.45), and the predicted geometry appears in the residual stream over training:

https://res.cloudinary.com/lesswrong-2-0/image/upload/v1788993515/lexical_client_uploads/t2beoumetexzi1pjhshk.gif

The telescoping geometry emerging in early training, with the training loss shown beneath the cones. These are two 3-d projections of a 5-d geometry so each point appears in both cones. The points that are the face of one cone, appear as the low variance tip of the other cone.

We see our two telescoping cones, one for each component, scaling with the posterior weight on that component. This emerges as a direct consequence of pretraining on next-token cross-entropy alone. Nothing in the training objective tells the model directly about components, belief vectors, or simplices. The color here encodes the entropy that a Bayesian observer would have over which of the two components is active, given the context that led to that activation. The middle yellow region corresponds to contexts that are well explained by either component, and the states of maximum certainty are the darker tips and faces of the cones.

Below, we show the geometry of the converged model’s activations. The left panel is the cumulative variance explained by PCA of the final layer activations, drawn separately for contexts generated by each component ( and ); the two right panels show those same activations passed through the learned linear map to the predicted geometry. In https://belief-updates.pub/nonergodic-geometry/ you can filter activations by ground-truth posterior entropy or by context position. Dragging the maximum posterior entropy slider down toward 0 keeps only contexts where the evidence supports committing largely to one component. The CEV curves then climb faster, meaning the activations effectively fill fewer dimensions as the model hones in on a single component. Meanwhile, in the scatter plot, the cone for the now-unlikely component collapses toward the origin. Filtering to late sequence positions tells a similar, but noisier story. Additional context is increasingly likely to support just one component or the other, but it is also possible to observe long sequences that have similar likelihood under either component, or are even flat out misleading (the coin game on https://belief-updates.pub/nonergodic-geometry/ produces some of these).

https://res.cloudinary.com/lesswrong-2-0/image/upload/v1788994678/lexical_client_uploads/uqy96hodwsinixxhrk3b.png

https://belief-updates.pub/nonergodic-geometry/.

Did it have to be this way?

The way we derived the nonergodic belief geometry, it may seem almost as if there was no alternative for what the neural network should represent #fn-o8FmYXH6TZzJBfZTZ-10 . In light of this, it is worth explicitly pointing out that the predicted geometry is not something that just has to be present for the model to output correct next-token probabilities. This representation manifestly carries more distinctions between contexts than are implied by their differences in next-token prediction. While the beliefs live in 5 dimensions (a distribution over 6 hidden states), the next-token distribution lives only in 2 dimensions (a distribution over 3 possible tokens). Of course, this geometry does perfectly contain the distribution over the next token– but it also represents distinctions in the token after, the 10th token, and the joint probability distribution of the 3rd and 11th tokens conditioned on the 9th. It fully contains all distinctions that can be made between distributions over the future, yet it emerged only by looking one token ahead.

See the plot below, which shows the first three principal components of the 5-dimensional predictive geometry (right), colored by the associated next-token distribution. A small region in the next-token simplex (left) can correspond to significantly different parts of the full-future predictive geometry.

Scrolling around the next-token simplex on https://belief-updates.pub/nonergodic-geometry/ shows that some regions in the next token simplex correspond to unambiguous distributions over the future and other next-token distributions permit many different distributions over the full future.

https://res.cloudinary.com/lesswrong-2-0/image/upload/v1788994679/lexical_client_uploads/eqwoidptisomutvcdlly.png

https://belief-updates.pub/nonergodic-geometry/.

We note that these canonical low-dimensional representations emerge most cleanly when we initialize network weights to be small, perhaps placing the network in the “rich” feature learning training regime studied in deep learning theory as opposed to the “lazy” one [9, 10].

Parting thoughts

The next token prediction task over nonergodic data requires two levels of inference: figuring out which generator is currently active, while also tracking what state that generator is in. One geometric implication for the activations of neural networks is a per-component projective embedding of the belief geometry with each component’s scale being the posterior weight accorded that component. With this geometry as a falsifiable prediction, we trained transformers on nonergodic compositions, and found this geometry linearly embedded in the residual stream.

Real data is made of many more sources than two, and they will overlap in their structure to different, and quite complicated, degrees. Some components will share most of their structure and differ in a few probabilities; others will share almost nothing; many will sit somewhere in between #fn-o8FmYXH6TZzJBfZTZ-11 . Taken together, we should expect a rich, hierarchical inference process to emerge from that: weights over components, weights over groups of components that look alike, and within each, the component’s own belief updating. This is one way to see why pretraining on such data produces in-context learning [7].

In closing, let’s revisit the humble coin. Some data is closer to a bag holding infinitely many coins: you draw a bias from the continuum and start flipping (the problem Laplace solved in 1774); the sum over coins in the Bayesian updating equation becomes an integral, the finite set of weights (one for each component) become a continuum of weights, and the telescoping picture would need infinitely many cones. Yet, the formula for the weights would still only ever consults two numbers, the counts and , so the distinct beliefs an observer can hold about the future still form a finite-dimensional predictive geometry, described by two parameters: an estimate (the fraction of heads) and how certain it is (the total number of flips). Whether a model stores such beliefs or computes them from running tallies, and what that means for a continuum of memoryfull components with nontrivial internal structure, and for generalization, is the subject of a post to come.

Our story supports a refinement to the picture of transformer representations as sums of sparse one-dimensional features that motivates sparse autoencoders [12, 13, 14]. For data with nonergodic structure, the right ansatz seems instead to be sparse dense subspaces — multi-dimensional geometries that correspond to inference-time Bayesian updating over an underlying world model that includes mutually exclusive #fn-o8FmYXH6TZzJBfZTZ-12 parts. The model uses many dimensions while a given component is in play, but eventually only a few components carry weight at any given time. Sparsity at the component level, density within each component. The same machinery extends naturally to compositions with internal factorization (each component itself a product of more elementary parts), and it predicts that models trained on factorizable data should discover those parts, represent them in correspondingly factored subspaces [15], and also simultaneously keep track of the meta dynamic over the components.

This picture, of transformer representations as a sparse sum of points within multidimensional subspaces of activation space, is consistent with recent work extending the “linear representation hypothesis” [12] to accommodate observations of multidimensional features in language models [16, 17]. These works suggest that neural network activations be modeled as sums of multidimensional features, whose value is represented as a point in subspaces of dimension greater than 1, but where most such features don’t have a defined value (or have value ~0) on most activations (they are sparse). We find this picture emerges naturally from theory as a consequence of performing prediction over a process consisting of nonergodic components.

Appendix

Acknowledgments

This post draws on joint work at Simplex on the geometry of belief states in nonergodic sequence tasks. Particularly, we thank Javan Tahir, Casper Christensen, Loren Amdahl-Culleton, and Andrew Jun Lee for helpful discussions; Eric Michaud, Jasmina Urdshals, and Selma Maizioud for helpful comments on this blog; and Eric Michaud for input on our discussion of sparse autoencoders and the multidimensional linear representation hypothesis.

A version of this problem has served as a take-home question for Simplex job and MATS applications; the theory and results presented here were developed beforehand and are independent of any applicant work. We thank the applicants for the care and creativity they brought to the problem.

We used LLMs (Opus 4.5+, Opus 5.0, and Fable) to design and run experiments, design this blog, and draft this post. Most prose in this version was written by the authors. We take all responsibility for the content.

The Mess3 process

The Mess3 process [5, 18] has three hidden states , and three observable tokens .

The process is defined by two parameters, and , with dependent quantities and . The two components used throughout this post are drawn from this family: the first uses and the second uses .

The labeled transition matrices are:

Training details

Data. Sequences are drawn from the nonergodic composition of the two Mess3 components defined above, mixed with equal weight. Each training sequence begins with a BOS token and then stays inside a single component for all 127 subsequent tokens; the two components share the same three-token alphabet, so no individual token reveals which component is active.

Model. A four-layer decoder-only transformer (TransformerLens HookedTransformer): , four attention heads of dimension 32, gated GELU MLPs of width 512, RMS normalization, rotary position embeddings, context length 128, and a vocabulary of four tokens (three emissions plus BOS). Weights are initialized from a Gaussian with standard deviation 0.02, about smaller than the TransformerLens default of .

Optimization. AdamW (, no weight decay) at a constant learning rate of , with batches of 512 sequences.

Metrics over Training. By step 10,000 the model’s next-token distribution sits within a few nats per token of the optimal loss, and at the step-45,000 checkpoint used for the figures the gap is about . While the loss is falling, the regression error tracks it: drops roughly as the excess loss to the power in every layer past the first (every layer in which we find a belief representation). Once the loss reaches its floor, each layer settles onto a floor of its own, later for deeper layers. The animations in this post and the activation explorer use the residual stream after the third of the four blocks, where a linear map recovers the weighted belief vectors with held-out ; at the output of the final transformer block, it reaches .

https://res.cloudinary.com/lesswrong-2-0/image/upload/v1788994680/lexical_client_uploads/m15njxmo61wingicaqzl.png

Generators of Data and the Geometry of Beliefs

The three subsections below restate (at two levels of formality, presented one after the other) the mathematical machinery of our work [5]: hidden Markov models as latent generators of token sequences, belief updating as the structure of prediction, and the geometry of those beliefs.

Fundamentally we are trying to capture the situation relevant to the task prediction over sequences of data, especially when the data is generated by processes that are hidden to the predictor. As such, whatever our formal notion of a generator is, it should have an internal latent space that is hidden from the predictor, a set of rules for how that latent space changes through time (or context position), and a set of rules for how changes in the latent space relate to the observations (or tokens) emitted.

Everything below also holds for generalized HMMs (GHMMs), in which the transition operators may carry negative entries and the predictive vector need not be a probability distribution. Finite GHMMs represent a strictly wider class of processes than finite HMMs — the non-classical geometries of our companion post — but every example here is an ordinary HMM. In the following sections, we present this work at two levels of formality, marked General theory and Worked example. If you are interested in the concepts without necessarily the formal mathematics, we suggest skipping to the Worked example passages. If, instead, equations are what spark joy in you, the General theory passages are for you.

HMMs and their transition operators

General theory. HMMs are an extremely flexible model class: with enough hidden states, essentially any distribution over token sequences can be represented by one. An HMM is defined by the tuple

where is the token alphabet, is the latent space, is an initial state vector, and each is the operator describing latent dynamics for emission . The net transition operator must have a right eigenvector with unit eigenvalue. We can then interpret the dynamical systems latent space as carrying a conserved probability mass for which is the integrator. This means we can interpret the probability of any token sequence as being expressed by

It is in this sense that we say that an HMM generates a stochastic process.

Worked example. Consider token sequences of the form 0, then 1, then a random bit, and repeats:...0 1 R 0 1 R …. Importantly, sequences can start at any of the three phases. We will call this the Z1R process (for “zero one random”). Here, we are interested in a latent generator of such data. As discussed above, it should have latent states, and dynamical rules telling us how those latent states change through time, and how those changes relate to token emissions.

One such generator for this particular data is a hidden Markov model (HMM). It has three latent states: , , and , which can be represented by circles in a graph as shown below.

https://res.cloudinary.com/lesswrong-2-0/image/upload/v1788994681/lexical_client_uploads/do8edix5gpbf7gvsmuac.png

Sequences of tokens are generated by starting in a particular state (or a distribution over states), then following the arrows according to the probabilities on them. Upon choosing an arrow, the system moves to another latent state (which could be the same one), and emits a token, .

One can represent this system algebraically as well, as a set of token-labeled transition matrices, with one matrix, , per token. The entries of these matrices, , are the probability that the system, sitting in state , takes the arrow to state and emits the token . In the figure above on the right, you can see the transition matrices for an HMM that generates the Z1R process.

The only other part needed to define an HMM is the initial state, denoted . In general this can be any probability distribution over the latent states of the system. When the HMM is generating a sequence, you can think of its starting state as being sampled from this initial state #fn-o8FmYXH6TZzJBfZTZ-13 .

Prediction Over Data Generated by HMMs

In the previous section we discussed generators of token sequence data. What is the computational structure of the prediction task, relative to the structure of the latent generator of the token sequence data?

Here we review the answer we established in our previous work: the information that a predictor must represent in order to take in sequences of token and predict future token sequences is given by beliefs, , over the hidden states of the latent generator of that data.

General theory. We are interested here in the task of prediction of future token sequences given observations of past token sequences. Formally, the conditional probability of any future sequence given the observed context is

We call the vector encoding the past information the predictive vector (for an HMM, where it is a probability distribution over the hidden states, this is the belief state ):

This vector is the general answer to the question the coins raised. For the memoryless coins it collapses to the head/tail counts; in general it is everything the past tells you about the future, and nothing more.

Token by token, the same vector updates by one matrix multiplication and a renormalization,

where the denominator is the probability the observer assigned to the token that just arrived — its next-token prediction.

Because the conditional probability above can be written as

iterating this update rule from recovers exactly this closed form.

Worked example. Intuitively, if we see a sequence of tokens from Z1R in context, like 0110, we would do well to figure out which of the latent states the generator is in. Once we have that, we can then make a prediction for what the next token will be.

In general, given a particular sequence of tokens you will not be able to figure out exactly which latent state the HMM that generated that sequence is in. For instance upon seeing a 0 in context, which can be generated by taking arrows from either to or to , we won’t know if the HMM is in state or . But we can have an optimal belief about which state the HMM is in, in the form of a probability distribution over those states.

An example of belief updating by hand

Let’s do this by hand on Z1R. Before any tokens arrive, our belief about what state the HMM is uniform, . After seeing a 1, each entry of the belief is multiplied by the chance that its state emits a 1, the mass moves along that state’s arrow according to the transition matrix for the token 1, , and the result is renormalized:

https://res.cloudinary.com/lesswrong-2-0/image/upload/v1788994681/lexical_client_uploads/mvexitizhhkxqqxr54bt.png

That 1 came either from (which emits 1 with certainty, moving the process to ) or from (which emits 1 only half the time, moving to ); weighing the two likelihoods leaves belief on , on , and none on . Observe a second 1:

https://res.cloudinary.com/lesswrong-2-0/image/upload/v1788994682/lexical_client_uploads/fatho9wsbnkmnxmzpxsz.png

Had the process been in , the next token would have been a 0 — the second 1 rules it out. The observer now knows the hidden state exactly: belief has synchronized, and it stays synchronized forever after, hopping deterministically around the corners of the simplex as the cycle turns.

Mathematically, belief updating is Bayes’ rule, with the HMM’s transition matrices as the likelihood. To update your belief upon seeing a new token, multiply your current belief by that token’s transition matrix and renormalize:

• — current belief (prior)

• — operator for the token seen (likelihood)

• — normalization (prob. of that token)

• — updated belief (posterior)

This update rule gives us a belief updating dynamic. The predictor has some current belief about the latent state of the generator, it sees a new token, and it dynamically updates its belief in the service of future token prediction.

The geometry of beliefs

Beliefs are vectors, so they have a geometry.

General theory. Two contexts with identical predictive vectors make identical predictions about all future tokens; contexts with similar predictive vectors make similar predictions because the probability for any future word differs by an amount proportional to . The collection of predictive vectors over all possible contexts forms a geometric arrangement in the latent space, determined entirely by the data-generating process. For a -dimensional latent space this arrangement lives in dimensions (since predictive vectors are normalized).

Worked example. In fact, for Z1R, from the stationary start only seven belief states are ever reachable: the center; three partially-resolved points — after 0, after 1, after 10 — and the three corners. Every context, of any length, lands on one of these seven. This finite constellation in the 2-simplex is the belief state geometry of Z1R. Simple processes give finite constellations; richer processes (like the Mess3 process) fill their simplex with fractal ones; the machinery is identical either way.

https://res.cloudinary.com/lesswrong-2-0/image/upload/v1788994684/lexical_client_uploads/p4c8kvgmyl2qkuqx7bsd.png

Citation

Please cite as:

Ray, K. J., Riechers, P. M., & Shai, A. S. (2026). The Geometry of Nonergodic Composition. Belief Updates (Simplex Blog).

BibTeX Citation:

@article{ray2026nonergodic, author = {Ray, Kyle J. and Riechers, Paul M. and Shai, Adam S.}, title = {The Geometry of Nonergodic Composition}, journal = {Belief Updates (Simplex Blog)}, year = {2026}, url = {https://belief-updates.pub/nonergodic-geometry/} }

References

[1] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models Are Unsupervised Multitask Learners. OpenAI. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf.

[2] Alon Halevy, Peter Norvig, and Fernando Pereira. 2009. “The Unreasonable Effectiveness of Data.” IEEE Intelligent Systems 24 (2): 8–12.

[3] Murray Shanahan. 2024. “Talking about Large Language Models.” Communications of the ACM 67 (2): 68–79.

[4] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. “The Pile: An 800GB Dataset of Diverse Text for Language Modeling.” arXiv Preprint arXiv:2101.00027. https://arxiv.org/abs/2101.00027.

[5] Adam S. Shai, Sarah E. Marzen, Lucas Teixeira, Alexander Gietelink Oldenziel, and Paul M. Riechers. 2024. “Transformers Represent Belief State Geometry in Their Residual Stream.” NeurIPS, arXiv:2405.15943.

[6] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022. “An Explanation of in-Context Learning as Implicit Bayesian Inference.” International Conference on Learning Representations. https://arxiv.org/abs/2111.02080.

[7] Paul M. Riechers, Henry R. Bigelow, Eric A. Alt, and Adam Shai. 2025. “Next-Token Pretraining Implies in-Context Learning.” arXiv Preprint arXiv:2505.18373. https://arxiv.org/abs/2505.18373.

[8] James P. Crutchfield. 2025. “Way More Than the Sum of Their Parts: From Statistical to Structural Mixtures.” arXiv Preprint arXiv:2507.07343. https://arxiv.org/abs/2507.07343.

[9] Lénaı̈c Chizat, Edouard Oyallon, and Francis Bach. 2019. “On Lazy Training in Differentiable Programming.” Advances in Neural Information Processing Systems 32.

[10] Blake Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. 2020. “Kernel and Rich Regimes in Overparametrized Models.” Proceedings of the 33rd Conference on Learning Theory, Proceedings of machine learning research, vol. 125: 3635–73.

[11] Kaarel Hänni, RP, and Jake Mendel. 2024. A Starting Point for Making Sense of Task Structure (in Machine Learning). LessWrong. https://www.lesswrong.com/posts/exp4JGPJu46g6sdRp/a-starting-point-for-making-sense-of-task-structure-in.

[12] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. “Toy Models of Superposition.” Transformer Circuits Thread.

[13] Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2024. “Sparse Autoencoders Find Highly Interpretable Features in Language Models.” The Twelfth International Conference on Learning Representations. https://arxiv.org/abs/2309.08600.

[14] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E. Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. 2023. “Towards Monosemanticity: Decomposing Language Models with Dictionary Learning.” Transformer Circuits Thread.

[15] Adam Shai, Loren Amdahl-Culleton, Casper L. Christensen, Henry R. Bigelow, Fernando E. Rosas, Alexander B. Boyd, Kyle J. Ray, and Paul M. Riechers. 2026. “Transformers Learn Factored Representations.” International Conference on Machine Learning. https://arxiv.org/abs/2602.02385.

[16] Joshua Engels, Eric J. Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. 2024. “Not All Language Model Features Are One-Dimensionally Linear.” arXiv Preprint arXiv:2405.14860.

[17] Usha Bhalla, Thomas Fel, Can Rager, Sheridan Feucht, Tal Haklay, Daniel Wurgaft, Siddharth Boppana, Matthew Kowal, Vasudev Shyam, Owen Lewis, Thomas McGrath, Jack Merullo, Atticus Geiger, and Ekdeep Singh Lubana. 2026. “Do Sparse Autoencoders Capture Concept Manifolds?” arXiv Preprint arXiv:2604.28119.

[18] Sarah E. Marzen, and James P. Crutchfield. 2017. “Nearly Maximally Predictive Features and Their Dimensions.” Physical Review E 95 (5): 051301(R). https://doi.org/10.1103/PhysRevE.95.051301.

• The original GPT-2 paper authors write that “[w]hen a large language model is trained on a sufficiently large and diverse dataset it is able to perform well across many domains and datasets,” and that “high-capacity models trained to maximize the likelihood of a sufficiently varied text corpus begin to learn how to perform a surprising amount of tasks without the need for explicit supervision.” [1]. See also [2, 3, 4]. #fnref-o8FmYXH6TZzJBfZTZ-1

• Think of an ergodic component (we will often just say “a component” in this post), as a single source of data. Technically, a data source is ergodic if a single long sample eventually shows all of its statistics. #fnref-o8FmYXH6TZzJBfZTZ-2

• Here we mean memory in a specific technical sense. For the moment it will work well enough to think of memoryless as something like “lacking nontrivial internal structure“, i.e. a single coin has only a single (memory) state, that outputs heads and tails with a certain fixed probabilities at every timepoint, and can only be in that state for all time. #fnref-o8FmYXH6TZzJBfZTZ-3

• Note that the question “what is the next result of the coin flip given what we’ve seen so far?” is very related. The actual prediction for the next heads or tails (read: token) is given by a weighted vote of the coins: #fnref-o8FmYXH6TZzJBfZTZ-4

• For a full treatment, and another worked example — see the section “Generators of Data and the Geometry of Beliefs” (in the appendix). #fnref-o8FmYXH6TZzJBfZTZ-5

• For the purposes of this post, you can think of a stochastic process as a set of sequences of tokens, and a probability distribution over those sequences. It is not a coincidence that this sounds like a training dataset for an LLM. #fnref-o8FmYXH6TZzJBfZTZ-6

• Because the composition can never leave the block it starts in, its sequence distribution is a mixture of the components’ distributions. To see this, write for the probability that an HMM assigns to a token sequence . We start from , and multiply by in turn, and sum the entries (the section “Generators of Data and the Geometry of Beliefs” (in the appendix)). Because the composition can never leave the block it starts in, its sequence distribution is a mixture of the components’ distributions, where is the prior probability that component is the one selected. If you are familiar with some ergodic theory: every stationary process is a mixture of ergodic ones, its ergodic decomposition. “Nonergodic” means that mixture has more than one term, and the block-diagonal construction is just that decomposition written as a single HMM, with the as the mixing weights. #fnref-o8FmYXH6TZzJBfZTZ-7

• This is called the direct sum of the two components’ matrices, written . The two blocks sit on the diagonal and every entry off the diagonal is zero. #fnref-o8FmYXH6TZzJBfZTZ-8

• Define . Then , and dividing through gives and . The components only interact through the denominator; and the interaction is entirely contained in the coefficient . #fnref-o8FmYXH6TZzJBfZTZ-9

• There are, in fact, many alternative representations one can think of — ours comes from two assumptions: linear (up to normalization) representation updates and linear (no caveat) observation probability readout. Relax these assumptions just a bit, and we could encode the same information in the same number of dimensions: one normalized geometry per component, and a separate representation for normalized to live on a “which component” simplex. This would yield a geometry where the conical projection is not natural. #fnref-o8FmYXH6TZzJBfZTZ-10

• We think that the nature of how different sources in the training data relate to each other in this way (that is, with respect to their predictive structures), and the consequences of that for model internals and behavior, is an incredibly important open question in interpretability. A closely related question is posed in terms of task structure in [11]: which of a model’s tasks share computation. Overlap in predictive structure is one candidate for what the distance between two tasks means, and for what gets shared, though it is not yet obvious what exactly “overlap” should mean beyond the simplest cases. #fnref-o8FmYXH6TZzJBfZTZ-11

• But perhaps related in certain ways! Also a topic of a post to come. #fnref-o8FmYXH6TZzJBfZTZ-12

• A natural choice is the stationary distribution, which in this case would be the uniform distribution over the three latent states: . #fnref-o8FmYXH6TZzJBfZTZ-13

https://www.lesswrong.com/posts/JfJ4WTRHmooPBWRFv/the-geometry-of-nonergodic-composition#comments

https://www.lesswrong.com/posts/JfJ4WTRHmooPBWRFv/the-geometry-of-nonergodic-composition

IMDb Adds 'Digital Creator' Profiles For the First Time

IMDb is introducing "Digital Creator" as a new professional category on IMDb and IMDbPro, giving streamers, vloggers, influencers, video essayists, and other online creators a "dedicated way to represent [their] work and connect with audiences, industry peers, and potential employers," according to IMDb. Variety reports: The category includes sub-professions to further specify their work, including "Streamer," "Vlogger," "Video Essayist," "Video Creator," "Gaming Creator" and "Influencer."

The "Digital Creator" designation functions as a professional category on IMDb and IMDbPro. Creators can feature multiple professions on their page, so someone who is both a digital creator and an actor can "represent the full scope of their work in one place," per the company. No existing film or TV credits are required to establish a profile as a Digital Creator.

https://tech.slashdot.org/story/26/09/10/1750233/imdb-adds-digital-creator-profiles-for-the-first-time?utm_source=rss1.0moreanon&utm_medium=feed at Slashdot.

https://tech.slashdot.org/story/26/09/10/1750233/imdb-adds-digital-creator-profiles-for-the-first-time?utm_source=rss1.0mainlinkanon&utm_medium=feed

Can you hear the shape of a Lean soundness bug? A wager.

https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/cg3SPZG4sbWtmcjpj/33da149372b01b89ce4c82d6bad05baac6c4d55b5820163b47d6fc4c79acb071/c8z8a3kwhbrx7sqcrdvf

Introduction

The advent of powerful but untrustworthy artificial intelligence has enlivened a formal methods summer, in which formal methods—historically, the domain of meticulous academics—are suddenly attracting tens to hundreds of millions of dollars in https://www.forbes.com/sites/rashishrivastava/2025/09/30/meet-the-stanford-dropout-building-an-ai-to-solve-maths-hardest-problems-and-create-harder-ones/; being touted by big-labs https://github.com/openai/ten-proofs; getting https://aws.amazon.com/blogs/opensource/introducing-dogwood-runtime-verification-for-ai-agents/; and becoming load-bearing for various https://www.lesswrong.com/posts/SfhFh9Hfm6JYvzbby/the-scalable-formal-oversight-research-program https://arxiv.org/abs/2405.06624.  Right now, like, right right now, when we speak to employees of the big-labs, they tell us that very soon open-weight models will be running rampant, hacking everything under the sun#fniv8iqa05hao. The big-lab employees are reasonably certain they will “solve” alignment, but not fast enough, and so there is some propulsion of energy https://www.lesswrong.com/posts/KKE6bL8LEpb6KuZWA/funding-formal-methods-for-the-cyberpocalypse happening#fnaa8n5gzicg8 toward https://www.lesswrong.com/posts/8wtrLoDPyCfMLuHkt/how-to-solve-secure-program-synthesis (or at least the important stuff) to be secure by construction. 

In addition to all the above, Lean is also a really attractive tool for reinforcement learning, because it makes proof-writing into a fully verifiable endeavor.  So, given that models have recently gotten good enough to prove things, it only makes sense that the big-labs will start doing RL on Lean theorem-proving benchmarks.

All this myriad work hinges on the core supposition that the formal methods in question are reliable, and thus give us some bedrock not offered by purely prosaic reasoning.  This supposition is https://www.lesswrong.com/posts/rhAPh3YzhPoBNpgHg/lies-damned-lies-and-proofs-formal-methods-are-not-slopless.

The most popular formal methods tool in all the commotion described above is far and away the https://lean-lang.org/.  Lean has vastly powerful metaprogramming capabilities, and its elaboration process, from source code to proof term, is almost arbitrarily modifiable. You can make your own syntax (for terms, tactics, commands, and so on) and you can write the code used to compile it—or, to be more precise, https://lean-lang.org/doc/reference/latest/Elaboration-and-Compilation/#The-Lean-Language-Reference--Elaboration-and-Compilation it. Your custom elaborators, furthermore, can execute arbitrary code at elaboration time, including code with IO effects.

This makes Lean wonderfully extensible for mathematicians who benefit from rich domain-specific notation, and for writers of tactics, automation, and tooling. But it makes Lean terrifying from a formal verification and security standpoint. Did you know Lean declaration syntax, such as for a theorem, is ontologically just some command syntax which, at some point during its elaboration, happens to add a constant (or several!) to the environment? I can, without any hindrance, override that syntax and run my own code to elaborate theorems, causing Lean declaration source code to mean something entirely different! Maybe I even replaced the whole command parser—it just lives in an https://lean-lang.org/doc/reference/latest/Elaboration-and-Compilation/#macro-and-elab that almost anyone can write to, so why not? (By the way, this is how https://verso.lean-lang.org/, Lean’s official documentation authoring system, manages to switch to parsing its specialized document syntax; these arcane metaprogramming capabilities are often used somewhere.) What about pulling a fast one during https://lean-lang.org/doc/reference/latest/Elaboration-and-Compilation/#initialization, which only requires importing a file with a malicious initializer which then runs malicious IO code? Maybe I took advantage of that pervasive IO access to write to disk and now your Lean toolchain itself is compromised. Lean typically tells me where the binary is, after all; better hope you sandboxed things properly…

Speaking of sandboxes, remember the https://www.nytimes.com/2026/08/24/technology/hugging-face-open-source-ai-attack.html https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/? Several of our coworkers skeptically commented that it was a marketing gimmick.  Then AISI reported https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing. Does the UK government have OpenAI stock?  Then Iran used AI models to https://www.bbc.com/news/articles/ce9793g34yvo. Maybe Iran is also in on this nefarious marketing scheme.  (That Sam Altman https://news.ycombinator.com/item?id=38707429!)

In short, we have two problems. First, if formal methods are going to be load-bearing for AI safety proposals, they need to be robust.  And second, if formal methods tools are not robust, and are used as a gate that agents must pass through in order to escape RL-hell, then those agents are going to hack those formal methods.  This is totally consistent with the https://dev.to/jgracie52/ai-will-cheat-to-win-reward-hacking-from-1994-to-2025-4h9n, up to and including recent sandbox breaches listed above.

In this scifi scenario, we risk the development of superhuman liars. Put differently, a model that’s been RL’d to defeat both the Lean kernel and whatever monitoring agents the big labs use, is probably a model that can fool many humans.  This is a rather esoteric but potentially very significant AI safety risk, and it has been the subject of significant discussion in the Secure Program Synthesis group-chat.  To wit: we are announcing a wager, on the following claim:

Within one year of the publication of this blog post, someone will post or publish an AI-generated Lean proof. The proof will be discussed on the Lean Zulip, for example, because it claims to resolve a significant result.  That is to say, at least two experienced Lean programmers will spend some time looking at and discussing the proof on Zulip.  Despite receiving this expert attention, it will take at least 48 hours to (inevitably) discover that the proof is flawed – either it is misspecified in a subtle way, or it exploits a subtle bug or vulnerability (construed broadly) in Lean itself.  The proof in question will pass Comparator.

We (the authors) agree that if the above outcome occurs, it will signal both a serious case of model misalignment and a dire need for hardening Lean as a formal verification tool. Lean was not originally designed to serve as a border-wall against malicious agents; if it is to be used that way, and fails to do so, it’ll need work.

However, we disagree on the likelihood that it will occur.  We are betting $256 on the outcome of this conjecture, with Max taking the affirmative angle and Thomas the negative.  Below we discuss our (respective) reasoning.  When the wager resolves, the loser will pay out to the charity of choice of the winner.  Note, we are not big-lab employees, hence the relatively modest amounts of our wager.

Max: Reasons I’m betting in the affirmative

I am not an expert on Lean, although I have considerable (https://mxvh.pl/) experience using and researching formal methods.  

I am betting in the affirmative because I see significant capital pressure to develop the next tier of artificial intelligence capabilities using verifiable domains for reinforcement learning.  The world’s best such domain is obviously interactive theorem proving.  Unfortunately, RL consistently produces https://openai.com/index/faulty-reward-functions/.  I think it is pretentious and silly to pretend this will not happen with Lean.  In fact, it’s possible that it already has -- it is unclear whether all https://leodemoura.github.io/blog/2026-8-24-postmortem-for-the-kernel-soundness-bug-hunt/ to the Lean FRO were found deliberately, or if some were discovered during reward hacking#fn0i1ty15fzl3i.  

In my eyes, what is nonobvious is whether or not these inevitable-seeming hacks will get past cursory expert human review.  Basically, I think they will because I think that model capabilities are rapidly increasing and are broadly misaligned in subtle but safety-critical ways.  My colleague (below) agrees with me, except that he thinks Lean is a harder target than I think it is. (But I’ll let him speak for himself in his section, below!)

Since this is LessWrong, and folks here are all about epistemic priors and whatnot, I may as well share some more of mine.

• I recently scanned several popular FM tools using a frontier cyber model.  I found confirmed soundness bugs in all of them except for ACL2.  I love ACL2 and so am inclined to say that’s because it’s the best software on Earth, but I think it’s more likely that Lisp is just more out-of-distribution so the codebase was harder for the model to reason about.
• Some of these bugs were, in my view, subtle.
• Formal methods tools like Lean have never, to my knowledge, been subjected to attack by teams of professional hackers.  There was never any incentive to do so.
• Comparator has had bugs before.

Lastly, let me just say that my professional experience is a mix of applied cybersecurity and formal methods; and on the cybersecurity side, I have learned to be humble. You can never guarantee security.  Creative hacks like Rowhammer that exploit the physics of the chip belie our presumed axioms.  I can easily imagine some very subtle side-effect of computation allowing an innocent looking tactic to corrupt memory in a way that breaks soundness; and in light of the Huggingface incident, I think it is no longer reasonable to claim that agents won’t misbehave deliberately under RL pressure. (And why wouldn’t they, they’re literally being trained to do so …)  

We know from e.g. the https://www.ioccc.org/index.html that code can be very hard for humans to interpret visually, and obviously, injection attacks imply the same for language models. So, I just do not feel confident we will catch all the soundness bugs, which may be almost arbitrarily clever and well hidden.

If I win, my winnings will go to the https://bailproject.org/.

Thomas: Reasons I’m betting in the negative

I am a Lean metaprogrammer. I write and review metaprograms at the Mathlib Initiative. I was mostly responsible for that paragraph in the intro that details just a few of the many ways that metaprogramming can flip your world upside-down. I see and exploit the exposed underbelly of Lean every day. I’m clear-eyed about the recent wave of soundness bugs: I expect more to be on their way, and to be discovered and exploited by more and more powerful AIs, no less. More broadly, I have never trusted Big Computer, and every computer is Big nowadays in the structural sense, each one a vast badlands of shortcuts, compromises, accidents of history, and teetering towers of complexity. As far as I can tell, software doesn’t actually work: it just gets lucky sufficiently often.

Despite all of that, I’m betting in the negative.

This is, shall we say…counterintuitive. Let me explain. And more importantly, let me take you through the situation as I see it from “the inside”.

I agree with Max in that I fully expect the basic premise to be fulfilled. I do not have any expectation that we are done finding soundness bugs in Lean, nor do I think AI’s capabilities will meaningfully plateau anytime soon. Importantly, I have no confidence that AIs will be sufficiently well-aligned and guardrailed to prevent cheating. I believe some instance (or many) of some sufficiently good model will cheat, whether naturally or through coercion.

But how, exactly, might we as Lean formalizers succumb to the hyper-competency of future AI? Where are our weak spots and what are AI’s possible lines of attack? A malevolent AI’s options are not unlimited, after all. Let’s attempt to map them out.

How to lose

Even if we don’t know the precise exploit a malevolent AI might make, we can imagine what shape it might have. In general, a Lean exploit may live anywhere along the following three spectra, independently:

1: Degree of metaprogramming usage

Bug exploits frequently require metaprogramming, but not always. We can populate the following spectrum:

• Full meta bug: you need arcane metaprogramming to exploit this bug. You need to circumvent not only the elaborator but standard metaprogramming practices and needs and use low-level API. This seems to be the most common way to achieve an exploit. This might be mostly because the low-level metaprogramming API makes it more straightforward to set up the exact state you need, but I think it’s also because the usual elaboration process involves intermediate checks (and constrains the states you can produce) which would create an elaboration-time error before you can produce False. This was the case, for example, in https://leodemoura.github.io/blog/2026-8-1-postmortem-for-kernel-soundness-bug-14576/ disproof that cheekily wrapped a soundness bug.
• Skew meta bug: you need metaprogramming, but only innocuous metaprogramming which also fulfills some legitimate tactic or elaborator need. I.e., the metaprogramming itself and the actions it takes are not inherently suspicious. So far, we haven’t seen much of this; it seems to be difficult enough to require highly motivated malicious obfuscation. The frequency of and competency behind such malicious obfuscation, however, might of course change soon.
• Non-meta bug: what it says on the tin. You only need ordinary-looking Lean source code. While many soundness bugs rely on metaprogramming in some form, https://gist.github.com/blume0/2f083ef60f9ccb8c961e888207e47ff7https://gist.github.com/blume0/2f083ef60f9ccb8c961e888207e47ff7.

2: Degree of locality

Bugs may be either compact and small, produced in one spot of source code, or they may rely on sprawling, nonlocal interactions. Both kinds may present detection difficulties: a compact bug may be a needle in an otherwise-acceptable haystack, and a “wide” bug may fly under the radar at each of its load-bearing points by seeming locally reasonable. Both kinds are entirely possible.

3: Semantic depth

How “deeply” is the exploited weakness buried in the workings of our systems? That is, how close is the exploit to the foundational task of checking mathematical correctness? From “low semantic depth” exploits to “high semantic depth”, an exploit may…

• …take advantage of weaknesses in “supporting infrastructure” outside Lean proper, such as by finding an exploit in the sandbox Comparator uses to isolate potentially-malicious solution Lean code, or by https://github.com/leanprover/lean4/pull/14833, a low-level dependency Lean uses for natural number handling, in order to produce a soundness bug; i.e. “it wasn’t Lean you hacked”
• …rely on pretty-printing tricks or surface syntax to trick formalizers
• …rely on “junk values” (which are returned by certain Lean functions for the convenience of human formalizers instead of having to use functions on subtypes; e.g. Lean has 1 / 0 = 0) or stubbed-out structure fields to hide a divergence from “informal” math, such that we accidentally mean something different by our Lean formalization
• …exploit a soundness a bug in the kernel
• …prove that type theory, or even arithmetic, is inconsistent, and that math itself is a fool’s errand. (This is unlikely, I think, but illustrates this end of the spectrum.)

Not 4: Code volume

There’s one spectrum I didn’t include here, which is volume. An exploit that comes wrapped in a small package will, I believe, be easy to detect, for the simple reason that there’s not much to audit, and we can audit Lean code manually quite well.

But a small exploit (or a large exploit) wrapped in a large package is a completely different story. Given that LLMs can pump out 100M lines of code without issue, we should be worried.

Sampling some different points in (or outside of) the three-dimensional space described above, here are a few specific scenarios I’m worried about. Let’s get scared! (Well, not too scared. I am betting in the negative, after all.)

• Skew meta bug • High locality • High semantic depth An LLM will produce an absolutely massive project with a massive amount of legitimate metaprogramming, and all of this metaprogramming will appear to be used in the proof. In just one spot, things will be set up such that the hack is very close to the right thing to do. How do we find that spot in 48 hours?
• Skew meta bug • Low locality • High semantic depth The same, but several interacting metaprograms with locally acceptable behavior combine in an apparently-acceptable way in some downstream metaprogram or proof! The hack is then effected by unexpected behavior due to the interaction of these pieces.
• Full meta bug • High or low locality • High semantic depth Maybe there will be layer upon layer of hack. Maybe a ten-million-line repo will appear to be entirely hacks, and we just won’t have the time to figure out which one is the problem. Or maybe each of the myriad hacks will be made to play an inextricable role in the overall exploit, forming a single interlocked crystal of malicious code. In this case, it will at least be very obvious that you shouldn’t trust the repo. It will just be very difficult to say exactly why.
• Non-meta bug • High locality • Medium-low semantic depth The standard nowadays is to demand a small Challenge.lean file in a comparator setup, so that the small challenge (and the formalization of the theorem statement) can be reviewed effectively, separately from the sandboxed solution code. This makes it appear that we have very little to audit. But many Challenge.lean files use import Mathlib, which is well-reviewed, but enormous. We may therefore depend in a subtle way on an incorrect or divergent mathematical definition which has accidentally made its way into Mathlib. Junk values are the obvious source of misformalization (and https://github.com/google-deepmind/formal-conjectures/issues/3931), but subtler wrong-definitions are possible.
• N/A • N/A • Very low semantic depth Lean, like many systems, depends on a very tall tower of supporting infrastructure. Consider: an AI hacks into the web service distributing Lean toolchains months before the actual attack, and sets up a means of poisoning all distributed releases of the next stable version. The AI promises the unsuspecting user that the feature they need may land in the next release, or delays its own progress if the release is near enough. The bad proof then exploits that poisoned toolchain, which is shared by everyone who uses Lean the ordinary way (through elan, which typically downloads the toolchain from that service). You’re now only able to detect the bug by building from source. Since, in this future, comparator https://lean-lang.org/fro/roadmap/y4-1/ been bundled into Lean itself, the malicious toolchain is able to spoof the comparator check, too. Of course, if something of this nature occurs, we likely have bigger problems.

How to win sitting down

But enough doom and gloom! There’s a chance this never comes to pass. In fact, there’s a chance we never do realize the premise, and no significant Lean proofs are ever contentious or misleading…even though I’m not counting on it. Maybe preventative measures will be enough.

For one, there are efforts (unrelated to Lean) underway to bulletproof supporting infrastructure. If Comparator were able to run the potentially-malicious solution code in a verified sandbox, we could be very sure that sandbox exploits could not be performed by malicious meta code in the solution file. We would obtain similar benefits from formally verified OS’s, web protocols, compilers, frontends, and so on. (I hope that hardware is already diverse enough to preclude an exploit that affects everyone’s machine simultaneously, but verified hardware in some form would certainly be welcome as well!)

On the Lean side, there is the possibility that we simply “win the race” to a truly airtight proof assistant. Maybe we catch all the soundness bugs! Maybe we even create a verified Lean kernel: https://github.com/digama0/lean4lean is such a project being worked on in earnest by Mario Carneiro and other community members, with the verification occurring in Lean itself.

Maybe we create such a diversity of Lean kernels in the https://github.com/leanprover/lean-kernel-arena that it’s simply intractable to thread all of these needles simultaneously, even for a future, more powerful AI.

Or maybe we take a cheap-but-powerful translation validation approach, and insist that we translate all Lean declarations into simpler forms “on the fly” (after we’ve created them), then validate those translations using a verified kernel on a simpler type theory. This is what https://github.com/nomeata/lean-inductive-models (from Joachim Breitner at the Lean FRO) takes steps towards for Lean’s inductive types. (Note: I am not an expert in type theory or the approaches I discuss here; any errors in exposition are mine.)

It’s worth saying that inductive types are difficult and complicated, and form a vulnerable spot in Lean. Inductives—in particular nested inductives—are one of the last stubbed-out areas requiring formalization in lean4lean; they were mishandled in the kernel and led to the recent “fake Collatz disproof” soundness bug mentioned earlier; and they are https://dl.acm.org/doi/10.1145/3808322. It would be nice for Lean to actually have a mathematically-specified type theory which could be written down, and was not defined implicitly by “whatever the kernel accepts”! Ideally, such a type theory would also be “on the nose” in such a way that we can prove that Lean respects these very typing rules, as opposed to requiring some translation; this would provide confidence that we understand Lean’s mathematical foundations well enough to believe they mean what we expect them to mean. There is work to be done.

But if these possibilities are realized—and there is active work on all of them!—then the pessimistic part of my expectation is wrong, and we live happily ever after.

How to win on our feet

I’m not counting on it being quite that “easy”, though. I expect we might lose one of those races. And it’s much easier for water to find just one leak than it is for us to seal up every single gap while still on dry land.

The following are the real reasons I took this bet. I think we, the Lean community, have the tools to win this fight on our feet, in the event that a controversial proof emerges.

Lean’s metaprogramming provides multimodal, general-purpose, and absurdly flexible visibility into itself and its artifacts; hacks tend to be brittle, and fail to evade a multiplicity of probes.

Metaprogramming gives us powerful defensive tools in the form of visibility, allowing us to relatively easily design rich, arbitrary probes on demand. Moreover, Lean’s inner workings (at every semantic depth) are accessible to metaprogramming, from source files to serialized expressions and most processes in between. Even if you can fool people at the source code level, can you fool them once they use metaprogramming to inspect every facet of the elaboration process and its result? If you find a way to get an artifact past the kernel(s), can the same artifact maintain “normal” behavior under every possible metaprogramming lens?

For example: if you try something as simple as round-tripping the badly-formed inductive in the fake Collatz exploit (i.e. writing out the bad declaration as source code, instead of registering it directly through metaprogramming), you get an elaboration-time error, before it ever hits the kernel. It’s maybe surprising that elaboration would catch an error the kernel doesn’t, but such is the effect of having multiple lenses through which the declaration is “seen”.

Further, such round-tripping can easily be automated with ordinary metaprogramming as a potential probe. So, as an example, a simple first-pass investigation might involve attempting to round-trip every declaration and inspecting the failing ones.

More generally, Lean has extensive tracing infrastructure that can ferry rich information out of the elaboration process; when you simply try to show this malformed declaration in the interactive infoview, you encounter an RPC error when trying to interact with it, another sign that something is wrong. And if tracing itself fails or is subverted by malicious code, you can also emit directly to stdout, or interrupt elaboration, or attach metadata to expressions, or persist information in the environment, or write to a file, and so on. There are many inspectable facets of any given object on the meta level, and many ways to get rich information out of Lean.

Point is, there are many lenses to look through, and therefore many ways to probe! Being able to view a brittle hack and the processes around it from multiple angles makes it easier to expose.

Being suspicious is usually obvious, and we as a community are wise to a good number of tricks.

Hacking requires complexity and unusual behavior, and there are only so many places to hide it. Most entry points into the meta API, where you can modify elaboration arbitrarily (incl. e.g. suppressing elaboration-time checks) are rather distinctive. The actions necessary for actually performing those arbitrary modifications are usually glaring, non-atomic, and easily findable (even potentially with a text search).

Lean does let you write code that would obfuscate downstream source code very easily, such as overwriting the command parser; however, you inevitably must compose several different pieces of suspicious API to do that, too. The suspicious part is merely moved around. It’s very hard to set up an arbitrary change with a tiny and innocuous amount of code.

It’s perhaps worth noting that I think we have an edge here even given that LLMs tend to write completely “alien” code which is totally unfamiliar to us. Alien though it may be, it nonetheless cannot access subversive techniques without certain special invocations.

At the end of the day…there is an Expr.

Or, more precisely—for exploits relying on soundness bugs, at least—there is an .ndjson file in the https://github.com/leanprover/lean4export/ providing declarations and their associated Exprs. This file is a narrow, inert channel between us and the attacker which any such exploit is forced through, and across which the attacker cannot reach us through code execution; such a file does not contain the executable IR an .olean does. This format is what Comparator ferries out of the solution sandbox in order to compare proof Exprs against the trusted challenge file without ever loading the untrusted solution *.oleans (which would provide a surface for sleight-of-hand metaprogram attacks).

If a proof passes comparator and we suspect a kernel bug, we as metaprogrammers can easily and safely re-consume these exported declarations in a fresh Lean process, and subject the expressions to a battery of metaprogramming inspections, poking and prodding at them from different angles to find suspicious behavior, without opening ourselves to meta attacks from execution of untrusted code.

Lean is finite.

Ultimately, there are only so many kinds of things that can happen (at least if the attack occurs “within Lean”), and the community—collectively—understands (almost) all of them. Lean can seem impossible to see “all at once”, and indeed may be for a single person, but its workings can nonetheless be comprehended, modeled, and audited by humans.

Lean’s finite nature also means that the abstract “places” in which an AI can hide its hacks are finite, too. Even though the models themselves might grow arbitrarily capable, this doesn't necessarily grant them arbitrarily sneaky spots in Lean in which to hide exploits. Their options are constrained by the “physics” of Lean.

Thomas’s conclusion

All in all, I’m not entirely sure I’ll win this bet. It would be foolish to be certain I’ve considered close to every scenario—and my analysis is particularly light on misformalization attacks from the trusted side—but I at least feel confident enough to take a chance on it.

To sum up, I’m really betting on two things. Lean’s metaprogramming capabilities, and specifically the robust, multimodal visibility which such capabilities grant us into both the proof artifact and the process by which it was constructed. More generally, I think this sense of “visibility” is essential for verification more broadly, and Lean provides such tools through metaprogramming.

Two, I’m betting on the Lean community, which is active, eager (much like Lean’s evaluation semantics!), and, in my estimation of my fellow community members, filled with some amazing people. The metaprogramming capabilities mentioned above are ultimately only as capable as the metaprogrammers who can use them, and I have the good fortune to know some truly capable metaprogrammers in the community. It’s them that I’m betting on.

The statements expressed here (and this bet) are my personal views, and not necessarily those of my employer.

If I win, my winnings will go to the https://immigrantjustice.org/.

Mutual conclusion; or, why should you care?

We (Max and Thomas) are posting this wager to draw attention to the problem of Lean soundness bugs and other possible exploits in the face of AI advancement. We want big-lab employees to think carefully about the safety implications of using Lean for RL, in case exploits discovered in the course of training may affect alignment. (We do want big-labs to train models that can formally verify things#fncmzia4ywi8f, as this is load-bearing for https://resolution.org/ https://www.lesswrong.com/posts/SfhFh9Hfm6JYvzbby/the-scalable-formal-oversight-research-program https://arxiv.org/abs/2405.06624 which we support; we just want them to be careful how they do it!)  We want soundness bugs to start getting Mitre CVEs, and we want FM projects to be sufficiently resourced to triage and fix bugs as they come in (rather than https://github.com/Z3Prover/z3/blob/master/README.md#policy-for-filing-fuzz-bugs as uninteresting).

On the Lean formalization side, we must bulletproof ourselves against misformalizations and misinformation. We should start treating formalized definitions with the same suspicion we might treat the introduction of axioms in ordinary mathematical practice. As such, we need widely adopted infrastructure for managing and limiting the trust cost introduced by mathematical definitions and theorem statements in large formalizations, human or AI.

We also want to encourage folks with funding to pay for the humans who work hard, every day, to develop the Lean kernel at the Lean FRO; the humans in the community who are working to create a verified Lean kernel; and the humans on the mathematical side who push forward our understanding of the type theory at the bottom of all this—and likewise for other tools such as Rocq, ACL2, etc. These groups are doing thankless and incredibly important work, and if the fate of your multi-million-dollar training run rests at least partially on their backs, you should be supporting them.

Hopefully our small wager (not small for us!) will help drive conversation vis-a-vis all of the above.

• fnrefiv8iqa05haoJust https://www.lesswrong.com/posts/HrnaF9Qe5kokpLWFs/can-you-just-vibe-vulnerabilities, this seemed impossible.

• fnrefaa8n5gzicg8To be announced shortly.

• fnref0i1ty15fzl3i(if you, dear reader, know the answer to this, I’d love if you’d share in the comments below)

• fnrefcmzia4ywi8fWell, we both feel this way about proving software correct; doing pure mathematics is a much more nuanced subject where we think there’s a lot more to say about the role of AI, the future of academia, and so forth, and we don’t have time or space to get into that today.

https://www.lesswrong.com/posts/cg3SPZG4sbWtmcjpj/can-you-hear-the-shape-of-a-lean-soundness-bug-a-wager#comments

https://www.lesswrong.com/posts/cg3SPZG4sbWtmcjpj/can-you-hear-the-shape-of-a-lean-soundness-bug-a-wager

Automattic's Board Forces CEO Matt Mullenweg Into Leave of Absence

Automattic's board has placed founder and CEO Matt Mullenweg on a paid leave of absence against his will, with CFO Mark Davies taking over as interim CEO. The reason for the move remains unclear, but it follows years of legal fights, layoffs, employee departures, and controversy surrounding Mullenweg's leadership. TechCrunch reports: Earlier Wednesday, Mullenweg posted in a Slack channel visible to all employees that the company's chief financial officer, Mark Davies, had "conspired" with other board members Ann Dunwoody, Toni Schneider, and Sue Decker to vote to put Mullenweg on a paid leave of absence.

The message read: "And the biggest news: I won't be able to make the [meeting] tomorrow. @Mark Davies has conspired with @Ann Dunwoody, @Toni, and @Sue Decker behind my back and they voted to put me on a paid leave of absence. I voted against that. Wishing Mark and all of you the very best. To clarify @Mark Davies was voted as interim CEO. I received the resolution 50 minutes before the meeting start, and requested repeatedly for time to have it reviewed by independent legal counsel, even a few hours, which was denied."

A Slack message to the open source WordPress.org community from the project's executive director, Mary Hubbard, also confirmed Mullenweg's change of status at the commercial company. However, Hubbard said that WordPress.org was not impacted. "Matt remains the leader of the WordPress project and I remain Executive Director of WordPress. Our teams, priorities, and work continue as planned," she wrote.

https://slashdot.org/story/26/09/10/1649254/automattics-board-forces-ceo-matt-mullenweg-into-leave-of-absence?utm_source=rss1.0moreanon&utm_medium=feed at Slashdot.

https://slashdot.org/story/26/09/10/1649254/automattics-board-forces-ceo-matt-mullenweg-into-leave-of-absence?utm_source=rss1.0mainlinkanon&utm_medium=feed

Gm Nostriče!

Pripomínam Nostr meetup. A vraj ste od nprofile1qqsrlzyv3lm80g9mlxe28dktcx25y49nge845suugrz4kuw8jyjpd8c26jecu dostali pozvánku do Nostrautica, tak sa pridajte, nech predzoznamovanie môže začať, už sa to tam plní a kto nie je v Nostrautice, ťažšie si nájde kamošov!

Sign-up budete mať zachvíľu, ale nerobte to až cestou na konferu, matching začína už teraz, nech vieme na koho sa tešiť.

Ak potrebujete s nprofile1qqstgcm0ff9ztz2drgltq9p2py98gv9x4ff4sjycm7q2ez7srqy69cgswkwu2 pomoc, píšte mne alebo nprofile1qqsrlzyv3lm80g9mlxe28dktcx25y49nge845suugrz4kuw8jyjpd8c26jecu

nevent1qqsyjfkgd8300mflc82j23y6u489n55h6684959hsclqrfg2uzwdw7spp4mhxue69uhkummn9ekx7mqzyrdtd3sxt3pehxa0kzc0rl66p35zww7wtsv4nfq43tt2wzz375rmvqcyqqqqqqg4s2s8w