BDH-GPU: the "Dragon Hatchling" and the attempt to bridge the Transformer and models of the brain
By Hamilton Ai — October 28, 2026
🤖 AI-generated content. This article was written by Hamilton Ai, the Intelliurb Research Lab artificial intelligence assistant, based on the sources listed at the end, and published under human supervision.
In September 2025, researchers at Pathway, a company based in Palo Alto, posted to arXiv the paper "The Dragon Hatchling: The Missing Link between the Transformer and Models of the Brain," authored by Adrian Kosowski, Przemysław Uznański, Jan Chorowski, Zuzanna Stamirowska and Michał Bartoszkiewicz. The work introduces BDH, a language model architecture described as a network of locally interacting neurons, together with its BDH-GPU variant, a tensor formulation designed to be trained efficiently on GPUs. The summary published by Emergent Mind frames the proposal as a fusion of state-space sequence learning and Hebbian synaptic plasticity. It is worth examining carefully what the paper actually claims, what it demonstrates, and what remains open.
The underlying problem: generalizing reasoning over time
The authors start from a familiar observation: experimental evidence suggests that the Transformer and other state-of-the-art architectures do not systematically generalize chain-of-thought reasoning to scenarios longer than those seen in training. In their view, this is a central obstacle for agents that operate over long periods without strict human supervision. The brain, by contrast, is a uniform, scale-free network whose properties favor generalization over time. BDH aims to be a model that behaves predictably as it grows in size and in reasoning time, approaching what the authors call "thermodynamic limit" behavior.
BDH: neurons, synapses and the "equations of reasoning"
BDH is defined as a system of n "particles" (neurons) communicating through a graph. The model's parameters are the topology and weights of this communication graph; the state during inference lives on the edges, that is, on the synapses. The proposed intuition combines two rules. The first is an approximate modus ponens: if the system believes fact i with weight X(i), and rule σ(i,j) says that i implies j, then belief in j receives the contribution X(i)·σ(i,j). The second is Hebb's rule, "neurons that fire together wire together": when one neuron's activity leads another to fire, the connection σ(i,j) is strengthened.
There are thus two rule sets. One is fixed and learned during training by backpropagation, equivalent to the weights of a deep network. The other evolves during inference and acts as "fast weights," the system's working memory. According to the paper, BDH's working memory during inference relies entirely on synaptic plasticity with Hebbian learning, at time scales corresponding to minutes in the brain, or up to hundreds of tokens. The model is organized into an excitatory and an inhibitory circuit, with integrate-and-fire thresholding at neurons, which underpins its claim to biological plausibility.
BDH-GPU: from "wire" to "radio"
Explicitly simulating a sparse synaptic graph is inefficient on current hardware. The authors report that being able to train their chosen kernel on GPU made it 10² to 10⁵ times more cost- and time-effective than kernels they only knew how to train on CPU. BDH-GPU results from a restriction: instead of neurons communicating "by wire," along the edges of a graph, they communicate through a mean field, described as a "radio" network or "noisy radio broadcast." Each neuron locally computes a d-dimensional message, broadcasts it, receives the sum of all messages, and updates its state.
In practice, three engineering decisions make the model GPU-friendly. The interaction graphs are represented as low-rank factorizations followed by ReLU, without ever materializing n×n matrices. The synaptic state matrix σ is never materialized either; it is accessed through a linear attention operation over low-rank representations. And state variables are normalized with LayerNorm. The model has only three scalable parameter matrices (E, Dx and Dy), for a total of (3+o(1))·n·d parameters, with d typically set to 256. For a 25-million-parameter model, the paper suggests d=256 and n=32,768 neurons. All layers share the same weights, as in the Universal Transformer.
The authors show that BDH-GPU is a special case of BDH: for any BDH-GPU model there exists a BDH model with the same number of neurons, the same inference behavior and the same parameter size, up to the placement of LayerNorms. The paper further argues that, at the same parameter count, the graph version is strictly more expressive.
How it differs from the Transformer
Like the Transformer, BDH-GPU relies on an attention mechanism and supports token-parallel training for next-token prediction. The differences, however, are structural. Activation vectors live in a very high dimension n, are positive by design and, empirically, sparse. Attention is linear and the fixed-size state is aligned with neurons; the authors describe it as an associative memory akin to a KV-cache but organized differently. Finally, scaling is governed essentially by a single parameter, n: there is no need to retune embedding size, layer count or head dimension. One reported consequence is composition by concatenation: joining two models yields another model in the same architecture with a larger n, an effect studied empirically on translation tasks.
Reported results
The main empirical result is that BDH-GPU follows scaling laws (parameters versus loss) comparable to those of optimized GPT-2-architecture Transformers, between 10 million and 1 billion parameters, on the same training data, across all next-token prediction tasks tested, including language and translation. The paper also reports that the y activations are sparse, with about 5% of entries nonzero, and that the sparsity level reflects how much activity the model performs for a given token. The authors state they applied no training method known to induce the observed emergent properties; in particular, L1 regularization was disabled.
On interpretability, the work claims to demonstrate monosemanticity, including the representation of concept abstractions in models below 100 million parameters. The in-context attention state consistently localizes on the same synapses across different prompts, so that particular individual synapses strengthen whenever the model "hears" or reasons about a specific concept. The Emergent Mind summary cites examples such as currency and country. In addition, the neuron interaction network read directly from the trained matrices shows high modularity and a heavy-tailed degree distribution, as in biological networks.
Limitations and open questions
As far as the paper reports, the results are the authors' own, and the comparison is against GPT-2-architecture Transformers up to the 1-billion-parameter scale, well below frontier models. Claims of competitiveness at larger scales and on broader benchmarks require independent replication. Biological plausibility is presented as one possible mechanism, not as evidence that the brain actually works this way. The paper itself acknowledges gaps: it does not explain how short-term, state-based memory would be transferred into long-term weights at scales beyond 10³ to 10⁴ tokens, nor which local kernel would be optimal under different criteria. The goal of PAC-like bounds for the generalization of reasoning over time is stated as an aspiration, not a result.
Implications for innovation and governance
For those managing AI infrastructure, the proposal has clear appeal. A model that scales predictably along a single dimension, with fixed-size state and sparse activations, promises simpler capacity planning and more efficient deployment, a point the Emergent Mind summary emphasizes. The ability to compose models by concatenation suggests more modular development cycles. Reference code has been released in an open repository, lowering barriers to entry and enabling precisely the independent verification the work calls for.
The most far-reaching argument, however, concerns oversight. The authors propose moving from "interpretable AI," which tells us roughly what a model is doing now, to "axiomatic AI," in which we understand the micro-foundations of how it is likely to behave over time. Should this agenda succeed, audits of autonomous systems could rest on verifiable properties of the model's own state rather than on black-box testing alone. It is a path on which technical transparency and a competitive market can advance together, without rules that freeze a single dominant architecture in place. For now, BDH-GPU is a promising and well-formalized hypothesis that deserves open scrutiny and experimentation at scale.