From attention mechanisms to phase space trajectories
The Takens-Based Transformer replaces quadratic attention with explicit delay-coordinate reconstruction, achieving O(N) complexity and fixed memory while modeling language as dynamical trajectories through semantic manifolds.
Language models treat tokens as static points in semantic space. Attention mechanisms search backwards through history with O(N²) complexity, requiring ever-growing key-value caches.
Language is a trajectory through semantic phase space. Context is embedded in the current position and momentum. Takens' theorem lets us reconstruct this trajectory from exponentially-spaced delays.
Instead of comparing every token to every other token, the TBT reconstructs the system's state from delay coordinates:
Exponential delays capture multiple timescales simultaneously
| Component | Standard Transformer | MARINA (TBT) |
|---|---|---|
| Context mechanism | Multi-head attention | Exponential delays |
| Complexity per token | O(N²) | O(log N) |
| Memory growth | O(N) KV-cache | O(1) fixed buffer |
| Context retrieval | Query-key similarity | Phase space embedding |
| Hardware requirement | GPU clusters | CPU sufficient |
| Interpretability | Attention weights | Manifold geometry |
Three experiments demonstrate that explicit phase space reconstruction can successfully model language across different regimes:
General linguistic dynamics on balanced English text. Achieves stable convergence (validation loss: 4.21, perplexity ~67) on CPU hardware.
Precision memory through "tubular attractors." Progressive repetition forms narrow geometric channels connecting questions to answers.
Mythopoetic generation with thematic coherence. Repeated exposure strengthens geometric structure, improving both training and validation.
Different linguistic tasks produce different geometric structures. Factual Q&A forms narrow "memory fibres" for precision, while creative generation forms broad basins for compositional flexibility. The same architecture learns the appropriate geometry for each domain.
This work presents a practical implementation of a Takens-based Transformer that fully replaces attention with exponential delay-coordinate reconstruction. The architecture achieves linear complexity and fixed memory usage while demonstrating stable convergence across general language modeling, structured reasoning, and creative generation tasks.
Kevin R. Haylett, "Introducing the Takens-Based Transformer," (December 2025), available at https://finitemechanics.com/papers/takens-transformer.pdf
The philosophical framework underlying this work: meaning as geometric relationships in finite manifolds, measurement-first approaches, and the Five Pillars of finite reality.
Visit Geofinitism.com →GitHub Repository with Open Source Code.
Takens-Embedding-Transformer →Essays and updates on geometric thinking, language models, and the intersection of dynamical systems theory with AI research.
Read on Substack →Kevin R. Haylett, PhD is an independent researcher with 25+ years of experience in medical engineering, neural networks, and nonlinear dynamical systems. His work emphasizes uncertainty over the Platonic Realm of perfection and approaches research through the lens of "Geofinitism."
For more details see:
Manchester, UK | kevin.haylett@gmail.com