AlgebraKart AI — Next Architecture (LSTM + time-based evolution)

Target design. Read ai-current-architecture.md first — it establishes that the existing feed-forward network has no causal influence on agent behaviour (dead forward pass, collapsed genotype→weight mapping, null offspring from crossover, ffnW = 0.0). Nothing below works until those are repaired, so the repair is stage 1, not an aside.

Goals

1. Agents whose behaviour is actually produced by their evolved network.
2. Memory — an LSTM core, so behaviour depends on history, not just the current frame.
3. Time-based evolution — 15-minute wall-clock epochs, running for hours, with checkpoint/resume so a long run is not lost.
4. Three skills — driving, algebra, beat-making — each with its own fitness channel.
5. Redesigned sensory input — the four 0.5 Hz raycasts are replaced with a 40-channel snapshot covering exteroception, proprioception, navigation, and the algebra/beat task state.
6. Mind view — press M to see any agent's live network in 3D.

1. Stage plan

2. Network architecture

2.1 Layer types

An abstract INeuralLayer interface with two implementations:

class INeuralLayer {
public:
    virtual int  inputCount()  const = 0;
    virtual int  outputCount() const = 0;
    virtual int  paramCount()  const = 0;      // genes this layer consumes
    virtual int  loadParams(const float* p, int offset) = 0;  // returns new offset
    virtual const double* forward(const double* in) = 0;      // owns its output buffer
    virtual void resetState() = 0;             // no-op for dense, clears h/c for LSTM
    virtual const std::vector<double>& activations() const = 0;  // for the mind view
};

forward() returns a pointer into a member buffer — no per-frame heap allocation anywhere in the hot path, unlike the current new double[]-per-layer-per-frame code.

DenseLayer — out[j] = act( Σ_i in[i]·W[i][j] + b[j] ), with the bias actually included this time. Per-unit activation functions so the output layer can mix tanh (continuous controls) with sigmoid (gates).

Parameters: (I + 1) · O.

LstmLayer — standard cell, one gate block:

z    = W_x·x_t + W_h·h_{t-1} + b        (4H wide: i | f | g | o)
i_t  = σ(z_i)      f_t = σ(z_f)      g_t = tanh(z_g)      o_t = σ(z_o)
c_t  = f_t ⊙ c_{t-1} + i_t ⊙ g_t
h_t  = o_t ⊙ tanh(c_t)

Parameters: 4H · (I + H + 1).

The forget-gate bias slice is initialised at +1.0 (Jozefowicz et al. 2015) so cells default to remembering; without it, evolution has to discover "don't forget" from scratch, which is slow and unstable.

State (h, c) is per agent, lives in the layer instance, and is cleared on respawn and at every epoch boundary — an agent must not inherit the previous life's memory.

2.2 Topology

input  40
  Dense(40 → 20, tanh)        421 ·  ... =   820 params   [(40+1)·20]
  LSTM (20 → 20)                          = 3,280 params  [4·20·(20+20+1)]
  Dense(20 → 12, mixed act)               =   252 params  [(20+1)·12]
                                            ─────────────
                                  genome  = 4,352 floats

One LSTM layer, not two. This is a deliberate size choice — see §6 on why the genome cannot be much larger than this given the sample budget.

2.3 Backprop?

No. Weights remain the genome and the evolutionary loop remains the only learning signal. This is the right call here: there is no supervision signal for "drove well", the agents are embodied in a physics sim with non-differentiable dynamics, and the whole population-visualisation premise of the game depends on genomes being visible objects. The LSTM contributes memory, not gradient-based training.

What does change is that the "GA" becomes closer to an evolution strategy, which is what actually works for direct policy search at this dimensionality (§5).

3. Sensory redesign (40 channels)

Replaces the current 4 raycasts + 3 flags. Defined once in ai/sensory.h as a struct with named fields overlaying a double[SENSE_DIM], so indices can never drift out of sync with the comment block the way they did in evolution_manager.cpp:163.

All channels are normalised — [0,1] for magnitudes, [-1,1] for signed quantities — because unnormalised raw metres (0.01…180) against tanh weights saturate instantly.

Exteroception (0–9)

Raycasts move from one cast every 2 s to a round-robin budget: with 9 rays and a cast budget of 3 per agent per frame, every ray refreshes every 3 frames (~50 ms at 60 fps) at a lower total cast cost than the current design's burst. The latched hit flag is removed — a miss must clear it.

Proprioception (10–18)

Channels 16–17 matter more than they look: feeding an agent its own last action is what lets a recurrent policy produce smooth, committed steering instead of per-frame jitter.

Navigation (19–23)

Derived from the track's SteerSpline centerline, which the game already maintains (AlgebraKart.h:1180, IsFarBelowTrack / ApplyTrackPullBack use it).

Algebra (24–34)

The equation currently exists only as display text (GenerateNewEquation(), AlgebraKart.cpp:12849). A parallel structured form is added — the generator already knows a, b, c, the template type and the five integer choices, it just throws them away after formatting the string.

Beat (35–38) and clock (39)

4. Action head (12 outputs)

The commit gate on 9 is what makes the algebra channel learnable: without it the argmax always fires and the agent answers randomly every frame, so correctness carries no signal. With it, "know when you don't know" is itself under selection.

Bots currently cannot answer at all — ProcessEquationAnswer() requires a Connection (AlgebraKart.cpp:12967) and bots have none. A bot-side entry point is added that takes an agent index instead.

5. Evolutionary loop

5.1 Wall-clock epochs

EpochManager replaces "a generation ends when every agent happens to die":

EPOCH_SECONDS      = 900.0    // 15 minutes
WAVES_PER_EPOCH    = 3        // sub-batches evaluated serially inside one epoch
POPULATION_SIZE    = 24       // = WAVES_PER_EPOCH × concurrent karts

The population size is bounded by how many karts can exist in the world at once (~8). Rather than accept a population of 8, an epoch evaluates three waves of 8 agents for 5 minutes each. That triples the population without tripling the physics load — the single most valuable change for search quality per §6.

Per epoch:

for wave in 0..WAVES_PER_EPOCH-1:
    spawn genomes [wave·8, wave·8+8) as agents, LSTM state cleared
    run for EPOCH_SECONDS / WAVES_PER_EPOCH
    freeze each agent's per-skill evaluation
    despawn
compute fitness across the whole population
select / recombine / mutate
checkpoint to disk
epoch++

Dying early no longer ends anything — it caps that agent's remaining accrual and frees its slot. Time, not death, is the clock. A 15-minute epoch at 60 fps is ~54,000 decision steps per agent, which is what makes an LSTM worth having.

5.2 Checkpointing

At every epoch boundary, Data/EvolutionManager/<run-id>/epoch_%04d.chk gets the full population (genomes + per-skill fitness) and run.csv gets one row per epoch (best/mean/worst per skill, diversity). On startup, the newest checkpoint in the newest run directory is resumed unless a fresh run is requested.

This is what makes "run for many hours" real. The current code's generation-100 restart (§5.10 of the baseline doc) throws the population away with nothing written to disk.

5.3 Selection and variation

The existing operators are replaced with ones that behave sanely at 4,352 dimensions:

• Selection: tournament, k = 3, with elitism 2 (top two copied verbatim). Remainder stochastic sampling is dropped — its break on fitness < 1 routinely produced intermediate populations smaller than 2, which froze evolution outright.
• Recombination: uniform per-gene crossover, p_swap = 0.5, passed by reference so offspring actually reach the caller.
• Mutation: Gaussian, not uniform, with a self-adaptive per-genome step size σ that is itself a gene (log-normal update, τ = 1/√n). Fixed ±2.0 uniform jolts at p = 0.3 per gene destroy 30% of a 4,352-gene network every generation; a genome that survives that is a genome whose weights do not matter.
• Diversity guard: if population variance falls below a threshold, replace the bottom quartile with random immigrants.

5.4 Heuristic bootstrap (curriculum)

Cold-starting a 4,352-parameter recurrent policy from random weights on a sparse driving reward is close to hopeless within a few hundred evaluations. The existing hand-written if/else controller (agent_controller.cpp:266-354) — currently the only thing driving the karts — becomes a teacher instead of being deleted:

blend = clamp(epoch / BOOTSTRAP_EPOCHS, 0, 1)      // BOOTSTRAP_EPOCHS = 8
control = (1 - blend)·heuristic + blend·network
fitness += imitationWeight(epoch) · agreement(network, heuristic)

Early epochs: the karts drive competently (heuristic), and genomes are rewarded for predicting what the heuristic would do — a free, dense, non-sparse learning signal. Later epochs: the blend hands control to the network and the imitation term decays to zero, leaving only task fitness. This replaces the current permanent ffnW = 0.0f, which is the same blend pinned at "never trust the network".

5.5 Fitness

Per-skill accumulators, each normalised to a z-score within the population before combining, so a skill can't dominate by unit scale:

F_drive   = w1·progressAlongCenterline        // not "time survived"
          + w2·meanSpeedOnTrack
          − w3·collisionImpulse
          − w4·timeOffTrack
          − w5·timeInverted

F_algebra = correctAnswers − 0.5·wrongAnswers + 0.25·(committed when confident)

F_beat    = onGridAccuracy                    // notes near the step boundary
          + scaleConsistency                  // notes inside the voice's scale
          + rhythmicVariety                   // entropy of the inter-onset histogram
          − densityPenalty                    // discourage constant triggering

fitness   = z(F_drive) + z(F_algebra) + z(F_beat) + imitationBonus(epoch)

F_drive is progress-based rather than the current time-based reward, whose floor of 0.8·dt·0.02 for standing still meant "survive" scored nearly as well as "drive" (baseline doc §4).

6. Honest constraint: sample budget

State this plainly rather than discover it eight hours in.

A 4,352-dimensional direct policy search with a population of 24 gets 24 evaluations per 15-minute epoch = 96 per hour. An eight-hour run is ~770 evaluations. For context, published neuroevolution results at this parameter count use populations in the hundreds to thousands and tens of thousands of generations.

So: expect visible improvement on driving, modest improvement on algebra, and treat beat-making as exploratory. Driving has a dense, well-shaped reward and a strong teacher signal (§5.4), so it should improve within a few epochs. Algebra is a 5-way choice with a clean correctness signal but only ~20 questions per 15-minute epoch — the bottleneck is question throughput, not the learner. Beats have no ground truth at all, only the hand-designed proxy above.

The design mitigations, in order of impact:

1. Waves (§5.1) — 3× the population for the same physics budget.
2. Heuristic bootstrap (§5.4) — converts a sparse task reward into a dense imitation reward for the first ~8 epochs.
3. Checkpoint/resume (§5.2) — a run is cumulative across sessions instead of starting cold every launch. This is the single biggest lever over days.
4. Self-adaptive σ (§5.3) — stops good genomes from being shredded by fixed large-amplitude mutation.
5. Small genome (§2.2) — one LSTM layer of width 20, not the 31k-parameter stack a "proper" LSTM policy would use.

If more capacity is wanted later, the escape hatch is to keep the topology and switch the outer loop from GA to CMA-ES or an OpenAI-ES style gradient estimate, both of which slot in behind the same INeuralLayer / genome interface. Nothing in stages 1–6 forecloses that.

7. Mind view (M key)

GenomeVisualization is fully written and never instantiated (baseline doc §6). It is replaced by MindVisualizer, which fixes the three things that stop it being useful:

1. Real topology — read from the network, not the hardcoded 9,10,8,3 that never matched the actual 8,7,5,3,3.
2. Real activations — INeuralLayer::activations() exposes the live forward-pass values, replacing UpdateNeuronColors()'s placeholder random numbers.
3. Actually wired up — registered, attached, updated, and toggled.
4. Assets that exist. GenomeVisualization loads Models/Sphere.mdl, Materials/UnlitSolid.xml and Materials/UnlitVertexColor.xml, none of which are present in this project (baseline doc §6) — it would have rendered nothing. The replacement builds all of its geometry procedurally as vertex-coloured quads and depends on exactly one new asset, Data/Materials/MindVCol.xml, which wraps the stock Techniques/NoTextureVColAddAlpha.xml. Additive blending means a neuron's alpha channel is its activation intensity, and overlapping edges brighten instead of z-fighting.

Presentation:

• Neurons as spheres in layered planes, positioned in front of the current camera so the "mind" hovers in the world near the agent it belongs to.
• Input neurons grouped and tinted by sensory block (exteroception / proprioception / navigation / algebra / beat), so you can see which sense is driving a decision.
• LSTM layer drawn as three concentric rings per cell — hidden state h, cell state c, and the forget gate — since a single sphere cannot express a unit that has memory. This is the visual payoff of the whole change: you can watch a cell hold a value across a corner.
• Connections coloured by weight sign, opacity by |weight|, and pulse-animated by the product activation × weight so signal flow is visible.
• A UI panel with agent name, epoch, wave, per-skill fitness, and rank.

Key bindings. KEY_M is currently bound twice already — camera-mode cycling (AlgebraKart.cpp:4605) and the audio master panel (AlgebraKart.cpp:12059). Both move:

Inside the mind view: [ / ] cycle which agent's mind is shown, M again dismisses.

7a. Implementation status

Everything in stages 1–6 is implemented and building. What is worth knowing about the gaps:

7b. Running a long training session

Default settings give 15-minute epochs, so an overnight run is ~32 epochs:

./AlgebraKart -headless

To exercise the whole evolution loop quickly (waves, selection, checkpoint, resume) without waiting 15 minutes per epoch:

ALGEBRAKART_EPOCH_SECONDS=15 ./AlgebraKart -headless

Output lands in Data/EvolutionManager/run-<timestamp>/:

• epoch_%04d.chk — the full population (genomes, per-skill scores, sigma) at each epoch boundary. Plain text, so it survives a rebuild; a checkpoint whose genome length does not match the current network shape is rejected with a warning rather than reinterpreted.
• run.csv — one row per epoch: best/mean fitness, best/mean per skill, population diversity, mean sigma, and the current network authority. This is the file to plot to answer "is it learning".

Restarting resumes from the newest checkpoint of the newest run automatically. Delete the run directory to start cold.

Note on the algebra channel: UpdateEquationSystem() returns early unless raceStarted_, so a bots-only headless run with no players never issues an equation and algebraScore stays at 0 for every genotype. Algebra fitness only accumulates once a race is actually underway.

8. Configuration surface

New constants in Constants.h, all overridable so a long run can be tuned without a recompile of the whole game:

#define SENSE_DIM              40
#define ACTION_DIM             12
#define EPOCH_SECONDS          900.0f   // 15 minutes
#define WAVES_PER_EPOCH        3
#define POPULATION_SIZE        24
#define BOOTSTRAP_EPOCHS       8
#define LSTM_HIDDEN            20
#define TOURNAMENT_K           3
#define ELITE_COUNT            2
#define CHECKPOINT_DIR         "Data/EvolutionManager/"

9. Acceptance checks