# Clean Heuristic Learning Playground

This repository is intentionally blank.

It is not a copy of `learning-beyond-gradients`, and it does not include starter
code. Use this space to learn the method from first principles by asking Codex
to create each piece as you need it.

## Purpose

Reproduce the workflow, not the original implementation:

```text
observe the environment
-> write a small experiment
-> run it
-> inspect evidence
-> ask Codex to update the code
-> record what changed
-> repeat
```

## Suggested Starting Prompt

Ask Codex something like:

```text
We are building a Pong heuristic learning system from scratch.

Do not copy code from learning-beyond-gradients.
Start by creating the smallest environment inspection script:
- create EnvPool Pong-v5
- reset once
- print obs shape
- print action space
- print info keys
- print the first 64 RAM bytes
- step 20 frames with no-op
- print reward, done, and a few RAM bytes each step

After writing it, run it and summarize what we learned.
```

## Recreate The Environment

This repo intentionally has no dependency files yet. Create them when Codex
generates the first experiment. Until then, the minimal Pong environment can be
rebuilt manually like this.

Use Python 3.11:

```bash
cd /path/to/hl-clean-playground
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip setuptools wheel
```

Install the minimum packages for Atari/Pong experiments:

```bash
python -m pip install envpool==1.2.0 numpy==1.26.4 pillow
```

On macOS, EnvPool's wheel may need Homebrew Qt 5:

```bash
brew install qt@5
```

If EnvPool is slow to download on macOS arm64, download the wheel on a faster
machine and install from a local wheelhouse. Example target wheel:

```text
envpool-1.2.0-cp311-cp311-macosx_11_0_arm64.whl
```

A local install looks like:

```bash
python -m pip install \
  --no-index \
  --find-links /path/to/wheelhouse \
  /path/to/wheelhouse/envpool-1.2.0-cp311-cp311-macosx_11_0_arm64.whl
python -m pip install numpy==1.26.4 pillow
```

For faster CPU rollout on Windows, use WSL2 and keep the repository inside the
Linux filesystem, not under `/mnt/c`. A minimal Ubuntu setup is:

```bash
sudo apt update
sudo apt install -y python3.11 python3.11-venv python3.11-dev build-essential git

cd ~/work/hl-clean-playground
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip setuptools wheel
python -m pip install envpool==1.2.0 numpy==1.26.4 pillow
```

Verify WSL reproducibility with a no-video smoke run:

```bash
PYTHONPATH=. python tests/test_state_policy.py
python scripts/run_eval.py \
  --episodes 80 \
  --seed 0 \
  --max-steps 30000 \
  --policy-name ram_intercept_v3_28_wsl_check \
  --notes "WSL reproducibility check for retained policy" \
  --no-video
```

The retained v3.28 baseline over seeds `0..79` is mean `19.025`, min `18`,
max `20`; a matching WSL run should reproduce those seed-level scores.

After Codex creates the first inspection script, verify that it can:

```text
import envpool
create Pong-v5
reset once
step a few frames
read info["ram"]
```

## Learning Order

1. Inspect the environment.
2. Probe the action meanings.
3. Probe which RAM bytes correspond to ball and paddle state.
4. Cross-check RAM state against RGB diagnostics when the policy fails.
5. Write a naive policy.
6. Add trial logging.
7. Add a summary CSV.
8. Analyze failures.
9. Improve the policy.
10. Add GIF/video recording only after the loop works.

## Current Experiment Workflow

Use `scripts/run_eval.py` as the normal evaluation entry point. It should:

- append trial records to `trials.jsonl`
- append per-episode records to `episodes.jsonl`
- refresh `summary.csv`
- save per-episode RGB GIFs under `videos/` for human inspection
- save per-episode OBS GIFs under `videos/` for checking what the policy sees
- overlay RAM-derived state on RGB GIFs by default: inferred ball, paddles,
  action, reward, and score
- record one GIF frame per environment step by default
- preserve duplicate visual frames in GIFs with a tiny one-pixel frame marker
- rebuild `report.html`

Keep `report.html` open in the browser while experimenting. After each eval run,
refresh the page to see the latest scores, RGB videos, and OBS videos.

Do not generate alternate-speed GIFs during eval runs; it slows down the
experiment loop. Generate speed variants later with `scripts/make_video_speeds.py`
when needed. In the Go report server, the report shows on-demand controls for
`0.25x`, `0.5x`, `1x`, `2x`, and `3x`; clicking a missing speed generates it
from the original GIF without rerunning the episode.

For an always-on local report, run the Go server:

```bash
go run ./server
```

Then open `http://127.0.0.1:8080/`. The server rebuilds `report.html` on page
refresh, so it does not need to restart after each experiment. It also serves
local GIFs and can generate speed variants from existing recordings on demand:

```bash
curl -X POST \
  'http://127.0.0.1:8080/api/speed?path=videos/example_rgb.gif&label=2x'
```

This creates the derived GIF and refreshes the report links without rerunning
the episode.

The report is a history view, not only the latest run. It should show all
rounds, the round number, what changed in that round from `--notes`, the score
summary, a suggested next step, and the videos for each episode in that round.
Keep this report in Chinese by default, with a compact/high-density layout so
many rounds and videos can be scanned on one page.

## Iteration Loop

Use this repository as a closed loop:

```text
probe actions and observations
-> update `pong_state.py`
-> update `pong_policy.py`
-> run full episodes with `scripts/run_eval.py`
-> inspect `trials.jsonl`, `summary.csv`, and report videos
-> cross-check RAM assumptions with `scripts/probe_visual_ram.py` if RGB and RAM disagree
-> record failure notes with `scripts/record_failure.py`
-> make one targeted policy/state change
-> run regression tests
-> simplify code when behavior stabilizes
```

Keep state detection separate from policy logic. Policy code should consume
`PongState` instead of directly indexing RAM whenever possible. Record failure
diagnoses in `failures.jsonl`; it is local experiment evidence and should stay
out of commits unless intentionally curated.

Current RGB/RAM calibration from `scripts/probe_visual_ram.py`: the visible ball
is approximately `x = ram[49] - 48.5`, `y = ram[54] - 12.5`. Use this for video
overlays and visual diagnostics; keep policy decisions in RAM coordinates unless
a failure note shows that comparison is wrong.
The visible right-paddle center is approximately `y = ram[51] + 2.75`.
Reward-event probes over seeds 0-2 show `ram[14]` increments on `reward=+1`
(right/player score) and `ram[13]` increments on `reward=-1` (left/opponent
score). `PongState` now exposes these as `self_score` and `opponent_score`.

Use `scripts/trace_points.py` for point-level diagnostics before changing the
policy. It cuts an episode by score changes, writes each point window to JSONL,
and summarizes repeated loss patterns:

```bash
python scripts/trace_points.py \
  --seeds 0,1,3,4,5,8,9 \
  --max-steps 30000 \
  --window 80 \
  --tail 0 \
  --jsonl diagnostics/point_traces_seeds_0_1_3_4_5_8_9.jsonl
```

Current point-level diagnosis from seeds 0, 1, 3, 4, 5, 8, and 9:

- Seeds 0 and 9 repeat two alternating late-right losses: a high/downward ball
  near `(x=203,y=173,dy=8)` with paddle about 9 RAM units below target, and a
  mid/upward ball near `(x=203,y=164,dy=-8)` where the paddle is numerically
  near target but still misses.
- Seed 1 repeats high/downward losses near `(x=203,y=185,dy=8)` and mid/downward
  losses near `(x=202,y=86,dy=4)`, both with the paddle above target by about
  9-16 RAM units.
- Seeds 5 and 8 repeat the same mid/downward `(x=202,y=86,dy=4)` loss plus a
  near-target `(x=202,y=130,dy=4)` loss.
- Seeds 3 and 4 survive to one very late loss only, around step 16300, so they
  are less useful for first policy changes than the highly repeated patterns.

Candidate hypotheses, not yet policy changes:

- The residual failures are phase-specific late-return misses, not a missing RAM
  byte or global coordinate offset.
- Several repeated losses have target and paddle within roughly 0-7 RAM units at
  the final sampled frame, so a pure target offset can be misleading; the Atari
  collision point may depend on paddle edge, action phase, or the last few
  frames of movement.
- The next safe experiment should target one repeated family at a time, starting
  with the `(x≈202,y≈86,dy=4)` low/mid downward miss because it appears in seeds
  1, 5, and 8.

Current policy direction: use a small stateful RAM policy that estimates ball
velocity from consecutive `PongState` values and aims at the predicted right-side
intercept. Treat this as an experiment, not a solved controller.

Current RAM policy constants were chosen from no-video sweeps over seeds 0-4:
`MIN_PLAY_Y_RAM=46`, `MAX_PLAY_Y_RAM=210`, `SELF_PADDLE_X_RAM=192`,
`PADDLE_DEADBAND=3`, target clamp `46..196`, and inactive/serve-ready center
`100`.
The policy also has a narrow low-serve pattern rule: when a point-reset serve is
detected around `x=126..138`, `y=116..128`, moving up-right, it temporarily aims
at `LOW_SERVE_TARGET_Y=50` for 24 policy steps. This was added because the
generic intercept controller repeatedly failed that serve pattern on seed 0.
The current retained policy is `ram_intercept_v3_28_chain2_score00_initial_up`: it keeps
the `ram_intercept_v3_3` controller and adds narrow action-phase corrections.
At `0-0`, if a low `dy=8` ball reaches `x=184..188,y=130..145` while the right
paddle is recovering upward, the policy executes one `UP` instead of baseline
`DOWN`; this improved the 10-seed gate from mean `0.5`, min `-1`, max `9` to
mean `0.9`, min `-1`, max `9`. A later low-serve correction changes only
`target=50` `NOOP` decisions at `x=189..201,y=55..67,paddle=35..65` to `UP`;
this fixed expanded seed 13 and improved the 20-seed gate from mean `0.2`, min
`-21`, max `9` to mean `1.2`, min `-1`, max `9`. The latest retained correction
changes only `dy=4` pre-contact `UP` decisions at `x=186..194,y=68..82,paddle=65..90`
to `NOOP`; this improves the 20-seed gate to mean `1.5`, min `-1`, max `9`.
The current newest correction changes only `dy=-8` mid-up pre-contact `UP`
decisions at `x=187..191,y=188..196,paddle=190..205` to `NOOP`; this improves
the 20-seed gate to mean `4.6`, min `-1`, max `20`. The latest high-down setup
correction changes only `dy=8`, `x=121..153,y=103..167,paddle=155..195`,
`target=165..185` baseline `NOOP` decisions to `DOWN`; this improves the
20-seed gate to mean `9.4`, min `-1`, max `20`. The late low-loop correction
changes a late low-loop `dy=3..4`, `x=169..177,y=49..57,paddle=80..105`,
`target=60..75` baseline `UP` to `NOOP`, capped at 64 uses per episode to match
the validated counterfactual probe; this improves the 20-seed gate to mean
`12.9`, min `9`, max `19`. Expanded validation over seeds `0..39` found a new
seed `33` collapse, so the current `ram_intercept_v3_10_low_up_score01_noop`
adds one score-gated correction at `0-1`, `dy=-8`, `x=176..178,y=172..176`,
`paddle=132..138,target=135..150`, changing baseline `DOWN` to `NOOP` once per
episode. The 40-seed gate is now mean `11.65`, min `-1`, max `19`. Expanded
validation over seeds `40..79` found seed `44` and seed `66` share the same
score `-1` weak family. The current `ram_intercept_v3_11_low_mid_score00_noop`
adds a score-`0-0` low/mid correction for `dy=4`,
`x=177..181,y=53..57,paddle=45..55,target=65..70`, changing baseline `DOWN` to
`NOOP`, capped at 16 uses per episode to match the validated probe. The 80-seed
gate over seeds `0..79` became mean `12.75`, min `9`, max `19`. The latest
`ram_intercept_v3_12_low_up_y111_down` correction changes only low-up `dy=-8`,
`x=187..195,y=127..143,paddle=130..150,target=125..135` baseline `UP` decisions
to `DOWN`, capped at 64 uses per episode. The 80-seed gate over seeds `0..79`
became mean `13.1875`, min `9`, max `19`. The latest
`ram_intercept_v3_13_y86_score22_setup_down` correction changes one score-`2-2`
y86 setup action at `dy=4,x=172..176,y=56..60,paddle=84..90,target=74..78`
from baseline `UP` to `DOWN`. The 80-seed gate over seeds `0..79` is now mean
`13.5125`, min `10`, max `19`. The latest
`ram_intercept_v3_14_y86_score42_setup_down` correction applies the same one-shot
y86 setup `UP` to `DOWN` change at score `4-2`. The 80-seed gate over seeds
`0..79` became mean `13.8375`, min `11`, max `19`. The latest
`ram_intercept_v3_15_y86_score62_early_setup_down` correction applies the same
one-shot y86 setup change at score `6-2` only during step window `1500..1700`;
the ungated score-`6-2` correction was rejected because it collapsed hidden
seeds. The 80-seed gate over seeds `0..79` became mean `14.1625`, min `11`,
max `19`. The latest `ram_intercept_v3_16_y86_score20_serve_initial_up`
correction changes one score-`2-0` point-reset first active frame at
`x=124..128,y=124..128,paddle=96..106` from baseline `DOWN` to `UP`; the
`DOWN -> NOOP` variant was rejected. The 80-seed gate over seeds `0..79` is now
mean `14.3125`, min `12`, max `19`. The latest
`ram_intercept_v3_17_y86_score82_early_setup_down` correction applies the same
one-shot y86 setup `UP` to `DOWN` change at score `8-2`, but only during step
window `2050..2110`; the ungated score-`8-2` correction was rejected because it
improved early y86 seeds while collapsing later hidden phase seeds. The 80-seed
gate over seeds `0..79` became mean `14.6375`, min `12`, max `19`. The latest
`ram_intercept_v3_18_high_down_score40_bounce_noop` correction changes only the
score-`4-0` high-down bounce frame `dy=9,x=144..148,y=50..56,paddle=143..150,target=154..162`
from baseline `DOWN` to `NOOP`; the ungated version improved mean but regressed
score-`0-0` hidden seeds, and `DOWN -> UP` collapsed hidden high-down seeds. The
80-seed gate over seeds `0..79` became mean `14.9125`, min `13`, max `19`.
The latest `ram_intercept_v3_19_y86_after_score22_bounce_up` correction adds a
small state marker: after an episode has seen the score-`2-2` y86 x166 bounce
family, later `dy=5,x=164..168,y=48..52,paddle=76..80,target=80..85` baseline
`DOWN` actions are changed to `UP`. Direct score-`2-2` only was inert, ungated
`DOWN -> UP` improved one y86 family while collapsing another, and excluding only
score `6-0` still exposed score `8-1` regressions. The stateful 80-seed gate over
seeds `0..79` became mean `16.8625`, min `13`, max `19`. The latest
`ram_intercept_v3_20_y86_score60_serve_initial_noop` correction changes one
pure y86 score-`6-0` point-reset first active frame at
`x=124..128,y=124..128,paddle=96..106` from baseline `DOWN` to `NOOP`. The
corresponding first x166 bounce `DOWN -> NOOP` was inert and `DOWN -> UP`
collapsed the pure family, but the earlier serve-initial `DOWN -> NOOP` and
`DOWN -> UP` probes both improved the same 22 floor seeds without regressions;
the retained policy uses `NOOP` as the smaller action change. The 80-seed gate
over seeds `0..79` became mean `17.1375`, min `14`, max `19`. The latest
`ram_intercept_v3_21_high_down_score80_bounce_noop` correction changes one
score-`8-0` high-down bounce frame at
`dy=9,x=144..148,y=50..56,paddle=143..150,target=154..162` from baseline `DOWN`
to `NOOP`. This mirrors the retained score-`4-0` high-down bounce correction;
the one-shot `DOWN -> NOOP` probe improved the same 22 floor seeds without
regressions, while `DOWN -> UP` was inert. The 80-seed gate over seeds `0..79`
became mean `17.4125`, min `15`, max `19`. The latest
`ram_intercept_v3_22_y86_score100_serve_initial_noop` correction changes one
score-`10-0` y86 point-reset first active frame at
`x=124..128,y=124..128,paddle=96..106` from baseline `DOWN` to `NOOP`. Round 49
setup diagnostics showed this score-`10-0` chain shares the same reset-to-x166
geometry as the retained score-`6-0` serve-initial family; 80-seed
counterfactuals for `DOWN -> NOOP` and `DOWN -> UP` both improved the same 22
floor seeds without regressions, so the retained policy uses `NOOP` as the
smaller action change. The 80-seed gate over seeds `0..79` is now mean
`17.6875`, min `16`, max `19`. The latest
`ram_intercept_v3_23_high_down_score120_bounce_noop` correction changes one
score-`12-0` high-down bounce frame at
`dy=9,x=144..148,y=50..56,paddle=143..150,target=154..162` from baseline `DOWN`
to `NOOP`. The score-`16` floor trace showed 22 of 24 floor seeds had repeated
high-down y165 losses at score chain `12-0,14-1,16-2,18-3,20-4`; the one-shot
`DOWN -> NOOP` probe improved those 22 seeds without regressions, while
`DOWN -> UP` was inert. The 80-seed gate over seeds `0..79` is now mean
`17.9625`, min `16`, max `19`; the min is now held by the separate mixed
early-loss seeds `2` and `15`. The latest
`ram_intercept_v3_24_score00_mid_down_y154_noop` correction changes one opening
score-`0-0` mid-down final approach at
`dy=4,x=198..204,y=148..156,paddle=150..170,target=150..156` from baseline `UP`
to `NOOP`, with an additional `previous_paddle_delta >= 10` guard and one-shot
limit. The mixed seed `2`/`15` trace showed this is the shared earliest loss
anchor; full 80-seed counterfactuals improved only seeds `2` and `15` by `+2`
with no regressions, while `UP -> DOWN` was inert. The 80-seed gate over seeds
`0..79` became mean `18.0125`, min `17`, max `19`. The latest
`ram_intercept_v3_25_y86_score140_serve_initial_noop` correction changes one
score-`14-0` y86 point-reset first active frame at
`x=124..128,y=124..128,paddle=96..106` from baseline `DOWN` to `NOOP`. Setup
diagnostics showed this score-`14-0` chain shares the same reset-to-x166
geometry as the retained score-`6-0` and score-`10-0` y86 serve-initial
families; 80-seed counterfactuals for `DOWN -> NOOP` and `DOWN -> UP` both
improved the same 22 y86 floor seeds without regressions, so the retained policy
uses `NOOP` as the smaller action change. The 80-seed gate over seeds `0..79`
became mean `18.2875`, min `17`, max `19`. The latest
`ram_intercept_v3_26_low_up_score10_final_noop` correction changes one score-`1-0`
low-up final frame at `dy=-8,x=202..204,y=118..122,paddle=113..117,target=118..122`
from baseline `DOWN` to `NOOP`, with a `previous_paddle_delta <= -10` guard and
one-shot limit. The v3.25 score-`17` floor trace showed the remaining 11 floor
seeds all shared `loss/low_up/near_target` at `x=204,y=120,paddle=115,target=120`;
the final-frame `DOWN -> NOOP` counterfactual improved those 11 seeds without
regressions, while final `DOWN -> UP` was weaker and offset-2 `NOOP -> DOWN` was
inert. The 80-seed gate over seeds `0..79` became mean `18.7`, min `18`, max
`20`. The latest `ram_intercept_v3_27_high_down_score160_bounce_noop`
correction changes one score-`16-0` high-down y165 bounce frame at
`dy=9,x=144..148,y=50..56,paddle=143..150,target=154..162` from baseline
`DOWN` to `NOOP`. The v3.26 score-`18` floor trace showed 22 seeds share a
repeated high-down y165 chain at pre-loss scores `16-0,18-1,20-2`; separate
score-gated counterfactuals at all three links were equivalent, improving those
22 seeds by `+1` without regressions, so the retained policy uses only the
earliest score-`16-0` one-shot guard. The 80-seed gate over seeds `0..79`
became mean `18.975`, min `18`, max `20`. The latest
`ram_intercept_v3_28_chain2_score00_initial_up` correction changes one opening
score-`0-0` first active frame for the 2-seed residual chain at
`x=125,y=131,paddle=83..87,target=129..133`, with `previous_paddle_delta=-23..-19`,
from baseline `DOWN` to `UP`. Rounds 63-65 first rejected y171/y203 endpoint
patches and then showed this chain has a specific pre-outcome family marker;
the full 80-seed counterfactual improved only seeds `2` and `15` from `18` to
`20`, with no regressions. The formal 80-seed gate over seeds `0..79` is now
mean `19.025`, min `18`, max `20`.

## What Belongs Here

Code generated during your learning process belongs here.

Generated artifacts should stay local unless you intentionally want to commit
them:

- virtual environments
- trial logs
- summaries
- screenshots
- GIFs
- videos
