# HalfCheetah-v5 heuristic policy log

Constraint: no neural network.  Target: **>= 12,000 episode return**.  The optional staged tree MPC path now reaches the 12k target on seed100; the stretch target is **>= 13,000** and is not met yet.  The non-MPC asymmetric-phase scalar PD controller remains a baseline only.

Search is only over explicit scalar gait / target-angle / reflex parameters.  The script's JSONL CEM mode records per-iteration sampled frames, sampled episodes, scores, returns, and the search action.

| Iteration | Sampled frames | Sampled episodes | Score / result | What changed |
| --- | ---: | ---: | --- | --- |
| 0 | 0 | 0 | random/zero baselines <= ~1 return; naive sin best ~545 | Confirmed Gymnasium `HalfCheetah-v5` runs locally; action order is `[bthigh, bshin, bfoot, fthigh, fshin, ffoot]`; reward is forward velocity minus control cost. |
| 1 | 2,400,000 | 2,400 | train best 3,069; validation 2,927 +/- 56 on 10 held-out seeds | Ran CEM over a pure open-loop 2-harmonic Fourier torque gait: bias + sin/cos + second harmonic per actuator. |
| 2 | 2,401,000 | 2,401 | train best 4,008; validation 2,185 +/- 1,677 on 10 seeds | Added local joint angle/velocity + torso pitch/pitch-rate reflexes.  Rejected this single-seed-selected best because it overfit reset noise and sometimes ran backward. |
| 3 | 3,459,000 | 3,459 | 3-seed train mean 3,650; independent 30-episode mean 3,310, max 3,680, min 48 | Re-ran reflex CEM with 3 reset seeds per candidate and score = mean - 0.20 * std.  This is the current checked-in reflex parameter set; it is higher scoring but still has rare bad starts. |
| 4 | 1,800,000+ | 1,800+ | fallback portfolio not adopted | Tried switching from reflex to open-loop on low speed / bad attitude.  Simple switching often made failures worse because oscillator phase changed mid-fall. |
| 5 | 900,000 | 900 | 100-seed vector check: reflex mean 3,401 / max 3,735 / 6 starts below 2,000; action blends all lower mean | Compared open-loop, reflex, and action-level blends on the same 100 resets.  Kept full reflex as the high-score default; do not use naive action interpolation. |
| 6 | ~2,905,000 | ~2,905 | rejected: linear ARS-from-zero best ~2,099 | Tried a no-hidden-layer affine observation controller.  It learned too slowly and never caught the reflex baseline. |
| 7 | ~1,991,000 | ~1,991 | rejected: periodic linear/reflex ARS candidates stayed <= ~2,430 | Initialized a linear periodic-feedback matrix from the reflex gait.  Perturbations destroyed the gait instead of improving it; stopped instead of continuing to burn episodes. |
| 8 | ~1,025,000 | ~1,025 | rejected: 20-step open-loop torque table CEM best ~1,009 | Used `mujoco.rollout` to batch periodic torque-table episodes.  Phase-averaging the reflex gait was a bad initializer for a 20-step table. |
| 9 | ~11,200 model decision-frames | 0 full episodes | 200-step iLQR smoke: 658.9 -> 667.9 on the short-horizon objective | Verified MuJoCo rollout / finite-difference dynamics path.  Full-horizon terminal-distance adjoints were numerically unstable, so this is not the current policy path. |
| 10 | 971,000 | 971 | single-seed PD-CGP CEM train best 4,839; 10-seed mean for that overfit best 4,242 with one fall | Fitted a PD target-angle CPG to imitate the reflex gait, then searched target-angle Fourier coefficients.  This was the first policy family that clearly improved speed, but single-seed selection overfit. |
| 11 | 1,370,000 | 1,370 | selected robust PD-CGP: 10-seed mean 4,569 / std 143 / min 4,157 / max 4,665 | Re-ran the same 10-iteration PD-CGP search with extra 10-seed validation of the top candidates and selected by validation mean.  This remains as a fallback. |
| 12 | 1,382,400 model planning frames | 0 training episodes; 300 executed steps | rejected: naive 30-step shooting MPC got 1,137 return in 300 executed steps; linear 1000-step projection ~3,791 | Used `mujoco.rollout` online with horizon 30, block 5, population 256, 3 CEM rounds per replan.  It was slower than the checked-in PD gait and should not be continued in this form. |
| 13 | 3,841,000 | 3,841 | rejected: low-frequency open-loop residual CEM best 4,863 on one reset, below the closed-loop PD baseline | Froze one rollout of the PD-CGP policy, added a spline-like open-loop action residual, and optimized with CEM.  Open-loop residuals did not stay on the closed-loop gait attractor. |
| 14 | 2,026,000 | 2,026 | selected asymmetric PD-CGP: 10-seed mean 4,800 / std 64 / min 4,652 / max 4,892 | Added a two-rate phase oscillator: one frequency for one half-cycle, another for the other half-cycle.  CEM validation selected the current checked-in default `asym-pd-cpg`. |
| 15 | 928,000 sampled before stop | 928 sampled before stop | rejected: phase-table search around `asym-pd-cpg` had no validation improvement | Tried replacing the Fourier target with a phase table initialized from the current asymmetric gait.  Stopped early because the first batches degraded return. |
| 16 | 736,000 | 736 | rejected: linear feedback residual around `asym-pd-cpg` did not beat zero feedback | Ran a small feedback-CEM smoke on an affine observation residual.  The baseline residual remained best. |
| 17 | 1,952,000 completed scored frames | 1,952 completed scored episodes | rejected: expert-compression probes; best completed finite-state expert-template controller scored 1,855; full-dataset KNN max 2,544 on 10 seeds; nearest-state expert+PD oracle scored 838 | Downloaded a public HalfCheetah expert dataset for diagnosis only.  Found the expert runs a 6-env-step high-frequency gait.  Tested action replay, 6-step torque tables, 6-state template FSM, nearest-expert-state action, full 1M-state KNN action lookup, nearest-expert-state action+PD, and 1000-step expert target PD; all failed from Gym reset and are not checked in. |
| 18 | 710,000 | 710 | rejected: best 10-seed target/velocity tweak mean 4,801 vs baseline 4,800 | Swept named target-angle biases, target gains, and a single root-velocity feedback gain around `asym-pd-cpg`.  Some single resets reached ~4,970, but 10-seed validation did not improve reliably. |
| 19 | 1,401,000 | 1,401 | rejected: ARS linear residual train candidates up to 4,963; selected residual stayed zero, baseline 4,838 on its seed | Ran a conservative antithetic ARS smoke for a no-hidden-layer observation residual on top of `asym-pd-cpg`.  Validation never beat the fixed gait, so the residual is not checked in. |
| 20 | 3,961,000 | 3,961 | rejected as current policy: normalized linear ARS best single-seed validation 2,835 after 60 iterations | Fixed the earlier broken linear-ARS-from-zero attempt: added running observation normalization, antithetic directions, top-direction update, and validation.  It learns a running controller from zero, but it is still below the checked-in asymmetric PD gait. |
| 21 | 4,000,000 warm-start frames | 4,000 warm rollouts | rejected: 20-step target-angle table improved from 2,463 to 3,551 from the warm state, still below the Fourier feedback baseline's 4,920 warm-state rollout | Optimized a periodic PD target table from the current policy's t=500 state.  Direct target tables lost the closed-loop Fourier attractor and did not become a faster orbit. |
| 22 | 4,000,000 warm-start frames | 4,000 warm rollouts | rejected as reset policy: 2-rate Fourier raw improved warm rollout 4,920 -> 5,359, but reset 10-seed mean fell to 2,113 / min -487 | Re-optimized the existing `asym-pd-cpg` raw parameters while starting every candidate from the current policy's t=500 state.  This finds a slightly faster cruise orbit, but it cannot start itself from reset. |
| 23 | 80,000 | 80 | rejected: best baseline-then-warm-raw state machine 10-seed mean 4,725, below the 4,800 baseline | Tested switch times 100..800 that start with the checked-in baseline and then switch to the warm-optimized raw parameters.  Several starts fell off the warm-only orbit. |
| 24 | ~350,000 model decision-frames + 1,000 executed frames | 1 executed episode | added as optional diagnostic policy: `mpc-asym-pd-cpg` scored 5,741.5 on seed 100 | Implemented a pure heuristic model-based overlay.  At each env step it scores small perturbations to the current baseline action by copying the MuJoCo state, taking that first action, then following the baseline closed-loop policy for a 14-step tail. |
| 25 | ~1,500,000 model decision-frames + 1,500 executed frames | 5 partial executed rollouts | rejected: greedy shooting variants scored 4.7..64 return over 300 steps, versus baseline 1,384 over 300 | Tested one-step and repeated-action random-shooting MPC without the closed-loop tail.  It optimizes immediate motion and destroys the gait. |
| 26 | 1,536,000 model decision-frames + 300 executed frames | 1 partial executed rollout | rejected: open-loop sequence CEM-MPC scored -54 over 300 steps | Optimized a 25-step open-loop action sequence with CEM, executed 5 actions, and replanned.  The planner found low-speed folded trajectories; do not continue this unregularized sequence-MPC version. |
| 27 | ~1,200,000 model / env frames | diagnostic only | rejected: expert joint-angle PD <= 113 on periodic probes; expert action permutation/sign replay <= 386 over 300 steps | Checked whether the public expert data failed because of action order/sign or because actions were the only non-portable part.  Neither target joint angles nor transformed expert actions reproduce a v5 gait. |
| 28 | 2,048,000 warm-start frames before manual stop | 2,048 warm rollouts | rejected: 5-harmonic target-angle Fourier warm search reached 5,217, below the simpler 2-harmonic warm best 5,359 | Widened the PD target from 2 harmonics to 5 and optimized from the same warm state.  The higher-dimensional search plateaued early, so it was stopped. |
| 29 | ~576,000 model/env scoring frames + 300 executed frames | 1 partial executed rollout | rejected: 4-action-prefix MPC with baseline tail scored 1,368 over 300, slightly below baseline 1,384 | Optimized a short 4-action prefix in vector env, then scored the baseline closed-loop tail.  The optimizer repeatedly selected the baseline candidate and did not improve execution. |
| 30 | 5,000 executed frames + online model scoring | 5 executed episodes | superseded: narrow-candidate `mpc-asym-pd-cpg` 5-seed mean 5,725 / std 14.6 / min 5,705 / max 5,741 | Validated the first-action / closed-loop-tail MPC overlay over seeds 100..104.  It is robustly better than the baseline but still far below target velocity. |
| 31 | ~9,000,000+ scoring frames | diagnostic only | rejected / not checked in: reset-identical P=5 open-loop torque table optimized 5,190 -> 5,740 on seed100, but 10-seed mean was 2,923 / min -376; baseline->table switches peaked at 10-seed mean 5,066 | Revisited periodic bang-bang torque tables after discovering that `gym.make_vec(...).reset(seed=s)` assigns different reset seeds per sub-env.  Fixed table evaluation by setting every sub-env to the same qpos/qvel.  The table can be fast on one reset but is not robust. |
| 32 | ~2,000,000 model decision-frames + 3,000 executed frames | 3 executed episodes | superseded: widened `mpc-asym-pd-cpg` 3-seed smoke mean 5,938.2 / min 5,918.0 / max 5,958.6 for seeds 100..102 | Widened the online MPC action candidate set from one coordinate delta to deltas `+/-0.15` and `+/-0.35`, increased random perturbations to 32 with std 0.20, and kept the 14-step baseline tail.  A scratch RNG variant reached 5,944.0 on seed100; the checked script returned 5,918.0 on seed100. |
| 33 | ~4,000,000 online model decision-frames + 2,000 executed frames | 2 executed episodes | current best checked script path: `mpc-asym-pd-cpg` returned 6,041.1 / 6,058.5 on seeds 100/101 | Increased the online MPC candidate set again: coordinate deltas are now `+/-0.10`, `+/-0.25`, `+/-0.50`; random candidates are now two blocks, 32 at std 0.15 and 32 at std 0.35.  300-step smoke improved from ~1,675 to ~1,714 before running full episodes. |
| 34 | ~1,000 executed frames + online model scoring | 1 executed episode | rejected: phase-binned mean residual fitted from a 5,918-point MPC rollout gave 3-seed means below baseline for tested gains | Collected widened-MPC actions and averaged action deltas into 16 oscillator phase bins.  Cheap playback of that table was unstable, confirming that MPC gains are state/model-conditional and not a fixed phase correction. |
| 35 | diagnostic only | 0 full episodes | rejected / calibration only: early free-state orbit smoke produced impossible-looking speeds until root-x bookkeeping was fixed | Started a free initial-state + periodic torque-table search.  Found two harness bugs before trusting it: the MuJoCo rollout control sequence must cover all `H * frame_skip` simulator steps, and baseline warm states must reset or subtract `qpos[0]` before computing reward. |
| 36 | ~2,700,000 corrected model decision-frames | 0 full episodes | rejected: best free-state P=6 orbit candidate scored 683 over 120 env steps, then only 811 over 300 and 1,449 over 1000 | Ran corrected CEM over 17 initial-state scalars + 6x6 periodic torque table, initialized from checked baseline states and the public expert observations/actions.  The search improved the short-horizon periodic objective but did not find a stable high-speed limit cycle. |
| 37 | ~2,000 executed frames + reduced online model scoring | partial rollouts only | rejected: reduced-candidate MPC terminal-velocity scan peaked at 807 over 160 steps; high terminal-velocity weights were worse | Swept first-action MPC terminal root-velocity bonus and horizon with a cheap candidate set.  Horizon 14 remained the local best; terminal weights 10/20/40 made the 160-step smoke worse rather than accelerating toward a 12k gait. |
| 38 | ~660 executed frames + vectorized open-loop-sequence model scoring | 2 partial executed sequence-MPC rollouts | rejected: residual sequence MPC returned 1,027 and 908 over 220 steps; baseline was 971 over the same 220 steps | Tested vectorized residual sequence MPC around a closed-loop baseline action tape.  It can predict short bursts but the fixed tape tail is not a good value function; it did not beat the existing one-action / closed-loop-tail MPC. |
| 39 | ~1,750 executed frames + vectorized closed-loop-tail first-action scoring | 4 partial executed planner rollouts | rejected for now: wide/global vectorized first-action smoke got 1,764 / 1,814 / 1,879 over 350 steps; baseline was 1,645 over 350 | Implemented a scratch vectorized closed-loop-tail scorer for many first-action candidates.  Wider/global candidate sets gave small 350-step improvements but are not a breakthrough; also do not check this in until it is parity-tested against the existing Python scratch-data scorer. |
| 40 | ~500 executed frames + vectorized closed-loop-tail prefix scoring before manual stop | partial rollouts only | rejected: 3-action-prefix + closed-loop-tail MPC returned 1,487 over 300; 4-action-prefix was killed at 962 over 200 | Tested coordinated 3-5 step random residual prefixes, then a per-candidate closed-loop baseline tail.  The first completed prefix planner was below both the checked one-action MPC 300-step smoke and the wider one-action scratch smoke. |
| 41 | ~4,100 executed frames + online model scoring | 2 full executed planner episodes plus partial smokes | new best checked script path: widened/vectorized `mpc-asym-pd-cpg` returned 6,615.1 / 6,565.2 on seeds 100/101 | Parity-tested a vectorized `mujoco.rollout` implementation of the checked one-action / closed-loop-tail scorer: 300-step old/new action mismatch = 0 and return matched at 1,713.5467.  Fixed the scratch-score contamination bug by using an independent `MjData` rollout template.  Replaced the Python per-candidate loop with the vectorized scorer, then increased the candidate set to 445 actions: coordinate deltas now include `+/-0.75`, `+/-1.00`; random blocks are 64 at std 0.15, 128 at std 0.35, and 192 at std 0.70.  A 250-step horizon sweep kept horizon 14: H8 1,410, H10 992, H14 1,529, H18 1,471. |
| 42 | diagnostic only | 0 full episodes | rejected: ballistic / constant-action high-root-velocity probes dissipated speed instead of making reward | From reset posture, set qvel[0] to 0/6/10/14 and rolled out zero / constant saturated actions for 200 env steps.  Even with qvel[0]=14, the body falls into contact and qvel[0] decays to ~0 by the end; reward per step stayed far below target.  High score needs repeated contact impulses, not initial velocity or sliding. |
| 43 | ~500 executed oracle frames + fast policy evals | 1 partial oracle rollout; many cheap fitted-controller evals | rejected: best ridge-fitted explicit residual roughly matched baseline; best 3-seed mean seen was 4,809 | Collected 500 steps from the 445-candidate MPC oracle and ridge-fit a phase/state/action-feature linear residual controller with phase-state interactions.  Training RMSE was ~0.28 action units, but execution was unstable or collapsed back toward baseline.  The oracle residual is a saturated/discrete action selection, not a smooth single linear residual around the baseline. |
| 44 | ~7,500,000 warm-start model frames + diagnostics; 0 new training episodes | 0 full executed episodes | rejected: no offline/warm-start route beat the saved 6.6k MPC trajectory | Saved seed100 445-candidate MPC nominal trajectory at `/tmp/halfcheetah_mpc_nominal_seed100_v2.npz` and verified it replays open-loop in Gym with return 6,615.1.  Model chunk replay from recorded states/actions matches recorded reward/state.  Rejected local action-window CEM because it selected the nominal chunk.  Rejected repeated torque-table compression of the t=500 MPC tail: nominal 500:800 is 2,035.7, but repeated P=5..100 prefixes/phase-averages scored <= 902 over 300 steps.  Rejected warm 32-param asym-PD search from the t=500 MPC state: 500-step objective improved 2,500.7 -> 2,810.8, but 1000-step warm validation was 4,928.5 vs baseline raw 4,990.7 and MPC nominal 6,773 projected.  Rejected a warm P=6 torque-table CEM smoke: it reached only 2,466 over 500 steps before stop, versus nominal 3,386.9.  Rechecked the public expert dataset from reconstructed expert states: one-step dynamics can match, but long open-loop diverges and feedforward+joint-PD trajectory tracking scored only hundreds, so do not use it as a shortcut policy. |
| 45 | ~2,400 executed frames + online model scoring; 100 offline scoring states | 2 full executed planner episodes plus partial smoke | new best checked script path: all-bang-augmented `mpc-asym-pd-cpg` returned 6,831.7 / 6,846.4 on seeds 100/101 | Added the 64 fully saturated `{-1,+1}^6` torque vectors to the online first-action MPC candidate set.  Offline scorer diagnostic on 100 saved nominal states: existing 445-candidate set still won 83 states, all-bang won 15, uniform random won 2; mean extra score margin was 0.12 / max 2.74.  Despite the small margin, online smoke improved seed100 400-step return 2,541.8 -> 2,672.4, so the candidate set was checked by full episodes.  Candidate count is now 509; optional MPC is slower (~90s/episode on this laptop). |
| 46 | ~9,000 executed frames + online model scoring; video/telemetry diagnostics | 2 full executed planner episodes plus partial smokes | new best checked script path: slower-tail `mpc-asym-pd-cpg` returned 7,053.0 / 7,114.4 on seeds 100/101 | Rendered `/tmp/halfcheetah_seed100_baseline_vs_mpc.mp4` and inspected a contact sheet.  The MPC video showed the same low flat bounding gait, not a qualitatively new high-speed run.  Telemetry confirmed the bottleneck is speed, not control cost: baseline avg velocity/cost was 5.21/0.37, all-bang MPC was 7.25/0.42, and foot contact duty fell from 26.3% none-free to 19.8% contact.  Horizon sweep kept H14 as best over 250 steps; terminal-velocity weights 0.0..8.0 and planning control-cost coefficients 0.0..0.2 all failed to beat the current H14/0.5/0.1 setting.  Rejected `{ -1,0,+1 }^6` ternary actions and sparse offline winner subsets: full ternary only improved 300-step 1,951.1 -> 1,961.5 at 1,174 candidates, while cheaper subsets were worse.  Rejected repeated-first-action scoring: repeat 2/3 scored 1,645/1,192 over 300.  A first-principles CPG phase sweep found that the MPC tail was too fast for useful contact impulse; lowering the two MPC-only phase raw values by 0.35 changed the tail frequencies from 4.50Hz to 4.32Hz and improved 300-step seed100 to 2,026.1, then full seeds 100/101 to 7,053.0 / 7,114.4.  The default `asym-pd-cpg` baseline raw is unchanged. |
| 47 | ~10,000 executed frames + ~hundreds of millions of online model frames | 2 full executed planner episodes plus partial smokes | new best checked script path: two-step tree `mpc-tree-asym-pd-cpg` returned 7,601.7 / 7,578.7 on seeds 100/101 | Added an optional two-step tree MPC overlay.  First it scores the existing 509 first-action candidates with the closed-loop tail, keeps the top 4, steps each candidate once in copied MuJoCo state, then rescores a lightweight second-action set (base + coordinate deltas + 64 bang-bang actions) before executing only the selected first action.  This improved seed100 240-step return 1,578.9 -> 1,686.4 and 500-step return 3,450.9 -> 3,677.1 before full validation.  Rejected deeper and wider variants: three-step beam scored 1,175.8 vs two-step 1,218.0 over 180 steps, first-action local refine scored 1,167.0, mixed immediate-reward/qvel first-branch selection scored <= 1,199.2, and second-action subsets using only bang or only coordinate deltas scored 1,152.1 / 1,171.8.  Rechecked tree-specific low-dimensional knobs: current tail frequency `-0.35/-0.35`, terminal velocity weight 0.5, horizon 14, top-K 4, and coordinate+bang second candidates remain the best short-rollout settings; adjacent top-K 3/4/5/6 scored 875.6 / 972.4 / 946.9 / 933.7 over 150 steps. |
| 48 | ~20,000 executed frames + online model scoring; render/telemetry diagnostics | 4 full executed planner episodes plus warm-state smokes | new best checked script path: staged two-step tree `mpc-staged-tree-asym-pd-cpg` returned 8,220.4 / 8,128.8 on seeds 100/101 | Rendered and inspected `/tmp/halfcheetah_seed100_tree_mpc.mp4` plus `/tmp/halfcheetah_seed100_tree_mpc_contact_sheet.jpg` for the two-step tree.  Telemetry showed return 7,601.7 = avg velocity 8.06 minus avg control cost 0.44; velocity p50/p90/p99/max was 8.14/9.98/10.91/11.75, and 803/1000 steps had no foot contact.  The target 12k still needs avg reward near 12, so the gap is sustained contact impulse, not control-cost trimming.  Rejected terminal posture value shaping: pitch, pitch-rate, and height penalties all lowered 160-step return versus pure terminal qvel.  Rejected contact-readiness terminal bonuses based on MuJoCo foot geometry; they did not materially increase contact duty or return.  Rejected periodic compression of the 7.6k tree action tape: P=10..200 mean/first tables scored from -386 to only 1,403.  Rejected target-angle amplitude/bias sweeps around the tree raw; all tested global scale, hind/front scale, and single-joint bias shifts were below the current raw over 140 steps.  Rejected short sequence CEM around baseline and around tree nominal: the former scored 993.8 vs tree 1,050.7 over 160, and the latter only improved the first 80 steps before falling below tree at 160.  Rejected sparse ternary candidates and larger/smaller random blocks; current candidate mix remained best.  The useful first-principles signal came from warm-state frequency analysis: from the tree t=500 state, a much slower cruise tail improved 240-step suffix return, with raw frequency delta -1.0/-1.0 scoring 2,152.3 versus -0.65/-0.65 scoring 1,898.8.  Added staged tree MPC: use the existing -0.35/-0.35 tail for startup, then switch at step 400 to a -1.0/-1.0 cruise tail.  Checkpoint replay over seed100 found switch 300/400/500/600/700 returns 4,899.2 / 8,220.4 / 8,047.1 / 7,971.9 / 7,916.9, so switch 400 was checked in and full CLI-validated. |
| 49 | ~59,000 executed frames + substantially larger online model scoring | 59 full executed planner episodes plus telemetry / objective sweeps | new best checked script path: staged two-step tree `mpc-staged-tree-asym-pd-cpg` returned 8,500.5 / 8,290.3 on seeds 100/101 | Tuned the staged cruise from first principles around the actual speed gap.  A direct cruise-frequency sweep showed the useful signal was not just slower frequency but asymmetric half-cycle timing: symmetric -1.1/-1.1 scored 8,204.0, symmetric -1.2/-1.2 collapsed to 6,066.7, asymmetric -1.2/-1.0 scored 8,500.5, and asymmetric -1.0/-1.2 scored 8,408.7 on seed100.  Finer full-episode sweeps did not beat -1.2/-1.0: -1.3/-0.9 scored 8,481.4, -1.3/-1.0 scored 8,394.7, -1.15/-1.0 scored 8,377.4, switch 375/425 scored 8,331.0 / 8,357.4, and switch phase offsets +/-0.5 scored 8,357.0 / 8,269.3.  Telemetry on the new best showed return 8,500.5 = avg velocity 8.93 minus avg control cost 0.43; 100-step velocity bins after the switch were around 9.3-10.0, p50/p90/p99/max velocity was 9.24/10.74/11.53/12.48, and 815/1000 steps still had no foot contact.  Re-rendered the new best with `mujoco.Renderer` at `/tmp/halfcheetah_seed100_staged_asym8500_mujoco.mp4` and `/tmp/halfcheetah_seed100_staged_asym8500_mujoco_contact_sheet.jpg`; the video still shows a low airborne/bounding gait rather than a new long ground-contact run.  Therefore the remaining 12k gap is still sustained 12+ m/s velocity, not action-cost trimming.  Rejected late multi-stage schedules: the best 400->-1.2/-1.0, 700->-1.3/-0.9 scored only 8,501.7.  Rejected terminal velocity weights 0.0/0.25/0.75/1.0/2.0/4.0; current 0.5 remained best.  Rejected horizon 10/12/16/18/22; current H14 remained best and H10 collapsed to 2,790.8.  Rejected top-K 1/2/3/5/6/8; current top-K 4 remained best.  Rejected velocity-threshold schedules, contact-triggered phase locking, and an online two-raw bank planner; the raw bank chose the alternate raw 328/600 cruise steps but scored only 7,459.2, showing short-sighted raw switching destabilizes the limit cycle.  Checked in cruise raw deltas -1.20/-1.00 and validated the CLI on seeds 100/101; scratch seed102 was 7,776.7. |
| 50 | ~22,500 executed frames plus ~2,457,600 sequence-MPC model frames and online tree scoring | 21 full-equivalent executed rollouts plus partial smokes | no new best; current best remains staged two-step tree 8,500.5 / 8,290.3 on seeds 100/101 | Diagnosed the new 8.5k trajectory more deeply.  Contact is still sparse: 815/1000 steps had no foot or shin contact, bfoot contact averaged 8.55 velocity over 77 steps, ffoot contact averaged 9.52 velocity over 105 steps, and the highest velocity steps were mostly airborne immediately after contact impulses.  The contact-conditioned mean action was already saturated in the same directions the planner chooses, so the next hypothesis was whether the action library lacked useful intermediate stance pushes.  Rejected adding fixed stance prototypes: contact means scored 7,272.5, contact+top-velocity prototypes scored 8,290.5, blended prototypes scored 8,366.5, and hand-mirrored pushes scored 8,406.8.  Rejected a pure short-sequence MPPI/CEM planner: 300-step return was only 1,164.8 versus current tree 2,135.8, because open-loop sequence optimization without the closed-loop tail loses the gait attractor.  Rejected a repeated-second-action macro branch: 300-step return was 2,042.8 versus current 2,135.8.  Rejected state-dependent virtual-model-control candidates from foot Jacobians: 300-step return was 2,096.0 versus current 2,135.8.  Ran another local frequency grid around the best cruise; none beat -1.20/-1.00, with closest -1.25/-1.00 at 8,468.5.  Ran target-angle bias sweeps to lower/advance stance feet; none beat current, with closest front-positive bias at 8,477.6.  The first-principles conclusion is stronger now: this low airborne CPG/tree attractor can briefly hit 12+ m/s but does not generate enough sustained ground impulse; single-step action enrichment, repeated second actions, simple VMC force candidates, and small target-bias changes do not create the missing long-contact running gait. |
| 51 | ~590,800 executed/search frames plus online tree/KNN model scoring | ~1,181 full-equivalent cheap/search episodes plus partial/full planner rollouts | no new best; current best remains staged two-step tree 8,500.5 / 8,290.3 on seeds 100/101 | Tested four non-NN alternatives aimed at escaping the 8.5k attractor.  First, a KNN terminal-value table built from the 8.5k trajectory gave a small 500-step proxy gain but failed the full episode: value weight 0.12 returned 8,352.8 on seed100 and took 211s, so table value shaping was rejected.  Second, a contact-state FSM residual policy searched 4 contact-conditioned residual action vectors plus 4 contact-conditioned phase-speed multipliers with CEM for 576,500 frames / 1,153 500-step episodes; the best 500-step score was 1,949.3 and full validation was only 2,687.6 / 1,868.8 on seeds 100/101, so cheap contact heuristics cannot replace the MPC attractor.  Third, action-library MPC candidates extracted from the 8.5k trajectory looked promising in an initial scratch run but failed reproducibility: the checked CLI path scored only 6,884.3, the current-function 300-step replay scored 1,770.1 versus the 2,135.8 staged-tree baseline, and deterministic 200-step subset sweeps over top-reward, top-value, late-only, phase-spread, and reward+value action libraries were all negative.  The exemplar policy was therefore removed instead of checked in.  Fourth, diverse top-K first-branch selection with min action L2 0.05 improved 300/500-step proxies to 2,218.4 / 3,901.3 versus 2,135.8 / 3,787.8, but full validation was not robust: seed100 only rose to 8,502.1 while seed101 fell to 8,276.6, so the two-seed mean worsened and the change was reverted.  This reinforces that naive lookup/value/action-library additions and short-horizon branch-diversity fixes bias the planner toward locally attractive actions that disrupt the long cruise gait. |
| 52 | ~4,819,600 model frames plus 2,100 online tree suffix frames | ~4,416 full-equivalent open-loop tapes plus 803 no-MPC suffix candidates and 7 tree suffixes | no new best; current best remains staged two-step tree 8,500.5 / 8,290.3 on seeds 100/101 | Tried leaving the online-MPC family and optimizing deterministic trajectories.  First verified that the saved seed100 action tape replays exactly open-loop: 8,500.46 return, matching the saved reward sum.  Then swept 803 no-MPC CPG suffix raw variants from the t=500 state for 500 steps; the proxy strongly preferred faster frequency deltas around -0.85/-0.80, but true online tree suffix validation rejected that proxy: current -1.20/-1.00 scored 2,808.1 over the t=500 300-step suffix, while -0.85/-0.80, -0.70/-0.95, -0.85/-0.55, -0.75/-1.00, and -0.75/-0.80 scored only 2,484.1, 2,590.5, 2,419.3, 2,613.5, and 2,550.9.  This shows cruise raw is optimized for the MPC scorer, not for pure CPG playback.  Next tried open-loop action-tape residual optimization around the exact 8.5k tape using vectorized MuJoCo full-episode rollouts.  Global smooth residuals at std 0.001..0.02 over 256 candidates each all collapsed, with best returns only 474..745.  Local residuals restricted to windows 800:1000, 900:1000, 950:1000, 700:800, and 500:600 over 512 candidates each also had zero improvements; the closest was 8,436.3 from perturbing only the final 50 steps.  The open-loop tape is therefore a narrow, highly sensitive trajectory rather than a smooth optimizable basin. |
| 53 | ~4,608,000 sequence-MPC model frames plus 200 executed frames | 1 partial executed rollout with 40 replans | rejected: long-horizon spline shooting scored 504.5 over 200 steps versus staged-tree 1,370.9 | Tested a genuinely different receding-horizon controller: optimize 8 residual action nodes, linearly interpolate them into a 40-step action sequence, score with vectorized MuJoCo rollout plus terminal velocity and action-smoothness terms, run 3 CEM iterations with population 192, execute the first 5 actions, then replan.  This was intended to give a longer-horizon optimizer a chance to find sustained contact impulse instead of a one-step/tree action.  It failed immediately: seed100 200-step return was only 504.5 after 40 replans.  The failure mode matches previous sequence-MPC attempts: without a closed-loop value/tail that preserves the gait attractor, long open-loop shooting finds short-horizon actions that destabilize the body. |
| 54 | ~3,083,000 executed/search frames plus ~75,000,000 online tree model decision-frames | 3,073 full linear-controller search episodes plus 24 partial online-tree rollouts/diagnostics | no new best; current best remains staged two-step tree 8,500.5 / 8,290.3 on seeds 100/101 | Pushed several first-principles alternatives around the 12k gap.  First rechecked longer closed-loop-tail scoring on reset execution: over 220 steps, H10/w0.5 scored 1,117.6; H14 with terminal velocity weights 0.0/0.5/1.0 scored 1,475.4 / 1,529.8 / 1,450.3; H18 with weights 0.25/0.5 scored 1,405.3 / 1,410.9, so H14/w0.5 remained best and the longer tail was stopped before H24/H32.  Second, tried a non-NN oscillator plus single affine state-feedback controller initialized exactly to the current ASYM PD-CPG: 134 parameters, population 256, 12 CEM iterations, 3,073,000 frames.  It never beat the 4,837.7 seed100 initialization; by iteration 12 the batch p90 was only 2,260.7, so a broad linear feedback controller did not escape the attractor.  Third, tested whether the video's sparse contact came from too-short torque impulses by holding chosen MPC actions for 2-5 env steps; 220-step returns fell from 1,529.8 to 563.3, 181.6, 119.1, and -24.3.  Fourth, reconstructed contacts from the saved 8.5k MuJoCo states: foot contact occurred on 168/1000 steps, concentrated at phase 3.7-4.7, while high-reward steps were also near phase 4.0.  Direct contact-triggered phase resets to 3.7/4.1/4.5 over 240 steps scored 1,404.0 / 1,495.5 / 773.4 versus 1,686.4 baseline, so naive phase locking reduced velocity and did not increase useful contact.  Fifth, from the saved t=500 state, cruise suffix terminal-weight 0.25 looked locally better over 180 steps (1,681.2 versus current 0.5 at 1,654.1), but the signal did not transfer from reset: 500-step dynamic schedules startup 0.5, cruise 0.15/0.25 scored 3,769.9 / 3,732.3 versus current 3,787.8.  These results reject longer horizon alone, affine phase feedback, action hold, hard contact phase reset, and single-state cruise terminal-weight tuning as routes to 12k. |
| 55 | ~100,000 executed/partial planner frames plus ~2,000,000,000 online tree model decision-frames | ~151 partial/full online planner rollouts | new best checked script path: high-gain all-cruise top-K8 `mpc-staged-tree-asym-pd-cpg` returned 11,422.1 on seed100 and scratch-validated 11,042.0 on seed101 | Found the first large post-8.5k breakthrough by retuning the closed-loop PD tail used inside MPC.  A 220-step gain sweep showed `Kp=2.0,Kd=0.02` at 1,555.2 and `Kp=1.0,Kd=0` at 1,548.4 versus the current 1,529.8.  The signal transferred to 500 steps: `Kp=2.0,Kd=0` scored 4,168.4 versus 3,787.8, and full seed100 reached 9,218.3.  Increasing stiffness further gave `Kp=4,Kd=0` at 9,358.9 / 9,340.6 on seeds 100/101.  With the stronger tail, the old staged startup was no longer useful: switch 400 with raw -1.7/-1.1 scored 6,455.2 over 700 steps, switch 250 scored 6,753.2, and switch 0 scored 7,273.0.  Retuning the all-cruise raw found `-1.8/-1.1`: full seed100 scored 10,805.6 at Kp4 and then 11,051.0 at Kp8.  Raising first-branch tree width from top-K4 to top-K8 improved the same configuration from 11,051.0 to 11,422.1 on seed100; seed101 scratch validation was 11,042.0.  Rejected higher Kp values 10/12/16, nonzero damping 0.005/0.01, alternative raw deltas around -1.8/-1.1, terminal velocity weights 0.0/0.25/0.75/1.0/1.5/2.0/4.0, planner cost coefficients 0.0..0.2, horizon 10/12/16/18/22, top-K10/12, initial phase offsets over 12 bins, random-candidate RNG offsets, and a third-step tree beam; none beat the checked Kp8/Kd0/switch0/raw -1.8/-1.1/top-K8/H14/w0.5 configuration.  The current gap to 12k is about 0.58 reward/step: average velocity is 11.93 and true average action cost is 0.51. |
| 56 | ~25,000 executed/partial planner frames plus ~300,000,000 online/planner model decision-frames | ~70 partial/full planner rollouts plus vectorized action-tape batches | no new best; current best remains high-gain all-cruise top-K8 `mpc-staged-tree-asym-pd-cpg` at 11,422.1 on seed100 | Re-targeted the harder 13k objective from first principles after saving the 11,422.1 trajectory to `/tmp/hc_best11422_seed100_traj.npz`.  Telemetry showed the best 100-step segment was only 1,255.2 reward from steps 600:700, then velocity drifted down from 13.06 to 12.54 m/s by the final segment; average velocity/cost was 11.93/0.51, so 13k needs a qualitatively faster gait, not just lower action cost.  Rejected second-step random candidates: adding the full random candidate set to the second tree layer scored 1,258.9 over 160 steps versus the current 1,276.6 and cost 151s.  Rejected startup schedules under Kp8/top-K8: old fast startup, old 8.5k raw, midfast, and slower startups for 40..80 steps all scored below the current 160-step baseline, with the closest old-startup switch-40 at 1,265.7.  Rejected phase-speed profiles based on the reward/phase histogram: slowing phase 2.8..4.5 and speeding bad phase bins peaked at 1,274.0 over 160, then fell to 2,805.8 over 300 versus the 2,858.2 baseline.  Rejected phase-dependent stiffness profiles: push-phase Kp10/12 and recovery Kp6 variants all scored below 1,276.6 over 160.  Rejected trajectory-prototype action candidates from the current best tape: 62 high-reward/contact/phase prototypes scored only 1,215.9 over 160.  A t=600 suffix raw sweep gave a small local proxy signal for -1.8/-0.9 over 200 steps, but the restored-state baseline did not reproduce the saved tape due to RNG/control-state mismatch and the gain was too small to chase toward 13k.  Tested a new tape-tail MPC that scores first-action candidates with the saved 11.4k action tape as the 14-step tail.  It improved the first 300 steps to 2,977.7 versus 2,858.2, but full reset fell to 11,274.6; hybrid switches back to CPG-tree at 50/200/300 scored only 10,967.9 / 11,004.3 / 11,163.0, and direct switch back to the open-loop tape collapsed.  Trust-region and score-penalized tape-tail variants also failed: penalty 0.1 scored 2,946.8 over 300 but only 11,039.1 full, while stronger penalties, small candidate sets, and blend execution were worse.  Adding time-indexed joint-PD tracking back to the saved 11.4k nominal tape after tape-tail deviation also failed hard: switch 50/100/200 variants stayed around -94..1,805, so simple tape stabilizers cannot re-capture the high-speed path.  Finally, CPG-tree base-deviation penalties and one-round local first-branch refine were negative over 300 steps.  The useful conclusion is that short-tail values can find locally better early actions, but tiny state deviations destroy the later high-speed trajectory; reaching 13k likely requires a stabilizing value/feedback model around a faster gait, not action-tape nudges or more candidate widening. |
| 57 | ~45,000 executed/partial planner frames plus online tree model scoring | ~90 partial/full planner rollouts plus restored-state suffix sweeps | new best checked script path: staged swing-amplitude tree `mpc-staged-tree-asym-pd-cpg` returned 12,041.2 on seed100 | Used the 4.8k / 7.8k / 11.4k comparison video and joint-range telemetry to test the hypothesis that higher reward comes from a more natural, larger-amplitude swing rather than lower control cost.  Actual joint ranges increased from 4.8k to 11.4k, especially front shin/foot and hind thigh, so the search moved from frequency/Kp only to target-angle geometry.  Simple single-joint amplitude changes were mostly negative over 160 steps, but coordinated whole-body non-bias Fourier amplitude and front shin/foot bias were positive: `all_amp110_bias_p10` scored 3,010.1 over 300 versus the old 2,858.2 and validated full at 11,710.4.  Finer amplitude/bias sweeps found `amp=1.15,bias=0.15` at 11,736.9, then staged schedules improved further: `1.10/0.10 -> 1.18/0.20 @700` scored 11,826.0, `@500` scored 11,847.7, and adding a final `1.15/0.15 @800` scored 11,941.9.  The final breakthrough was keeping large `1.15/0.15` swing for startup through step 300 and switching to `1.18/0.20` through step 900, then back to `1.15/0.15` for the final 100 steps; the CLI-validated checked script returns 12,041.189857475818 on seed100.  Rejected variants included per-joint-only amplitude increases, excessive whole-body amplitude such as 1.20 early, early `1.15` only until step 200, switching from `1.15` to `1.10` around step 300, and several tail Kp/raw changes that produced local suffix gains but did not transfer full episode. |
| 58 | ~7,400 completed executed/partial planner frames plus online tree model scoring; several aborted coarse probes | ~30 partial reset/checkpoint/suffix planner rollouts | no new best; current best remains staged swing-amplitude tree at 12,041.2 on seed100 | Followed the render-based hypothesis that 13k may require a more natural large-swing gait.  First tested replacing the abrupt 300/900 swing-amplitude switches with cosine-smoothed 250:400 and 900:1000 transitions.  It slightly improved the first 300 steps to 3,050.1 versus 3,042.3, but by 500 steps it fell to 5,571.4 versus the current best path's ~5,612, so simple smoothing was stopped.  Then built exact step-300 checkpoints and compared true top-K8 MPC suffixes for fast-stage naturalness changes.  Larger fast amplitudes were negative: `1.19/0.21`, `1.20/0.22`, and `1.17/0.18` scored 1,218.0, 1,215.9, and 1,225.2 over the 300:400 suffix versus baseline 1,264.4.  Two local positives appeared: front-leg phase shift `-0.05 rad` scored 1,277.5, and hind-thigh harmonic scale `1.05` scored 1,278.4.  Their combination and nearby values were not additive: `hind105_front_m05` scored only 1,216.1, `hind103_front_m05` 1,216.9, `hind105_front_m03` 1,221.6, `front_phase_m08` 1,227.5, `hind103` 1,229.0, and `hind108` 1,240.7.  Full/partial reset validation rejected the local positives: fast-stage `hind_thigh_scale=1.05` was behind by step 700 after initially gaining at 300:400; fast-stage `front_phase=-0.05` was farther behind by step 700; limiting `hind_thigh_scale=1.05` to only steps 300:400 still fell behind after the window.  Rechecked planner-level knobs under the new 12k swing schedule: terminal velocity weight 0.75 and 0.25 were both far behind by step 700, and horizon 12 collapsed after step 300.  The useful conclusion is that the video intuition is real but narrow: local swing timing/shape can raise one 100-step segment, but without a stabilizing longer-horizon value it moves the state off the better high-speed attractor and loses more later. |

## Current status

Best checked-in baseline:

```bash
python examples/heuristic_halfcheetah_v5.py --policy asym-pd-cpg --eval-episodes 10 --eval-seed 100
```

Fallback baselines:

```bash
python examples/heuristic_halfcheetah_v5.py --policy mpc-asym-pd-cpg --eval-episodes 1 --eval-seed 100
python examples/heuristic_halfcheetah_v5.py --policy mpc-tree-asym-pd-cpg --eval-episodes 1 --eval-seed 100
python examples/heuristic_halfcheetah_v5.py --policy mpc-staged-tree-asym-pd-cpg --eval-episodes 1 --eval-seed 100
python examples/heuristic_halfcheetah_v5.py --policy pd-cpg --eval-episodes 10 --eval-seed 100
python examples/heuristic_halfcheetah_v5.py --policy reflex --eval-episodes 10 --eval-seed 100
python examples/heuristic_halfcheetah_v5.py --policy open-loop --eval-episodes 10 --eval-seed 100
```

Continue scalar-only CEM search and write a per-iteration JSONL ledger:

```bash
python examples/heuristic_halfcheetah_v5.py --search --policy asym-pd-cpg --search-iterations 20 --population 64 --repeats 3
```

JSONL records include `iteration`, `sampled_frames_this_iter`, `sampled_episodes_this_iter`, totals, batch best/mean score, best-so-far score, raw per-repeat returns, and the search action taken.

Current checked-in non-MPC 10-seed result: mean 4,799.7 / std 64.1 / min 4,651.8 / max 4,892.0 for seeds 100..109.  The optional staged tree MPC path has reached the 12,000 target on seed100; a follow-up 5-seed eval over seeds 100..104 returned mean 11,836.7 / std 109.5 / min 11,735.0 / max 12,041.2.  The stretch target is 13,000+.

Best checked script path seen so far: staged swing-amplitude top-K8 tree `mpc-staged-tree-asym-pd-cpg` returned 12,041.2 on seed100 and 5-seed mean 11,836.7 over seeds 100..104 with schedule `1.15/0.15` through step 300, `1.18/0.20` through step 900, and `1.15/0.15` for the final 100 steps.  The previous high-gain all-cruise staged tree returned 11,422.1 on seed100 and scratch-validated 11,042.0 on seed101.  The best rejected tape-tail MPC variant improved the first 300 steps to 2,977.7 but fell to 11,274.6 full episode.  The previous Kp4/all-cruise/top-K4 candidate returned 10,805.6 on seed100.  The previous staged two-step tree returned 8,500.5 / 8,290.3 on seeds 100/101.  The previous symmetric-cruise staged tree returned 8,220.4 / 8,128.8 on seeds 100/101.  The previous two-step tree `mpc-tree-asym-pd-cpg` returned 7,601.7 / 7,578.7 on seeds 100/101.  The previous slower-tail `mpc-asym-pd-cpg` returned 7,053.0 / 7,114.4 on seeds 100/101.  The previous all-bang-augmented MPC returned 6,831.7 / 6,846.4 on seeds 100/101.  The previous 445-candidate MPC returned 6,615.1 / 6,565.2 on seeds 100/101.  The previous 101-candidate MPC returned 6,041.1 / 6,058.5 on seeds 100/101.  The earlier/narrow MPC candidate set reached 5-seed mean 5,725.2 / min 5,705.4 over seeds 100..104.  MPC is model-based online planning around the baseline, is much slower than the default, and is not a 10-seed-selected replacement.

Current planner conclusion: the planner variants that clearly improved reset execution are the checked one-action / closed-loop-tail MPC overlay, the two-step tree, staged asymmetric cruise timing, high-gain undamped MPC tail control, immediate all-cruise switching, top-K8 first-branch retention, and coordinated target-angle swing-amplitude staging.  Free-state periodic orbit CEM, fixed-tape residual sequence MPC, closed-loop-tail prefix MPC, high terminal-velocity bonus, public expert replay/targets/KNN, warm-only raw-parameter optimization, simple stance prototypes, repeated-action macro branches, short-sequence MPPI/CEM, long-horizon spline shooting, longer closed-loop-tail horizons, broad affine phase-state feedback, action holds, hard contact-triggered phase resets, virtual-model-control candidate actions, contact-state residual FSMs, trajectory KNN terminal-value shaping, exemplar action libraries, no-MPC suffix raw proxies, open-loop action-tape residual optimization, single-state terminal-weight retuning, initial phase offsets, RNG-candidate stream changes, third-step tree beams, small cruise target-angle bias changes, second-step random widening, startup schedules, phase-speed profiles, phase-dependent stiffness, trajectory-prototype candidates, tape-tail MPC/hybrids, trust-region tape-tail variants, time-indexed nominal-tape joint tracking, CPG-tree base-deviation penalties, first-branch local refine, per-joint-only amplitude increases, smoothed amplitude switches, fast-stage phase shifts, fast-stage hind-thigh-only boosts, and retuned terminal-velocity/horizon values are rejected in the forms tested above.

Next high-upside direction: stop widening generic first-action candidate sets around the low-speed attractor and stop trying fixed periodic torque tables.  The vectorized one-action scorer is now parity-tested and checked in.  Offline local action-window search around the saved 6.6k MPC trajectory did not improve nominal chunks, but video/telemetry-guided CPG tail timing helped.  The latest diagnostics show that contact phase is identifiable, but hard phase locking hurts; the missing piece is likely a soft, state-dependent value/stabilization term rather than a direct reset rule.  A plausible next step is a contact-aware DDP/iLQR-style local trajectory optimizer with a closed-loop stabilizing tail, or a hand-derived value function fitted from controlled perturbations around the 8.5k trajectory and validated from reset.  Blindly adding more random candidates, setting high root velocity, fitting one global linear residual, holding actions, or compressing the MPC tape into a short torque period have diminishing returns.
