cascade-broad-v1 — wall-saturating, cluster-broad candidate generator

cascade SN91 submission repo. Three files are the contract (generator.py, config.json, requirements.txt); this README is documentation.

/tmp/.../venv/bin/cascade verify /workspace/cascade-rig/gen/candidate
/tmp/.../venv/bin/python /workspace/cascade-rig/rig/bench_throughput.py \
        /workspace/cascade-rig/gen/candidate --seconds 20

Measured (this box, single core, seed 9993167447787770989)

Why it is fast

generate() never touches a single series in Python. It draws a block of B series of one identical, 32-aligned length L as one (B, L) float32 array, entirely with vectorised numpy plus scipy.signal.lfilter for the recursive parts, then yields the rows one at a time. Every recursion is a filter along the time axis of the whole batch (C-level), never a Python loop over time or over series.

• points_per_block = 1,048,576 ⇒ B = 1M / L (256 rows at L=4096, 4096 rows at L=256), rounded to a multiple of 64 = the trainer's batch_size, so p-buckets flush without long-lived partials.
• float32 blocks. check_series accepts any floating dtype and the trainer canonicalises to float64 losslessly. float32 halves memory traffic and takes the SIMD math path: cos is 2.5 ns/pt vs 21.7 ns/pt in float64 on this box.
• interleave_slots = 6. Rows are drawn round-robin from 6 concurrently-open blocks, so a trainer minibatch is a mixture of families instead of 64 rows cut from one block. Costs ~nothing; removes the intra-batch correlation that block-sequential emission would otherwise hand the optimiser.
• Per-row gates. Optional components (trend, level shift, bump, outage, stuck sensor, negative-price regime, …) are switched by a (B,1) 0/1 mask folded into the amplitude, not by an if around the whole block — full per-series diversity at the cost of one multiply.
• Cheap smooth components. _slow() draws control knots at L/k resolution and linearly interpolates instead of filtering white noise at full rate (~10× cheaper for the same low-frequency content).

Family cost, measured at B=512, L=2048 (ns per emitted point → equivalent Mpts/s if that family ran alone):

Dominant remaining costs are rng.standard_normal (≈16 ns/pt even in float32) and the float64 round-trip that lfilter needs. If more headroom is ever wanted: switch the AR innovations of energy_price/kernel_mix from gaussian to the uniform innovation path (_innov kind 1, 4.3 ns/pt) — but there is 3.8× of slack today, and past the wall further speed is worth exactly zero.

Length choice

lengths = [4096, 2048, 1024, 512, 256] — all multiples of 32, so the trainer's p = min(L//32, 128) slice s[-p*32:] consumes 100 % of every emitted point (measured 32-align waste 0.00 %). Only 5 distinct lengths ⇒ only 5 p-buckets, each filling to 64 rows fast.

length_weights are the fraction of series at each length (the generator divides by that length's rows-per-block to get the block-draw probability). Defaults [0.30, 0.26, 0.22, 0.14, 0.08] put 63 % of emitted points at L = 4096, the exact context width the evaluator feeds, while keeping shorter contexts in the mix. Per-family overrides skew minute-cadence families long and daily families short, matching how long those feeds actually are.

The mixture — families, weights, target clusters

A family's weight is its block-draw probability, which equals its share of emitted points (every block is ~1 M points regardless of family/length), i.e. its share of the training token budget and therefore of the loss. Measured point share over 200k series is within sampling noise of the configured weight.

Pool facts these weights are aimed at (pool/metadata_0726.json, 1746 series, 214 source clusters):

• cluster share (what the duel LCB counts): energy 33.2 %, econ_fin 19.6 %, nature 17.3 %, healthcare 10.7 %, transport 7.9 %, web_cloudops 7.5 %, sales 3.7 %
• cluster share by cadence: 15min/sp 96 → 29 %, D/sp 7 → 28.5 %, min/sp 60 → 17.8 %, H/sp 24 → 13 %, 5min/sp 288 → 5 %, 30min/sp 48 → 3.3 %
• window share (what the heat ranks): transport 68.3 %, of which GBFS bike-share is 57 % of all windows from just 9 clusters

So the mixture has to serve two different objectives at once: heat = minute-cadence bounded counts, duel = breadth over energy / econ_fin / nature.

Coverage check against the eight shapes the brief asked for: bounded integer counts → dock_counts; degenerate/frozen/intermittent → degenerate (+ the pad family, + power_load outages, + geo_smooth stuck sensors); daily-seasonal business with trend and level shifts → daily_business; 15min/5min electricity with heavy tails, negatives and spikes → energy_price; smooth geophysical → geo_smooth; step functions with long plateaus → step_plateau; GP/kernel mixture and OU → kernel_mix; plus power_load, hydro_burst and fin_walk for the rest of the energy / nature / econ_fin cluster mass.

The pad family (eval-geometry match)

cascade's evaluator left-pads every history with h[0] out to the full 4096 before the forward pass (toto2_trainer.py:596-613); training otherwise never sees that shape, and causal_standardize pins scale = 1e-5 on a flat context — a log-space cliff.

pad in config.json is applied orthogonally to every family after the affine step: a fraction of rows get their first R samples overwritten with s[R], i.e. a genuine constant head equal to the first live value.

"pad": { "fraction": 0.12, "run_frac_lo": 0.05, "run_frac_hi": 0.80, "run_frac_beta": 1.6 }

run_frac_beta > 1 skews R/L short, < 1 skews it long; fraction is the sweep knob. Measured: 16.3 % of corpus series carry a flat head ≥ 5 % of L (pad + naturally-frozen families), and 7.9 % of all standardized steps sit on the eps scale floor — the model trains on the cliff instead of meeting it first at eval. Verified end to end: an eval-style pad (512 real samples inflated to 4096) goes through causal_standardize finite, max |z| 4.8.

Safety rails

One bad series aborts the whole round (sandbox_stream_failed), so _finalize runs on every block before any row is yielded:

1. np.nan_to_num(..., copy=False) — NaN → 0, ±inf → ±max_abs_value.
2. np.clip(±max_abs_value) — default 1e9, far under CAST_SAFE_MAX_FLOAT32 (3.4e38); measured clip rate 0.000 % on every family, so the rail never distorts the prior.
3. finite_check — one np.isfinite pass, any survivor zeroed.
4. Lengths are drawn from a fixed 32-aligned table inside [64, 4096] and validated in __init__ (raises if a config edit breaks the band).
5. Flat series always carry a continuous level, so two of them can never be byte-identical — the corpus dup cap is 5 % and measured duplicates are 0.007 %.
6. A malformed config.json is swallowed and the built-in DEFAULTS are used; an unknown family name in config is ignored rather than fatal.

Determinism

• np.random.default_rng(seed) only, one generator, drawn in a fixed order. np.random.seed appears nowhere — production's generation_seed is a full 64-bit blake2b value (e.g. 9993167447787770989) and np.random.seed rejects anything ≥ 2³², which would pass cascade verify (seed 0) and die in the round.
• Verified: cascade verify prints [deterministic]; two Generators at 9993167447787770989 produce identical arrays, and seed±1 produces different ones.
• Grouped lfilter iterates group ids 0..n-1 in fixed order; no reduction-order-dependent parallelism, no threads, no wall-clock, no os.urandom.
• generate(n) depends only on (seed, n) and its first k outputs are independent of n — with the interleave pool, blocks are still built in a fixed rng-driven order.

Sweeping

Everything in the table above is in config.json; nothing needs a code edit:

• mixture: families.<name>.weight (= point/token share), length_weights
• eval geometry: pad.fraction, pad.run_frac_lo/hi/beta
• magnitudes: affine.log10_scale_lo/hi, offset_sigma, sign_flip_prob
• throughput/quality trade: points_per_block, interleave_slots, ar_groups
• every per-family parameter listed in the table

Note the ratchet: config-only forks are kept today (dedup_config_only_enforce=false) but that is the named next change, so the code is written to make real edits cheap too — a new family is one _fam_*(rng, B, L, p) function returning a (B, L) float32 array plus one entry in _FAMILY_FN and one block in DEFAULTS.

Dependencies

generator.py imports only numpy and scipy.signal.lfilter (plus stdlib json/math/pathlib), both allowlisted in chain.toml [dependencies] and both baked into deploy/Dockerfile (numpy 2.4.6, scipy 1.17.1) — remember requirements.txt is never installed server-side. No numba is needed: the mixture already runs at 3.8× the wall on pure numpy + scipy. Nothing on the static-guard blocklist is imported, and no weight file of any kind ships.