Section VI Self-Evaluation Figures 11–12

FutureMarket

An autonomous equity-research loop — ingestion, scoring, execution, and self-grading — described from the source

The Audit

Marking the homework, nightly, in a fixed order

Four jobs run in the first half hour after every restart, and the order they run in is the architecture. Reconciliation must precede the tournament; the tournament must precede calibration; calibration must precede the weight update. Each depends on what the one before it wrote.

reconciliation_service · strategy_tournament calibration_tracker · backtest_engine Nightly, then weekly

A prediction is a claim until something checks it. The reconciliation job is what converts claims into evidence, and everything else in this section is downstream of that conversion. It selects prediction-log rows that are approximately one, five, or twenty days old — with a twelve-hour tolerance either side — that carry an entry price and do not yet have a return recorded for that horizon. The null check on the return column is the idempotency guard: a row already filled is never selected again.

The return is the simplest possible expression: current price minus entry price, over entry price. Direction correctness is scored only for the one-day horizon, by comparing the predicted direction against the sign of the realised return.

Two mechanical caveats follow from the implementation and should be stated. All three horizons read the same live quote rather than a historical close at the correct as-of date; the five- and twenty-day figures are “return from entry to today” for a row that happens to be that old, which is correct only because the row was selected by age and only if the job actually ran that day. And if the job misses a day, that horizon's twelve-hour window never matches again, so the column stays null permanently.

The job also computes rolling hit rates over seven, thirty and ninety days. These are pooled across all strategies and written only to the log. Nothing persists them, and nothing reads them.

What happens next is the interesting part. Fifteen minutes later, the tournament pairs every two predictions made about the same symbol on the same day by different strategies, decides which was better, and updates both Elo ratings. Five minutes after that, the calibration tracker rebuilds each strategy's reliability curve. Five minutes after that, the compositor recomputes its source weights.

None of these three write anything the next boot will read. The tournament writes ratings to a table, which survives; the calibration tracker writes nothing at all and recomputes from scratch on every call; and the weight update writes a history row that is never loaded, so the weights themselves reset to their defaults on restart.

A prediction is a claim until something checks it. One nightly job does the converting, and everything else here is downstream of it.

On reconciliation as the root of the feedback loop

Figure 11 · Lifecycle

The life of one prediction, and the nightly chain that grades it

Figure 11 Prediction lifecycle and the ordered nightly feedback chain
Prediction lifecycle and nightly reconciliation chain An upper timeline follows a single prediction from the moment it is written, through reconciliation at one, five and twenty days, showing which columns are filled at each point and which twelve-hour tolerance window admits it. A lower panel shows the four nightly jobs in dependency order with their first-run offsets, what each reads and writes, and which of their outputs survive a process restart. PART A — ONE ROW IN prediction_log, OVER TWENTY DAYS T + 0 T + 1 day T + 5 days T + 20 days ±12 h window ±12 h window ±12 h window ROW WRITTEN direction · magnitude confidence · strategy_id entry_price captured here FILL 1-DAY actual_return_1d correct_direction the only horizon scored FILL 5-DAY actual_return_5d same live quote as above — not a historical close FILL 20-DAY actual_return_20d miss the window and the column stays null forever actual_return = (current_price − entry_price) ÷ entry_price — then correct_direction = (predicted_direction == sign of that return), for the 1-day horizon only PART B — THE NIGHTLY CHAIN, IN DEPENDENCY ORDER 1 · RECONCILIATION T+2 min reads prediction_log + quotes writes actual_return_1d/5d/20d correct_direction SURVIVES RESTART — it is a table idempotent via the null-column guard 2 · ELO TOURNAMENT T+17 min reads all reconciled rows writes strategy_ratings pairs same symbol, same day SURVIVES RESTART — it is a table no watermark — replays all history 3 · CALIBRATION T+22 min reads prediction_log writes nothing at all Brier score, Beta posterior NOTHING SURVIVES — recomputed the nightly run only warms logs 4 · WEIGHT UPDATE T+27 min reads 30-day accuracy per source writes signal_weight_log needs ≥5 samples per source ROW SURVIVES, EFFECT DOES NOT nothing reloads it at boot WEEKLY, MUCH LATER Alpha factor lifecycle — T+155 min evaluate candidates → retire decayed → generate exactly as many as were retired Walk-forward backtests — T+185 min four strategies, 90/30-day windows writes backtest_runs · see Figure 12 Deep analysis — T+125 min the five-stage pipeline, all tracked names batches of five, five-second spacing Two schedulers — the offsets above are the in-process scheduler's. A Celery beat configuration also exists and schedules three of these same jobs on cron. Running both would double-execute reconciliation, the screener, and forecast precompute. Nothing in the codebase prevents that, and the two do not share a cache or a lock. Offsets are deliberate — the source comments explain the stagger as avoiding a thundering herd of upstream data requests, and the ordering as a dependency chain.
Figure 11. The lifecycle and the chain. The persistence annotations on each job matter more than the arrows: of four nightly jobs, two write durable state, one writes nothing, and one writes a record whose effect is discarded on the next restart. The feedback loop is real while the process lives, and partially amnesiac across deployments. backend/services/reconciliation_service.py:19–127, :130–159 · services/strategy_tournament.py:135–202 · services/calibration_tracker.py:58–156 · services/signal_compositor.py:303–401 · scheduling at backend/main.py:85–198

Rating and reliability

Two different questions about the same rows

Elo: which strategy was better?

The tournament groups reconciled predictions by calendar day and symbol, then plays every unique cross-strategy pair within each group. The rating update is textbook: expected score from the rating difference over four hundred, then the rating moves by the K-factor times the difference between actual and expected. K is thirty-two for strategies with fewer than thirty comparisons and sixteen thereafter.

Winner determination has three rules in order. If exactly one strategy called the direction correctly, it wins. If both were right or both wrong, the smaller absolute magnitude error wins. If those errors are within a tenth of a per cent of each other, it is a draw.

Two implementation notes. Because the two competitors can carry different K-factors — a new strategy at thirty-two against an established one at sixteen — rating mass is not conserved across the pool, and total ratings drift. And there is no watermark of any kind: every nightly run reprocesses the entire reconciled history from the beginning, so the same matchups compound their effect indefinitely.

Calibration: was the stated confidence honest?

Five bins spanning fifty to one hundred per cent stated confidence. Within each bin, the tracker forms a Beta posterior over the realised hit rate, using a prior of ten virtual observations. The posterior is then the prior count plus the successes over the total, which requires roughly ten real observations per bin before evidence outweighs the prior.

The prior is centred on the bin's own stated confidence — not on one half. In effect the tracker begins by assuming each strategy is perfectly calibrated, and demands evidence to move it. A comment beside the code describes a neutral prior centred on one half; the code does not do that.

Two summary numbers come out. A Brier score, the mean squared difference between stated confidence and binary outcome, rescaled to a nought-to-hundred calibration score where a coin flip maps to zero and perfection to one hundred. And an adjustment factor: the count-weighted mean ratio of realised accuracy to stated confidence, hard-clamped between one half and one and a half.

That adjustment factor is the value the compositor multiplies into its source weights and the executor applies to confidence before sizing. Predictions with stated confidence below one half fall into no bin and are silently excluded from the curve.

backend/services/calibration_tracker.py:114–119 — the Beta posterior, per confidence bin
# stated_mid is the bin midpoint, e.g. 0.75 for the 70-80% bin
alpha_prior = PRIOR_STRENGTH * stated_mid           # 10 * 0.75 = 7.5
beta_prior  = PRIOR_STRENGTH * (1 - stated_mid)     # 10 * 0.25 = 2.5
alpha_post  = alpha_prior + correct
beta_post   = beta_prior  + (total - correct)
bayesian_accuracy = alpha_post / (alpha_post + beta_post)
backend/services/calibration_tracker.py:158–177 — the shrinkage actually applied to live confidence
if report["total_reconciled"] < 10:
    # cold start: halve the distance from 0.5 regardless of history
    return 0.5 + (raw_confidence - 0.5) * 0.5

adjusted = 0.5 + (raw_confidence - 0.5) * adjustment_factor   # factor ∈ [0.5, 1.5]
return max(0.01, min(0.99, adjusted))

Figure 12 · Backtesting

Windowed scoring, honestly labelled

The module is called a walk-forward backtester and the name overstates what it does. It slices the reconciled prediction history into overlapping train-and-test windows, then computes out-of-sample metrics on each test window. Nothing is ever fitted to the training window — it is measured for its row count and otherwise unused. The correct description is windowed out-of-sample scoring of already-logged predictions, which is a useful thing to have, but it is not walk-forward optimisation.

Figure 12 Window geometry, aggregate metrics, and the graduation gate
Walk-forward backtest window geometry and graduation criteria A date axis with four successive windows, each composed of a ninety-day training segment and a thirty-day test segment, stepping forward thirty days at a time so that training segments overlap. An annotation marks that the training segment is counted but never trained on. Below, the aggregate metric formulas are listed, followed by the four graduation conditions that must all hold for a strategy to be marked as graduated. WINDOW GEOMETRY — train 90 d, test 30 d, step 30 d first prediction most recent win 1 train 90 d — counted, never fitted test 30 d win 2 train 90 d test 30 d win 3 train 90 d test 30 d win 4 train 90 d test 30 d step 30 d — test windows tile without overlap at the default step WHY THE NAME MISLEADS The train slice is passed into the metric function and used only as a row count. No model is refit, no parameter is chosen from it. It is out-of-sample scoring, not optimisation. AGGREGATE METRICS OVER ALL TEST RETURNS strategy_return = sign(direction) × actual_return_1d FLAT is signed −1.0 — treated as a short sharpe = mean ÷ std × √252 no risk-free rate max_drawdown = max(running_max − cumsum) additive, not compounded; return-fraction units profit_factor = Σ gains ÷ |Σ losses| capped at 999 annualised = total_return × (252 ÷ n) n is a prediction count, not a day count consistency = 1 − std(per-window win rates) defaults to 0.5 with a single window GRADUATION GATE — all four must hold ≥ 20 predictions minimum sample Sharpe > 0 any positive value passes win rate ≥ 0.55 directional accuracy max drawdown ≤ 0.20 in return-fraction units
Figure 12. The backtest geometry. Four strategy identifiers are eligible — the technical, alpha-factor, quantitative and compositor sources — and the language-model and sentiment sources are deliberately excluded because their calls cannot be replayed. A run needs at least ten reconciled predictions and a date range longer than one full window, or it returns an error rather than a result. backend/services/backtest_engine.py:46–51 (eligible strategies), :53–229 (window loop), :278–303 (window metrics), :305–363 (aggregates), :365–394 (graduation)

The other graduation check, and why it is advisory

A separate module answers a bigger question — is this system ready for live trading? — with five criteria: ninety days of paper trading, a positive Sharpe over at least twenty realised returns, two strategies with a proven edge, evidence the circuit breaker has fired, and a human opt-in.

Three of the five are weaker than they read. The strategy criterion substitutes tournament win rate for the walk-forward efficiency it names, and the source comment says so. The circuit-breaker criterion depends on in-memory trigger history, so it resets to false on every restart. And the human opt-in is bypassed outright by a configuration flag that defaults to true, annotated as skipping the check for automated systems. The column that would record an acknowledgement is read through a safe attribute lookup and does not exist on the user model at all.

Most importantly: nothing anywhere gates live trading on the result. The check is exposed as a read-only report. Whether orders go to paper or production is decided entirely by a separate configuration boolean.

Test coverage

One hundred and thirty-four tests, unevenly distributed

The suite runs against an in-memory database with fixtures that create every table, insert a sample instrument, and generate a hundred days of synthetic price history. Coverage is concentrated on the modules that are hardest to reason about, which is the right instinct, and absent from most of the rest.

backend — pytest 
$ python -m pytest --collect-only -q
tests/test_alpha_factor_service.py ::test_...   28 tests
tests/test_api_endpoints.py        ::test_...   12 tests
tests/test_backtest_engine.py      ::test_...   20 tests
tests/test_bitemporal.py           ::test_...   15 tests
tests/test_entity_neutering.py     ::test_...   26 tests
tests/test_signal_compositor.py    ::test_...   23 tests
tests/test_weight_adjustment.py    ::test_...   10 tests

134 tests collected

$ ls backend/services/*.py | wc -l
44

# 7 test modules · 44 service modules.
# Untested and load-bearing: reconciliation_service, calibration_tracker,
# strategy_tournament, trade_executor, circuit_breaker, factor_scoring_engine,
# universe_service, openbb_service, nim_agents_service.
# The tournament and calibration tracker appear only as mocks inside
# test_signal_compositor.py — their own logic is never exercised.

Counts are exact, obtained by counting test function definitions per module. The frontend has no test framework and no test files at all.