A forecast that grades itself

Publishing a probability is cheap. Publishing one that survives being checked is not. Beat the Streak posts a number for every likely MLB starter each morning — the chance they get at least one hit — and then, the following night, looks up what actually happened, scores itself, and refits its own weights against the answer. This post is the whole pipeline: the shrinkage estimator that makes small samples honest, the log5 matchup algebra, why the lineup card matters more than anyone gives it credit for, why every adjustment happens in logit space, and the grading loop that makes the whole thing falsifiable.

The model matters less than the loop around it. But the model is where the transferable ideas are, so start there.

Step one: a base rate that respects sample size

A hitter batting .310 over 40 at-bats is not a .310 hitter; he is a hitter about whom you know very little. Every input rate is therefore shrunk toward the league mean by a fixed prior weight before it is used for anything:

def reg(rate, n, denom):
    if rate is None or n is None:
        return LEAGUE["avg"]
    return (rate * n + LEAGUE["avg"] * denom) / (n + denom)

This is an empirical-Bayes posterior mean with a Beta prior, written in the form a baseball analyst would recognise: denom is the prior's effective sample size, in the same units as the data. Batting average is regressed with a denominator of 60 at-bats; pitcher batting-average-against with 120, because a pitcher's BAA stabilises more slowly and is contaminated by defence.

The 30-AB row is the whole point. A naive board on 5 May is dominated by hot-start small samples; this one isn't, and the cost is that it will be slow to recognise a genuine breakout. That is the right trade for a daily forecast that gets graded, because being early on ten breakouts does not compensate for being fooled by a hundred hot weeks.

Step two: log5, for the matchup

Now combine a regressed hitter against a regressed pitcher. The classic tool is Bill James's log5, which asks what happens when a batter of quality b faces a pitcher who allows q, in a league averaging L:

def log5(b, q, L):
    x = b * q / L
    y = (1 - b) * (1 - q) / (1 - L)
    return x / (x + y) if (x + y) > 0 else b

The intuition: convert both rates to odds relative to league average, multiply the odds, convert back. It is exactly a logistic model with unit coefficients and the league rate as intercept, which is why it composes correctly at the extremes where an arithmetic mean does not.

Park factor then scales the result, clamped to ±6% so that Coors Field — nominally 1.10 — cannot do more work than the entire matchup. The park table is hand-maintained and includes both current and former names for the same building, because the feed changes sponsor names mid-season and a silently-missed park factor is indistinguishable from a neutral one.

Step three: at-bats, where the lineup card earns its place

All of that yields a per-at-bat hit probability. But the question is about the game, so it needs the number of chances:

exp_ab = 4.35   if batting 1st–3rd
         4.05   if batting 4th–6th
         3.75   if batting 7th or lower

P₀ = 1 − (1 − p_ab) ** exp_ab

Leadoff hitters get roughly half an extra plate appearance per game over the bottom of the order, and for a question of the form "did it happen at least once today", that half-appearance is worth more than most of the fancy adjustments below:

Five points of probability is larger than the entire spread of most of the nine feature adjustments combined. This is the single most under-weighted factor in casual versions of the problem: people compare hitters and forget that one of them bats ninth. The corollary is a data dependency — the board is materially better after lineups are posted, which is why the page prints whether a given hitter's slot is confirmed or assumed.

Step four: features, in logit space

Nine adjustments then modify the base rate — strikeout rate, BABIP, on-base percentage, the pitcher's K/9 and WHIP, recent form over the last 15 games, the platoon split, home/road, and batter-versus-pitcher history. Each is expressed as a standardised deviation from league average and clamped to ±2.5, so no single feature can run away with the estimate:

DEFAULT_WEIGHTS = {"A": 0.0, "B": 1.0, "w": {
    "krate": 0.13, "babip": 0.06, "obp":  0.06, "pk9": 0.09, "whip": 0.10,
    "recent": 0.09, "platoon": 0.11, "homeaway": 0.05, "bvp": 0.07,
}}

def calibrate(P0, feats, W):
    raw = logit(min(0.97, max(0.03, P0)))
    z   = W["A"] + W["B"] * raw + sum(W["w"].get(k, 0.0) * feats.get(k, 0.0) for k in W["w"])
    return min(0.93, sigmoid(z)), raw

The features are applied in logit space, not to the probability directly, and this is the part worth stealing regardless of what you're forecasting. Adding adjustments to a raw probability lets it walk out of [0,1], and it treats a move from 0.50→0.55 as equivalent to 0.94→0.99, which it very much is not — the first halves nothing, the second cuts the failure rate by a factor of six. In logit space the same additive nudge is automatically gentler near the extremes, which is the correct behaviour and comes for free:

Two details in that function are doing quiet work. The clamp to [0.03, 0.97] before taking the logit stops a degenerate base rate from producing an infinite input. And batter-vs-pitcher history carries an extra multiplier of ab / (ab + 15), because 2-for-4 lifetime against a pitcher is noise and the feature should say so — the same shrinkage idea as step one, applied to a feature rather than to a rate.

The loop: grading, and refitting

Every board is archived the moment it is published, complete with each hitter's feature vector and raw logit. This is the part that makes everything above falsifiable: the prediction is written down before the outcome exists, in a form precise enough to score.

The next night, a second job pulls the box scores, marks each hitter 1 or 0 for whether they got a hit, appends the labelled rows to a history file, and computes the day's Brier score — mean squared error between predicted probability and outcome:

brier = mean((p - y) ** 2)

# reference points for a board where the base rate is ~0.72
#   always predict 0.72 (the climatology)   → 0.202
#   a well-calibrated model with real skill →  0.19 or below
#   overconfident model, same accuracy      → worse than climatology

Brier is a proper scoring rule, which is the property that matters: it is minimised by reporting your true belief, so there is no way to game it by shading predictions toward the extremes. Note it is a calibration check as much as an accuracy one — a model that says 0.95 for everyone and is right 72% of the time scores appallingly, which is the intended verdict.

Then it refits, on the entire accumulated history:

def refit(rows, warm):
    A, B, w = warm["A"], warm["B"], dict(warm["w"])
    lr, lam, epochs = 0.05, 0.02, 400
    for _ in range(epochs):
        gA = gB = 0.0; gw = {k: 0.0 for k in w}
        for r in rows:
            z = A + B * r["raw"] + sum(w[k] * r["f"].get(k, 0.0) for k in w)
            e = sigmoid(z) - r["y"]                       # logistic gradient
            gA += e; gB += e * r["raw"]
            for k in w: gw[k] += e * r["f"].get(k, 0.0)
        A -= lr * (gA / n)
        B -= lr * (gB / n + lam * (B - PRIOR["B"]))       # ← shrink toward the prior,
        for k in w:                                       #   not toward zero
            w[k] -= lr * (gw[k] / n + lam * (w[k] - PRIOR["w"][k]))
    B = min(2.5, max(0.2, B))                             # hard stops
    return {"A": A, "B": B, "w": {k: min(0.6, max(-0.6, v)) for k, v in w.items()}, "n": n}

The B coefficient is a calibration slope on the base rate itself, and it is the single most informative number in the weights file:

Watching B drift is a better diagnostic than watching the Brier score, because Brier moves with the difficulty of the day's slate and B does not.

Using it yourself

The two scripts are stdlib-only Python and run on any machine with network access:

python3 modules/streak/predict.py              # today's board  → picks.json + archive/
python3 modules/streak/predict.py 2026-07-20   # any date
python3 modules/streak/grade.py               # grade yesterday, append history, refit

# state, all plain text and all inspectable
modules/streak/picks.json        today's board, as the page reads it
modules/streak/archive/*.json    every board ever published, with features
modules/streak/history.jsonl     one graded row per hitter per day
modules/streak/weights.json      A, B, the nine weights, and n

To repoint the machinery at a different question, three things need replacing and one does not: swap the data source and the LEAGUE constants, define your own feature vector, and set hand priors that encode what you already believe. Keep calibrate, refit and the archive-then-grade loop exactly as they are — that trio is domain-independent, and it is the actual reusable artefact here. Delete weights.json to reset to priors; the daily grade will rebuild it from the archive.

What this is not

It is a box-score model. No batted-ball data, no pitch-level information, no weather, no bullpen modelling, no injury awareness beyond the posted lineup, no defensive positioning. When a lineup has not been published it falls back to the nine hitters with the most at-bats, which will occasionally be wrong in a way that is obvious to anyone watching the game.

The cap at 0.93 is an admission rather than a modelling choice: no hitter is ever 97% to get a hit, and a model that says so is telling you about its own overconfidence rather than about baseball. The cap is where that overconfidence gets truncated instead of published.

What it does have is the property most published forecasts lack — a scoreboard. Every prediction is recorded before the outcome is known, graded against reality, and folded back into the weights, in public, every day, where being wrong is visible. That loop is the actual artefact. The baseball is a convenient excuse to run it.

Related: the same instinct applied to a domain with no ground truth at all — valuing a hockey player from a box score, where there is no labelled dataset to fit against and the coefficients have to be defended on their merits.