How to Evaluate an AI Football Prediction Service

Start with the claim, not the interface

Write down the exact claim being evaluated. “Accurate predictions” is too vague. A testable claim names the market, probability or selection rule, publication cutoff, period, competition coverage, metric and comparison.

Ten-part audit

scikit-learn's model-evaluation documentation sets out metrics for classification and probabilities. Calibration guidance explains the separate test of whether probability groups correspond to observed frequencies.

Recalculate one published period

1. Export every eligible forecast before examining results.
2. Verify fixture identity, target and kickoff time.
3. Preserve missing forecasts as coverage failures or declared exclusions.
4. Settle from a documented result source and market rule.
5. Recompute the published metric and denominator.
6. Compare the result with a baseline on the same fixtures.
7. Inspect calibration and league or time slices.

Use a later period that was not repeatedly used for tuning. TimeSeriesSplit documents chronological splitting; a public live record is stronger than a retrospective reconstruction when the underlying pipeline changes.

Check product attribution

Providers may supply probabilities that another product maps and displays. Sportmonks documents a probabilities feed with particular markets and predictability metadata. A service integrating such output should say so and distinguish provider modelling from its own mapping, comparison or reporting.

Check generated explanations

An explanation can sound specific while being disconnected from the actual model. OpenAI's accuracy guidance notes that generated outputs can be wrong or include fabricated citations. Ask whether the explanation is derived from stored model inputs and whether each factual statement has a source.

Decision record

Classify each material claim as verified, partly verified, unsupported or not testable from public evidence. Record the source, access date and missing artifact. “Not testable” is a meaningful conclusion; it should not be silently converted into trust or distrust.

Work one sample through the audit

Choose a predeclared date range and export the service record before settling it. The audit file should include every eligible fixture, published probability, timestamp, target, model or provider attribution and price record where a price claim is made. scikit-learn's model-evaluation guide supports aligning the target, prediction output and scoring input.

scikit-learn's model-evaluation guide supports the metric definitions, and its calibration guide supports reliability reporting.

Grade evidence, not polish

Give each claim a status: verified, partly verified, unsupported or not publicly testable. A clear interface and detailed explanation do not substitute for immutable forecasts. Conversely, private model code does not prevent prospective evaluation if outputs, attribution and coverage are preserved.

Repeat the audit after a version change. Do not combine records across materially different targets, providers or selection rules without exposing the boundary.

Continue the workflow

Continue with Premier League accuracy measurement for a concrete denominator, coverage and calibration reporting template.

Continue learning

• Next guide: Can AI Beat Bookmakers? What the Evidence Must Show
• Related guide: Elo Rating System for Football

Assumptions and limitations

This checklist does not identify a current best service. Public evidence can lag production systems, and proprietary code may remain private. A service can still be evaluated through prospective forecasts, clear attribution, reproducible outputs and honest limits without revealing all source code.