PortfolioToledo Football IQ
Computer VisionSports AnalyticsActive BuildUniversity of Toledo Athletics

Toledo Football IQ

A drone-based computer vision platform for Toledo Football that converts overhead practice footage into clip-linked player tracking, football labels, self-scout insights, and coach-correctable analytics.

10
Stage CV pipeline
50–100
Phase 0 evaluation clips
90%+
Field-marking visibility target
85%
Usable-clip target for metrics
18+
Database schema tables
<10 min
Same-session feedback target
Toledo Football IQ Analytics Platform
01 — Overview

Trust is the product

Toledo Football IQ is an active computer vision and sports analytics platform for University of Toledo Football. It is designed to convert drone-based overhead practice footage into structured, coach-verifiable football intelligence.

The core principle is evidence-first analytics. Every output connects to the exact clip, overlay, confidence score, correction path, and model version. If a coach cannot verify it, correct it, and teach from it — it should not be treated as a production metric.

The system is built as a coach-in-the-loop football intelligence platform. It starts with reliable practice-film infrastructure — video ingestion, play segmentation, field calibration, player tracking, clip-linked metrics, coach correction — and grows toward formation and motion analysis, route and coverage analytics, self-scout, pose-lite biomechanics, player development profiles, similar-rep search, zero-shot concept discovery, and same-session feedback.

Product north star: a Toledo-owned CV platform that converts drone and practice film into verified football intelligence — faster coaching feedback, better self-scout, better opponent prep, objective player development, and health-aware workload context.

02 — Problem

Raw film is valuable but expensive to review

Football programs already use film heavily. Coaches and analysts manually tag plays, identify formations, evaluate routes, track player effort, study tendencies, and create cutups. This works — but it is time-intensive, and the bottleneck gets worse as the volume of drone and practice footage grows.

The challenge is not just automation. The challenge is trust. A football computer vision system must handle:

Drone camera movement

Overhead drone footage involves dynamic homography — the camera moves, drifts, and reorients, meaning field calibration must be recomputed continuously, not once.

Occlusion at the line

Players in crowded line-of-scrimmage situations overlap heavily. Standard detection and tracking methods degrade significantly in high-occlusion football scenarios.

Re-identification difficulty

Players wear helmets and similar uniforms. Jersey numbers blur at distance or are never visible from overhead. Identity association requires multiple signal types.

Coach-specific terminology

Generic CV labels like "mesh" or "outside zone" may not match Toledo's internal call language. A system that speaks wrong terminology loses coach adoption immediately.

Overconfident outputs

A model that reports confident metrics on a poorly calibrated clip is worse than no model. Calibration confidence must gate downstream analytics.

Health and workload sensitivity

Effort, load, and biomechanics outputs touch athlete welfare. Role-based access, conservative labeling, and no automated medical inference are non-negotiable.

Commercial tools can provide broad AI tagging, but Toledo can create a local advantage through overhead drone footage, Toledo-specific terminology, a coach-correction flywheel, and athlete-development integration. That private, corrected, domain-specific data is the moat.

03 — CV Pipeline

10 stages from raw video to coach dashboard

01
Video Ingest

Probe FPS, resolution, codec, duration, corruption, and metadata. Store raw video and queue processing jobs.

02
Play Segmentation

Split continuous practice video into per-play clips with editable boundaries. Manual correction is a first-class feature, not an afterthought.

03
Field Calibration

Detect sidelines, yard lines, hash marks, field numbers, and boundaries. Estimate homography into standardized football field coordinates. Output calibration confidence and analytics-safe status — weak calibration suppresses downstream metrics.

04
Player Detection

YOLO or RF-DETR detects players, officials, and the ball. Detection confidence feeds the calibration-safe gating layer.

05
Player Tracking

ByteTrack for speed and throughput. BoT-SORT for identity-aware tracking. SAM + CSRT-style approaches for heavy occlusion cases at the line of scrimmage.

06
Identity Association (Re-ID)

Combine jersey OCR, roster priors, height/weight/position priors, equipment cues, biometric ratios, and movement signatures. Every identity assignment carries a confidence score and a correction path.

07
Event Detection

Detect snap, motion start/end, handoff, throw, catch, contact, tackle, and end-of-play events. Manual correction is supported early — automated event detection improves as corrections accumulate.

08
Football Labels

Propose offensive formation, defensive front, coverage shell, motion, shift, route family, and pressure candidates. Separate generic labels from Toledo-specific terminology.

09
Metric Computation

Compute cushion, separation, distance traveled, max speed, return-to-huddle speed, time to throw, dropback depth, gap width proxy, pass-set depth, pursuit effort, and downfield blocking participation.

10
Overlay Rendering & Dashboard Indexing

Render clips with field grids, tracks, route paths, motion arrows, coverage shells, confidence warnings, and metric callouts. Index for dashboard search and coach review.

04 — Architecture

Evidence-first system design

Every layer of the stack is chosen to serve the evidence-first contract: every metric must be traceable to a clip, a model version, a confidence score, and a correction path.

FE
React Frontend
Practice Inbox — upload status, QA flags, calibration warningsClip Review — overlays, boundary correction, identity assignment, label correctionCoach Dashboards — formations, effort, tempo, clip-linked chartsLabeling Queue — low-confidence plays, identity conflicts, regression examples
API
FastAPI Backend
Videos APIClips APIMetrics APILabels APICorrections APIJobs APIPlayers APIModel versions API
DB
Managed Postgres + pgvector
18+ relational tablespgvector for play embeddings and similar-rep searchAlembic migrationsFull evidence lineage per output
GPU
GPU Processing
Local NVIDIA workstation — Phase 0/1Burst cloud GPU workers — Phase 2+CUDA + cuDNNFFmpeg NVDEC/NVENCPyTorch model pipelineZero-copy GPU pipelines for high-res video
STORE
Object Storage
Cloudflare R2 or S3-compatibleRaw videosPer-play clipsOverlay rendersModel artifactsTraining dataset exports
QUEUE
Job Queue
Cloudflare Queues / BullMQ / CeleryVideo processing queueNightly training export queueSame-session lightweight queue
MLR
Model Registry
MLflow or internal registryEvery output tied to model version + dataset snapshotTraining data export from coach correctionsVersion-gated metric computation
05 — Database Design

Schema built around evidence and lineage

Every table answers one question for any output: where did this metric come from, what model produced it, what evidence supports it, and has a coach corrected it?

videos
video_id
filename
duration
fps
resolution
upload_at
status
clips
clip_id
video_id
start_frame
end_frame
play_number
corrected
analytics_safe
field_calibrations
cal_id
clip_id
homography_matrix
confidence
analytics_safe
model_version_id
tracklets
tracklet_id
clip_id
player_id
tracker
confidence
identity_source
corrected
trackpoints
point_id
tracklet_id
frame
pixel_x
pixel_y
field_x
field_y
events
event_id
clip_id
event_type
frame
confidence
corrected
labels
label_id
clip_id
label_type
value
toledo_term
confidence
model_version_id
coach_corrections
correction_id
target_type
target_id
field
old_value
new_value
corrected_by
created_at
metrics
metric_id
clip_id
player_id
metric_type
value
confidence
model_version_id
suppressed
players
player_id
roster_name
number
position
height
weight
active
player_profiles
profile_id
player_id
role_summary
dev_goals
coach_notes
workload_visible
model_versions
version_id
model_name
version_tag
trained_on
deployed_at
active
training_datasets
dataset_id
created_at
correction_count
source_clips
exported_at
play_embeddings
embedding_id
clip_id
model_version_id
vector
created_at
processing_jobs
job_id
video_id
stage
status
started_at
finished_at
error
users
user_id
name
role
department
active
quantum_experiments
exp_id
name
hypothesis
status
result_summary
created_at

The coach_corrections table is the flywheel engine — every correction to any output type is recorded, timestamped, and exported to training_datasets for model retraining. The metrics.suppressed flag prevents analytics from surfacing when calibration confidence is below threshold.

06 — Key Features

From reliable foundations to Toledo differentiators

Coach-Correction Flywheel

Coaches and analysts correct play boundaries, formation labels, route labels, coverage labels, player identity, and metrics. Every correction becomes structured Toledo-specific training data. This is the competitive moat: private, corrected, domain-specific data beats any off-the-shelf model at Toledo football terminology.

Toledo Terminology Layer

The platform separates generic football labels from Toledo-specific call language. A concept can have a generic label like "mesh" while also mapping to Toledo's internal terminology. This keeps ML portable while making the product feel native to staff.

Calibration Confidence Gating

Each clip receives an analytics-safe score based on field visibility, homography stability, player scale, motion blur, drone drift, and occlusion density. Weak calibration suppresses precise speed, separation, and distance metrics before they can mislead coaching decisions.

Self-Scout Exposure Index

Identifies what Toledo is unintentionally revealing: formation-to-play tendencies, motion-to-play tendencies, field-zone tendencies, personnel tendencies, and pre-snap tells. Shows what a prepared opponent would see.

Pose-Lite Biomechanics

Coarse, coach-useful measures from overhead footage: pad level, hip flexion, torso angle, pass-set depth, strike timing, shoulder-over-knee alignment, deceleration step count, hip sink depth, stride symmetry, QB shoulder-hip separation. These remain experimental until reviewed by the relevant position coach.

Individualized Player Profiles

Longitudinal development profiles: roster bio, role summary, development goals, coach notes, best teaching clips, best recruiting clips, corrected metrics, position benchmarks, player-facing summaries, workload visibility controls, and weekly snapshots. This turns the platform into a player-development system, not just a film tagger.

Similar-Rep Search

Coaches select a clip and find similar reps based on movement, spacing, formation, route concept, or play embeddings (pgvector). Valuable when labels are incomplete or when scouting opponent concepts.

Same-Session Feedback

A later phase targets period-break feedback: clip-ready metrics to position coaches within 5–10 minutes between periods. Full sessions processed within 60 minutes of practice ending. Lightweight models run during practice; full-quality runs run overnight.

Technical stack

YOLORF-DETRByteTrackBoT-SORTRTMPoseFastAPIReactPostgrespgvectorCloudflare R2MLflowPyTorchFFmpegCUDABullMQ
07 — Phased Roadmap

Foundations before frontier analytics

Phase 0Capture & Validation Sprint
  • Drone capture protocol and naming/upload process
  • 50–100 representative evaluation clips
  • Toledo label taxonomy definition
  • Baseline detection and calibration report
  • Coach review checkpoint and MVP scope freeze
Exit criteria: 90%+ of clips have visible field markings. Coaches approve MVP direction.
Phase 1Foundation MVP
  • Video ingestion and job tracking
  • Play segmentation with manual boundary correction
  • Field calibration and analytics-safe scoring
  • Player detection and tracking
  • Clip-linked metrics
  • Coach correction UI
  • Model and data versioning
  • Role-based access
Exit criteria: 90% of plays have acceptable boundaries. 85% of clips yield useful tracking. Metric → evidence clip in ≤2 clicks.
Phase 2Coach-Trusted Football Layer
  • Formation and motion recognition
  • Route classification
  • Coverage shell and leverage
  • Offensive-line spacing and blocking metrics
  • QB decision metrics
  • Practice tempo and effort metrics
  • Self-scout exposure dashboard
  • Correction analytics
Exit criteria: Weekly coaching workflow integration. Analysts running correction exports.
Phase 3Toledo Differentiator Layer
  • Pose-lite biomechanics (one position group pilot)
  • Individualized player profiles
  • Similar-rep search (pgvector)
  • Zero-shot concept discovery
  • Playbook overlay
  • Assignment execution scoring
  • Workload proxy dashboard
  • Same-session feedback pipeline
Exit criteria: Toledo owns a private football intelligence layer no vendor can replicate.
Phase 4Frontier R&D
  • xSep, xYards, xPressure, xCompletion models
  • Defensive intent modeling
  • Counterfactual simulation
  • Opponent concept matching
  • Advanced health fusion
Exit criteria: Only begins after tracking, labels, corrections, and adoption are reliable.
08 — Lessons & Design Principles

What this project reinforces

Trust beats novelty

A coach who does not trust the system will not use it. Building trust — through evidence links, correction paths, confidence scores, and conservative suppression — is more important than adding advanced features early.

Correction workflows come before advanced models

The coach-correction flywheel must exist before pose-lite biomechanics, similar-rep search, or zero-shot discovery. Without corrections, there is no Toledo-specific training data. Without Toledo data, the models are generic.

Calibration confidence is a safety gate

A model that reports confident metrics on a poorly calibrated clip is actively harmful. Suppressing metrics when calibration is weak is a feature, not a limitation.

Drone footage is powerful but technically messy

Dynamic homography, camera drift, high-occlusion football scenarios, and helmeted players with similar uniforms make drone footage harder than broadcast footage. These constraints need to be part of the design, not discovered during development.

Speed matters as much as accuracy

A system that takes 24 hours to process a practice session cannot change practice behavior. Same-session feedback — even lightweight — is worth more than overnight perfect analysis.

R&D must not destabilize the MVP

Frontier experiments — zero-shot discovery, quantum AI sandbox, advanced health fusion — need isolation from the production coaching workflow. Experimental outputs must be clearly labeled and role-gated.

Health outputs require governance, not just disclaimers

Effort, load, and biomechanics data touch athlete welfare. Role-based access, conservative labeling, and a strict no-automated-medical-inference rule are non-negotiable from the start, not retrofitted later.

In football analytics, trust is the product.

Toledo Football IQ demonstrates that computer vision in sports is not a model problem — it is a product problem. The best moat is private, corrected, domain-specific data. Every coach correction is a training example. Every trusted output extends what the system can do next.