CineTrace AI — protect the dailies

—frame samples

—render jobs

—render hosts

—days of history

—on disk

—rollup rows (MV)

Dailies at risk

Projecting the shot queue against each show's review time…

— shots will miss review

— recoverable by freeing stuck slots

— GPU slots held by zombies & idle queue

The SQL behind this

Impact

Burning right now

—

Recovered by approval

—

Same pricing over the farm's full history

—

Waste by category

Live counts from the deterministic detection layer

The three agents

Detect → decide → dry-run, in that order every time. The pipeline is an ADK SequentialAgent, so no stage can be skipped; the reasoning inside each stage is the model's.

No supervisor run in this session.

This run · SQL the Sentinel wrote itself

The Sentinel is given the schema and the goal, not a list of queries. Run the supervisor to capture the run_query calls it composed.

Root cause

ASOF LEFT JOIN pairs every OOM failure with the nearest telemetry sample recorded before it died. No other engine expresses "the row just before this moment" in one join, and pulling a quarter-billion samples into Python to find it is not an option.

job	show / shot	host	VRAM at death	lead time	frame

The SQL behind this

Repeat failures per host

lagInFrame over failures partitioned by host. Two failures inside 90 minutes on the same node is a node problem, not six unlucky jobs.

host	show	job	error	gap

Have we seen this before?

Error text is embedded with Vertex AI text-embedding-005; the archive of past incidents lives in ClickHouse and is searched by cosineDistance. It matches meaning, so a description that shares no keywords with the original ticket still finds it.

Detection layer

Five deterministic queries over the job_waste view. Thresholds are per-cohort quantileTDigest fences, not constants: a job is an overrun relative to other jobs on the same show and renderer, never at a fixed hour count.

ClickHouse query_log

live cluster

Recent queries on this service that touched the farm tables.

when (UTC)	user	rows read	ms	query

Farm activity

Served from farm_minute, an AggregatingMergeTree kept current by a materialized view. The quarter-billion-row fact table is never touched by this chart.

day	jobs	cpu h	gpu h

Live farm

Open waste first, then the freshest activity. 198k jobs in the table; these are the ones that matter.

job	show / shot	status	class	error	retries	cpu h	gpu h	frames	waste

Remediation proposals

Every proposal is dry-run and append-only. Nothing reaches a render host until a human approves it here — and the Impact card only credits a saving once you do.

when (UTC)	job	action	protects	evidence	decision