Agent capability toolkit

OverlayMotion specifies capabilities, not vendors. An agent may use local CLI tools, hosted services or a human-assisted step, provided the returned evidence meets these contracts. The canonical machine-readable list is AGENT_CAPABILITIES in src/agent/policy.ts.

Required capabilities

Media probe

Input: source media. Output: duration, display dimensions, rotation, frame rate, codec, color/HDR information and audio streams. ffprobe is one suitable implementation. Never assume file extension describes the decoded video.

Word transcription

Input: speech audio. Output: language, segments, each word's start/end time and confidence where available. Whisper implementations are suitable. Segment-only transcription is a degraded fallback; evenly divided words are not “exact sync.”

Frame sampling

Input: media and timestamps. Output: a timeline contact sheet plus full-resolution frames at edit checkpoints. Sample the whole clip once, then densely around gestures, overlays, cuts, crops and semantic inserts.

Visual inspection

Input: frames plus transcript intent. Output: padded protected regions, crop focus, gesture anchor and confidence. Multimodal inspection is enough for stable placements. It must examine time, not one convenient still.

Validation and preview QA

Validate both the Edit Decision Plan and Edit Spec. Inspect low-cost checkpoints before full render; verify source audio, captions, spatial safety and semantic content. The MCP server exposes the guide, prompt, templates and validators.

Conditional capabilities

Face/hand landmarks or tracking

Use a detector such as MediaPipe only when the face, hand or fingertip moves enough that sampled inspection cannot keep an anchor safe. Return time-series landmarks and confidence. Smooth noisy coordinates, keep padding, and fall back to a static safe region when confidence drops. Installing a large model for a stable three-second gesture is wasteful and creates extra failure modes.

Asset acquisition

Use when the requested image/video is absent. Search sources that expose usage rights, download an editor-friendly file, and return source URL, license and required credit. Reject a visually good asset when its rights cannot be established. Cache it locally so preview and render do not depend on a remote URL.

Citation verification

Use when displaying a quote, statistic, identity, date or other factual claim. Return verified copy, an authoritative source and any edition/translation that changes wording. Search snippets are discovery, not verification.

Media normalization

Use only when the source cannot be decoded predictably. Bake rotation and make an editor-friendly proxy while preserving the original reference. Do not silently alter speed, color or audio sample timing.

Escalation ladder

1. metadata + transcript + sparse contact sheet;
2. targeted full-resolution frames;
3. denser sampling around uncertain motion;
4. landmarks/tracking when spatial meaning still cannot be preserved;
5. ask the user only when evidence cannot resolve a consequential choice.

This ladder keeps tokens, model downloads and renders proportional to actual uncertainty.