Editing agent playbook

This is the canonical behavior for an agent editing with OverlayMotion. It is designed for the common case where the user supplies footage and a weak prompt such as “make this good,” as well as for footage that contains spoken editing instructions.

The agent produces two artifacts:

1. an edit decision plan recording evidence, assumptions, protected subjects, assets and QA checkpoints;
2. an Edit Spec v1 describing the render.

The plan explains judgment. The spec remains a deterministic rendering input. Machine-readable defaults and the plan schema live in src/agent/. Required and conditional tool contracts are in Agent capability toolkit.

Prime directive

Improve comprehension without contradicting the speaker, hiding the important subject or fabricating facts. Follow this priority order:

1. explicit user request;
2. observable transcript and visual evidence;
3. subject, message, rights and platform safety;
4. established brand and platform defaults;
5. decorative style.

When an explicit request conflicts with subject safety, satisfy both through layout. For example, split the screen and reframe the speaker into the open panel instead of placing secondary media over their face.

What the agent may decide

Make reversible aesthetic decisions without interrupting the job: caption style, conservative motion, exact safe placement, crop focus, phrase grouping, and the shortest faithful copy. Record these as assumptions.

Verify or ask before asserting identity, changing meaning, making a factual claim, selecting unlicensed media, publishing a quote whose wording or attribution is uncertain, or making an ambiguous destructive cut. A low-risk draft may keep an unresolved item as a clearly labeled placeholder.

Source analysis before effects

Create one normalized source record before authoring the spec:

• probe container, codec, dimensions, rotation, frame rate, duration, color range and audio streams;
• transcribe speech with word start/end timestamps and language;
• inspect a contact sheet across the full timeline;
• inspect frames around each named person, object, gesture, scene change and requested edit beat;
• note face, mouth, active-hand, demonstrated-object and platform-UI regions;
• preserve the original; use a rotation-correct, editor-friendly proxy only when the renderer needs one.

Treat spoken edit commands as commands only when the context supports it. Keep the original words in the transcript either way.

Default edit for a weak prompt

For a spoken social video, the useful default is deliberately modest:

• trim only obvious dead head/tail space; do not rewrite the speaker;
• add word-timed captions;
• apply brand tokens and real supplied brand assets;
• use restrained emphasis overlays only where the speech supplies a clear semantic beat;
• preserve faces, hands and demonstrated objects;
• keep source audio primary and avoid decorative sound under important speech;
• deliver a debug preview or checkpoints before paying for a full render.

Do not fill silence with effects. Do not turn every sentence into a card.

Captions

Spoken video gets captions by default unless the user opts out.

• Prefer word timestamps from Whisper or another transcriber. Never infer exact cadence by evenly dividing a phrase when real timings exist.
• Group about 3–8 words into a semantic phrase, normally no more than two lines and about 32 characters per line. Break at clauses, not mechanically.
• Preserve pauses as empty time. A caption must disappear during a meaningful silence rather than stretching across it.
• Keep casing and punctuation readable. Correct obvious transcription mistakes only when audio makes the intended word clear.
• Put captions above platform controls and away from the face, mouth, active hands and any media carrying the current meaning.

caption-classic accepts phrase start/end and word start/end, all relative to its overlay window. Text-only lines remain a compatibility fallback.

Faces, split screens and secondary media

Faces, mouths, eyes, active hands, a demonstrated object and the current gesture target are protected subjects. Add padding; do not merely avoid their exact bounding boxes.

When showing an image or secondary video:

1. decide which subject is primary at that beat;
2. choose the secondary media region;
3. reframe the base source into the complementary region with source.reframes;
4. inspect the subject at the entrance, midpoint and exit, not only one frame;
5. place captions in the remaining safe region.

Prefer a stable crop with modest easing. Use tracking only when a static crop cannot keep the subject safe; unnecessary tracking looks nervous and costs more to verify.

{
  "source": {
    "type": "video",
    "src": "speaker.mp4",
    "position": "center 28%",
    "reframes": [
      {
        "time": { "start": "9s", "duration": "8s" },
        "region": { "x": 4, "y": 8, "w": 44, "h": 84 },
        "position": "58% center",
        "transitionSec": 0.3
      }
    ]
  },
  "overlays": [
    {
      "template": "b-roll",
      "region": { "x": 52, "y": 8, "w": 44, "h": 84 },
      "time": { "start": "9s", "duration": "8s" },
      "props": { "src": "secondary.mp4", "frame": "flush", "divider": "left" }
    }
  ]
}

Gesture-aware placement

Words such as “here,” “there,” “this” and “right here” are spatial only when resolved with nearby frames.

1. Find the transcript time of the deictic word.
2. Inspect roughly 0.35 seconds before through 0.65 seconds after it.
3. Identify the active hand, fingertip and direction of the pointing ray.
4. Place the overlay just beyond the fingertip, usually with about 3% frame offset, while keeping hand, face and intended target visible.
5. Inspect the entire overlay window. Track only if the anchor moves enough to break the meaning.

Use a custom percentage region for the resolved location. The region is the result of observation; a named corner is only a fallback.

Quotes and factual inserts

A quote is content, not decoration.

• Select wording that directly supports the nearby speech and fits the safe region at a readable size.
• Verify it against an authoritative source. Preserve exact wording, author or speaker, work/reference, and translation/edition when applicable.
• Do not use an attractive paraphrase as a quotation. If only the idea is known, label it as a paraphrase or use original editorial copy without quote marks.
• Prefer a shorter verified excerpt over shrinking a long quote until it is unreadable.
• Record the source in the decision plan even when the on-screen design shows only the concise attribution.

Assets and logos

Use assets in this order: user-supplied, brand library, licensed source, then generated asset when generation is appropriate and disclosed.

Record source URL, license and required credit. Keep visible credit concise; keep complete provenance in the decision plan. Never reconstruct or hallucinate a real company logo when the asset is unavailable. logo-sting accepts a real transparent logo; its monogram is an intentional fallback, not an imitation.

Efficient QA

Quality comes from targeted checks, not repeated full renders.

1. Validate the decision plan and Edit Spec.
2. Preview the timeline or render low-cost stills at: just before the edit, entrance, midpoint, exit and just after.
3. Add checkpoints for every spatial decision (face/split/gesture) and semantic decision (quote, number, name, attribution).
4. Check caption onset, a middle word, phrase disappearance and a pause.
5. Check source audio once, especially around wrappers and duplicated video layers.
6. Watch one complete preview at normal speed. Full-render once after these checks pass.

For a stable static placement, three frames may be enough. For a moving hand, crop, face or tracked anchor, inspect more frames. Spend QA where uncertainty is.

Completion report

Report what was inferred, what was verified, asset provenance/credit, what the current template system could not express, and which QA checks ran. Never claim “perfect sync,” “face safe” or “verified quote” without the corresponding evidence.