How to edit video with Claude, Codex, or any AI agent

What the agent can and cannot do

Because the transcript only captures speech, the agent edits based entirely on what people say. It can strip out false starts, filler, repeated takes, tangents, and long pauses, or pull specific sentences to build a short clip. It cannot see the video itself. If you need visual edits, like choosing which camera angle to show, you will need to make those calls yourself or pass video frames to the agent.

Agents that run terminal commands (like Claude Code, Codex CLI, or a Cursor agent) can call the API and run ffmpeg on their own. A standard chat assistant without code execution cannot run the tools, though it can still write the scripts for you.

The prompt to paste into your agent

This prompt points the agent to our machine-readable API docs and forces it to reply with segment numbers instead of raw timestamps. That way, the model never hallucinates times.

Edit talk.mp4 down to the parts worth keeping, using the KolWrite API for the transcript.
Read https://console.kolwrite.com/agent-instructions.md first. The key is in KOLWRITE_API_KEY.
1. Extract the audio with ffmpeg and transcribe it with model "oracle", output_format "json", word_timestamps true.
2. Print the segments as numbered lines: S<number> [start-end] text.
3. Choose the segments to keep (remove false starts, filler, repeated takes and tangents) and write
   their numbers to keep.json as {"keep": [0, 2, 4]}. Do not invent timestamps; use segment numbers only.
4. Cut the video from those segments' word times with ffmpeg and write edited.mp4.
5. Make edited.srt for the edited video and attach it as a subtitle track.
Show me the list of removed segments before you cut anything.

To keep the API available in every session, drop the rule from the tool guide into your AGENTS.md or CLAUDE.md file.

Step 1: extract the audio and transcribe it

Signed uploads take files up to 100 MiB, so if you have a long video, strip out the audio track first. One hour of mono audio at 48 kbit/s comes out to about 21 MB.

ffmpeg -i talk.mp4 -vn -ac 1 -ar 16000 -b:a 48k talk.mp3

This helper script uploads a local file, starts the transcription job, and polls until the result is ready:

kolwrite.py

# kolwrite.py — three small functions around the KolWrite REST API
import os, time, requests

API = "https://app.kolwrite.com"
HEADERS = {"x-api-key": os.environ["KOLWRITE_API_KEY"]}


def upload(path):
    """Upload a local file (up to 100 MiB) and return a URL the API can read."""
    name = os.path.basename(path)
    form = requests.post(f"{API}/upload", headers=HEADERS, json={"filename": name}, timeout=30)
    form.raise_for_status()
    form = form.json()
    with open(path, "rb") as media:
        # No x-api-key on this request: it goes to the storage address.
        sent = requests.post(form["uploadUrl"], data=form["uploadFields"],
                             files={"file": (name, media)}, timeout=600)
    sent.raise_for_status()
    return form["fileUrl"]


def transcribe(audio_url, timeout_s=3600, **options):
    """Start a job, wait for it, return the result: a dict for json, a string for srt and txt."""
    body = {"audio_url": audio_url, "model": "oracle", **options}
    job = requests.post(f"{API}/transcribe", headers=HEADERS, json=body, timeout=30)
    job.raise_for_status()
    job_id = job.json()["jobId"]
    deadline = time.time() + timeout_s
    while time.time() < deadline:
        time.sleep(5)
        reply = requests.get(f"{API}/transcribe/{job_id}", headers=HEADERS, timeout=30)
        reply.raise_for_status()
        reply = reply.json()
        if reply["status"] == "COMPLETED":
            return reply["result"]
        if reply["status"] == "FAILED":
            raise RuntimeError(reply.get("error", "transcription failed"))
    raise TimeoutError(f"job {job_id} did not finish in {timeout_s} seconds")

In edit.py below, we request JSON output with word timestamps (both are on by default, though we write them out here so it is clear). That gives you segments[], where each item includes start, end, text, an optional speaker, and words[]:

Here is a shortened view of the response:

{
  "jobId": "34c63d45-4cb5-473b-bc60-25c175e69c57",
  "status": "COMPLETED",
  "result": {
    "language": "en",
    "segments": [
      {
        "start": 1.0,
        "end": 3.75,
        "text": "Hello everyone and welcome to the show.",
        "speaker": "Speaker 1",
        "words": [
          { "start": 1.0, "end": 1.35, "word": "Hello" },
          { "start": 1.4, "end": 1.75, "word": "everyone" }
        ]
      }
    ]
  }
}

Step 2: let the agent choose what to keep

Print the transcript as a simple list of numbered lines for the agent to read. Sending plain text uses far fewer tokens than feeding it the raw JSON with every single word object.

view.py

# view.py — print the transcript the way an agent can read it
import json, sys

result = json.load(open(sys.argv[1]))
for number, segment in enumerate(result["segments"]):
    who = f'{segment["speaker"]}: ' if segment.get("speaker") else ""
    print(f'S{number} [{segment["start"]:.1f}-{segment["end"]:.1f}] {who}{segment["text"]}')

The agent replies with just the segment numbers:

keep.json

{"keep": [0, 2, 4]}

Have the agent list what it dropped and why before you run the cut. Re-running a bad cut is cheap, but scanning a text list is much faster than watching the whole video back.

Step 3: cut with ffmpeg from the word times

The script looks up the first and last word of each chosen segment to get the exact start and end times. It pads each end by a few hundredths of a second, merges adjacent segments, and re-encodes the video in a single pass for frame-accurate cuts. At the same time, it shifts the word timestamps to match the new timeline so your subtitles stay in sync.

cut.py

# cut.py — ranges from segment numbers, one ffmpeg pass, word times for the new timeline
import subprocess

PAD = 0.08  # seconds of room kept around every range


def ranges_from_ids(segments, keep_ids, pad=PAD):
    """Turn the segment numbers the agent chose into [start, end] ranges in seconds."""
    ranges = []
    for i in sorted(set(keep_ids)):
        segment = segments[i]
        words = segment.get("words") or []
        start = words[0]["start"] if words else segment["start"]
        end = words[-1]["end"] if words else segment["end"]
        start, end = max(0.0, start - pad), end + pad
        if ranges and start <= ranges[-1][1]:
            ranges[-1][1] = max(ranges[-1][1], end)
        else:
            ranges.append([start, end])
    return ranges


def cut(video, ranges, out):
    """Re-encode the kept ranges into one file (frame accurate)."""
    parts = []
    for i, (start, end) in enumerate(ranges):
        parts.append(f"[0:v]trim=start={start:.3f}:end={end:.3f},setpts=PTS-STARTPTS[v{i}]")
        parts.append(f"[0:a]atrim=start={start:.3f}:end={end:.3f},asetpts=PTS-STARTPTS[a{i}]")
    joined = "".join(f"[v{i}][a{i}]" for i in range(len(ranges)))
    parts.append(f"{joined}concat=n={len(ranges)}:v=1:a=1[v][a]")
    subprocess.run(["ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-i", video,
                    "-filter_complex", ";".join(parts), "-map", "[v]", "-map", "[a]", out], check=True)


def remap(words, ranges):
    """Move word times onto the edited timeline, so subtitles match the cut video."""
    moved, offset = [], 0.0
    for start, end in ranges:
        for word in words:
            if word["start"] >= start and word["end"] <= end:
                moved.append({**word, "start": word["start"] - start + offset,
                              "end": word["end"] - start + offset})
        offset += end - start
    return moved

edit.py

# edit.py — the whole pipeline: transcript in, edited video and subtitles out
import json
from kolwrite import upload, transcribe
from cut import ranges_from_ids, cut, remap
from srt_tools import cues_from_words, write_srt

result = transcribe(upload("talk.mp3"), output_format="json", word_timestamps=True)
with open("talk.json", "w", encoding="utf-8") as out:
    json.dump(result, out, ensure_ascii=False)
# ... the agent reads talk.json (python view.py talk.json) and writes keep.json ...

keep = json.load(open("keep.json"))["keep"]
ranges = ranges_from_ids(result["segments"], keep)
cut("talk.mp4", ranges, "edited.mp4")

words = [w for segment in result["segments"] for w in segment.get("words", [])]
write_srt(cues_from_words(remap(words, ranges)), "edited.srt")

While copying streams without re-encoding (-c copy) is faster, ffmpeg can only cut on keyframes that way, which can throw your cut points off by a few seconds. Re-encode whenever exact timing matters.

Step 4: subtitles for the edited video

edit.py already used the shifted word times to write edited.srt. If you want to see how the cues are put together, check the SRT guide.

srt_tools.py

# srt_tools.py — neat subtitle cues from word timings
def srt_time(seconds):
    ms = round(seconds * 1000)
    h, ms = divmod(ms, 3_600_000)
    m, ms = divmod(ms, 60_000)
    s, ms = divmod(ms, 1000)
    return f"{h:02}:{m:02}:{s:02},{ms:03}"


def cues_from_words(words, max_chars=42, max_seconds=6.0, max_gap=0.8):
    """Group timed words into subtitle cues: (start, end, text)."""
    cues, line = [], []
    for word in words:
        text = word["word"].strip()
        if not text:
            continue
        if line:
            length = sum(len(w["word"]) + 1 for w in line) + len(text)
            if (length > max_chars
                    or word["end"] - line[0]["start"] > max_seconds
                    or word["start"] - line[-1]["end"] > max_gap):
                cues.append(line)
                line = []
        line.append({"start": word["start"], "end": word["end"], "word": text})
    if line:
        cues.append(line)
    return [(c[0]["start"], c[-1]["end"], " ".join(w["word"] for w in c)) for c in cues]


def cues_from_result(result):
    """Cues for a whole JSON result; a segment without word times becomes one cue."""
    cues = []
    for segment in result["segments"]:
        if segment.get("words"):
            cues += cues_from_words(segment["words"])
        elif segment["text"].strip():
            cues.append((segment["start"], segment["end"], segment["text"].strip()))
    return cues


def write_srt(cues, path):
    with open(path, "w", encoding="utf-8") as out:
        for number, (start, end, text) in enumerate(cues, 1):
            out.write(f"{number}\n{srt_time(start)} --> {srt_time(end)}\n{text}\n\n")

From here, you can either burn the subtitles directly into the video or attach them as a soft track that viewers can toggle on and off:

ffmpeg -i edited.mp4 -vf "subtitles=edited.srt" -c:a copy burned.mp4

ffmpeg -i edited.mp4 -i edited.srt -c copy -c:s mov_text -metadata:s:s:0 language=eng soft.mp4

Other ways to edit with word timestamps

• Shorten dead air: keep continuous stretches of speech and cut out any pause longer than your threshold (covered in the word timestamps guide).
• Pull a single clip: search the word list for a specific phrase and cut just that time range.
• Strip filler words: ask the agent to flag segments that only contain filler, or search the word list for them directly.
• Build chapters and highlights by having the agent pick the segment numbers while your script converts them into timestamps (see the podcast and YouTube guide).
• Edit a speaker's part out: add include_diarization and keep only the segments with the speaker you want.

Pricing

We bill per second with no minimum charge. If one section has rough audio and needs a stronger model, you can slice out just that clip and re-run it so you only pay for those seconds. Full rates are in /llms/pricing.md.

Things to keep in mind

• Word times are good enough to cut between words, though they are slightly off at the very edge of a word. The padding in cut.py covers that, so just listen to the cut points before publishing.
• Automated transcripts make mistakes. Always double-check proper names and numbers before publishing. KolWrite does not publish per-language accuracy numbers for this API.
• Hold on to your original video file. The script re-encodes from the source, and you will want it if you decide to change the edit later.
• Passing hundreds of time ranges creates a massive ffmpeg filter graph. If you have more than a few hundred cuts, run them in batches.

Questions and answers