How to give an AI agent a transcription tool

Do you need a tool definition at all?

If your agent can run shell commands (Claude Code, Codex CLI, Cursor, Windsurf), it does not need a schema: it runs curl or Python, and the instructions in agent-instructions.md tell it how. A tool definition is for agents you build with an API and a tool loop, or for frameworks that call functions.

Tool definition: Anthropic tool-use format

{
  "name": "transcribe_media",
  "description": "Transcribe an audio or video file with the KolWrite API. Returns timed text. Use output_format \"json\" for word timestamps and speakers, \"srt\" for subtitles, \"txt\" for plain text. Billed per second of audio.",
  "input_schema": {
    "type": "object",
    "properties": {
      "path_or_url": { "type": "string", "description": "Local file path or a public URL of an audio or video file." },
      "model": { "type": "string", "enum": ["base", "oracle_mini", "oracle", "oracle_pro", "oracle_max"], "description": "Default oracle. base is cheapest for clear audio; translate_to needs an Oracle model." },
      "language": { "type": "string", "description": "Language code such as en, he, ar, es, or auto to detect." },
      "translate_to": { "type": "string", "description": "Language code to translate into (Oracle models only)." },
      "output_format": { "type": "string", "enum": ["json", "txt", "srt"] },
      "word_timestamps": { "type": "boolean", "description": "Word times in json. For srt, false gives one cue per segment." },
      "include_diarization": { "type": "boolean", "description": "Label who is speaking." },
      "num_speakers": { "type": "integer", "minimum": 1 }
    },
    "required": ["path_or_url"]
  }
}

Tool definition: OpenAI function-calling format

{
  "type": "function",
  "function": {
    "name": "transcribe_media",
    "description": "Transcribe an audio or video file with the KolWrite API. Returns timed text. Use output_format \"json\" for word timestamps and speakers, \"srt\" for subtitles, \"txt\" for plain text. Billed per second of audio.",
    "parameters": {
      "type": "object",
      "properties": {
        "path_or_url": { "type": "string", "description": "Local file path or a public URL of an audio or video file." },
        "model": { "type": "string", "enum": ["base", "oracle_mini", "oracle", "oracle_pro", "oracle_max"] },
        "language": { "type": "string", "description": "Language code such as en, he, ar, es, or auto to detect." },
        "translate_to": { "type": "string", "description": "Language code to translate into (Oracle models only)." },
        "output_format": { "type": "string", "enum": ["json", "txt", "srt"] },
        "word_timestamps": { "type": "boolean" },
        "include_diarization": { "type": "boolean" },
        "num_speakers": { "type": "integer", "minimum": 1 }
      },
      "required": ["path_or_url"]
    }
  }
}

Both formats use the exact same JSON Schema for their parameters. The model enum only lists current models, leaving out the legacy core and edge options.

The function that runs the tool

This executor handles uploading local files, starting the transcription job, polling until it finishes, and returning a short response. When requesting JSON, it writes the full output to a local file and only returns numbered lines to the agent, since the raw word list from a long recording is much larger than the model needs.

kolwrite.py

# kolwrite.py — three small functions around the KolWrite REST API
import os, time, requests

API = "https://app.kolwrite.com"
HEADERS = {"x-api-key": os.environ["KOLWRITE_API_KEY"]}


def upload(path):
    """Upload a local file (up to 100 MiB) and return a URL the API can read."""
    name = os.path.basename(path)
    form = requests.post(f"{API}/upload", headers=HEADERS, json={"filename": name}, timeout=30)
    form.raise_for_status()
    form = form.json()
    with open(path, "rb") as media:
        # No x-api-key on this request: it goes to the storage address.
        sent = requests.post(form["uploadUrl"], data=form["uploadFields"],
                             files={"file": (name, media)}, timeout=600)
    sent.raise_for_status()
    return form["fileUrl"]


def transcribe(audio_url, timeout_s=3600, **options):
    """Start a job, wait for it, return the result: a dict for json, a string for srt and txt."""
    body = {"audio_url": audio_url, "model": "oracle", **options}
    job = requests.post(f"{API}/transcribe", headers=HEADERS, json=body, timeout=30)
    job.raise_for_status()
    job_id = job.json()["jobId"]
    deadline = time.time() + timeout_s
    while time.time() < deadline:
        time.sleep(5)
        reply = requests.get(f"{API}/transcribe/{job_id}", headers=HEADERS, timeout=30)
        reply.raise_for_status()
        reply = reply.json()
        if reply["status"] == "COMPLETED":
            return reply["result"]
        if reply["status"] == "FAILED":
            raise RuntimeError(reply.get("error", "transcription failed"))
    raise TimeoutError(f"job {job_id} did not finish in {timeout_s} seconds")

tool.py

# tool.py — run a transcribe_media call for the model and keep the reply small
import json
from kolwrite import upload, transcribe  # the helper from the video-editing guide


def numbered_line(number, segment):
    who = f'{segment["speaker"]}: ' if segment.get("speaker") else ""
    return f'S{number} [{segment["start"]:.1f}-{segment["end"]:.1f}] {who}{segment["text"]}'


def run_transcribe_media(path_or_url, output_format="json", **options):
    url = path_or_url if path_or_url.startswith(("http://", "https://")) else upload(path_or_url)
    result = transcribe(url, output_format=output_format, **options)
    if output_format != "json":
        return result  # srt and txt are already strings
    # Long recordings: save the full JSON and hand the model numbered lines, not 500 KB of words.
    with open("transcript.json", "w", encoding="utf-8") as out:
        json.dump(result, out, ensure_ascii=False)
    lines = [numbered_line(i, s) for i, s in enumerate(result["segments"])]
    return "Full JSON saved to transcript.json (words[] has word times).\n" + "\n".join(lines)

A minimal tool loop (Anthropic Messages API)

# loop.py — Anthropic Messages API tool loop around run_transcribe_media
import anthropic
from tool import run_transcribe_media

client = anthropic.Anthropic()          # reads ANTHROPIC_API_KEY
TOOLS = [...]                           # the transcribe_media definition above
messages = [{"role": "user", "content": "Transcribe interview.mp3 and list the questions asked, with times."}]

while True:
    reply = client.messages.create(model="YOUR_MODEL", max_tokens=2000, tools=TOOLS, messages=messages)
    messages.append({"role": "assistant", "content": reply.content})
    if reply.stop_reason != "tool_use":
        print("".join(block.text for block in reply.content if block.type == "text"))
        break
    results = []
    for block in reply.content:
        if block.type == "tool_use" and block.name == "transcribe_media":
            results.append({"type": "tool_result", "tool_use_id": block.id,
                            "content": run_transcribe_media(**block.input)})
    messages.append({"role": "user", "content": results})

Swap YOUR_MODEL for the model id you use. The loop keeps running until the model stops calling tools. You can use this same executor with any framework that can call a Python function.

The rule for AGENTS.md and CLAUDE.md

Drop this into the project file your coding agent reads so every session uses the right endpoint, key handling, and formats:

## Transcription
For speech-to-text, subtitles and word timings use the KolWrite API (https://app.kolwrite.com).
Before writing code read https://console.kolwrite.com/agent-instructions.md and /llms-full.txt.
- Key: KOLWRITE_API_KEY in the environment, sent as the x-api-key header. Never print it or commit it.
- POST /transcribe {"audio_url", "model": "oracle", "output_format": "json"}, then poll GET /transcribe/{jobId}.
- Local files: POST /upload first and use fileUrl as audio_url. Extract audio from video with ffmpeg to stay under 100 MiB.
- Subtitles: output_format "srt" with "word_timestamps": false (one cue per segment). Formats are json, txt, srt only.
- Always send "model". Ask me before transcribing anything longer than one hour: it costs money.

Cost control for agents

• Transcription is billed per second of audio, starting at US$0.48 per hour on base. Because of that, the rule above makes the agent check with you before running long files.
• Save the JSON file right next to the media file so you can reuse it instead of paying to transcribe the same audio twice.
• Pick the cheapest model that fits what you're doing. Use base if the audio is clear, oracle when you need production quality, and oracle_mini if you also need translation.
• Agents can't sign up on their own, so a human has to set up the account and create the API key. Once you have the key, pass it to the agent as an environment variable.

Questions and answers