Skip to content
Collimate

RL quickstart

collimate-rl is the Python SDK for RL and post-training rollouts. It drives the same fleet the agentic API does, with the ergonomics a trainer wants: a warm sandbox per rollout, deterministic replay, and per-fork egress.

Terminal window
pip install collimate-rl

The SDK is dependency-free (stdlib only). The surfaces you'll touch:

  • connect() — the one entry point. Give it a col_* API key (or set COLLIMATE_API_KEY) and it returns a client for the managed cloud. Everything downstream runs over it unchanged.
  • Sandbox — the in-worker hot path: fork a clone, run the rollout, discard. A context manager.
  • grade / grade_batch — turn a candidate completion (or a whole GRPO group) into a reward in one call.

Template registration and capacity are platform-managed: you bake a template once (see Templates) and the platform keeps it warm for you; there is nothing to size.

Grade one rollout

Section titled “Grade one rollout”

connect() returns a client that speaks the API directly. rollout_once is the one-shot convenience: fork an ephemeral clone, run once, discard.

from collimate_rl import connect, rollout_once


client = connect()   # reads COLLIMATE_API_KEY


r = rollout_once(
    "python",
    code="print(sum(range(1000)))",
    seed="c0ffee…",          # pin the guest CRNG for a reproducible rollout
    client=client,
)
assert r.ok and r.stdout.strip() == "499500"
print(r.seed, r.seed_ack)   # the exact seed used, and the seed-ack verdict

The seed used is always echoed back — even when you pass seed=None and let the platform draw fresh entropy — so any graded rollout is replayable. See Deterministic replay.

A stateful rollout with Sandbox

Section titled “A stateful rollout with Sandbox”

For multi-step episodes, hold a clone across execs. Sandbox is a context manager: enter forks a fresh clone, exit destroys it, even on error.

from collimate_rl import Sandbox, connect


client = connect()   # reads COLLIMATE_API_KEY


with Sandbox("python", client=client) as sb:       # ~ms warm fork
    sb.upload_bytes("/work/task.py", task_source)     # byte-exact file in
    r = sb.exec(command="cd /work && python task.py", timeout_seconds=120)
    if r.ok:
        artifact = sb.download("/work/out.json")       # byte-exact file out
    print("warm hit:", sb.warm_hit)                     # served warm?

Sandbox.reset() swaps in a fresh clone for the next episode (a fork from the pristine template is the cheapest, most reliable reset). Sandbox.attach(client, session_id) re-adopts an existing clone for worker-crash recovery.

Reward grading

Section titled “Reward grading”

The SDK ships grader helpers so a candidate completion becomes a reward in one call:

from collimate_rl import UnitTestReward, grade, extract_code


spec = UnitTestReward(
    template="python",
    tests="assert solve([3,1,2]) == [1,2,3]\n",
)
code = extract_code(model_completion)     # pull code out of an LLM completion
result = grade(code, spec, client=client) # runs in an isolated VM
print(result.reward)                       # 1.0 / 0.0

grade_batch runs a whole GRPO group; PropertyReward does property/fuzz grading with common-random-numbers seeding.

Next steps

Section titled “Next steps”