Skip to content
Collimate

Cookbook: live model server

04 / Live service

For tool-use and agent RL, the environment is often a server the agent calls: a small inference endpoint, a retrieval service, a mock third-party API, or a tool gateway. Its startup — loading weights or an index, warming a framework — is slow, and a rollout should not pay it. Bake a live-service ready state: start the server at bake time and leave it running, so every fork wakes with it already listening and a rollout just hits it.

The idea

Section titled “The idea”
bake:  boot guest → start server → block until GET /health is 2xx → SNAPSHOT → publish
fork:  restore snapshot → server is already serving → rollout hits it

The warm command starts your server detached, then blocks until GET /health returns a 2xx — the real "loaded and serving" signal, not merely that the process started. See the warm-command contract.

The warm script

Section titled “The warm script”

/usr/local/bin/collimate-warm launches your server detached and polls its health endpoint until it is green. Swap the launch line for your own server command — the readiness shape stays the same:

#!/bin/sh
set -e


ip link set lo up 2>/dev/null || true


# 1. Launch YOUR server DETACHED so it outlives this script and lands in the
#    snapshot. Replace this line with your real command — e.g. a FastAPI app
#    under uvicorn, a vLLM/TGI endpoint, or a tool gateway. Bind localhost.
setsid python3 -m your_server --host 127.0.0.1 --port 8000 \
  >/var/log/server.log 2>&1 < /dev/null &


# 2. Block until GET /health returns 2xx — the real "loaded and serving" signal,
#    so the snapshot captures a server that answers, not one still loading weights.
for i in $(seq 1 120); do
  code=$(python3 -c "import urllib.request as u; \
    print(u.urlopen('http://127.0.0.1:8000/health', timeout=1).status)" 2>/dev/null || echo 000)
  if [ "$code" -ge 200 ] && [ "$code" -lt 300 ] 2>/dev/null; then
    echo server-ready; exit 0
  fi
  sleep 1
done
echo server-timeout >&2; exit 1

Bake it:

from collimate_rl import Ready, Template, connect


client = connect()   # a col_pro_* key via COLLIMATE_API_KEY


handle = Template.bake(
    client,
    name="tool-server",
    image="ghcr.io/you/tool-server@sha256:…",          # pin by digest
    ready=Ready(cmd="/usr/local/bin/collimate-warm"),  # launch detached → block → exit 0
    width=64,
)
handle.wait(timeout=600)

Run the warm script at build time as a self-test so a server that never turns /health green is a red build, not a bad bake.

Using it from the SDK

Section titled “Using it from the SDK”

A rollout attaches to the live server — the weights/index load is already paid — and the agent hits it as its tool or model endpoint:

from collimate_rl import connect


client = connect(api_key="col_...")


# The rollout body runs inside the sandbox. It calls the server the ready-state
# snapshot is already running — no per-fork server startup, no weight load.
ROLLOUT = r"""
import json, urllib.request


def call_tool(payload):
    req = urllib.request.Request(
        "http://127.0.0.1:8000/v1/tool",
        data=json.dumps(payload).encode(),
        headers={"content-type": "application/json"},
    )
    return json.load(urllib.request.urlopen(req, timeout=30))


# One tool step of an agent episode — grade the result downstream.
print(call_tool({"query": "capital of France"}))
"""


sb = client.create_sandbox("tool-server")
result = client.exec(sb["id"], commands=[["python3", "-c", ROLLOUT]], timeout_seconds=60)
print(result["stdout"])
client.delete_sandbox(sb["id"])

Fan a warmed parent out across many rollouts with fork — every child inherits the same live server with its weights or index already resident:

# count is the per-call fan-out, capped by tier (Pro 32, Ent 64); call again for more.
children = client.fork(sb["id"], count=32)["children"]
for child in children:
    client.exec(child["id"], commands=[["python3", "-c", ROLLOUT]], timeout_seconds=60)

Reach it from outside

Section titled “Reach it from outside”

The recipe above drives the server in-guest — the rollout runs inside the sandbox and hits 127.0.0.1:8000. To call it from an HTTP client on your machine instead — the connect-from-outside access model — make two bake-time changes and forward the port locally:

  • Bind all interfaces. The warm script above launches the server on --host 127.0.0.1; change that to --host 0.0.0.0 (e.g. uvicorn app:app --host 0.0.0.0 --port 8000) so the per-fork NIC reaches it. A server bound to loopback is in-guest only.
  • Expose the port. Bake with ready=Ready(cmd="/usr/local/bin/collimate-warm", ports=[8000]) — ports gives the fork a guest NIC with 8000 exposed.

Then open a session, forward the port, and hit it like any local endpoint:

import urllib.request
from collimate_rl import connect


client = connect(api_key="col_...")


with client.session("tool-server") as sb:          # a live Session, carries the mint methods
    with sb.port_forward(8000) as fwd:
        # fwd.url == http://127.0.0.1:<local> — the tunnel carries raw HTTP
        print(urllib.request.urlopen(fwd.url + "/health", timeout=10).read())
# leaving the `with` deletes the sandbox

Next steps

Section titled “Next steps”