Mid-trajectory forks & RL trees

Fork a running rollout, not just a fresh template. At any decision point, branch one live trajectory into N children in single-digit milliseconds, each resuming from the exact same state. This is the primitive tree search needs: explore many continuations from one expensive prefix instead of paying to recreate it.

The replay tax

Tree search over an agentic environment has a hidden, quadratic cost. Most frameworks can't clone a live environment, so to explore a branch at depth d they reset to the start and replay every prior action to get back to where they were. The deeper the search, the more replay. This overhead scales with depth² × branches, while your LLM and search logic sit idle.

REPLAY-BASED SEARCH · re-run the whole prefix for every branch

  branch @ depth 3:   replay s0 → s1 → s2 → try a'
  branch @ depth 6:   replay s0 → s1 → s2 → s3 → s4 → s5 → try a''
  branch @ depth 9:   replay s0 → s1 → s2 → s3 → s4 → s5 → s6 → s7 → s8 → try a'''

  cost ∝ depth² × branches × step_time      // a depth-10, branch-3 task ≈ 300s of pure replay (modeled)

Fork the environment, not the trajectory

Collimate clones the live environment itself. At the decision node, fork the running session N ways; each child continues from the parent's exact instant: files written, processes running, in-memory state, browser tabs and cookies, all intact. Run a different action in each child, score them, keep the winner, discard the losers. No replay, ever.

FORK-BASED SEARCH · branch the live state in place

                                   ┌─► fork · try a'    score 0.31
   s0 ─ s1 ─ s2 ─ ● live node ──┼─► fork · try a''   score 0.82  ◄── keep
                  │                └─► fork · try a'''  score 0.44
                  │
                  └─ continue from the winner, fork again ─► …

  cost ∝ depth × branches × fork_time       // ~ms per branch, not ~seconds

Branch a live session

Open a session, drive it to an interesting state, then fork it. The parent keeps running; the child is an independent, isolated VM that started life as a perfect copy.

POST /v1/sessions/sess_019cf68f.../fork
Authorization: Bearer col_live_...

201 Created
{
  "session_id": "sess_01a2b3c4...",
  "parent_session_id": "sess_019cf68f...",
  "template_id": "webarena",
  "fork_time_ms": 6.8,
  "total_time_ms": 9.9
}

The child is a normal session: drive it with /v1/sessions/{child}/exec, fork it again to go deeper, or DELETE it to prune the branch.

A tree-search loop

Forking turns the search into a tight loop: propose actions, fork once per action, execute, score, keep the best, prune the rest.

def fork_search(agent, task, branching=3, max_depth=15):
    session = open_session(template="webarena")
    for _ in range(max_depth):
        actions = agent.propose_actions(state(session), task, n=branching)

        # fork the live state once per candidate, ~ms each, no replay
        forks = [fork_session(session) for _ in actions]

        scored = []
        for child, action in zip(forks, actions):
            result = agent.act(action, session=child)
            scored.append((child, agent.score(result)))

        best = max(scored, key=lambda x: x[1])
        for child, _ in scored:
            if child is not best[0]:
                delete_session(child)   # prune losing branches
        session = best[0]               # continue from the winner

Why it matters for RL

Related