Compute¶

A node is a machine you connected to your AutoLab account with autolab serve. It belongs to you, stays connected, and can be attached to any number of climbs. A climb runs nothing until at least one node is attached to it; once one is, the climb's agent runs its commands and the official evaluations there and decides itself how to use the hardware.

Connect this machine¶

curl -fsSL https://app.autolab.ai/install.sh | sh   # installs the autolab CLI
autolab login                                       # browser sign-in (or --token)
autolab serve                                       # connects the machine to your account

serve starts a background daemon and returns once the node is online. The node survives closing the terminal or the SSH session and reconnects on its own after a control-node restart. On Linux with a per-user systemd it runs as a systemd user service that also comes back after a reboot and updates itself to the control node's build while idle; where there is none (cluster login nodes, macOS) it runs as a plain background process instead and tells you so. Re-running serve on a connected machine just reports it.

Manage the background node¶

autolab serve status     # connection state, control-node view, recent log
autolab serve log        # the node's full runner log (-f follows, -n N tails)
autolab serve stop       # disconnect this machine (removes the service)
autolab serve --daemon   # plain background process, no systemd (not back after a reboot)
autolab update           # sync the CLI to the control node's build by hand

Options: --name picks the node name (default: derived from the machine and its GPU allocation, so two allocations on one host become two nodes), and --autolab-home (or AUTOLAB_HOME, default ~/.autolab) sets where everything the node writes lives. Each attached climb gets its own checkout under <AUTOLAB_HOME>/<hostname>/nodes/<node>/projects/; only the download cache is shared.

Slurm clusters¶

Run autolab serve on a login node. The node then represents the whole cluster: the agent submits its own sbatch/srun jobs for anything heavy, and official evaluations normally run under an srun … launcher the agent chooses (partition, GPUs, time limit); the agent may also evaluate on the login node itself when that is the faster path for a light CPU-only evaluation. The node's page lists the cluster's partitions with idle counts and time limits. While an evaluation waits for an allocation, the jobs list and the job page show waiting in the Slurm queue with the Slurm job id; that wait pauses the run's stall and time limits. The agent sees the same queue state and is asked to optimize time to results, cancelling and resubmitting elsewhere when that helps.

Kubernetes clusters¶

A namespace you can create pods in is compute too. From a machine with kubectl pointed at the cluster:

autolab serve --kubernetes autolab --name venice

This deploys a small launcher pod (a service account scoped to pods in that namespace, a persistent volume for code and caches, a secret with your token and one Deployment). The pod installs the CLI and joins your account as a node named venice with a kubernetes badge; it has no GPU of its own. The climb's agent runs each official evaluation as its own GPU pod by choosing

autolab kube run --gpus 1 -- …

as the launcher (add --memory 32Gi, --cpu 8, --deadline 120, --gpus 2 as it sees fit). The job pod mounts the launcher's volume at the same path, so the code, the hill bundle, uv's caches and the report are shared without copying. Two GPUs mean two evaluations can run at once; a third waits Pending and the jobs list shows waiting for a GPU pod with the pod name, with the run's stall and time limits paused until it starts. The node page lists the GPU nodes the namespace can see and how many GPUs are in use, refreshed on every check-in, and the agent gets the same numbers in every step. kubectl get pods / describe / logs work from the agent's shell in the launcher, for that namespace only.

Options: --image (default python:3.12-bookworm, used for the launcher and the job pods; the NVIDIA driver comes from the host, PyTorch wheels bring their own CUDA), --storage 200Gi, --storage-class, --shared-volume (a ReadWriteMany claim, so job pods may run on any node — otherwise they are pinned to the launcher's node so a ReadWriteOnce volume can follow them). autolab kube install is the same with every knob (image, volume size and class, launcher CPU and memory); autolab kube manifest … prints the YAML instead of applying it; autolab kube uninstall --name venice -n autolab removes everything (add --keep-volume to keep the caches).

Attach nodes to climbs¶

The Compute tab lists all your nodes: hardware, what is running on each one right now (per GPU, across all your climbs), which climbs use it, and its runner log. Open a node to attach it to a climb, or attach from the climb's Nodes page with one click. When you start a climb from a hill's Start a climb page, your nodes are listed right on the form — tick the ones it should run on and they are attached the moment the climb goes live. From the CLI:

autolab nodes                                        # your nodes, and which climbs use them
autolab nodes attach <node> --project <owner>/<slug> # inside a workspace --project is implied
autolab nodes detach <node> --project <owner>/<slug>
autolab nodes rm <node>                              # disconnect + forget

Attaching hands the climb's agent a shell as you on that machine, so only your own nodes can be attached. A node attached to nothing goes dormant (it checks in every few minutes); attached nodes stay hot so commands land instantly.

Tell the agent about the machine¶

The agent sees the node's hardware and the current GPU occupancy in every step, and three kinds of notes:

Your notes on the node (Compute page) — facts about the machine: "home dir is capped at 100 GB, keep datasets under /scratch", "GPUs 0-1 belong to another team on weekdays".
Per-climb notes (a climb's Nodes page, or the node's Compute page) — how to share the machine in that climb: "use only GPU 3 here; GPUs 0-2 are used by my other climbs".
The agent's own notes — what it learned about the machine (quotas, env activation snippets, OOM thresholds). Shown on the Compute page; clear them if they go stale.

The agent picks the GPU(s) each evaluation runs on and how many evaluations the climb runs at once (set_parallel_runs; also editable under the climb's Settings → Advanced → Parallel evaluations).

Let AutoLab provide the compute¶

If you would rather not attach a machine at all, or want more than you have, Autolab can rent one for you, billed to your credit. Drive it from the CLI:

autolab compute                          # the catalog: machine ids, specs, price per hour
autolab compute rent h100-80 --max 1     # let this climb rent up to one H100
autolab compute rent cpu-small --max 4   # and up to four small CPU boxes; caps stack
autolab compute stop                     # rent nothing; release live rentals now

Inside a climb's workspace autolab compute also shows that climb's current cap and live rentals; from elsewhere pass --project <owner>/<slug>. Each rent sets one machine's ceiling and leaves the others alone; --max 0 drops a shape.

The same picker is on the web, on the Start a climb form and under a climb's Nodes and Settings pages, as Autolab compute.

The agent reads the evaluation and puts its recommendation first: a CPU-only box for a climb that only calls a hosted model, a big GPU for one that trains at scale, each card saying why and what it costs per hour. Use the − / + buttons to say how many of each machine AutoLab may run at once. Show all 30 machines opens the rest of the catalog, with a search box (try 4090, 80GB, amd) and CPU / GPU / multi-GPU filters.

Nothing is selected by default, so a climb rents nothing until you pick something.

Browsing the catalog

The Compute tab in the sidebar lists every machine AutoLab can rent and what each costs, without picking anything. You choose machines on a climb — that's where the agent knows what's being optimized and can scale them with the work.

Under the cards you get the arithmetic: how many nodes you allowed and what they'd cost per hour with all of them running at once. That is the worst case, and you should expect to pay well under it — the agent starts a node only when there's work for it and hands it back the moment there isn't. Rented hours are billed to your AutoLab credit at 1.2x the underlying provider price; both numbers are on every card.

Your own machines never evict a rental¶

The number you pick per machine is a ceiling, not a fleet: AutoLab rents only what the queue needs beyond what your own attached machines can absorb. A machine already rented stays while there is work for it, even after you attach one of your own; it goes back when the queue empties, when you lower the cap, or when you pause.

Rented nodes are released when you pause¶

This is the point of the picker: you never pay for idle rented compute.

Pause or stop the climb — every rented node is handed back. One that's mid-experiment is released the moment that run finishes, without waiting for you to resume.
The agent pauses itself (stop policy met, spend cap hit, needs your input) — same thing.
Fewer nodes needed than selected — the surplus goes back on its own, and the machine the agent recommended is the last to go.
Turn a machine down or off — takes effect immediately, and releases that machine rather than whichever node happened to be idle.
Delete the climb — everything is released first.

Machines you attached with autolab serve are never touched by any of this, and work they can already absorb is never a reason to rent.

Headless / remote GPU box¶

A remote machine has no browser. Mint a dedicated token and inject it over SSH:

TOK=$(autolab token create --name gpu-box)
ssh gpubox "curl -fsSL https://app.autolab.ai/install.sh | sh && AUTOLAB_TOKEN=$TOK ~/.local/bin/autolab serve"
autolab nodes attach gpu-box --project <owner>/<slug>

Revoke the token later with autolab token revoke gpu-box; the node keeps its own credential, so a laptop logout never disconnects a node.

If a node shows offline¶

Nothing can revive a dead node from the dashboard. On the machine, autolab serve status shows what happened (with the log tail), and autolab serve reconnects it. A node running as a systemd service comes back on its own after a crash or a reboot; a plain background process does not. Runs that were in flight are requeued once the control node sees the node offline.

Rental limits and release retries¶

AutoLab compute uses both your selected rental limits and the climb's maximum parallel runs. Available machines you own count before additional machines are rented. CPU and NVIDIA GPU rentals are supported; AMD/MI300X rental choices are currently unavailable while the ROCm runtime is being validated.

When you pause a climb, idle rentals are released and busy rentals drain after their current work. AutoLab retries a release if the provider is temporarily unavailable. The compute panel keeps showing unresolved rentals until release is confirmed. If deleting a climb reports that compute is still being released, the climb stays paused; retry deletion after cleanup finishes.

A rental keeps the hourly rate quoted when it was provisioned. Charges and billing checkpoints are saved together, so a restart does not charge the same rental interval twice.