Core concepts¶

A handful of ideas explain almost everything the CLI and the hub do.

Hills¶

A hill is one task with the scoring frozen: an evaluator (eval.py), a README that states what to build, and the held-out data the score runs on. Autolab uses the open-source hills format, which is why the CLI verbs are autolab hills ….

Two properties make a hill trustworthy. The score always comes from running the evaluator, so the agent never grades its own work. And a hill is identified by the hash of its contents, so once a climb pins a version, that version never shifts underneath it. Change the evaluator and you get a new version with its own hash and its own leaderboard.

Build your own with autolab hills new/check/commit/push, or climb one that already exists.

Climbs¶

A climb is one attempt to top a hill: a starting point, a model, compute, a budget, and stopping rules. Several climbs can share a hill, and the hill's page ranks them on a leaderboard, best per climb or best per person.

Under the hood a climb is a project. The CLI and dashboard still say "project" in a few places, and autolab init creates one; read it as "a climb of a hill." Every climb has a display name ("My Cool Climb") and a namespaced slug (alice/my-cool-climb), and the slug is what commands and URLs use. It is deduped to …-v2 if the name is taken.

The hub: Explore, hills, and lists¶

Explore is where hills live. It has two tabs:

Hills, every published evaluation, searchable and tagged.
Lists, hills someone grouped and wrote up, so a newcomer knows which are worth the time. A list has a cover, a readme, and an ordered set of hills.

You can star a hill or a list, share either with a link (the page renders a preview when pasted), and open anyone's profile at /u/<username> to see what they built and climbed. A platform-wide search (press Ctrl/Cmd+K) jumps to any hill, list, person, or climb.

Forking a hill¶

Found a hill that is close but not quite your task? Fork it. A fork copies the hill's current version into one you own, so you can change the evaluator, the README, or the data and publish your own version:

autolab hills fork alice/nanogpt-10min --name nanogpt-30min

Forks start private. The web Fork button does the same thing. See Build a hill.

Climbing a hill locally¶

Every hill page has a Start a climb menu. Climb on Autolab runs the climb on the platform, on your compute or rented GPUs. Climb locally keeps everything on your machine: your own coding agent, the open-source hills tool, and the hill pulled down with autolab hills pull.

npx skills add autolab-ai/hills                 # the hills skill for your agent
curl -fsSL https://app.autolab.ai/install.sh | sh && autolab login
autolab hills pull alice/nanogpt-10min          # -> ./.autolab/hills/nanogpt-10min

The hill's owner gets the exact frozen hill, private data included, so local hills eval reports carry the official tree hash. Anyone else gets the public files and unofficial local scores. Nothing is billed.

The autonomous loop¶

Once a climb is live (autolab start), an agent drives a loop:

Generate. Propose hypotheses and queue experiments, if idea generation is on.
Code. Write the experiment's code.
Schedule. Run it on an available node.
Analyze. Read the run's metric curves (every run logs to the climb's built-in MLflow tracker) and its logs, and decide what worked. The same curves render live on the job's dashboard page.
Merge. Fold improvements into main; discard the rest.

You steer the loop from the CLI: pause and resume it, toggle idea generation, set the objective and constraints, or queue your own experiments. You watch it in the browser, where the logs, metrics, and the agent's written analysis live. See Drive the agent.

Experiments are commits¶

Autolab treats research like git: an experiment is a commit. When you submit, the CLI snapshots your working tree, commits it, and queues it as one experiment. The agent's experiments are commits too. autolab log is the history and autolab graph draws the branch tree; → marks the experiment you are in.

That is why the workspace behaves like a checkout: autolab diff shows your changes against the experiment you are in, and autolab checkout <id> resets your tree to any experiment's code.

Field locking¶

An experiment is a small bag of fields: a name, an idea or description, a node constraint, and the code. By default every field you set is locked, so the agent fills only the blanks and never overwrites your values. That makes submit predictable: what you typed is what runs.

--soft lets the agent refine the fields you set, revising your code or run command. Use it for a starting point rather than a fixed spec.
--nocode submits an idea only and lets the agent write the code.

You advance into the new experiment only when the code is final, meaning you pushed code and did not pass --soft. With --soft or --nocode you stay in your current experiment. See Run experiments.

Compute¶

The control node never runs experiments. Compute does, and it comes three ways:

A machine you connect with autolab serve. It stays yours and attaches to any climb. A laptop, a workstation, or an 8×H100 box all work.
A Slurm or Kubernetes cluster, served from a login node or a launcher pod.
A GPU Autolab rents for you with autolab compute, billed to your credit at 1.2x the provider price. Rented nodes attach on their own and release when the climb pauses, so you never pay for idle rented compute.

See Compute.

The LLM key¶

The agent needs a model to do its work. Every climb runs on your Autolab Key, platform credits with nothing to configure; top up and track spend under Credits in the dashboard. See Tokens & API keys.

The workspace (`.autolab/`)¶

A linked directory keeps its state in a hidden .autolab/:

.autolab/git is the workspace's version control. It is not a normal .git; the CLI drives it through autolab verbs and hides it, and any nested .git from your own code, from its work tree.
.autolab/config.json records which climb this directory is linked to. If it is lost, the CLI rebuilds it from the git remote.
.autolab/hills/<name> holds hills you scaffold or pull.

You do not edit these by hand; init, clone, submit, checkout, and the hills verbs manage them. Details in Configuration & environment.