Skip to content

Build a hill

A hill is a task with the scoring fixed: an evaluator, a README that states what to build, and the held-out data the score runs on. Build one when no existing hill measures what you care about. Anyone can then climb it, and every climb is scored the same way.

Hills use the open-source hills format, so the CLI verbs wrap the pinned OSS tool. If your coding agent has the Autolab skill, ask it to build the hill and it will follow the same steps.

Scaffold it

autolab hills new my-task        # -> ./.autolab/hills/my-task, registered locally

The scaffold is a working hill that already scores end to end, so hills check is green before you change anything. It contains:

  • hill.yaml, the manifest: the metric and its direction, typed parameters, and the watchdog timeout.
  • eval.py, the evaluator: eval(submission, *, final=False, **params) returns whether the submission passed, the metrics, and details. The climber can read this file, so anything that would reveal the answer belongs under private/.
  • README.md, the contract the climber reads: the task, the submission format, the metric, and the parameters. The first paragraph is the subtitle shown on the hub, so keep it under 200 characters.
  • private/, held-out data, never published in readable form.
  • examples/baseline/, a submission that scores, and tests/, checks that the evaluator scores it as expected.

Start from a bundled example instead of the blank template with -t:

autolab hills new my-task -t circle-packing    # `hills examples` lists them

Write and check

Fill in eval.py, README.md, hill.yaml, and put held-out data in private/. Then validate the whole contract:

autolab hills check my-task      # manifest, evaluator contract, and tests/

check runs the evaluator against examples/baseline/ and your tests, so a report that drifts from the declared metrics fails here, in your own run, before anyone climbs it. Iterate until it is green. The full OSS tool is available as hills for the rest: hills describe, hills eval <dir> -H my-task.

Commit and publish

autolab hills commit my-task -m "first version"   # freeze this version (its hash)
autolab hills push my-task --public               # publish to your control node

The first push creates the hill under your account; later pushes add versions. --public puts it on the Explore tab; leave it off to keep it private while you work. Add --source-repo <git-url> to tell climbs where to start their code from.

The New Hill wizard

Prefer the browser? On Explore, New hill walks the same ground with the agent: it asks what to measure, what is held out, and what a submission may change, builds the evaluator on a connected machine, and runs check before it publishes. You review the result before anything goes live.

Fork an existing hill

To start from someone else's hill and change the task, fork it:

autolab hills fork alice/nanogpt-10min --name nanogpt-30min

A fork copies the hill's current version, including private data if you can read it, into a hill you own, and starts private. Pull it down to edit it, then commit and push a new version:

autolab hills pull you/nanogpt-30min      # -> ./.autolab/hills/nanogpt-30min

Manage a published hill

autolab hills                                   # the hills on your control node
autolab hills settings my-task --public         # or --private
autolab hills settings my-task --tag ml --tag systems   # replace tags
autolab hills settings my-task --deadline 2026-09-15     # a competition deadline

Run autolab hills settings my-task with no flags to see the current settings. Delete a hill from its dashboard Settings page; the CLI refuses to delete one a climb is still using.

Verify before you publish

A hill is a promise that the score means something, so hold it to that:

  • Read your own eval.py as an adversary would. Can a submission score well without doing the task, by reading private/, by monkeypatching, or by writing files the evaluator later reads? Keep the untrusted submission and the scorer in separate processes when the task warrants it.
  • Confirm the baseline scores what you claim, and that a broken submission is rejected with a clear message, not a traceback.
  • State the metric and its direction in the README exactly as hill.yaml declares them.

The full authoring playbook, including the report shape and the checklist, ships with the CLI skill under references/hills.md and at the hills repository.

Next: Explore, hills & lists to publish it into a list, or Climb a hill to climb it yourself.