sagax
Introduction

Getting started

sagax is, at its core, a task execution engine with some fancy bells and whistles. You define a set of tasks representing some unit of work, configure how many times you want to run it, and you're off to the races. It's easy to do this. It's hard to make it nice.

sagax is the result of spending some effort thinking about the right way to architect such things, mainly to serve the purpose of evaluating agents. There's a lot of small details that have to come together for the entire system to work. sagax doesn't claim to know what all those small details are, so every part of the framework is designed to be extensible and pluggable, so if a component doesn't work for you, you can "eject" the core at any time. How?

Architecture

sagax is packaged as a library, not a binary. There are two main components that comprise the execution engine: a planner and a scheduler. Planners take task specifications and turn them into individual invocations that can be run by the scheduler.

This architecture allows you to build on top of our foundations and expand the framework to meet your needs, without forking.

sagax executions follow the hierarchy: task -> job -> trial. A task is the actual code you write which performs the work, the unit of work. A job is a task plus a set of configuration. This distinction exists so tasks and jobs can be one-to-many, and an execution can contain the same tasks with different configurations. A trial is an execution of each job, and this is what is actually run by the scheduler.

Frontends

To make the library usable without having to build your own frontends for it, sagax comes prepackaged with a few, the first of which is a CLI, enabling you to run tasks by running commands:

use sagax::{Engine, TaskExtensions, TaskRegistry};

#[tokio::main]
async fn main() {
  let tasks = TaskRegistry::new();
  let extensions = TaskExtensions::new();
  let engine = Engine::new(tasks, extensions);
  sagax::frontends::cli::run(engine).await?;
}
cargo run -p my-sagax-cli job create coding --config-json '{"image_id":"your-image-id","command":"echo hello"}' --trials 2

Remember sagax is a library. This CLI is yours. If you don't like it, you can build a different frontend.

Sandboxes

sagax is batteries-included. That means we seek to enable you to focus on building your evals, instead of having to build a set of tools first. One part of this is sandboxes. It's now common (and perhaps standard) to run evals in microVMs, and there's a new company providing this service every week. To give you freedom to choose between them and access them in Rust, sagax contains a set of sandbox libraries for each provider. Each one has a sans-IO API, exposing all client functionality as state machines that can be driven by asynchronous runtimes, as well as an ergonomic, native-feeling tokio client.

To make it seamless to swap between providers, sagax also offers a uniform sandbox interface that all the different providers implement. This means it only takes one line to swap sandbox providers, enabling you to do things like using one for local development (such as Docker sandboxes) and another in production.

On this page