REST API

The Maniac API provides openai compatible inference endpoints for interacting with both frontier models and your custom models. This reference details the available endpoints.

List evaluation runs

get

List evaluation runs for the authenticated project. Optionally filter by container and status.

Authorizations

AuthorizationstringRequired

API key in Authorization header using Bearer .

Query parameters

containerstring · nullableOptional

Container ID or label to filter by.

statusstring · nullableOptional

Filter by run status (e.g. 'running', 'completed', 'error').

limitinteger · min: 1 · max: 100OptionalDefault: 20

offsetintegerOptionalDefault: 0

Responses

200

Successful Response

application/json

objectconst: listOptional

Object type identifier.

Default: list

totalintegerRequired

Total number of items available for this resource.

400

Bad Request

application/json

401

Unauthorized

application/json

403

Forbidden

application/json

422

Validation Error

application/json

429

Too Many Requests

application/json

500

Internal Server Error

application/json

501

Not Implemented

application/json

503

Upstream Unavailable

application/json

get/v1/evaluation/runs

GET /v1/evaluation/runs HTTP/1.1
Host: platform.maniac.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

{
  "object": "list",
  "data": [
    {
      "created_at": "text",
      "finished_at": "text",
      "error_at": "text",
      "status": "text",
      "error": null,
      "object": "evaluation.run",
      "id": "text",
      "process_id": "text",
      "evaluators": [
        "text"
      ],
      "container": "text",
      "dataset_id": "text",
      "sample": {
        "range": "0:100",
        "type": "text",
        "dataset": "file_abc123"
      },
      "ground_truth": {
        "range": "0:100",
        "type": "text",
        "dataset": "file_abc123"
      },
      "baseline": {
        "range": "0:100",
        "type": "text",
        "dataset": "file_abc123"
      },
      "results": {
        "overall": {
          "avg_score": 1,
          "avg_accuracy": 1,
          "ANY_ADDITIONAL_PROPERTY": "anything"
        },
        "metadata": {
          "ANY_ADDITIONAL_PROPERTY": "anything"
        },
        "per_model": [
          {
            "model": {
              "ANY_ADDITIONAL_PROPERTY": "anything"
            },
            "per_eval": {
              "ANY_ADDITIONAL_PROPERTY": {
                "accuracy": 1,
                "avg_score": 1,
                "num_total": 1,
                "num_errors": 1,
                "num_failed": 1,
                "num_passed": 1,
                "num_scored": 1,
                "num_missing": 1,
                "avg_accuracy": 1,
                "break_reason": "text",
                "expected_count": 1,
                "ANY_ADDITIONAL_PROPERTY": "anything"
              }
            },
            "avg_score": 1,
            "avg_accuracy": 1,
            "launch_index": 1,
            "launch_call_id": "text",
            "ANY_ADDITIONAL_PROPERTY": "anything"
          }
        ],
        "launch_count": 1,
        "ANY_ADDITIONAL_PROPERTY": "anything"
      },
      "metrics": {
        "ANY_ADDITIONAL_PROPERTY": "anything"
      },
      "config": {
        "ANY_ADDITIONAL_PROPERTY": "anything"
      },
      "spend": 1,
      "metadata": {
        "ANY_ADDITIONAL_PROPERTY": "anything"
      }
    }
  ],
  "total": 0
}

Create an evaluation run

post

Launch an evaluation run. Validates access to the specified container, evaluators, data sources, and models, then dispatches the run through the backend gateway interface.

Authorizations

AuthorizationstringRequired

API key in Authorization header using Bearer .

Body

Request body for creating an evaluation run.

Each side of the evaluation (sample and ground_truth) is described by a single data-source object whose type discriminator determines how data is obtained:

• "dataset" — pull from a dataset.
• "container" — pull from the container's task logs.
• "generate" — generate completions using one or more models.

Both fields are optional, but at least one must be provided. When a side is omitted it defaults to the top-level container's task logs. At least one resolved side must not be type='generate' so there is seed input to evaluate against.

containerstringRequired

Container id or label.

evaluatorsstring[] · min: 1Required

Evaluator ids or labels.

sampleone of · nullableOptional

Sample-side data source. Omit to default to the container's task logs. Use type='dataset' to pull from a dataset, type='container' to pull from task logs, or type='generate' to generate completions with the specified models.

or

or

ground_truthone of · nullableOptional

Ground-truth-side data source. Omit to default to the container's task logs. Use type='dataset' to pull from a dataset, type='container' to pull from task logs, or type='generate' to generate completions with a model.

or

or

baselineone of · nullableOptional

Baseline-side data source for pairwise evaluation. Use type='dataset' to pull from a dataset, type='container' to pull from task logs, or type='generate' to generate completions with a model.

or

or

environmentstring · nullableOptional

Execution environment name (maps to Modal app suffix).

Default: main

Responses

201

Successful Response

application/json

Response model for an evaluation run.

created_atstringRequired

finished_atstring · nullableOptional

error_atstring · nullableOptional

statusstringRequired

errorany · nullableOptional

objectconst: evaluation.runOptional

Object type.

Default: evaluation.run

idstringRequired

Evaluation run id (run group id).

process_idstring · nullableOptional

Process id for lifecycle tracking.

evaluatorsstring[] · nullableOptional

Evaluator ids used in this run.

containerstring · nullableOptional

Container id.

dataset_idstring · nullableOptional

Dataset id (if a dataset was used).

sampleone of · nullableOptional

Resolved sample-side data source.

or

or

ground_truthone of · nullableOptional

Resolved ground-truth-side data source.

or

or

baselineone of · nullableOptional

Resolved baseline-side data source for pairwise evaluation.

or

or

spendnumber · nullableOptional

Estimated spend.

400

Bad Request

application/json

401

Unauthorized

application/json

403

Forbidden

application/json

404

Not Found

application/json

409

Conflict

application/json

422

Validation Error

application/json

429

Too Many Requests

application/json

500

Internal Server Error

application/json

501

Not Implemented

application/json

503

Upstream Unavailable

application/json

post/v1/evaluation/runs

POST /v1/evaluation/runs HTTP/1.1
Host: platform.maniac.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Content-Type: application/json
Accept: */*
Content-Length: 316

{
  "container": "text",
  "evaluators": [
    "text"
  ],
  "sample": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "ground_truth": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "baseline": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "metadata": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "environment": "main"
}

{
  "created_at": "text",
  "finished_at": "text",
  "error_at": "text",
  "status": "text",
  "error": null,
  "object": "evaluation.run",
  "id": "text",
  "process_id": "text",
  "evaluators": [
    "text"
  ],
  "container": "text",
  "dataset_id": "text",
  "sample": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "ground_truth": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "baseline": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "results": {
    "overall": {
      "avg_score": 1,
      "avg_accuracy": 1,
      "ANY_ADDITIONAL_PROPERTY": "anything"
    },
    "metadata": {
      "ANY_ADDITIONAL_PROPERTY": "anything"
    },
    "per_model": [
      {
        "model": {
          "ANY_ADDITIONAL_PROPERTY": "anything"
        },
        "per_eval": {
          "ANY_ADDITIONAL_PROPERTY": {
            "accuracy": 1,
            "avg_score": 1,
            "num_total": 1,
            "num_errors": 1,
            "num_failed": 1,
            "num_passed": 1,
            "num_scored": 1,
            "num_missing": 1,
            "avg_accuracy": 1,
            "break_reason": "text",
            "expected_count": 1,
            "ANY_ADDITIONAL_PROPERTY": "anything"
          }
        },
        "avg_score": 1,
        "avg_accuracy": 1,
        "launch_index": 1,
        "launch_call_id": "text",
        "ANY_ADDITIONAL_PROPERTY": "anything"
      }
    ],
    "launch_count": 1,
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "metrics": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "config": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "spend": 1,
  "metadata": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  }
}

Get an evaluation run

get

Retrieve a single evaluation run by ID within the authenticated project.

Authorizations

AuthorizationstringRequired

API key in Authorization header using Bearer .

Path parameters

run_idstringRequired

Responses

200

Successful Response

application/json

Response model for an evaluation run.

created_atstringRequired

finished_atstring · nullableOptional

error_atstring · nullableOptional

statusstringRequired

errorany · nullableOptional

objectconst: evaluation.runOptional

Object type.

Default: evaluation.run

idstringRequired

Evaluation run id (run group id).

process_idstring · nullableOptional

Process id for lifecycle tracking.

evaluatorsstring[] · nullableOptional

Evaluator ids used in this run.

containerstring · nullableOptional

Container id.

dataset_idstring · nullableOptional

Dataset id (if a dataset was used).

sampleone of · nullableOptional

Resolved sample-side data source.

or

or

ground_truthone of · nullableOptional

Resolved ground-truth-side data source.

or

or

baselineone of · nullableOptional

Resolved baseline-side data source for pairwise evaluation.

or

or

spendnumber · nullableOptional

Estimated spend.

400

Bad Request

application/json

401

Unauthorized

application/json

403

Forbidden

application/json

404

Not Found

application/json

409

Conflict

application/json

422

Validation Error

application/json

429

Too Many Requests

application/json

500

Internal Server Error

application/json

501

Not Implemented

application/json

503

Upstream Unavailable

application/json

get/v1/evaluation/runs/{run_id}

GET /v1/evaluation/runs/{run_id} HTTP/1.1
Host: platform.maniac.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

{
  "created_at": "text",
  "finished_at": "text",
  "error_at": "text",
  "status": "text",
  "error": null,
  "object": "evaluation.run",
  "id": "text",
  "process_id": "text",
  "evaluators": [
    "text"
  ],
  "container": "text",
  "dataset_id": "text",
  "sample": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "ground_truth": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "baseline": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "results": {
    "overall": {
      "avg_score": 1,
      "avg_accuracy": 1,
      "ANY_ADDITIONAL_PROPERTY": "anything"
    },
    "metadata": {
      "ANY_ADDITIONAL_PROPERTY": "anything"
    },
    "per_model": [
      {
        "model": {
          "ANY_ADDITIONAL_PROPERTY": "anything"
        },
        "per_eval": {
          "ANY_ADDITIONAL_PROPERTY": {
            "accuracy": 1,
            "avg_score": 1,
            "num_total": 1,
            "num_errors": 1,
            "num_failed": 1,
            "num_passed": 1,
            "num_scored": 1,
            "num_missing": 1,
            "avg_accuracy": 1,
            "break_reason": "text",
            "expected_count": 1,
            "ANY_ADDITIONAL_PROPERTY": "anything"
          }
        },
        "avg_score": 1,
        "avg_accuracy": 1,
        "launch_index": 1,
        "launch_call_id": "text",
        "ANY_ADDITIONAL_PROPERTY": "anything"
      }
    ],
    "launch_count": 1,
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "metrics": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "config": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "spend": 1,
  "metadata": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  }
}

Healthz

get

Health check endpoint for load balancers and uptime monitors.

Responses

200

Successful Response

application/json

Health check response.

okbooleanRequired

get/healthz

GET /healthz HTTP/1.1
Host: platform.maniac.ai
Accept: */*

200

Successful Response

{
  "ok": true
}