Evaluation

Evaluation and evaluator endpoints.

List evaluators

get

List evaluators for the authenticated project, optionally filtered by container.

Authorizations

ApiKeyAuth

AuthorizationstringRequired

API key in Authorization header using Bearer .

Query parameters

containerany ofOptional

stringOptional

or

nullOptional

limitinteger · min: 1 · max: 100OptionalDefault: 20

offsetintegerOptionalDefault: 0

Responses

200

Successful Response

application/json

objectconst: listOptional

Object type identifier.

Default: list

totalintegerRequired

Total number of items available for this resource.

400

Bad Request

application/json

401

Unauthorized

application/json

403

Forbidden

application/json

422

Validation Error

application/json

429

Too Many Requests

application/json

500

Internal Server Error

application/json

501

Not Implemented

application/json

503

Upstream Unavailable

application/json

get

/v1/evaluators

HTTPcURLJavaScriptPython

GET /v1/evaluators HTTP/1.1
Host: platform.maniac.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

{
  "object": "list",
  "data": [
    {
      "object": "evaluator.judge",
      "project_id": "text",
      "container_id": "text",
      "id": "text",
      "name": "text",
      "description": "text",
      "created_at": "text",
      "metadata": {
        "ANY_ADDITIONAL_PROPERTY": "anything"
      },
      "pass_threshold": 1,
      "mode": "pointwise",
      "api": "text",
      "container": "text",
      "model": "text",
      "prompt": "text",
      "inference_parameters": {
        "ANY_ADDITIONAL_PROPERTY": "anything"
      }
    }
  ],
  "total": 0
}

Create an evaluator

post

Create a new evaluator scoped to a project or container.

Authorizations

ApiKeyAuth

AuthorizationstringRequired

API key in Authorization header using Bearer .

Body

application/json

containerany ofOptional

Container label.

stringOptional

or

nullOptional

typestring · enumRequired

Evaluator type.

Possible values:

modelany ofOptional

Judge model slug.

stringOptional

or

nullOptional

promptany ofOptional

Judge prompt.

stringOptional

or

nullOptional

sourceany ofOptional

Code evaluator source.

stringOptional

or

nullOptional

requirementsany ofOptional

Optional requirements list.

string[]Optional

or

nullOptional

pass_thresholdany ofOptional

Optional pass threshold.

numberOptional

or

nullOptional

modestring · enumOptional

Evaluation mode.

Default: pointwisePossible values:

metadataany ofOptional

Optional metadata.

or

nullOptional

nameany ofOptional

Evaluator name.

stringOptional

or

nullOptional

descriptionany ofOptional

Evaluator description.

stringOptional

or

nullOptional

apiany ofOptional

API type.

stringOptional

or

nullOptional

inference_parametersany ofOptional

Optional inference parameters for judge evaluators (e.g. response_format, temperature).

or

nullOptional

Responses

201

Successful Response

application/json

400

Bad Request

application/json

401

Unauthorized

application/json

403

Forbidden

application/json

404

Not Found

application/json

409

Conflict

application/json

422

Validation Error

application/json

429

Too Many Requests

application/json

500

Internal Server Error

application/json

501

Not Implemented

application/json

503

Upstream Unavailable

application/json

post

/v1/evaluators

HTTPcURLJavaScriptPython

POST /v1/evaluators HTTP/1.1
Host: platform.maniac.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Content-Type: application/json
Accept: */*
Content-Length: 304

{
  "container": "text",
  "type": "judge",
  "model": "text",
  "prompt": "text",
  "source": "text",
  "requirements": [
    "text"
  ],
  "pass_threshold": 1,
  "mode": "pointwise",
  "metadata": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "name": "text",
  "description": "text",
  "api": "text",
  "inference_parameters": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  }
}

{
  "object": "evaluator.judge",
  "project_id": "text",
  "container_id": "text",
  "id": "text",
  "name": "text",
  "description": "text",
  "created_at": "text",
  "metadata": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "pass_threshold": 1,
  "mode": "pointwise",
  "api": "text",
  "container": "text",
  "model": "text",
  "prompt": "text",
  "inference_parameters": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  }
}

Get an evaluator

get

Fetch a single evaluator by id or name within the authenticated project.

Authorizations

ApiKeyAuth

AuthorizationstringRequired

API key in Authorization header using Bearer .

Path parameters

evaluatorstringRequired

Responses

200

Successful Response

application/json

or

400

Bad Request

application/json

401

Unauthorized

application/json

403

Forbidden

application/json

404

Not Found

application/json

409

Conflict

application/json

422

Validation Error

application/json

429

Too Many Requests

application/json

500

Internal Server Error

application/json

501

Not Implemented

application/json

503

Upstream Unavailable

application/json

get

/v1/evaluators/{evaluator}

HTTPcURLJavaScriptPython

GET /v1/evaluators/{evaluator} HTTP/1.1
Host: platform.maniac.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

{
  "object": "evaluator.judge",
  "project_id": "text",
  "container_id": "text",
  "id": "text",
  "name": "text",
  "description": "text",
  "created_at": "text",
  "metadata": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "pass_threshold": 1,
  "mode": "pointwise",
  "api": "text",
  "container": "text",
  "model": "text",
  "prompt": "text",
  "inference_parameters": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  }
}

Update an evaluator

patch

Update an existing evaluator by id or name.

Authorizations

ApiKeyAuth

AuthorizationstringRequired

API key in Authorization header using Bearer .

Path parameters

evaluatorstringRequired

Body

application/json

Patchable fields for an evaluator.

containerany ofOptional

Container label.

stringOptional

or

nullOptional

modelany ofOptional

Judge model slug.

stringOptional

or

nullOptional

promptany ofOptional

Judge prompt.

stringOptional

or

nullOptional

sourceany ofOptional

Code evaluator source.

stringOptional

or

nullOptional

requirementsany ofOptional

Optional requirements list.

string[]Optional

or

nullOptional

pass_thresholdany ofOptional

Optional pass threshold.

numberOptional

or

nullOptional

modestring · enum · nullableOptional

Evaluation mode.

Possible values:

metadataany ofOptional

Optional metadata.

or

nullOptional

nameany ofOptional

Evaluator name.

stringOptional

or

nullOptional

descriptionany ofOptional

Evaluator description.

stringOptional

or

nullOptional

apiany ofOptional

API type.

stringOptional

or

nullOptional

inference_parametersany ofOptional

Optional inference parameters for judge evaluators (e.g. response_format, temperature).

or

nullOptional

Responses

200

Successful Response

application/json

or

400

Bad Request

application/json

401

Unauthorized

application/json

403

Forbidden

application/json

404

Not Found

application/json

409

Conflict

application/json

422

Validation Error

application/json

429

Too Many Requests

application/json

500

Internal Server Error

application/json

501

Not Implemented

application/json

503

Upstream Unavailable

application/json

patch

/v1/evaluators/{evaluator}

HTTPcURLJavaScriptPython

PATCH /v1/evaluators/{evaluator} HTTP/1.1
Host: platform.maniac.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Content-Type: application/json
Accept: */*
Content-Length: 289

{
  "container": "text",
  "model": "text",
  "prompt": "text",
  "source": "text",
  "requirements": [
    "text"
  ],
  "pass_threshold": 1,
  "mode": "pointwise",
  "metadata": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "name": "text",
  "description": "text",
  "api": "text",
  "inference_parameters": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  }
}

{
  "object": "evaluator.judge",
  "project_id": "text",
  "container_id": "text",
  "id": "text",
  "name": "text",
  "description": "text",
  "created_at": "text",
  "metadata": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "pass_threshold": 1,
  "mode": "pointwise",
  "api": "text",
  "container": "text",
  "model": "text",
  "prompt": "text",
  "inference_parameters": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  }
}

Delete an evaluator

delete

Delete an evaluator by id.

Authorizations

ApiKeyAuth

AuthorizationstringRequired

API key in Authorization header using Bearer .

Path parameters

evaluator_idstringRequired

Responses

200

Successful Response

application/json

objectconst: evaluatorOptional

Object type.

Default: evaluator

idstringRequired

Evaluator id.

deletedconst: Optional

Deletion status.

Default: true

400

Bad Request

application/json

401

Unauthorized

application/json

403

Forbidden

application/json

404

Not Found

application/json

409

Conflict

application/json

422

Validation Error

application/json

429

Too Many Requests

application/json

500

Internal Server Error

application/json

501

Not Implemented

application/json

503

Upstream Unavailable

application/json

delete

/v1/evaluators/{evaluator_id}

HTTPcURLJavaScriptPython

DELETE /v1/evaluators/{evaluator_id} HTTP/1.1
Host: platform.maniac.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

{
  "object": "evaluator",
  "id": "text",
  "deleted": true
}

List evaluation runs

get

List evaluation runs for the authenticated project. Optionally filter by container and status.

Authorizations

ApiKeyAuth

AuthorizationstringRequired

API key in Authorization header using Bearer .

Query parameters

containerany ofOptional

Container ID or label to filter by.

stringOptional

or

nullOptional

statusany ofOptional

Filter by run status (e.g. 'running', 'completed', 'error').

stringOptional

or

nullOptional

limitinteger · min: 1 · max: 100OptionalDefault: 20

offsetintegerOptionalDefault: 0

Responses

200

Successful Response

application/json

objectconst: listOptional

Object type identifier.

Default: list

totalintegerRequired

Total number of items available for this resource.

400

Bad Request

application/json

401

Unauthorized

application/json

403

Forbidden

application/json

422

Validation Error

application/json

429

Too Many Requests

application/json

500

Internal Server Error

application/json

501

Not Implemented

application/json

503

Upstream Unavailable

application/json

get

/v1/evaluation/runs

HTTPcURLJavaScriptPython

GET /v1/evaluation/runs HTTP/1.1
Host: platform.maniac.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

{
  "object": "list",
  "data": [
    {
      "created_at": "text",
      "finished_at": "text",
      "error_at": "text",
      "status": "text",
      "error": null,
      "object": "evaluation.run",
      "id": "text",
      "process_id": "text",
      "evaluators": [
        "text"
      ],
      "container": "text",
      "dataset_id": "text",
      "sample": {
        "range": "0:100",
        "type": "text",
        "dataset": "file_abc123"
      },
      "ground_truth": {
        "range": "0:100",
        "type": "text",
        "dataset": "file_abc123"
      },
      "results": {
        "overall": {
          "avg_score": 1,
          "avg_accuracy": 1,
          "ANY_ADDITIONAL_PROPERTY": "anything"
        },
        "metadata": {
          "ANY_ADDITIONAL_PROPERTY": "anything"
        },
        "per_model": [
          {
            "model": {
              "ANY_ADDITIONAL_PROPERTY": "anything"
            },
            "per_eval": {
              "ANY_ADDITIONAL_PROPERTY": {
                "accuracy": 1,
                "avg_score": 1,
                "num_total": 1,
                "num_errors": 1,
                "num_failed": 1,
                "num_passed": 1,
                "num_scored": 1,
                "num_missing": 1,
                "avg_accuracy": 1,
                "break_reason": "text",
                "expected_count": 1,
                "ANY_ADDITIONAL_PROPERTY": "anything"
              }
            },
            "avg_score": 1,
            "avg_accuracy": 1,
            "launch_index": 1,
            "launch_call_id": "text",
            "ANY_ADDITIONAL_PROPERTY": "anything"
          }
        ],
        "launch_count": 1,
        "ANY_ADDITIONAL_PROPERTY": "anything"
      },
      "metrics": {
        "ANY_ADDITIONAL_PROPERTY": "anything"
      },
      "config": {
        "ANY_ADDITIONAL_PROPERTY": "anything"
      },
      "spend": 1,
      "metadata": {
        "ANY_ADDITIONAL_PROPERTY": "anything"
      }
    }
  ],
  "total": 0
}

Create an evaluation run

post

Launch a standalone evaluation run. Validates access to the specified container, evaluators, data sources, and models, then dispatches the evaluation to the Modal backend.

Authorizations

ApiKeyAuth

AuthorizationstringRequired

API key in Authorization header using Bearer .

Body

application/json

Request body for creating an evaluation run.

Each side of the evaluation (sample and ground_truth) is described by a single data-source object whose type discriminator determines how data is obtained:

• "dataset" — pull from a dataset.
• "container" — pull from the container's task logs.
• "generate" — generate completions using one or more models.

Both fields are optional, but at least one must be provided. When a side is omitted it defaults to the top-level container's task logs. At least one resolved side must not be type='generate' so there is seed input to evaluate against.

containerstringRequired

Container id or label.

evaluatorsstring[] · min: 1Required

Evaluator ids or labels.

sampleany ofOptional

Sample-side data source. Omit to default to the container's task logs. Use type='dataset' to pull from a dataset, type='container' to pull from task logs, or type='generate' to generate completions with the specified models.

anyOptional

or

or

or

nullOptional

ground_truthany ofOptional

Ground-truth-side data source. Omit to default to the container's task logs. Use type='dataset' to pull from a dataset, type='container' to pull from task logs, or type='generate' to generate completions with a model.

anyOptional

or

or

or

nullOptional

metadataany ofOptional

Optional metadata.

or

nullOptional

environmentany ofOptional

Execution environment name (maps to Modal app suffix).

Default: main

stringOptional

or

nullOptional

Responses

201

Successful Response

application/json

400

Bad Request

application/json

401

Unauthorized

application/json

403

Forbidden

application/json

404

Not Found

application/json

409

Conflict

application/json

422

Validation Error

application/json

429

Too Many Requests

application/json

500

Internal Server Error

application/json

501

Not Implemented

application/json

503

Upstream Unavailable

application/json

post

/v1/evaluation/runs

HTTPcURLJavaScriptPython

POST /v1/evaluation/runs HTTP/1.1
Host: platform.maniac.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Content-Type: application/json
Accept: */*
Content-Length: 249

{
  "container": "text",
  "evaluators": [
    "text"
  ],
  "sample": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "ground_truth": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "metadata": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "environment": "main"
}

{
  "created_at": "text",
  "finished_at": "text",
  "error_at": "text",
  "status": "text",
  "error": null,
  "object": "evaluation.run",
  "id": "text",
  "process_id": "text",
  "evaluators": [
    "text"
  ],
  "container": "text",
  "dataset_id": "text",
  "sample": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "ground_truth": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "results": {
    "overall": {
      "avg_score": 1,
      "avg_accuracy": 1,
      "ANY_ADDITIONAL_PROPERTY": "anything"
    },
    "metadata": {
      "ANY_ADDITIONAL_PROPERTY": "anything"
    },
    "per_model": [
      {
        "model": {
          "ANY_ADDITIONAL_PROPERTY": "anything"
        },
        "per_eval": {
          "ANY_ADDITIONAL_PROPERTY": {
            "accuracy": 1,
            "avg_score": 1,
            "num_total": 1,
            "num_errors": 1,
            "num_failed": 1,
            "num_passed": 1,
            "num_scored": 1,
            "num_missing": 1,
            "avg_accuracy": 1,
            "break_reason": "text",
            "expected_count": 1,
            "ANY_ADDITIONAL_PROPERTY": "anything"
          }
        },
        "avg_score": 1,
        "avg_accuracy": 1,
        "launch_index": 1,
        "launch_call_id": "text",
        "ANY_ADDITIONAL_PROPERTY": "anything"
      }
    ],
    "launch_count": 1,
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "metrics": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "config": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "spend": 1,
  "metadata": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  }
}

Get an evaluation run

get

Retrieve a single evaluation run by ID within the authenticated project.

Authorizations

ApiKeyAuth

AuthorizationstringRequired

API key in Authorization header using Bearer .

Path parameters

run_idstringRequired

Responses

200

Successful Response

application/json

Response model for an evaluation run.

created_atstringRequired

finished_atany ofOptional

stringOptional

or

nullOptional

error_atany ofOptional

stringOptional

or

nullOptional

statusstringRequired

errorany ofOptional

anyOptional

or

nullOptional

objectconst: evaluation.runOptional

Object type.

Default: evaluation.run

idstringRequired

Evaluation run id (run group id).

process_idany ofOptional

Process id for lifecycle tracking.

stringOptional

or

nullOptional

evaluatorsany ofOptional

Evaluator ids used in this run.

string[]Optional

or

nullOptional

containerany ofOptional

Container id.

stringOptional

or

nullOptional

dataset_idany ofOptional

Dataset id (if a dataset was used).

stringOptional

or

nullOptional

sampleany ofOptional

Resolved sample-side data source.

anyOptional

or

or

or

nullOptional

ground_truthany ofOptional

Resolved ground-truth-side data source.

anyOptional

or

or

or

nullOptional

resultsany ofOptional

Evaluation results (populated on completion).

or

nullOptional

metricsany ofOptional

Evaluation metrics (populated on completion).

or

nullOptional

configany ofOptional

Run configuration as submitted.

or

nullOptional

spendany ofOptional

Estimated spend.

numberOptional

or

nullOptional

metadataany ofOptional

Optional metadata.

or

nullOptional

400

Bad Request

application/json

401

Unauthorized

application/json

403

Forbidden

application/json

404

Not Found

application/json

409

Conflict

application/json

422

Validation Error

application/json

429

Too Many Requests

application/json

500

Internal Server Error

application/json

501

Not Implemented

application/json

503

Upstream Unavailable

application/json

get

/v1/evaluation/runs/{run_id}

HTTPcURLJavaScriptPython

GET /v1/evaluation/runs/{run_id} HTTP/1.1
Host: platform.maniac.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

{
  "created_at": "text",
  "finished_at": "text",
  "error_at": "text",
  "status": "text",
  "error": null,
  "object": "evaluation.run",
  "id": "text",
  "process_id": "text",
  "evaluators": [
    "text"
  ],
  "container": "text",
  "dataset_id": "text",
  "sample": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "ground_truth": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "results": {
    "overall": {
      "avg_score": 1,
      "avg_accuracy": 1,
      "ANY_ADDITIONAL_PROPERTY": "anything"
    },
    "metadata": {
      "ANY_ADDITIONAL_PROPERTY": "anything"
    },
    "per_model": [
      {
        "model": {
          "ANY_ADDITIONAL_PROPERTY": "anything"
        },
        "per_eval": {
          "ANY_ADDITIONAL_PROPERTY": {
            "accuracy": 1,
            "avg_score": 1,
            "num_total": 1,
            "num_errors": 1,
            "num_failed": 1,
            "num_passed": 1,
            "num_scored": 1,
            "num_missing": 1,
            "avg_accuracy": 1,
            "break_reason": "text",
            "expected_count": 1,
            "ANY_ADDITIONAL_PROPERTY": "anything"
          }
        },
        "avg_score": 1,
        "avg_accuracy": 1,
        "launch_index": 1,
        "launch_call_id": "text",
        "ANY_ADDITIONAL_PROPERTY": "anything"
      }
    ],
    "launch_count": 1,
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "metrics": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "config": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "spend": 1,
  "metadata": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  }
}