Ctrlk
DashboardSign up
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Evaluation

Evaluation and evaluator endpoints.

List evaluators

get

List evaluators for the authenticated project, optionally filtered by container.

Authorizations
AuthorizationstringRequired

API key in Authorization header using Bearer .

Query parameters
containerstring · nullableOptional
limitinteger · min: 1 · max: 100OptionalDefault: 20
offsetintegerOptionalDefault: 0
Responses
200

Successful Response

application/json
objectconst: listOptional

Object type identifier.

Default: list
totalintegerRequired

Total number of items available for this resource.

get/v1/evaluators
GET /v1/evaluators HTTP/1.1
Host: platform.maniac.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

{
  "object": "list",
  "data": [
    {
      "object": "evaluator.judge",
      "project_id": "text",
      "container_id": "text",
      "id": "text",
      "name": "text",
      "description": "text",
      "created_at": "text",
      "metadata": {
        "ANY_ADDITIONAL_PROPERTY": "anything"
      },
      "pass_threshold": 1,
      "mode": "pointwise",
      "api": "text",
      "container": "text",
      "model": "text",
      "prompt": "text",
      "inference_parameters": {
        "ANY_ADDITIONAL_PROPERTY": "anything"
      }
    }
  ],
  "total": 0
}

Create an evaluator

post

Create a new evaluator scoped to a project or container.

Authorizations
AuthorizationstringRequired

API key in Authorization header using Bearer .

Body
containerstring · nullableOptional

Container label.

typestring · enumRequired

Evaluator type.

Possible values:
modelstring · nullableOptional

Judge model slug.

promptstring · nullableOptional

Judge prompt.

sourcestring · nullableOptional

Code evaluator source.

requirementsstring[] · nullableOptional

Optional requirements list.

pass_thresholdnumber · nullableOptional

Optional pass threshold.

modestring · enumOptional

Evaluation mode.

Default: pointwisePossible values:
namestring · nullableOptional

Evaluator name.

descriptionstring · nullableOptional

Evaluator description.

apistring · nullableOptional

API type.

Responses
201

Successful Response

application/json
or
post/v1/evaluators
POST /v1/evaluators HTTP/1.1
Host: platform.maniac.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Content-Type: application/json
Accept: */*
Content-Length: 304

{
  "container": "text",
  "type": "judge",
  "model": "text",
  "prompt": "text",
  "source": "text",
  "requirements": [
    "text"
  ],
  "pass_threshold": 1,
  "mode": "pointwise",
  "metadata": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "name": "text",
  "description": "text",
  "api": "text",
  "inference_parameters": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  }
}
{
  "object": "evaluator.judge",
  "project_id": "text",
  "container_id": "text",
  "id": "text",
  "name": "text",
  "description": "text",
  "created_at": "text",
  "metadata": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "pass_threshold": 1,
  "mode": "pointwise",
  "api": "text",
  "container": "text",
  "model": "text",
  "prompt": "text",
  "inference_parameters": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  }
}

Get an evaluator

get

Fetch a single evaluator by id or name within the authenticated project.

Authorizations
AuthorizationstringRequired

API key in Authorization header using Bearer .

Path parameters
evaluatorstringRequired
Responses
200

Successful Response

application/json
or
get/v1/evaluators/{evaluator}
GET /v1/evaluators/{evaluator} HTTP/1.1
Host: platform.maniac.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

{
  "object": "evaluator.judge",
  "project_id": "text",
  "container_id": "text",
  "id": "text",
  "name": "text",
  "description": "text",
  "created_at": "text",
  "metadata": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "pass_threshold": 1,
  "mode": "pointwise",
  "api": "text",
  "container": "text",
  "model": "text",
  "prompt": "text",
  "inference_parameters": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  }
}

Update an evaluator

patch

Update an existing evaluator by id or name.

Authorizations
AuthorizationstringRequired

API key in Authorization header using Bearer .

Path parameters
evaluatorstringRequired
Body

Patchable fields for an evaluator.

containerstring · nullableOptional

Container label.

modelstring · nullableOptional

Judge model slug.

promptstring · nullableOptional

Judge prompt.

sourcestring · nullableOptional

Code evaluator source.

requirementsstring[] · nullableOptional

Optional requirements list.

pass_thresholdnumber · nullableOptional

Optional pass threshold.

modestring · enum · nullableOptional

Evaluation mode.

Possible values:
namestring · nullableOptional

Evaluator name.

descriptionstring · nullableOptional

Evaluator description.

apistring · nullableOptional

API type.

Responses
200

Successful Response

application/json
or
patch/v1/evaluators/{evaluator}
PATCH /v1/evaluators/{evaluator} HTTP/1.1
Host: platform.maniac.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Content-Type: application/json
Accept: */*
Content-Length: 289

{
  "container": "text",
  "model": "text",
  "prompt": "text",
  "source": "text",
  "requirements": [
    "text"
  ],
  "pass_threshold": 1,
  "mode": "pointwise",
  "metadata": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "name": "text",
  "description": "text",
  "api": "text",
  "inference_parameters": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  }
}
{
  "object": "evaluator.judge",
  "project_id": "text",
  "container_id": "text",
  "id": "text",
  "name": "text",
  "description": "text",
  "created_at": "text",
  "metadata": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "pass_threshold": 1,
  "mode": "pointwise",
  "api": "text",
  "container": "text",
  "model": "text",
  "prompt": "text",
  "inference_parameters": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  }
}

Delete an evaluator

delete

Delete an evaluator by id.

Authorizations
AuthorizationstringRequired

API key in Authorization header using Bearer .

Path parameters
evaluator_idstringRequired
Responses
200

Successful Response

application/json
objectconst: evaluatorOptional

Object type.

Default: evaluator
idstringRequired

Evaluator id.

deletedconst: Optional

Deletion status.

Default: true
delete/v1/evaluators/{evaluator_id}
DELETE /v1/evaluators/{evaluator_id} HTTP/1.1
Host: platform.maniac.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

{
  "object": "evaluator",
  "id": "text",
  "deleted": true
}

List evaluation runs

get

List evaluation runs for the authenticated project. Optionally filter by container and status.

Authorizations
AuthorizationstringRequired

API key in Authorization header using Bearer .

Query parameters
containerstring · nullableOptional

Container ID or label to filter by.

statusstring · nullableOptional

Filter by run status (e.g. 'running', 'completed', 'error').

limitinteger · min: 1 · max: 100OptionalDefault: 20
offsetintegerOptionalDefault: 0
Responses
200

Successful Response

application/json
objectconst: listOptional

Object type identifier.

Default: list
totalintegerRequired

Total number of items available for this resource.

get/v1/evaluation/runs
GET /v1/evaluation/runs HTTP/1.1
Host: platform.maniac.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

{
  "object": "list",
  "data": [
    {
      "created_at": "text",
      "finished_at": "text",
      "error_at": "text",
      "status": "text",
      "error": null,
      "object": "evaluation.run",
      "id": "text",
      "process_id": "text",
      "evaluators": [
        "text"
      ],
      "container": "text",
      "dataset_id": "text",
      "sample": {
        "range": "0:100",
        "type": "text",
        "dataset": "file_abc123"
      },
      "ground_truth": {
        "range": "0:100",
        "type": "text",
        "dataset": "file_abc123"
      },
      "baseline": {
        "range": "0:100",
        "type": "text",
        "dataset": "file_abc123"
      },
      "results": {
        "overall": {
          "avg_score": 1,
          "avg_accuracy": 1,
          "ANY_ADDITIONAL_PROPERTY": "anything"
        },
        "metadata": {
          "ANY_ADDITIONAL_PROPERTY": "anything"
        },
        "per_model": [
          {
            "model": {
              "ANY_ADDITIONAL_PROPERTY": "anything"
            },
            "per_eval": {
              "ANY_ADDITIONAL_PROPERTY": {
                "accuracy": 1,
                "avg_score": 1,
                "num_total": 1,
                "num_errors": 1,
                "num_failed": 1,
                "num_passed": 1,
                "num_scored": 1,
                "num_missing": 1,
                "avg_accuracy": 1,
                "break_reason": "text",
                "expected_count": 1,
                "ANY_ADDITIONAL_PROPERTY": "anything"
              }
            },
            "avg_score": 1,
            "avg_accuracy": 1,
            "launch_index": 1,
            "launch_call_id": "text",
            "ANY_ADDITIONAL_PROPERTY": "anything"
          }
        ],
        "launch_count": 1,
        "ANY_ADDITIONAL_PROPERTY": "anything"
      },
      "metrics": {
        "ANY_ADDITIONAL_PROPERTY": "anything"
      },
      "config": {
        "ANY_ADDITIONAL_PROPERTY": "anything"
      },
      "spend": 1,
      "metadata": {
        "ANY_ADDITIONAL_PROPERTY": "anything"
      }
    }
  ],
  "total": 0
}

Create an evaluation run

post

Launch an evaluation run. Validates access to the specified container, evaluators, data sources, and models, then dispatches the run through the backend gateway interface.

Authorizations
AuthorizationstringRequired

API key in Authorization header using Bearer .

Body

Request body for creating an evaluation run.

Each side of the evaluation (sample and ground_truth) is described by a single data-source object whose type discriminator determines how data is obtained:

  • "dataset" — pull from a dataset.
  • "container" — pull from the container's task logs.
  • "generate" — generate completions using one or more models.

Both fields are optional, but at least one must be provided. When a side is omitted it defaults to the top-level container's task logs. At least one resolved side must not be type='generate' so there is seed input to evaluate against.

containerstringRequired

Container id or label.

evaluatorsstring[] · min: 1Required

Evaluator ids or labels.

sampleone of · nullableOptional

Sample-side data source. Omit to default to the container's task logs. Use type='dataset' to pull from a dataset, type='container' to pull from task logs, or type='generate' to generate completions with the specified models.

or
or
ground_truthone of · nullableOptional

Ground-truth-side data source. Omit to default to the container's task logs. Use type='dataset' to pull from a dataset, type='container' to pull from task logs, or type='generate' to generate completions with a model.

or
or
baselineone of · nullableOptional

Baseline-side data source for pairwise evaluation. Use type='dataset' to pull from a dataset, type='container' to pull from task logs, or type='generate' to generate completions with a model.

or
or
environmentstring · nullableOptional

Execution environment name (maps to Modal app suffix).

Default: main
Responses
201

Successful Response

application/json

Response model for an evaluation run.

created_atstringRequired
finished_atstring · nullableOptional
error_atstring · nullableOptional
statusstringRequired
errorany · nullableOptional
objectconst: evaluation.runOptional

Object type.

Default: evaluation.run
idstringRequired

Evaluation run id (run group id).

process_idstring · nullableOptional

Process id for lifecycle tracking.

evaluatorsstring[] · nullableOptional

Evaluator ids used in this run.

containerstring · nullableOptional

Container id.

dataset_idstring · nullableOptional

Dataset id (if a dataset was used).

sampleone of · nullableOptional

Resolved sample-side data source.

or
or
ground_truthone of · nullableOptional

Resolved ground-truth-side data source.

or
or
baselineone of · nullableOptional

Resolved baseline-side data source for pairwise evaluation.

or
or
spendnumber · nullableOptional

Estimated spend.

post/v1/evaluation/runs
POST /v1/evaluation/runs HTTP/1.1
Host: platform.maniac.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Content-Type: application/json
Accept: */*
Content-Length: 316

{
  "container": "text",
  "evaluators": [
    "text"
  ],
  "sample": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "ground_truth": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "baseline": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "metadata": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "environment": "main"
}
{
  "created_at": "text",
  "finished_at": "text",
  "error_at": "text",
  "status": "text",
  "error": null,
  "object": "evaluation.run",
  "id": "text",
  "process_id": "text",
  "evaluators": [
    "text"
  ],
  "container": "text",
  "dataset_id": "text",
  "sample": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "ground_truth": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "baseline": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "results": {
    "overall": {
      "avg_score": 1,
      "avg_accuracy": 1,
      "ANY_ADDITIONAL_PROPERTY": "anything"
    },
    "metadata": {
      "ANY_ADDITIONAL_PROPERTY": "anything"
    },
    "per_model": [
      {
        "model": {
          "ANY_ADDITIONAL_PROPERTY": "anything"
        },
        "per_eval": {
          "ANY_ADDITIONAL_PROPERTY": {
            "accuracy": 1,
            "avg_score": 1,
            "num_total": 1,
            "num_errors": 1,
            "num_failed": 1,
            "num_passed": 1,
            "num_scored": 1,
            "num_missing": 1,
            "avg_accuracy": 1,
            "break_reason": "text",
            "expected_count": 1,
            "ANY_ADDITIONAL_PROPERTY": "anything"
          }
        },
        "avg_score": 1,
        "avg_accuracy": 1,
        "launch_index": 1,
        "launch_call_id": "text",
        "ANY_ADDITIONAL_PROPERTY": "anything"
      }
    ],
    "launch_count": 1,
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "metrics": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "config": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "spend": 1,
  "metadata": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  }
}

Get an evaluation run

get

Retrieve a single evaluation run by ID within the authenticated project.

Authorizations
AuthorizationstringRequired

API key in Authorization header using Bearer .

Path parameters
run_idstringRequired
Responses
200

Successful Response

application/json

Response model for an evaluation run.

created_atstringRequired
finished_atstring · nullableOptional
error_atstring · nullableOptional
statusstringRequired
errorany · nullableOptional
objectconst: evaluation.runOptional

Object type.

Default: evaluation.run
idstringRequired

Evaluation run id (run group id).

process_idstring · nullableOptional

Process id for lifecycle tracking.

evaluatorsstring[] · nullableOptional

Evaluator ids used in this run.

containerstring · nullableOptional

Container id.

dataset_idstring · nullableOptional

Dataset id (if a dataset was used).

sampleone of · nullableOptional

Resolved sample-side data source.

or
or
ground_truthone of · nullableOptional

Resolved ground-truth-side data source.

or
or
baselineone of · nullableOptional

Resolved baseline-side data source for pairwise evaluation.

or
or
spendnumber · nullableOptional

Estimated spend.

get/v1/evaluation/runs/{run_id}
GET /v1/evaluation/runs/{run_id} HTTP/1.1
Host: platform.maniac.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

{
  "created_at": "text",
  "finished_at": "text",
  "error_at": "text",
  "status": "text",
  "error": null,
  "object": "evaluation.run",
  "id": "text",
  "process_id": "text",
  "evaluators": [
    "text"
  ],
  "container": "text",
  "dataset_id": "text",
  "sample": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "ground_truth": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "baseline": {
    "range": "0:100",
    "type": "text",
    "dataset": "file_abc123"
  },
  "results": {
    "overall": {
      "avg_score": 1,
      "avg_accuracy": 1,
      "ANY_ADDITIONAL_PROPERTY": "anything"
    },
    "metadata": {
      "ANY_ADDITIONAL_PROPERTY": "anything"
    },
    "per_model": [
      {
        "model": {
          "ANY_ADDITIONAL_PROPERTY": "anything"
        },
        "per_eval": {
          "ANY_ADDITIONAL_PROPERTY": {
            "accuracy": 1,
            "avg_score": 1,
            "num_total": 1,
            "num_errors": 1,
            "num_failed": 1,
            "num_passed": 1,
            "num_scored": 1,
            "num_missing": 1,
            "avg_accuracy": 1,
            "break_reason": "text",
            "expected_count": 1,
            "ANY_ADDITIONAL_PROPERTY": "anything"
          }
        },
        "avg_score": 1,
        "avg_accuracy": 1,
        "launch_index": 1,
        "launch_call_id": "text",
        "ANY_ADDITIONAL_PROPERTY": "anything"
      }
    ],
    "launch_count": 1,
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "metrics": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "config": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "spend": 1,
  "metadata": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  }
}
PreviousModels
NextDatasets

Last updated