PHP API

The package is primarily driven through Artisan, but the same engine is exposed
as normal Laravel services for test suites, custom dashboards, and host
application workflows.

Eval engine

Resolve Padosoft\EvalHarness\EvalEngine from the container when you need to
register datasets and run a system under test directly.

use Padosoft\EvalHarness\EvalEngine;

$report = app(EvalEngine::class)
    ->registerDataset($dataset)
    ->run('rag.factuality', fn (string $input) => app(RagAgent::class)->answer($input));

The SUT can be a callable, a SampleRunner, or a SampleInvocation-aware
callable. Use the command path when you want exit-code behavior; use the service
path when another workflow owns orchestration.

Dataset builder

Programmatic datasets are useful for generated adversarial seeds, test fixtures,
or host applications that already store canonical cases elsewhere.

use Padosoft\EvalHarness\Datasets\DatasetBuilder;

$dataset = DatasetBuilder::make('rag.factuality')
    ->sample(
        id: 'refund-policy',
        input: 'How long do I have to request a refund?',
        expectedOutput: 'Refunds are available within 30 days.',
        metadata: ['tags' => ['policy', 'refunds']],
    )
    ->metric('contains')
    ->metric('llm-as-judge')
    ->build();

Metrics

Custom metrics implement Padosoft\EvalHarness\Metrics\Metric and return a
MetricScore between 0.0 and 1.0.

use Padosoft\EvalHarness\Datasets\DatasetSample;
use Padosoft\EvalHarness\Metrics\Metric;
use Padosoft\EvalHarness\Metrics\MetricScore;

final class StartsWithExpectedMetric implements Metric
{
    public function name(): string
    {
        return 'starts-with-expected';
    }

    public function score(DatasetSample $sample, mixed $actual): MetricScore
    {
        return new MetricScore(str_starts_with((string) $actual, (string) $sample->expectedOutput) ? 1.0 : 0.0);
    }
}

Register an instance in the dataset, pass an FQCN string, or bind it behind a
container alias consumed by MetricResolver.

Saved outputs

Use SavedOutputsLoader when generation and scoring are deliberately separated.
The loaded outputs can be passed through the same metrics and report renderers as
a live run.

use Padosoft\EvalHarness\Outputs\SavedOutputsLoader;

$outputs = app(SavedOutputsLoader::class)->load(base_path('storage/eval/outputs.json'));
$report = app(EvalEngine::class)->scoreOutputs('rag.factuality', $outputs);

Online monitor

OnlineMonitor::capture() samples production traffic according to config and
queues JudgeLiveSampleJob when the sample is selected.

use Padosoft\EvalHarness\Online\OnlineMonitor;

app(OnlineMonitor::class)->capture(
    dataset: 'rag.factuality',
    input: $question,
    expectedOutput: $expectedAnswer,
    actualOutput: $agentAnswer,
    metadata: ['tenant' => $tenantId],
);

Worked example

Gotchas and limits