Add a component

Every stage is a swappable component behind a small protocol, because the point of the project is comparing stages rather than blessing one configuration. Adding one means implementing a protocol, registering a name, and then earning the change through the harness.

The shape of a component

Register a class under a kind and a name; configuration then selects it by that name and the registry constructs it with the params from the file.

from typing import Any

from cairn.core.registry import register


@register("reranker", "recency")
class RecencyReranker:
    """Prefers recent documents when the question asks about the current state."""

    name = "recency"

    def __init__(self, half_life_days: float = 180.0) -> None:
        self._params = {"half_life_days": half_life_days}

    @property
    def params(self) -> dict[str, Any]:
        return dict(self._params)

    def rerank(self, question, candidates, k):
        ...

name and params are on every component so an index manifest and a report can record exactly what ran. Return a copy of params rather than the mutable attribute; a caller must not be able to edit what the manifest will record.

The kinds are fixed, so a typo in a configuration is an error at load time rather than a silent fallback: parser, chunker, embedder, vector_index, lexical_index, retriever, reranker, rewriter, provider, judge, confidence.

The contracts that are not in the type signature

Some invariants the harness relies on cannot be expressed as types, and breaking them produces wrong numbers rather than errors.

Chunkers. document.text[chunk.start:chunk.end] == chunk.text exactly. Every citation resolves through those offsets, so an off-by-one turns into a wrong quote in an answer. Chunks must not cross a page boundary when the page has real content.

Embedders. Rows are L2-normalised float32 of a fixed dim, and embed over a list must produce the same vectors as embed_query on each item, or an index built one way and queried the other silently degrades.

Vector indexes. Return chunk ids, never positions, so callers never depend on insertion order. save and load must round-trip, because a built index is a versioned artefact.

Rerankers. Return at most k, ranked from 1, with stage set to rerank. Scores from different stages live on different scales and must never be compared directly, which is what stage is for.

Providers. Own your pricing: cost_usd on the response comes from your price table, so the harness never has to know what a token costs where. Retry twice on retryable errors, never on a schema validation failure, which is a ProviderError(retryable=False).

Anything that puts corpus text in front of a model scrubs it first. See Injection.

Optional dependencies

A component whose library is missing still registers, and raises on construction with a message naming the extra:

from cairn.core.errors import MissingDependency


@register("reranker", "cross_encoder")
class CrossEncoderReranker:
    def __init__(self, model: str = "cross-encoder/ms-marco-MiniLM-L-6-v2") -> None:
        try:
            from sentence_transformers import CrossEncoder
        except ImportError as e:
            raise MissingDependency("the cross-encoder reranker", "local") from e

The name is always known and the error says what to install, which is what cairn doctor reports.

Making the change land

Import the module so its registration runs, add it to the built-ins list if it ships with the package, then write a configuration that differs from an existing one in exactly that component:

cp configs/hybrid-hashing.yaml configs/hybrid-recency.yaml
# edit retrieval.rerank to {name: recency, params: {half_life_days: 180}}
cairn bench --configs 'configs/hybrid-hashing.yaml configs/hybrid-recency.yaml' \
  --out reports/leaderboard.md
cairn gate --baseline reports/latest/hybrid-hashing.json \
           --candidate reports/latest/hybrid-recency.json

If the gate refuses it, the component did not earn its place. That is the mechanism working, and it applies to components shipped with Cairn exactly as it applies to yours.

Testing one

Test the contract, not the implementation. For a chunker, assert the offsets index back into the document. For a reranker, assert the ranks and the stage. For a provider, assert that a schema violation raises with retryable=False.

Components that need a network are marked so they are deselected by default:

import pytest


@pytest.mark.network
def test_the_hosted_model_answers() -> None:
    ...

Where a component cannot go

Two things are deliberately not extensible. The data model in cairn.core.types is the vocabulary every component exchanges, and the metric definitions in the evaluation harness are the protocol every configuration is measured by. If those could be swapped per configuration, two reports would stop being comparable, and comparability is the entire point.