Skip to content

Evaluation API

reid.RetrievalProtocol

Bases: EvaluationProtocol

Standard retrieval over crop folders or explicit benchmark splits.

Query sampling options apply only when deriving splits from crop folders. On crops cut from video, a query's nearest gallery images are often the same person a few frames apart, which inflates retrieval scores; use ReassociationProtocol to measure matching across a frame gap.

Parameters:

Name Type Description Default
queries_per_id int | None

Query images taken from each identity. Defaults to 1.

 None
min_crops int | None

Images an identity needs to be evaluated. Defaults to queries_per_id + 1, so every query keeps a correct match.

 None
seed int | None

Seed for choosing each identity's query images. Defaults to 0.

 None
distance str

"cosine" or "euclidean".

 'cosine'

reid.ReassociationProtocol

Bases: EvaluationProtocol

Rank same-sequence candidates at a fixed frame gap after each query.

Queries use frames divisible by query_stride and need a future positive among at least two candidates; other queries are skipped.

Parameters:

Name Type Description Default
gap_frames int

Frames between a query and its candidates.

 30
query_stride int

Queries are taken at frames divisible by this value.

 60
distance str

"cosine" or "euclidean".

 'cosine'

reid.ReIDResult dataclass

What reid.ReIDModel.evaluate returns.

metrics maps the protocol's metric names to percentages. distmat is empty unless return_distmat=True.

reid.distance_matrix(query_embeddings, gallery_embeddings, *, distance='cosine')

Distance from every query embedding to every gallery embedding.

These are the distances ReIDModel.evaluate ranks with. Smaller is closer, so distance_matrix(query, gallery).argmin(axis=1) gives the closest gallery embedding to each query, as long as the gallery is not empty.

Parameters:

Name Type Description Default
query_embeddings ndarray

Query embeddings of shape (Q, D), for example from ReIDModel.extract_embeddings.

 required
gallery_embeddings ndarray

Gallery embeddings of shape (G, D).

 required
distance str

"cosine" compares directions after L2-normalizing each embedding, from 0 (same direction) to 2 (opposite); an all-zero embedding has no direction and is at distance 1 from everything. "euclidean" compares the raw vectors.

 'cosine'

Returns:

Type Description
ndarray

Float32 array of shape (Q, G): row i holds query i's distance to
ndarray

each gallery embedding. Empty when either side has no embeddings, as for a
ndarray

frame with no detections.

Raises:

Type Description
ValueError

If distance is unknown, an input is not a 2-D array of finite values, the inputs have a different number of dimensions, or an embedding is too large to compare in float32.

reid.eval.EvaluationProtocol

Bases: ABC

How to evaluate: which images are compared and which metrics are reported.

A protocol holds settings only, so one instance can evaluate any number of datasets and models. build_query_gallery picks the images to compare, and must be deterministic: the same data and settings select the same images, which is what lets several models, or several training epochs, be scored on identical inputs.

Subclasses implement _build_query_gallery and metric_values.

Attributes:

Name Type Description
name str

Protocol name recorded in result metadata.
metric_names tuple[str, ...]

Metrics that metric_values reports.
distance str

"cosine" or "euclidean", used to compare embeddings.
exclude_same_pid_camid bool

Leave out gallery images with the query's person and camera ID, as the Market-1501 protocol does.

Pick the query images, gallery images, and candidates to compare.

Parameters:

Name Type Description Default
data ReIDSplit

Images to evaluate. The protocol splits them into query and gallery images; when gallery is provided, they are the query images.

 required
gallery ReIDSplit | None

Gallery images for the queries in data. When provided, both splits are used as given; re-association builds its own gallery and rejects one.

 None

Returns:

Type Description
ReIDSplit

(query, gallery, candidate_mask, metadata). candidate_mask
ReIDSplit

limits which gallery images each query is ranked against (None
ndarray | None

ranks against all of them); metadata records the protocol
dict

settings and counts.

metric_values(metrics) abstractmethod

Map computed statistics to the metrics this protocol reports.

reid.eval.ReIDMetrics dataclass

CMC / mAP / mINP scores (percentages) for one evaluation run.