docs: reconcile Candidate Pair resource contract
This commit is contained in:
fy59
parent
c27be7776b
commit
7712fe19fe
1 changed files with 475 additions and 285 deletions
|
|
@ -1,48 +1,71 @@
|
|||
# Sous-système Candidate Pair
|
||||
# Candidate Pair subsystem
|
||||
|
||||
## Vision
|
||||
|
||||
Le sous-système Candidate Pair répond uniquement à la question :
|
||||
|
||||
> « Quelles paires d'images valent la peine d'être présentées au Matcher ? »
|
||||
|
||||
Il ne répond **PAS** à :
|
||||
|
||||
> « Ces images ont-elle réellement des correspondances ? »
|
||||
|
||||
et ne contient **aucune** validation géométrique.
|
||||
|
||||
## Frontières
|
||||
## Status
|
||||
|
||||
```text
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
CANDIDATE_PAIR_MODEL=v1
|
||||
CANDIDATE_PAIR_MODEL_STATUS=IMPLEMENTED
|
||||
CANDIDATE_PAIR_TASK=candidate_pair.generate/1
|
||||
CANDIDATE_PAIR_TASK_STATUS=IMPLEMENTED
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
The Candidate Pair scientific model remains the Project DB v8 model. The durable Candidate Pair Task
|
||||
was added in Project DB v9. Later schema versions through v25 are additive and do not redefine
|
||||
Candidate Pair identity.
|
||||
|
||||
This document owns the Candidate Pair subsystem contract. Resource policy is governed by the Resource
|
||||
Governor and the canonical resource documents; Candidate Pair declares bounded demand and preserves
|
||||
deterministic scientific output.
|
||||
|
||||
## Purpose
|
||||
|
||||
The Candidate Pair subsystem answers one question:
|
||||
|
||||
> Which image pairs are worth presenting to the Matcher?
|
||||
|
||||
It does not answer whether two images actually have descriptor correspondences and it performs no
|
||||
geometric verification.
|
||||
|
||||
The downstream Matcher is implemented and consumes persisted Candidate Pairs, but Matcher science and
|
||||
persistence are outside this subsystem.
|
||||
|
||||
```text
|
||||
Visual Index
|
||||
↓
|
||||
|
|
||||
v
|
||||
Candidate Pair Generator
|
||||
↓
|
||||
|
|
||||
v
|
||||
Candidate Pair persistence
|
||||
↓
|
||||
Matcher — HORS SCOPE de ce ticket
|
||||
|
|
||||
v
|
||||
Matcher
|
||||
```
|
||||
|
||||
Le Matcher est un consommateur des paires persistées. Il n'est pas
|
||||
implémenté dans ce sous-système.
|
||||
## Core invariants
|
||||
|
||||
## Invariants fondamentaux
|
||||
| Invariant | Contract |
|
||||
| --- | --- |
|
||||
| Symmetry | `(A,B)` and `(B,A)` are the same scientific pair |
|
||||
| Canonical order | Persist with `image_id_a < image_id_b` |
|
||||
| No self-pairs | `image_id_a != image_id_b`, implied by the SQL ordering check |
|
||||
| Persistent uniqueness | `UNIQUE(image_id_a, image_id_b)` |
|
||||
| Bounded query result | `top_k <= LARDON3D_VISUAL_INDEX_TOP_K_MAX = 256` |
|
||||
| Determinism | Same inputs and options produce the same pair decisions in the same canonical order |
|
||||
| Idempotence | Repeating generation does not duplicate persisted pairs |
|
||||
| Persistent model | Candidate Pair rows were introduced by Project DB v8 |
|
||||
| Durable execution | `candidate_pair.generate/1` was introduced by Project DB v9 |
|
||||
|
||||
| Invariant | Description |
|
||||
|-----------|-------------|
|
||||
| **Symétrie** | A,B == B,A |
|
||||
| **Ordre canonique** | Stockage avec `image_id_a < image_id_b` |
|
||||
| **Self-pairs interdits** | `image_id_a != image_id_b` (implicite via CHECK SQL) |
|
||||
| **Unicité persistante** | `UNIQUE(image_id_a, image_id_b)` |
|
||||
| **Résultat borné** | `top_k <= LARDON3D_VISUAL_INDEX_TOP_K_MAX = 256` par requête |
|
||||
| **Déterminisme** | Mêmes entrées/configuration → mêmes paires dans le même ordre |
|
||||
| **Idempotence** | Répétition sans duplication |
|
||||
| **Persistance durable** | Paires persistées dans Project DB v8 |
|
||||
Candidate Pair identity is the canonical unordered image pair. Retrieval score, Visual Index provenance,
|
||||
Task ID, timestamps and operational resource choices do not enter that identity.
|
||||
|
||||
## Modèle persistant
|
||||
## Persistent model
|
||||
|
||||
### Table `candidate_pairs` (Project DB v8)
|
||||
### `candidate_pairs` — Project DB v8
|
||||
|
||||
```sql
|
||||
CREATE TABLE candidate_pairs(
|
||||
|
|
@ -57,17 +80,24 @@ CREATE INDEX candidate_pairs_image_a_idx ON candidate_pairs(image_id_a);
|
|||
CREATE INDEX candidate_pairs_image_b_idx ON candidate_pairs(image_id_b);
|
||||
```
|
||||
|
||||
The executable schema in `src/project_db.c` remains authoritative if prose and SQL excerpts ever
|
||||
diverge.
|
||||
|
||||
### Public Project DB API
|
||||
|
||||
The Candidate Pair persistence surface includes:
|
||||
|
||||
- `lardon3d_project_db_create_candidate_pair()`;
|
||||
- `lardon3d_project_db_load_candidate_pair()`;
|
||||
- `lardon3d_project_db_find_candidate_pair()`;
|
||||
- `lardon3d_project_db_list_candidate_pairs()`.
|
||||
|
||||
Creation canonicalizes the image order and persistence enforces uniqueness.
|
||||
|
||||
## Single-source generation
|
||||
|
||||
### API
|
||||
|
||||
- `lardon3d_project_db_create_candidate_pair()` — INSERT avec canonicalisation
|
||||
- `lardon3d_project_db_load_candidate_pair()` — SELECT par ID
|
||||
- `lardon3d_project_db_find_candidate_pair()` — SELECT par (image_a, image_b)
|
||||
- `lardon3d_project_db_list_candidate_pairs()` — SELECT paginé ORDER BY id
|
||||
|
||||
## Génération single-source
|
||||
|
||||
### Prototype
|
||||
|
||||
```c
|
||||
Lardon3DVisualIndexResult lardon3d_candidate_pair_generate(
|
||||
const char *project_path, Lardon3DProjectDb *database,
|
||||
|
|
@ -76,99 +106,117 @@ Lardon3DVisualIndexResult lardon3d_candidate_pair_generate(
|
|||
Lardon3DCandidatePairGenStats *stats);
|
||||
```
|
||||
|
||||
### Algorithme
|
||||
### Algorithm
|
||||
|
||||
1. Charger le FeatureSet source
|
||||
2. Obtenir `source_image_id`
|
||||
3. Interroger le Visual Index avec `query_options`
|
||||
4. Pour chaque candidat retourné :
|
||||
- Exclure les self-pairs
|
||||
- Canonicaliser l'ordre (image_a < image_b)
|
||||
- Chercher si la paire existe déjà (`find_candidate_pair`)
|
||||
- Créer si absente (`create_candidate_pair`)
|
||||
5. Retourner les statistiques `{generated, skipped, queried}`
|
||||
For one source Feature Set:
|
||||
|
||||
### Statistiques
|
||||
1. load the source Feature Set;
|
||||
2. obtain its `source_image_id`;
|
||||
3. query the Visual Index with the supplied options;
|
||||
4. for every returned candidate:
|
||||
- reject self-pairs;
|
||||
- canonicalize the image order;
|
||||
- find an existing Candidate Pair;
|
||||
- create the pair only when absent;
|
||||
5. return bounded generation statistics.
|
||||
|
||||
The generator does not perform descriptor matching and does not perform geometric verification.
|
||||
|
||||
### Statistics
|
||||
|
||||
```c
|
||||
typedef struct {
|
||||
uint32_t generated_count; // paires nouvellement créées
|
||||
uint32_t skipped_count; // paires déjà existantes (idempotence)
|
||||
uint32_t queried_count; // candidats retournés par le Visual Index
|
||||
uint32_t generated_count;
|
||||
uint32_t skipped_count;
|
||||
uint32_t queried_count;
|
||||
} Lardon3DCandidatePairGenStats;
|
||||
```
|
||||
|
||||
## Score et provenance
|
||||
`generated_count` counts newly persisted pairs. `skipped_count` counts pairs already present.
|
||||
`queried_count` counts candidates returned by the Visual Index query.
|
||||
|
||||
Le score de retrieval et la provenance Visual Index ne sont **PAS**
|
||||
persistés dans la table `candidate_pairs` pour les raisons suivantes :
|
||||
## Retrieval score and provenance
|
||||
|
||||
- L'identité Candidate Pair est纯粹 géométrique : (image_a, image_b)
|
||||
- Le score dépend de la configuration du Visual Index et peut changer
|
||||
- Le Matcher calculera ses propres scores de matching
|
||||
- La séparation des responsabilités est plus nette
|
||||
Retrieval score and Visual Index provenance are intentionally not stored in `candidate_pairs`.
|
||||
|
||||
Le score reste accessible via le Visual Index si nécessaire.
|
||||
Reasons:
|
||||
|
||||
## Déterminisme
|
||||
- Candidate Pair identity is only the canonical image pair;
|
||||
- retrieval score depends on Visual Index configuration;
|
||||
- a later Visual Index execution may score the same pair differently;
|
||||
- Matcher owns descriptor-level matching evidence;
|
||||
- keeping retrieval evidence out of Candidate Pair identity preserves subsystem separation.
|
||||
|
||||
### Déterministe
|
||||
A generation fingerprint describes the generation request. It does not change the identity of an
|
||||
already persisted Candidate Pair row.
|
||||
|
||||
- Mêmes entrées → mêmes paires
|
||||
- Même ordre de sélection top-K
|
||||
- Mêmes décisions de déduplication
|
||||
## Determinism
|
||||
|
||||
### Non déterministe
|
||||
### Deterministic inputs and decisions
|
||||
|
||||
- `created_at` (timestamp Unix, informatif uniquement)
|
||||
- `candidate_pair_id` (AUTOINCREMENT, identifiant technique)
|
||||
For identical immutable inputs and options, the subsystem preserves:
|
||||
|
||||
### Tie-breaks
|
||||
- the same Visual Index query contract;
|
||||
- the same top-K selection semantics;
|
||||
- the same canonical image ordering;
|
||||
- the same self-pair rejection;
|
||||
- the same deduplication decisions;
|
||||
- the same publication order for owner-published results.
|
||||
|
||||
En cas d'égalité de score dans le Visual Index, l'ordre est déterministe
|
||||
selon l'implémentation LSH (ordre des Feature Sets).
|
||||
### Non-scientific values
|
||||
|
||||
## Invalidation
|
||||
These values are not Candidate Pair scientific identity:
|
||||
|
||||
### Événements et impact
|
||||
- `created_at`;
|
||||
- `candidate_pair_id`;
|
||||
- Task ID;
|
||||
- resource reservation ID;
|
||||
- admitted CPU count;
|
||||
- admitted batch size.
|
||||
|
||||
| Événement | Ce qui devient invalide | Ce qui reste réutilisable | Ce qui doit être recalculé |
|
||||
|-----------|------------------------|--------------------------|---------------------------|
|
||||
| Nouvelle image | Rien (incrémental) | Paires existantes | Nouvelles requêtes Visual Index |
|
||||
| Nouveau FeatureSet | Rien | Paires existantes | Requête depuis ce FeatureSet |
|
||||
| FeatureSet remplacé | Paires basées sur ce FeatureSet | Autres paires | Nouvelle requête depuis ce FeatureSet |
|
||||
| Visual Index reconstruit | Toutes les paires (nouvelle config) | Rien | Tout recalcul |
|
||||
| Configuration top-K modifiée | Rien (borné par requête) | Paires existantes | Nouvelles requêtes avec nouveau top_k |
|
||||
| Filtre modifié | Rien | Paires existantes | Nouvelles requêtes avec nouveau filtre |
|
||||
| Relance après interruption | Rien | Paires déjà persistées | Suite du traitement |
|
||||
`candidate_pair_id` is a durable technical identity allocated by SQLite. It is not a scientific
|
||||
fingerprint.
|
||||
|
||||
### Politique
|
||||
## Generation fingerprint
|
||||
|
||||
L'invalidation est aussi locale que possible. On ne supprime jamais
|
||||
toutes les paires du projet suite à une modification locale.
|
||||
|
||||
### Vérification par fingerprint
|
||||
|
||||
Le fingerprint permet de vérifier si une génération doit être recalculée :
|
||||
### API
|
||||
|
||||
```c
|
||||
unsigned char fp_courant[32], fp_enregistre[32];
|
||||
lardon3d_candidate_pair_generation_fingerprint(..., fp_courant);
|
||||
// Si fp_courant != fp_enregistre → recalcul nécessaire
|
||||
void lardon3d_candidate_pair_generation_fingerprint(
|
||||
uint64_t visual_index_id, uint64_t source_feature_set_id,
|
||||
const Lardon3DVisualIndexQueryOptions *query_options,
|
||||
unsigned char fingerprint[32]);
|
||||
```
|
||||
|
||||
### Réutilisation
|
||||
### Included fields
|
||||
|
||||
- Même fingerprint → résultat réutilisable
|
||||
- Différent fingerprint → recalcul nécessaire
|
||||
- Les paires existantes sont conservées même si le fingerprint change
|
||||
The generation fingerprint includes:
|
||||
|
||||
## Batch projet
|
||||
- `visual_index_id`;
|
||||
- `source_feature_set_id`;
|
||||
- `query_options->top_k`;
|
||||
- `query_options->minimum_evidence_count`;
|
||||
- `query_options->scanset_filter`;
|
||||
- `query_options->exclude_same_asset`.
|
||||
|
||||
### Granularité
|
||||
### Excluded fields
|
||||
|
||||
La génération batch traite un ensemble de FeatureSets de manière
|
||||
bornée et déterministe.
|
||||
It excludes:
|
||||
|
||||
- `created_at`;
|
||||
- `candidate_pair_id`;
|
||||
- operational CPU/batch admission;
|
||||
- processing order of unrelated source Feature Sets.
|
||||
|
||||
### Reuse meaning
|
||||
|
||||
The same fingerprint means the same generation request may be reused.
|
||||
|
||||
A different fingerprint means the generation request must be evaluated again. Existing canonical
|
||||
Candidate Pair rows are not silently deleted merely because a different generation request is run;
|
||||
idempotent persistence may reuse rows that remain selected.
|
||||
|
||||
## Project batch generation
|
||||
|
||||
### API
|
||||
|
||||
|
|
@ -181,210 +229,352 @@ Lardon3DVisualIndexResult lardon3d_candidate_pair_generate_batch(
|
|||
uint64_t *last_feature_set_id);
|
||||
```
|
||||
|
||||
### Algorithme
|
||||
### Ordering and bounds
|
||||
|
||||
1. Lister les FeatureSets par pages de 64
|
||||
2. Pour chaque FeatureSet :
|
||||
- Appeler `lardon3d_candidate_pair_generate()`
|
||||
- Accumuler les statistiques
|
||||
- Mettre à jour le curseur
|
||||
3. Retourner les totaux et le dernier FeatureSet traité
|
||||
The project batch path:
|
||||
|
||||
### Ordre de traitement
|
||||
- pages Feature Sets in bounded pages;
|
||||
- processes source Feature Sets in increasing `feature_set_id`;
|
||||
- never assumes IDs are contiguous;
|
||||
- keeps top-K bounded by the Visual Index contract;
|
||||
- keeps query/result memory bounded;
|
||||
- returns the last processed Feature Set for restart.
|
||||
|
||||
Feature Sets traités en ordre croissant de `feature_set_id`.
|
||||
A pair selected from multiple sources is persisted once because canonical pair identity is unique.
|
||||
|
||||
### Déduplication
|
||||
The historical single-source and project-batch APIs remain valid. They are not, by themselves, the
|
||||
complete current resource-execution description of the durable Task.
|
||||
|
||||
Une paire produite depuis plusieurs sources n'existe qu'une fois.
|
||||
La deduplication est assurée par `find avant create`.
|
||||
|
||||
### Bornes
|
||||
|
||||
- Un seul FeatureSet traité à la fois
|
||||
- Top-K borné par requête
|
||||
- Mémoire bornée : allocation `top_k * sizeof(candidate)` par requête
|
||||
- Pagination bornée (64 FeatureSets par page)
|
||||
|
||||
### Reprise
|
||||
|
||||
Le curseur `after_feature_set_id` permet la reprise après interruption.
|
||||
La fonction retourne le dernier FeatureSet traité.
|
||||
|
||||
## Tâche durable
|
||||
## Durable Task
|
||||
|
||||
### Task Kind
|
||||
|
||||
`candidate_pair.generate` v1 — **IMPLEMENTED**.
|
||||
|
||||
### Unité de travail
|
||||
|
||||
Un membership FeatureSet source du Visual Index et sa requête associée. Chaque
|
||||
séquence traite un lot borné de memberships (1 à 64 selon le contrat Governor),
|
||||
en ordre croissant d'ID sans supposer des IDs contigus.
|
||||
|
||||
### Checkpoint
|
||||
|
||||
Curseur `after_feature_set_id` persisté dans `candidate_pair_generate_tasks`.
|
||||
Checkpoint sauvé après chaque lot via
|
||||
`lardon3d_project_checkpoint_candidate_pair_generate_task()`.
|
||||
|
||||
### Reprise
|
||||
|
||||
Reprise idempotente : le champ `after_feature_set_id` est rechargé depuis la
|
||||
DB, et les paires déjà persistées sont ignorées par `find avant create`.
|
||||
À l'ouverture du projet, la tâche est automatiquement restaurée via la
|
||||
registry production et resoumise à la queue.
|
||||
|
||||
Les anciens snapshots v1 produits avec l'estimation opérationnelle exacte
|
||||
128 Kio fixes, 64 Kio par item, lot 1–64, CPU 1, IO 1 et GPU 0 sont normalisés
|
||||
éphémèrement par la registry à la forme courante CPU64 avant admission. La
|
||||
forme historique immédiatement précédente CPU12/256 Kio fixes/64 Kio par item
|
||||
est elle aussi reconnue exactement. Le
|
||||
snapshot durable original reste la source du reconstructeur ; aucun checkpoint
|
||||
d'estimation seule n'est stagé, promu ou publié sous le même résumé. Une panne
|
||||
pré-terminale répète donc cette normalisation exacte. Aucun autre snapshot,
|
||||
curseur ou paramètre scientifique n'est réinterprété.
|
||||
|
||||
### Intégration Task/Queue/Governor
|
||||
|
||||
La tâche utilise le runtime générique via le pattern standard :
|
||||
- Estimation opérationnelle (256 Kio fixes, 8 Mio par item, lot 1–64).
|
||||
La Queue conserve un callback ; jusqu'à soixante-quatre participants CPU
|
||||
admis peuvent calculer une fenêtre interne bornée sans modifier l'identité
|
||||
scientifique.
|
||||
- CPU et lot sont essayés ensemble (`1/1`, `2/2`, puis les paliers sûrs) :
|
||||
un CPU supplémentaire avec un lot d'un seul membership ne peut exercer aucun
|
||||
participant supplémentaire et ne constitue donc pas une mesure de scaling.
|
||||
- Réservation CPU + IO avant exécution
|
||||
- `lardon3d_task_sequence_break()` entre chaque lot pour réadmission Governor
|
||||
- Callback terminal checkpoint après `COMPLETED`/`FAILED`/`CANCELLED`
|
||||
- Reconstruction depuis `Lardon3DProjectDbCandidatePairGenerateTask`
|
||||
|
||||
### API
|
||||
|
||||
```c
|
||||
Lardon3DTask *lardon3d_project_create_candidate_pair_generate_task(
|
||||
Lardon3DAppState *state, uint64_t visual_index_id,
|
||||
const Lardon3DVisualIndexQueryOptions *query_options, uint64_t *task_id);
|
||||
bool lardon3d_project_enqueue_candidate_pair_generate(
|
||||
Lardon3DAppState *state, uint64_t visual_index_id,
|
||||
const Lardon3DVisualIndexQueryOptions *query_options, uint64_t *task_id);
|
||||
bool lardon3d_candidate_pair_generate_reconstruct(
|
||||
const Lardon3DTaskDurableSnapshot *snapshot, void *context,
|
||||
Lardon3DTaskKindBinding *binding);
|
||||
```text
|
||||
candidate_pair.generate/1
|
||||
```
|
||||
|
||||
## Concurrence
|
||||
Status: **IMPLEMENTED**.
|
||||
|
||||
### Garantie actuelle
|
||||
### Durable unit
|
||||
|
||||
La Queue conserve un callback actif. À l'intérieur de la Task, jusqu'à soixante-quatre
|
||||
threads CPU admis calculent en parallèle une fenêtre d'au plus deux sources par
|
||||
thread. Chaque participant possède un handle DB de lecture privé. Le thread
|
||||
propriétaire publie ensuite seul et dans l'ordre canonique des sources. Voir le
|
||||
[contrat de parallélisme interne](internal_parallelism.md).
|
||||
The durable cursor is `after_feature_set_id` in `candidate_pair_generate_tasks`.
|
||||
|
||||
### Atomicité
|
||||
The Task consumes a bounded ordered set of Visual Index source memberships. A sequence handles an
|
||||
admitted bounded batch, publishes the canonical pair decisions, persists the cursor, checkpoints and
|
||||
returns through `lardon3d_task_sequence_break()` before the next Governor admission.
|
||||
|
||||
Les workers internes ne créent aucune paire. Après leur jointure, le
|
||||
propriétaire applique seul `find avant create`. La contrainte UNIQUE reste une
|
||||
protection persistante, pas un mécanisme d'ordonnancement parallèle.
|
||||
### Checkpoint and restart
|
||||
|
||||
### Limites
|
||||
The Task checkpoints through:
|
||||
|
||||
Le pattern `find + create` n'est pas atomique entre les deux appels. Le chemin
|
||||
de Task n'introduit aucun writer concurrent ; la sémantique existante reste
|
||||
inchangée pour les autres appelants éventuels.
|
||||
|
||||
## Bornes et ressources
|
||||
|
||||
### Top-K
|
||||
|
||||
`top_k <= 256` (LARDON3D_VISUAL_INDEX_TOP_K_MAX)
|
||||
|
||||
### Mémoire
|
||||
|
||||
Allocation de requête bornée par le top-K, plus un résultat de propositions
|
||||
borné par le même maximum. La Task limite sa fenêtre à 24 sources et annonce
|
||||
256 Kio fixes plus 64 Kio par item au Governor.
|
||||
|
||||
### Complexité
|
||||
|
||||
- O(top_k) par requête (lectures Visual Index)
|
||||
- O(1) par paire (DB write)
|
||||
- Pas de structure O(N²)
|
||||
|
||||
## Fingerprint de génération
|
||||
|
||||
### Composants
|
||||
|
||||
Le fingerprint identifie une génération Candidate Pair unique :
|
||||
|
||||
```c
|
||||
void lardon3d_candidate_pair_generation_fingerprint(
|
||||
uint64_t visual_index_id, uint64_t source_feature_set_id,
|
||||
const Lardon3DVisualIndexQueryOptions *query_options,
|
||||
unsigned char fingerprint[32]);
|
||||
```text
|
||||
lardon3d_project_checkpoint_candidate_pair_generate_task()
|
||||
```
|
||||
|
||||
### Éléments inclus
|
||||
Restart:
|
||||
|
||||
- `visual_index_id` : Visual Index utilisé
|
||||
- `source_feature_set_id` : Feature Set source
|
||||
- `query_options->top_k` : nombre de candidats par requête
|
||||
- `query_options->minimum_evidence_count` : filtre minimum
|
||||
- `query_options->scanset_filter` : filtre ScanSet
|
||||
- `query_options->exclude_same_asset` : exclusion même asset
|
||||
1. restores the generic Task snapshot;
|
||||
2. loads the typed Candidate Pair Task payload;
|
||||
3. restores `after_feature_set_id`;
|
||||
4. reconstructs the production binding through the Task Kind registry;
|
||||
5. resubmits through the normal Queue/Governor path;
|
||||
6. reuses already persisted Candidate Pairs idempotently.
|
||||
|
||||
### Éléments exclus (volontairement)
|
||||
A crash may therefore repeat work after the last durable cursor, but it must not invent a second
|
||||
scientific pair identity.
|
||||
|
||||
- `created_at` : informatif, pas fonctionnel
|
||||
- `candidate_pair_id` : identifiant technique
|
||||
- Ordre des Feature Sets traités en batch
|
||||
### Historical resource descriptors
|
||||
|
||||
### Stabilité
|
||||
Older durable snapshots are accepted only through exact compatibility shapes already recognized by the
|
||||
registry.
|
||||
|
||||
Le fingerprint est stable pour mêmes entrées et configuration.
|
||||
Un changement de configuration produit un fingerprint différent.
|
||||
|
||||
### Relation avec l'invalidation
|
||||
|
||||
Un fingerprint différent signifie que le travail doit être recalculé.
|
||||
Le même fingerprint signifie que le résultat peut être réutilisé.
|
||||
|
||||
### Relation avec la reprise
|
||||
|
||||
Le fingerprint permet de vérifier qu'une reprise utilise la même
|
||||
configuration que l'originale.
|
||||
|
||||
## Limites connues
|
||||
|
||||
1. Score non persistant (par design)
|
||||
2. La compaction de segments Visual Index n'est pas implémentée
|
||||
3. Aucun Matcher consommateur des paires
|
||||
|
||||
## Statut
|
||||
|
||||
**IMPLEMENTED** — génération single-source, persistance,
|
||||
canonicalisation, idempotence, batch projet, fingerprint et
|
||||
réutilisation/invalidation.
|
||||
|
||||
**IMPLEMENTED** — tâche durable `candidate_pair.generate` v1
|
||||
via le runtime/Queue générique, avec estimation immuable,
|
||||
checkpoint par curseur, reprise idempotente et intégration
|
||||
dans la registry production.
|
||||
|
||||
## Relation avec le pipeline
|
||||
|
||||
Le Candidate Pair Generator est l'étape E du pipeline de reconstruction :
|
||||
Historical forms include the exact earlier descriptors documented by the implementation, including:
|
||||
|
||||
```text
|
||||
128 KiB fixed
|
||||
64 KiB per item
|
||||
batch 1..64
|
||||
CPU1
|
||||
IO1
|
||||
GPU0
|
||||
```
|
||||
Feature Store (C)
|
||||
↓
|
||||
Visual Index (D)
|
||||
↓
|
||||
Candidate Pair Generator (E) ← CE DOCUMENT
|
||||
↓
|
||||
Matching (F) — HORS SCOPE
|
||||
|
||||
and the later exact historical CPU12 / 256 KiB fixed / 64 KiB-per-item form.
|
||||
|
||||
Those shapes are restart compatibility evidence. They are not the current resource model and must not
|
||||
be copied into new Task creation.
|
||||
|
||||
The original durable snapshot remains the source supplied to reconstruction. Compatibility
|
||||
normalization is ephemeral and does not rewrite the persisted checkpoint or Candidate scientific
|
||||
identity.
|
||||
|
||||
## Current resource contract
|
||||
|
||||
### Current Task estimate
|
||||
|
||||
The current validated Candidate Pair Task declares approximately:
|
||||
|
||||
```text
|
||||
fixed RAM 256 KiB
|
||||
per-item RAM 8 MiB
|
||||
batch range 1..64
|
||||
GPU demand 0
|
||||
IO demand bounded by the existing Task estimate
|
||||
CPU demand reducible and bounded by the host compute pool
|
||||
```
|
||||
|
||||
The exact implementation constants remain authoritative in source. This documentation records the
|
||||
current validated capability and intentionally does not retain the obsolete 24-source / 64-KiB-per-item
|
||||
description as current policy.
|
||||
|
||||
### Coupled CPU and batch admission
|
||||
|
||||
Candidate Pair generation has independent source work, but additional CPU cannot exercise additional
|
||||
participants if the admitted source batch remains one.
|
||||
|
||||
For this Task, CPU and batch scaling are therefore coupled during adaptation. Conceptually:
|
||||
|
||||
```text
|
||||
CPU1 / batch1
|
||||
CPU2 / batch2
|
||||
then larger safe coupled rungs
|
||||
```
|
||||
|
||||
subject to:
|
||||
|
||||
- the Task's declared maximums;
|
||||
- the host compute pool;
|
||||
- current Governor pressure;
|
||||
- measured usefulness;
|
||||
- current admission policy.
|
||||
|
||||
This coupling fixes an operational scaling defect. It does not modify Candidate Pair scientific
|
||||
identity, query options, top-K behavior, publication order or persistence.
|
||||
|
||||
### Canonical resource principles
|
||||
|
||||
```text
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
```
|
||||
|
||||
The interactive host reserve is preserved first. After that reserve and all safety constraints are
|
||||
satisfied, safe and useful compute capacity should not be left idle merely to preserve an old
|
||||
single-thread measurement.
|
||||
|
||||
Reference-host values are observations, not portable constants.
|
||||
|
||||
### Atomicity is not serialism
|
||||
|
||||
One source query and one Candidate Pair publication decision remain bounded scientific/transactional
|
||||
units.
|
||||
|
||||
That does not imply that independent source preparation must run serially.
|
||||
|
||||
```text
|
||||
PER_ITEM_ATOMICITY_REQUIRES_CROSS_ITEM_SERIALISM=NO
|
||||
OWNER_ONLY_PUBLICATION_REQUIRES_SERIAL_PREPARATION=NO
|
||||
```
|
||||
|
||||
## Internal concurrency
|
||||
|
||||
The Task Queue still owns one active callback. Candidate Pair uses bounded parallelism inside that
|
||||
callback.
|
||||
|
||||
The validated shape is:
|
||||
|
||||
```text
|
||||
one admitted Candidate owner Task
|
||||
-> bounded source window
|
||||
-> bounded CPU participants
|
||||
-> private read-side preparation
|
||||
-> join
|
||||
-> owner publishes in canonical source order
|
||||
```
|
||||
|
||||
Participants do not create Candidate Pair rows.
|
||||
|
||||
Each participant uses its allowed private read-side state. After participants join, the owner alone
|
||||
performs the canonical `find` / create publication sequence.
|
||||
|
||||
The SQL `UNIQUE(image_id_a, image_id_b)` constraint remains a persistent integrity guard, not a
|
||||
parallel scheduling primitive.
|
||||
|
||||
No second global scheduler, global worker pool or parallel SQLite writer subsystem is introduced.
|
||||
|
||||
## Persistence race boundary
|
||||
|
||||
The public pattern:
|
||||
|
||||
```text
|
||||
find candidate pair
|
||||
-> create candidate pair when absent
|
||||
```
|
||||
|
||||
is not a general atomic compare-and-insert primitive across arbitrary concurrent writers.
|
||||
|
||||
The production durable Candidate Task avoids introducing competing pair writers: participant work is
|
||||
read/preparation only and owner publication is serialized.
|
||||
|
||||
Other callers must not infer a stronger concurrency guarantee from the Task's owner-only publication
|
||||
model.
|
||||
|
||||
## Resource complexity
|
||||
|
||||
### Query bound
|
||||
|
||||
```text
|
||||
top_k <= LARDON3D_VISUAL_INDEX_TOP_K_MAX = 256
|
||||
```
|
||||
|
||||
### Memory
|
||||
|
||||
Current Task admission uses the current estimate:
|
||||
|
||||
```text
|
||||
256 KiB fixed + 8 MiB per admitted item
|
||||
```
|
||||
|
||||
with batch bounded to `1..64`.
|
||||
|
||||
Actual admission can be reduced by the Governor and host compute pool. Swap, zram and external scratch
|
||||
never enlarge admitted RAM.
|
||||
|
||||
Candidate Pair currently has no authoritative scratch consumer.
|
||||
|
||||
### Algorithmic shape
|
||||
|
||||
The subsystem does not allocate an O(N^2) project pair matrix.
|
||||
|
||||
Per source, bounded work is dominated by:
|
||||
|
||||
- bounded Visual Index query;
|
||||
- bounded top-K filtering;
|
||||
- canonical pair lookup/publication.
|
||||
|
||||
Project traversal is paged and restartable.
|
||||
|
||||
## GPU policy
|
||||
|
||||
Candidate Pair currently remains CPU.
|
||||
|
||||
The validated GPU audit classified it as:
|
||||
|
||||
```text
|
||||
CANDIDATE_GPU=REJECTED_WITH_MEASURED_REASON
|
||||
```
|
||||
|
||||
The workload is dominated by Visual Index access, filtering, branching and deterministic ordered SQLite
|
||||
publication, and no validated GPU primitive currently preserves the complete Candidate contract with a
|
||||
useful measured advantage.
|
||||
|
||||
This rejection does not authorize CPU serialism. Safe useful CPU parallelism remains required by the
|
||||
canonical resource policy.
|
||||
|
||||
## Real A6000 evidence
|
||||
|
||||
The retained real A6000 pre-SfM execution exercised the current Candidate path before Matcher, GV and
|
||||
Tracks.
|
||||
|
||||
It produced:
|
||||
|
||||
```text
|
||||
Candidate Pairs = 38,420
|
||||
Match Results = 38,420
|
||||
```
|
||||
|
||||
The later checkpoint is:
|
||||
|
||||
```text
|
||||
real-a6000-pre-sfm-2026-09-02
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
Candidate replay at the retained checkpoint created no new Candidate work. The proof continued through
|
||||
Matcher, Geometric Verifier v3 and Tracks, then stopped before real Sparse SfM.
|
||||
|
||||
This evidence validates the operational Candidate path on that project. It does not change Candidate
|
||||
Pair v1 identity or the Project DB v8/v9 historical contracts.
|
||||
|
||||
## Relationship to current Project DB
|
||||
|
||||
The Candidate Pair model was introduced in Project DB v8 and its durable Task payload in v9.
|
||||
|
||||
The current schema head is v25:
|
||||
|
||||
```text
|
||||
v22 selected scientific execution foundation
|
||||
v23 generic optical-context overlay
|
||||
v24 raw.develop.batch/1 persistence
|
||||
v25 features.extract.batch/1 persistence
|
||||
```
|
||||
|
||||
Those later additive migrations do not reinterpret Candidate Pair rows.
|
||||
|
||||
## Known limits
|
||||
|
||||
Current limits and non-goals include:
|
||||
|
||||
- retrieval score is not persisted in `candidate_pairs` by design;
|
||||
- Visual Index segment compaction remains separate work;
|
||||
- no generic DAG dependency scheduler is introduced by Candidate Pair;
|
||||
- no Candidate GPU backend is currently validated;
|
||||
- no scratch/spill path is authoritative for Candidate Pair;
|
||||
- arbitrary concurrent pair writers are not provided by the Task owner-publication model.
|
||||
|
||||
The downstream Matcher is implemented; it is not a missing Candidate Pair feature.
|
||||
|
||||
## Pipeline relationship
|
||||
|
||||
```text
|
||||
Feature Store
|
||||
|
|
||||
v
|
||||
Visual Index
|
||||
|
|
||||
v
|
||||
Candidate Pair Generator <- this document
|
||||
|
|
||||
v
|
||||
Matcher
|
||||
|
|
||||
v
|
||||
Geometric Verification
|
||||
|
|
||||
v
|
||||
Tracks
|
||||
```
|
||||
|
||||
Candidate Pair selects plausible image pairs. Matcher computes descriptor-level correspondence evidence.
|
||||
Geometric Verification validates geometry. Track Builder creates multi-view observation tracks. These
|
||||
scientific responsibilities remain separate.
|
||||
|
||||
## Summary
|
||||
|
||||
```text
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
|
||||
CANDIDATE_PAIR_MODEL=v1
|
||||
CANDIDATE_PAIR_MODEL_STATUS=IMPLEMENTED
|
||||
|
||||
CANDIDATE_PAIR_TASK=candidate_pair.generate/1
|
||||
CANDIDATE_PAIR_TASK_STATUS=IMPLEMENTED
|
||||
|
||||
CANDIDATE_PAIR_PERSISTENCE_VERSION=v8
|
||||
CANDIDATE_PAIR_TASK_PERSISTENCE_VERSION=v9
|
||||
|
||||
CANDIDATE_CURRENT_FIXED_RAM=256_KiB
|
||||
CANDIDATE_CURRENT_PER_ITEM_RAM=8_MiB
|
||||
CANDIDATE_CURRENT_BATCH_RANGE=1..64
|
||||
CANDIDATE_CPU_BATCH_ADAPTATION=COUPLED
|
||||
CANDIDATE_GPU=REJECTED_WITH_MEASURED_REASON
|
||||
CANDIDATE_SCRATCH_CONSUMER=NO
|
||||
|
||||
PER_ITEM_ATOMICITY_REQUIRES_CROSS_ITEM_SERIALISM=NO
|
||||
OWNER_ONLY_PUBLICATION_REQUIRES_SERIAL_PREPARATION=NO
|
||||
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
|
|
|||
Loading…
Reference in a new issue