docs: reconcile Candidate Pair resource contract
This commit is contained in:
fy59
parent
c27be7776b
commit
7712fe19fe
1 changed files with 475 additions and 285 deletions
|
|
@ -1,48 +1,71 @@
|
||||||
# Sous-système Candidate Pair
|
# Candidate Pair subsystem
|
||||||
|
|
||||||
## Vision
|
## Status
|
||||||
|
|
||||||
Le sous-système Candidate Pair répond uniquement à la question :
|
|
||||||
|
|
||||||
> « Quelles paires d'images valent la peine d'être présentées au Matcher ? »
|
|
||||||
|
|
||||||
Il ne répond **PAS** à :
|
|
||||||
|
|
||||||
> « Ces images ont-elle réellement des correspondances ? »
|
|
||||||
|
|
||||||
et ne contient **aucune** validation géométrique.
|
|
||||||
|
|
||||||
## Frontières
|
|
||||||
|
|
||||||
|
```text
|
||||||
|
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||||
|
CANDIDATE_PAIR_MODEL=v1
|
||||||
|
CANDIDATE_PAIR_MODEL_STATUS=IMPLEMENTED
|
||||||
|
CANDIDATE_PAIR_TASK=candidate_pair.generate/1
|
||||||
|
CANDIDATE_PAIR_TASK_STATUS=IMPLEMENTED
|
||||||
|
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||||
|
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||||
|
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The Candidate Pair scientific model remains the Project DB v8 model. The durable Candidate Pair Task
|
||||||
|
was added in Project DB v9. Later schema versions through v25 are additive and do not redefine
|
||||||
|
Candidate Pair identity.
|
||||||
|
|
||||||
|
This document owns the Candidate Pair subsystem contract. Resource policy is governed by the Resource
|
||||||
|
Governor and the canonical resource documents; Candidate Pair declares bounded demand and preserves
|
||||||
|
deterministic scientific output.
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
The Candidate Pair subsystem answers one question:
|
||||||
|
|
||||||
|
> Which image pairs are worth presenting to the Matcher?
|
||||||
|
|
||||||
|
It does not answer whether two images actually have descriptor correspondences and it performs no
|
||||||
|
geometric verification.
|
||||||
|
|
||||||
|
The downstream Matcher is implemented and consumes persisted Candidate Pairs, but Matcher science and
|
||||||
|
persistence are outside this subsystem.
|
||||||
|
|
||||||
|
```text
|
||||||
Visual Index
|
Visual Index
|
||||||
↓
|
|
|
||||||
|
v
|
||||||
Candidate Pair Generator
|
Candidate Pair Generator
|
||||||
↓
|
|
|
||||||
|
v
|
||||||
Candidate Pair persistence
|
Candidate Pair persistence
|
||||||
↓
|
|
|
||||||
Matcher — HORS SCOPE de ce ticket
|
v
|
||||||
|
Matcher
|
||||||
```
|
```
|
||||||
|
|
||||||
Le Matcher est un consommateur des paires persistées. Il n'est pas
|
## Core invariants
|
||||||
implémenté dans ce sous-système.
|
|
||||||
|
|
||||||
## Invariants fondamentaux
|
| Invariant | Contract |
|
||||||
|
| --- | --- |
|
||||||
|
| Symmetry | `(A,B)` and `(B,A)` are the same scientific pair |
|
||||||
|
| Canonical order | Persist with `image_id_a < image_id_b` |
|
||||||
|
| No self-pairs | `image_id_a != image_id_b`, implied by the SQL ordering check |
|
||||||
|
| Persistent uniqueness | `UNIQUE(image_id_a, image_id_b)` |
|
||||||
|
| Bounded query result | `top_k <= LARDON3D_VISUAL_INDEX_TOP_K_MAX = 256` |
|
||||||
|
| Determinism | Same inputs and options produce the same pair decisions in the same canonical order |
|
||||||
|
| Idempotence | Repeating generation does not duplicate persisted pairs |
|
||||||
|
| Persistent model | Candidate Pair rows were introduced by Project DB v8 |
|
||||||
|
| Durable execution | `candidate_pair.generate/1` was introduced by Project DB v9 |
|
||||||
|
|
||||||
| Invariant | Description |
|
Candidate Pair identity is the canonical unordered image pair. Retrieval score, Visual Index provenance,
|
||||||
|-----------|-------------|
|
Task ID, timestamps and operational resource choices do not enter that identity.
|
||||||
| **Symétrie** | A,B == B,A |
|
|
||||||
| **Ordre canonique** | Stockage avec `image_id_a < image_id_b` |
|
|
||||||
| **Self-pairs interdits** | `image_id_a != image_id_b` (implicite via CHECK SQL) |
|
|
||||||
| **Unicité persistante** | `UNIQUE(image_id_a, image_id_b)` |
|
|
||||||
| **Résultat borné** | `top_k <= LARDON3D_VISUAL_INDEX_TOP_K_MAX = 256` par requête |
|
|
||||||
| **Déterminisme** | Mêmes entrées/configuration → mêmes paires dans le même ordre |
|
|
||||||
| **Idempotence** | Répétition sans duplication |
|
|
||||||
| **Persistance durable** | Paires persistées dans Project DB v8 |
|
|
||||||
|
|
||||||
## Modèle persistant
|
## Persistent model
|
||||||
|
|
||||||
### Table `candidate_pairs` (Project DB v8)
|
### `candidate_pairs` — Project DB v8
|
||||||
|
|
||||||
```sql
|
```sql
|
||||||
CREATE TABLE candidate_pairs(
|
CREATE TABLE candidate_pairs(
|
||||||
|
|
@ -57,17 +80,24 @@ CREATE INDEX candidate_pairs_image_a_idx ON candidate_pairs(image_id_a);
|
||||||
CREATE INDEX candidate_pairs_image_b_idx ON candidate_pairs(image_id_b);
|
CREATE INDEX candidate_pairs_image_b_idx ON candidate_pairs(image_id_b);
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The executable schema in `src/project_db.c` remains authoritative if prose and SQL excerpts ever
|
||||||
|
diverge.
|
||||||
|
|
||||||
|
### Public Project DB API
|
||||||
|
|
||||||
|
The Candidate Pair persistence surface includes:
|
||||||
|
|
||||||
|
- `lardon3d_project_db_create_candidate_pair()`;
|
||||||
|
- `lardon3d_project_db_load_candidate_pair()`;
|
||||||
|
- `lardon3d_project_db_find_candidate_pair()`;
|
||||||
|
- `lardon3d_project_db_list_candidate_pairs()`.
|
||||||
|
|
||||||
|
Creation canonicalizes the image order and persistence enforces uniqueness.
|
||||||
|
|
||||||
|
## Single-source generation
|
||||||
|
|
||||||
### API
|
### API
|
||||||
|
|
||||||
- `lardon3d_project_db_create_candidate_pair()` — INSERT avec canonicalisation
|
|
||||||
- `lardon3d_project_db_load_candidate_pair()` — SELECT par ID
|
|
||||||
- `lardon3d_project_db_find_candidate_pair()` — SELECT par (image_a, image_b)
|
|
||||||
- `lardon3d_project_db_list_candidate_pairs()` — SELECT paginé ORDER BY id
|
|
||||||
|
|
||||||
## Génération single-source
|
|
||||||
|
|
||||||
### Prototype
|
|
||||||
|
|
||||||
```c
|
```c
|
||||||
Lardon3DVisualIndexResult lardon3d_candidate_pair_generate(
|
Lardon3DVisualIndexResult lardon3d_candidate_pair_generate(
|
||||||
const char *project_path, Lardon3DProjectDb *database,
|
const char *project_path, Lardon3DProjectDb *database,
|
||||||
|
|
@ -76,99 +106,117 @@ Lardon3DVisualIndexResult lardon3d_candidate_pair_generate(
|
||||||
Lardon3DCandidatePairGenStats *stats);
|
Lardon3DCandidatePairGenStats *stats);
|
||||||
```
|
```
|
||||||
|
|
||||||
### Algorithme
|
### Algorithm
|
||||||
|
|
||||||
1. Charger le FeatureSet source
|
For one source Feature Set:
|
||||||
2. Obtenir `source_image_id`
|
|
||||||
3. Interroger le Visual Index avec `query_options`
|
|
||||||
4. Pour chaque candidat retourné :
|
|
||||||
- Exclure les self-pairs
|
|
||||||
- Canonicaliser l'ordre (image_a < image_b)
|
|
||||||
- Chercher si la paire existe déjà (`find_candidate_pair`)
|
|
||||||
- Créer si absente (`create_candidate_pair`)
|
|
||||||
5. Retourner les statistiques `{generated, skipped, queried}`
|
|
||||||
|
|
||||||
### Statistiques
|
1. load the source Feature Set;
|
||||||
|
2. obtain its `source_image_id`;
|
||||||
|
3. query the Visual Index with the supplied options;
|
||||||
|
4. for every returned candidate:
|
||||||
|
- reject self-pairs;
|
||||||
|
- canonicalize the image order;
|
||||||
|
- find an existing Candidate Pair;
|
||||||
|
- create the pair only when absent;
|
||||||
|
5. return bounded generation statistics.
|
||||||
|
|
||||||
|
The generator does not perform descriptor matching and does not perform geometric verification.
|
||||||
|
|
||||||
|
### Statistics
|
||||||
|
|
||||||
```c
|
```c
|
||||||
typedef struct {
|
typedef struct {
|
||||||
uint32_t generated_count; // paires nouvellement créées
|
uint32_t generated_count;
|
||||||
uint32_t skipped_count; // paires déjà existantes (idempotence)
|
uint32_t skipped_count;
|
||||||
uint32_t queried_count; // candidats retournés par le Visual Index
|
uint32_t queried_count;
|
||||||
} Lardon3DCandidatePairGenStats;
|
} Lardon3DCandidatePairGenStats;
|
||||||
```
|
```
|
||||||
|
|
||||||
## Score et provenance
|
`generated_count` counts newly persisted pairs. `skipped_count` counts pairs already present.
|
||||||
|
`queried_count` counts candidates returned by the Visual Index query.
|
||||||
|
|
||||||
Le score de retrieval et la provenance Visual Index ne sont **PAS**
|
## Retrieval score and provenance
|
||||||
persistés dans la table `candidate_pairs` pour les raisons suivantes :
|
|
||||||
|
|
||||||
- L'identité Candidate Pair est纯粹 géométrique : (image_a, image_b)
|
Retrieval score and Visual Index provenance are intentionally not stored in `candidate_pairs`.
|
||||||
- Le score dépend de la configuration du Visual Index et peut changer
|
|
||||||
- Le Matcher calculera ses propres scores de matching
|
|
||||||
- La séparation des responsabilités est plus nette
|
|
||||||
|
|
||||||
Le score reste accessible via le Visual Index si nécessaire.
|
Reasons:
|
||||||
|
|
||||||
## Déterminisme
|
- Candidate Pair identity is only the canonical image pair;
|
||||||
|
- retrieval score depends on Visual Index configuration;
|
||||||
|
- a later Visual Index execution may score the same pair differently;
|
||||||
|
- Matcher owns descriptor-level matching evidence;
|
||||||
|
- keeping retrieval evidence out of Candidate Pair identity preserves subsystem separation.
|
||||||
|
|
||||||
### Déterministe
|
A generation fingerprint describes the generation request. It does not change the identity of an
|
||||||
|
already persisted Candidate Pair row.
|
||||||
|
|
||||||
- Mêmes entrées → mêmes paires
|
## Determinism
|
||||||
- Même ordre de sélection top-K
|
|
||||||
- Mêmes décisions de déduplication
|
|
||||||
|
|
||||||
### Non déterministe
|
### Deterministic inputs and decisions
|
||||||
|
|
||||||
- `created_at` (timestamp Unix, informatif uniquement)
|
For identical immutable inputs and options, the subsystem preserves:
|
||||||
- `candidate_pair_id` (AUTOINCREMENT, identifiant technique)
|
|
||||||
|
|
||||||
### Tie-breaks
|
- the same Visual Index query contract;
|
||||||
|
- the same top-K selection semantics;
|
||||||
|
- the same canonical image ordering;
|
||||||
|
- the same self-pair rejection;
|
||||||
|
- the same deduplication decisions;
|
||||||
|
- the same publication order for owner-published results.
|
||||||
|
|
||||||
En cas d'égalité de score dans le Visual Index, l'ordre est déterministe
|
### Non-scientific values
|
||||||
selon l'implémentation LSH (ordre des Feature Sets).
|
|
||||||
|
|
||||||
## Invalidation
|
These values are not Candidate Pair scientific identity:
|
||||||
|
|
||||||
### Événements et impact
|
- `created_at`;
|
||||||
|
- `candidate_pair_id`;
|
||||||
|
- Task ID;
|
||||||
|
- resource reservation ID;
|
||||||
|
- admitted CPU count;
|
||||||
|
- admitted batch size.
|
||||||
|
|
||||||
| Événement | Ce qui devient invalide | Ce qui reste réutilisable | Ce qui doit être recalculé |
|
`candidate_pair_id` is a durable technical identity allocated by SQLite. It is not a scientific
|
||||||
|-----------|------------------------|--------------------------|---------------------------|
|
fingerprint.
|
||||||
| Nouvelle image | Rien (incrémental) | Paires existantes | Nouvelles requêtes Visual Index |
|
|
||||||
| Nouveau FeatureSet | Rien | Paires existantes | Requête depuis ce FeatureSet |
|
|
||||||
| FeatureSet remplacé | Paires basées sur ce FeatureSet | Autres paires | Nouvelle requête depuis ce FeatureSet |
|
|
||||||
| Visual Index reconstruit | Toutes les paires (nouvelle config) | Rien | Tout recalcul |
|
|
||||||
| Configuration top-K modifiée | Rien (borné par requête) | Paires existantes | Nouvelles requêtes avec nouveau top_k |
|
|
||||||
| Filtre modifié | Rien | Paires existantes | Nouvelles requêtes avec nouveau filtre |
|
|
||||||
| Relance après interruption | Rien | Paires déjà persistées | Suite du traitement |
|
|
||||||
|
|
||||||
### Politique
|
## Generation fingerprint
|
||||||
|
|
||||||
L'invalidation est aussi locale que possible. On ne supprime jamais
|
### API
|
||||||
toutes les paires du projet suite à une modification locale.
|
|
||||||
|
|
||||||
### Vérification par fingerprint
|
|
||||||
|
|
||||||
Le fingerprint permet de vérifier si une génération doit être recalculée :
|
|
||||||
|
|
||||||
```c
|
```c
|
||||||
unsigned char fp_courant[32], fp_enregistre[32];
|
void lardon3d_candidate_pair_generation_fingerprint(
|
||||||
lardon3d_candidate_pair_generation_fingerprint(..., fp_courant);
|
uint64_t visual_index_id, uint64_t source_feature_set_id,
|
||||||
// Si fp_courant != fp_enregistre → recalcul nécessaire
|
const Lardon3DVisualIndexQueryOptions *query_options,
|
||||||
|
unsigned char fingerprint[32]);
|
||||||
```
|
```
|
||||||
|
|
||||||
### Réutilisation
|
### Included fields
|
||||||
|
|
||||||
- Même fingerprint → résultat réutilisable
|
The generation fingerprint includes:
|
||||||
- Différent fingerprint → recalcul nécessaire
|
|
||||||
- Les paires existantes sont conservées même si le fingerprint change
|
|
||||||
|
|
||||||
## Batch projet
|
- `visual_index_id`;
|
||||||
|
- `source_feature_set_id`;
|
||||||
|
- `query_options->top_k`;
|
||||||
|
- `query_options->minimum_evidence_count`;
|
||||||
|
- `query_options->scanset_filter`;
|
||||||
|
- `query_options->exclude_same_asset`.
|
||||||
|
|
||||||
### Granularité
|
### Excluded fields
|
||||||
|
|
||||||
La génération batch traite un ensemble de FeatureSets de manière
|
It excludes:
|
||||||
bornée et déterministe.
|
|
||||||
|
- `created_at`;
|
||||||
|
- `candidate_pair_id`;
|
||||||
|
- operational CPU/batch admission;
|
||||||
|
- processing order of unrelated source Feature Sets.
|
||||||
|
|
||||||
|
### Reuse meaning
|
||||||
|
|
||||||
|
The same fingerprint means the same generation request may be reused.
|
||||||
|
|
||||||
|
A different fingerprint means the generation request must be evaluated again. Existing canonical
|
||||||
|
Candidate Pair rows are not silently deleted merely because a different generation request is run;
|
||||||
|
idempotent persistence may reuse rows that remain selected.
|
||||||
|
|
||||||
|
## Project batch generation
|
||||||
|
|
||||||
### API
|
### API
|
||||||
|
|
||||||
|
|
@ -181,210 +229,352 @@ Lardon3DVisualIndexResult lardon3d_candidate_pair_generate_batch(
|
||||||
uint64_t *last_feature_set_id);
|
uint64_t *last_feature_set_id);
|
||||||
```
|
```
|
||||||
|
|
||||||
### Algorithme
|
### Ordering and bounds
|
||||||
|
|
||||||
1. Lister les FeatureSets par pages de 64
|
The project batch path:
|
||||||
2. Pour chaque FeatureSet :
|
|
||||||
- Appeler `lardon3d_candidate_pair_generate()`
|
|
||||||
- Accumuler les statistiques
|
|
||||||
- Mettre à jour le curseur
|
|
||||||
3. Retourner les totaux et le dernier FeatureSet traité
|
|
||||||
|
|
||||||
### Ordre de traitement
|
- pages Feature Sets in bounded pages;
|
||||||
|
- processes source Feature Sets in increasing `feature_set_id`;
|
||||||
|
- never assumes IDs are contiguous;
|
||||||
|
- keeps top-K bounded by the Visual Index contract;
|
||||||
|
- keeps query/result memory bounded;
|
||||||
|
- returns the last processed Feature Set for restart.
|
||||||
|
|
||||||
Feature Sets traités en ordre croissant de `feature_set_id`.
|
A pair selected from multiple sources is persisted once because canonical pair identity is unique.
|
||||||
|
|
||||||
### Déduplication
|
The historical single-source and project-batch APIs remain valid. They are not, by themselves, the
|
||||||
|
complete current resource-execution description of the durable Task.
|
||||||
|
|
||||||
Une paire produite depuis plusieurs sources n'existe qu'une fois.
|
## Durable Task
|
||||||
La deduplication est assurée par `find avant create`.
|
|
||||||
|
|
||||||
### Bornes
|
|
||||||
|
|
||||||
- Un seul FeatureSet traité à la fois
|
|
||||||
- Top-K borné par requête
|
|
||||||
- Mémoire bornée : allocation `top_k * sizeof(candidate)` par requête
|
|
||||||
- Pagination bornée (64 FeatureSets par page)
|
|
||||||
|
|
||||||
### Reprise
|
|
||||||
|
|
||||||
Le curseur `after_feature_set_id` permet la reprise après interruption.
|
|
||||||
La fonction retourne le dernier FeatureSet traité.
|
|
||||||
|
|
||||||
## Tâche durable
|
|
||||||
|
|
||||||
### Task Kind
|
### Task Kind
|
||||||
|
|
||||||
`candidate_pair.generate` v1 — **IMPLEMENTED**.
|
```text
|
||||||
|
candidate_pair.generate/1
|
||||||
### Unité de travail
|
|
||||||
|
|
||||||
Un membership FeatureSet source du Visual Index et sa requête associée. Chaque
|
|
||||||
séquence traite un lot borné de memberships (1 à 64 selon le contrat Governor),
|
|
||||||
en ordre croissant d'ID sans supposer des IDs contigus.
|
|
||||||
|
|
||||||
### Checkpoint
|
|
||||||
|
|
||||||
Curseur `after_feature_set_id` persisté dans `candidate_pair_generate_tasks`.
|
|
||||||
Checkpoint sauvé après chaque lot via
|
|
||||||
`lardon3d_project_checkpoint_candidate_pair_generate_task()`.
|
|
||||||
|
|
||||||
### Reprise
|
|
||||||
|
|
||||||
Reprise idempotente : le champ `after_feature_set_id` est rechargé depuis la
|
|
||||||
DB, et les paires déjà persistées sont ignorées par `find avant create`.
|
|
||||||
À l'ouverture du projet, la tâche est automatiquement restaurée via la
|
|
||||||
registry production et resoumise à la queue.
|
|
||||||
|
|
||||||
Les anciens snapshots v1 produits avec l'estimation opérationnelle exacte
|
|
||||||
128 Kio fixes, 64 Kio par item, lot 1–64, CPU 1, IO 1 et GPU 0 sont normalisés
|
|
||||||
éphémèrement par la registry à la forme courante CPU64 avant admission. La
|
|
||||||
forme historique immédiatement précédente CPU12/256 Kio fixes/64 Kio par item
|
|
||||||
est elle aussi reconnue exactement. Le
|
|
||||||
snapshot durable original reste la source du reconstructeur ; aucun checkpoint
|
|
||||||
d'estimation seule n'est stagé, promu ou publié sous le même résumé. Une panne
|
|
||||||
pré-terminale répète donc cette normalisation exacte. Aucun autre snapshot,
|
|
||||||
curseur ou paramètre scientifique n'est réinterprété.
|
|
||||||
|
|
||||||
### Intégration Task/Queue/Governor
|
|
||||||
|
|
||||||
La tâche utilise le runtime générique via le pattern standard :
|
|
||||||
- Estimation opérationnelle (256 Kio fixes, 8 Mio par item, lot 1–64).
|
|
||||||
La Queue conserve un callback ; jusqu'à soixante-quatre participants CPU
|
|
||||||
admis peuvent calculer une fenêtre interne bornée sans modifier l'identité
|
|
||||||
scientifique.
|
|
||||||
- CPU et lot sont essayés ensemble (`1/1`, `2/2`, puis les paliers sûrs) :
|
|
||||||
un CPU supplémentaire avec un lot d'un seul membership ne peut exercer aucun
|
|
||||||
participant supplémentaire et ne constitue donc pas une mesure de scaling.
|
|
||||||
- Réservation CPU + IO avant exécution
|
|
||||||
- `lardon3d_task_sequence_break()` entre chaque lot pour réadmission Governor
|
|
||||||
- Callback terminal checkpoint après `COMPLETED`/`FAILED`/`CANCELLED`
|
|
||||||
- Reconstruction depuis `Lardon3DProjectDbCandidatePairGenerateTask`
|
|
||||||
|
|
||||||
### API
|
|
||||||
|
|
||||||
```c
|
|
||||||
Lardon3DTask *lardon3d_project_create_candidate_pair_generate_task(
|
|
||||||
Lardon3DAppState *state, uint64_t visual_index_id,
|
|
||||||
const Lardon3DVisualIndexQueryOptions *query_options, uint64_t *task_id);
|
|
||||||
bool lardon3d_project_enqueue_candidate_pair_generate(
|
|
||||||
Lardon3DAppState *state, uint64_t visual_index_id,
|
|
||||||
const Lardon3DVisualIndexQueryOptions *query_options, uint64_t *task_id);
|
|
||||||
bool lardon3d_candidate_pair_generate_reconstruct(
|
|
||||||
const Lardon3DTaskDurableSnapshot *snapshot, void *context,
|
|
||||||
Lardon3DTaskKindBinding *binding);
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Concurrence
|
Status: **IMPLEMENTED**.
|
||||||
|
|
||||||
### Garantie actuelle
|
### Durable unit
|
||||||
|
|
||||||
La Queue conserve un callback actif. À l'intérieur de la Task, jusqu'à soixante-quatre
|
The durable cursor is `after_feature_set_id` in `candidate_pair_generate_tasks`.
|
||||||
threads CPU admis calculent en parallèle une fenêtre d'au plus deux sources par
|
|
||||||
thread. Chaque participant possède un handle DB de lecture privé. Le thread
|
|
||||||
propriétaire publie ensuite seul et dans l'ordre canonique des sources. Voir le
|
|
||||||
[contrat de parallélisme interne](internal_parallelism.md).
|
|
||||||
|
|
||||||
### Atomicité
|
The Task consumes a bounded ordered set of Visual Index source memberships. A sequence handles an
|
||||||
|
admitted bounded batch, publishes the canonical pair decisions, persists the cursor, checkpoints and
|
||||||
|
returns through `lardon3d_task_sequence_break()` before the next Governor admission.
|
||||||
|
|
||||||
Les workers internes ne créent aucune paire. Après leur jointure, le
|
### Checkpoint and restart
|
||||||
propriétaire applique seul `find avant create`. La contrainte UNIQUE reste une
|
|
||||||
protection persistante, pas un mécanisme d'ordonnancement parallèle.
|
|
||||||
|
|
||||||
### Limites
|
The Task checkpoints through:
|
||||||
|
|
||||||
Le pattern `find + create` n'est pas atomique entre les deux appels. Le chemin
|
```text
|
||||||
de Task n'introduit aucun writer concurrent ; la sémantique existante reste
|
lardon3d_project_checkpoint_candidate_pair_generate_task()
|
||||||
inchangée pour les autres appelants éventuels.
|
|
||||||
|
|
||||||
## Bornes et ressources
|
|
||||||
|
|
||||||
### Top-K
|
|
||||||
|
|
||||||
`top_k <= 256` (LARDON3D_VISUAL_INDEX_TOP_K_MAX)
|
|
||||||
|
|
||||||
### Mémoire
|
|
||||||
|
|
||||||
Allocation de requête bornée par le top-K, plus un résultat de propositions
|
|
||||||
borné par le même maximum. La Task limite sa fenêtre à 24 sources et annonce
|
|
||||||
256 Kio fixes plus 64 Kio par item au Governor.
|
|
||||||
|
|
||||||
### Complexité
|
|
||||||
|
|
||||||
- O(top_k) par requête (lectures Visual Index)
|
|
||||||
- O(1) par paire (DB write)
|
|
||||||
- Pas de structure O(N²)
|
|
||||||
|
|
||||||
## Fingerprint de génération
|
|
||||||
|
|
||||||
### Composants
|
|
||||||
|
|
||||||
Le fingerprint identifie une génération Candidate Pair unique :
|
|
||||||
|
|
||||||
```c
|
|
||||||
void lardon3d_candidate_pair_generation_fingerprint(
|
|
||||||
uint64_t visual_index_id, uint64_t source_feature_set_id,
|
|
||||||
const Lardon3DVisualIndexQueryOptions *query_options,
|
|
||||||
unsigned char fingerprint[32]);
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Éléments inclus
|
Restart:
|
||||||
|
|
||||||
- `visual_index_id` : Visual Index utilisé
|
1. restores the generic Task snapshot;
|
||||||
- `source_feature_set_id` : Feature Set source
|
2. loads the typed Candidate Pair Task payload;
|
||||||
- `query_options->top_k` : nombre de candidats par requête
|
3. restores `after_feature_set_id`;
|
||||||
- `query_options->minimum_evidence_count` : filtre minimum
|
4. reconstructs the production binding through the Task Kind registry;
|
||||||
- `query_options->scanset_filter` : filtre ScanSet
|
5. resubmits through the normal Queue/Governor path;
|
||||||
- `query_options->exclude_same_asset` : exclusion même asset
|
6. reuses already persisted Candidate Pairs idempotently.
|
||||||
|
|
||||||
### Éléments exclus (volontairement)
|
A crash may therefore repeat work after the last durable cursor, but it must not invent a second
|
||||||
|
scientific pair identity.
|
||||||
|
|
||||||
- `created_at` : informatif, pas fonctionnel
|
### Historical resource descriptors
|
||||||
- `candidate_pair_id` : identifiant technique
|
|
||||||
- Ordre des Feature Sets traités en batch
|
|
||||||
|
|
||||||
### Stabilité
|
Older durable snapshots are accepted only through exact compatibility shapes already recognized by the
|
||||||
|
registry.
|
||||||
|
|
||||||
Le fingerprint est stable pour mêmes entrées et configuration.
|
Historical forms include the exact earlier descriptors documented by the implementation, including:
|
||||||
Un changement de configuration produit un fingerprint différent.
|
|
||||||
|
|
||||||
### Relation avec l'invalidation
|
|
||||||
|
|
||||||
Un fingerprint différent signifie que le travail doit être recalculé.
|
|
||||||
Le même fingerprint signifie que le résultat peut être réutilisé.
|
|
||||||
|
|
||||||
### Relation avec la reprise
|
|
||||||
|
|
||||||
Le fingerprint permet de vérifier qu'une reprise utilise la même
|
|
||||||
configuration que l'originale.
|
|
||||||
|
|
||||||
## Limites connues
|
|
||||||
|
|
||||||
1. Score non persistant (par design)
|
|
||||||
2. La compaction de segments Visual Index n'est pas implémentée
|
|
||||||
3. Aucun Matcher consommateur des paires
|
|
||||||
|
|
||||||
## Statut
|
|
||||||
|
|
||||||
**IMPLEMENTED** — génération single-source, persistance,
|
|
||||||
canonicalisation, idempotence, batch projet, fingerprint et
|
|
||||||
réutilisation/invalidation.
|
|
||||||
|
|
||||||
**IMPLEMENTED** — tâche durable `candidate_pair.generate` v1
|
|
||||||
via le runtime/Queue générique, avec estimation immuable,
|
|
||||||
checkpoint par curseur, reprise idempotente et intégration
|
|
||||||
dans la registry production.
|
|
||||||
|
|
||||||
## Relation avec le pipeline
|
|
||||||
|
|
||||||
Le Candidate Pair Generator est l'étape E du pipeline de reconstruction :
|
|
||||||
|
|
||||||
|
```text
|
||||||
|
128 KiB fixed
|
||||||
|
64 KiB per item
|
||||||
|
batch 1..64
|
||||||
|
CPU1
|
||||||
|
IO1
|
||||||
|
GPU0
|
||||||
```
|
```
|
||||||
Feature Store (C)
|
|
||||||
↓
|
and the later exact historical CPU12 / 256 KiB fixed / 64 KiB-per-item form.
|
||||||
Visual Index (D)
|
|
||||||
↓
|
Those shapes are restart compatibility evidence. They are not the current resource model and must not
|
||||||
Candidate Pair Generator (E) ← CE DOCUMENT
|
be copied into new Task creation.
|
||||||
↓
|
|
||||||
Matching (F) — HORS SCOPE
|
The original durable snapshot remains the source supplied to reconstruction. Compatibility
|
||||||
|
normalization is ephemeral and does not rewrite the persisted checkpoint or Candidate scientific
|
||||||
|
identity.
|
||||||
|
|
||||||
|
## Current resource contract
|
||||||
|
|
||||||
|
### Current Task estimate
|
||||||
|
|
||||||
|
The current validated Candidate Pair Task declares approximately:
|
||||||
|
|
||||||
|
```text
|
||||||
|
fixed RAM 256 KiB
|
||||||
|
per-item RAM 8 MiB
|
||||||
|
batch range 1..64
|
||||||
|
GPU demand 0
|
||||||
|
IO demand bounded by the existing Task estimate
|
||||||
|
CPU demand reducible and bounded by the host compute pool
|
||||||
|
```
|
||||||
|
|
||||||
|
The exact implementation constants remain authoritative in source. This documentation records the
|
||||||
|
current validated capability and intentionally does not retain the obsolete 24-source / 64-KiB-per-item
|
||||||
|
description as current policy.
|
||||||
|
|
||||||
|
### Coupled CPU and batch admission
|
||||||
|
|
||||||
|
Candidate Pair generation has independent source work, but additional CPU cannot exercise additional
|
||||||
|
participants if the admitted source batch remains one.
|
||||||
|
|
||||||
|
For this Task, CPU and batch scaling are therefore coupled during adaptation. Conceptually:
|
||||||
|
|
||||||
|
```text
|
||||||
|
CPU1 / batch1
|
||||||
|
CPU2 / batch2
|
||||||
|
then larger safe coupled rungs
|
||||||
|
```
|
||||||
|
|
||||||
|
subject to:
|
||||||
|
|
||||||
|
- the Task's declared maximums;
|
||||||
|
- the host compute pool;
|
||||||
|
- current Governor pressure;
|
||||||
|
- measured usefulness;
|
||||||
|
- current admission policy.
|
||||||
|
|
||||||
|
This coupling fixes an operational scaling defect. It does not modify Candidate Pair scientific
|
||||||
|
identity, query options, top-K behavior, publication order or persistence.
|
||||||
|
|
||||||
|
### Canonical resource principles
|
||||||
|
|
||||||
|
```text
|
||||||
|
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||||
|
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||||
|
```
|
||||||
|
|
||||||
|
The interactive host reserve is preserved first. After that reserve and all safety constraints are
|
||||||
|
satisfied, safe and useful compute capacity should not be left idle merely to preserve an old
|
||||||
|
single-thread measurement.
|
||||||
|
|
||||||
|
Reference-host values are observations, not portable constants.
|
||||||
|
|
||||||
|
### Atomicity is not serialism
|
||||||
|
|
||||||
|
One source query and one Candidate Pair publication decision remain bounded scientific/transactional
|
||||||
|
units.
|
||||||
|
|
||||||
|
That does not imply that independent source preparation must run serially.
|
||||||
|
|
||||||
|
```text
|
||||||
|
PER_ITEM_ATOMICITY_REQUIRES_CROSS_ITEM_SERIALISM=NO
|
||||||
|
OWNER_ONLY_PUBLICATION_REQUIRES_SERIAL_PREPARATION=NO
|
||||||
|
```
|
||||||
|
|
||||||
|
## Internal concurrency
|
||||||
|
|
||||||
|
The Task Queue still owns one active callback. Candidate Pair uses bounded parallelism inside that
|
||||||
|
callback.
|
||||||
|
|
||||||
|
The validated shape is:
|
||||||
|
|
||||||
|
```text
|
||||||
|
one admitted Candidate owner Task
|
||||||
|
-> bounded source window
|
||||||
|
-> bounded CPU participants
|
||||||
|
-> private read-side preparation
|
||||||
|
-> join
|
||||||
|
-> owner publishes in canonical source order
|
||||||
|
```
|
||||||
|
|
||||||
|
Participants do not create Candidate Pair rows.
|
||||||
|
|
||||||
|
Each participant uses its allowed private read-side state. After participants join, the owner alone
|
||||||
|
performs the canonical `find` / create publication sequence.
|
||||||
|
|
||||||
|
The SQL `UNIQUE(image_id_a, image_id_b)` constraint remains a persistent integrity guard, not a
|
||||||
|
parallel scheduling primitive.
|
||||||
|
|
||||||
|
No second global scheduler, global worker pool or parallel SQLite writer subsystem is introduced.
|
||||||
|
|
||||||
|
## Persistence race boundary
|
||||||
|
|
||||||
|
The public pattern:
|
||||||
|
|
||||||
|
```text
|
||||||
|
find candidate pair
|
||||||
|
-> create candidate pair when absent
|
||||||
|
```
|
||||||
|
|
||||||
|
is not a general atomic compare-and-insert primitive across arbitrary concurrent writers.
|
||||||
|
|
||||||
|
The production durable Candidate Task avoids introducing competing pair writers: participant work is
|
||||||
|
read/preparation only and owner publication is serialized.
|
||||||
|
|
||||||
|
Other callers must not infer a stronger concurrency guarantee from the Task's owner-only publication
|
||||||
|
model.
|
||||||
|
|
||||||
|
## Resource complexity
|
||||||
|
|
||||||
|
### Query bound
|
||||||
|
|
||||||
|
```text
|
||||||
|
top_k <= LARDON3D_VISUAL_INDEX_TOP_K_MAX = 256
|
||||||
|
```
|
||||||
|
|
||||||
|
### Memory
|
||||||
|
|
||||||
|
Current Task admission uses the current estimate:
|
||||||
|
|
||||||
|
```text
|
||||||
|
256 KiB fixed + 8 MiB per admitted item
|
||||||
|
```
|
||||||
|
|
||||||
|
with batch bounded to `1..64`.
|
||||||
|
|
||||||
|
Actual admission can be reduced by the Governor and host compute pool. Swap, zram and external scratch
|
||||||
|
never enlarge admitted RAM.
|
||||||
|
|
||||||
|
Candidate Pair currently has no authoritative scratch consumer.
|
||||||
|
|
||||||
|
### Algorithmic shape
|
||||||
|
|
||||||
|
The subsystem does not allocate an O(N^2) project pair matrix.
|
||||||
|
|
||||||
|
Per source, bounded work is dominated by:
|
||||||
|
|
||||||
|
- bounded Visual Index query;
|
||||||
|
- bounded top-K filtering;
|
||||||
|
- canonical pair lookup/publication.
|
||||||
|
|
||||||
|
Project traversal is paged and restartable.
|
||||||
|
|
||||||
|
## GPU policy
|
||||||
|
|
||||||
|
Candidate Pair currently remains CPU.
|
||||||
|
|
||||||
|
The validated GPU audit classified it as:
|
||||||
|
|
||||||
|
```text
|
||||||
|
CANDIDATE_GPU=REJECTED_WITH_MEASURED_REASON
|
||||||
|
```
|
||||||
|
|
||||||
|
The workload is dominated by Visual Index access, filtering, branching and deterministic ordered SQLite
|
||||||
|
publication, and no validated GPU primitive currently preserves the complete Candidate contract with a
|
||||||
|
useful measured advantage.
|
||||||
|
|
||||||
|
This rejection does not authorize CPU serialism. Safe useful CPU parallelism remains required by the
|
||||||
|
canonical resource policy.
|
||||||
|
|
||||||
|
## Real A6000 evidence
|
||||||
|
|
||||||
|
The retained real A6000 pre-SfM execution exercised the current Candidate path before Matcher, GV and
|
||||||
|
Tracks.
|
||||||
|
|
||||||
|
It produced:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Candidate Pairs = 38,420
|
||||||
|
Match Results = 38,420
|
||||||
|
```
|
||||||
|
|
||||||
|
The later checkpoint is:
|
||||||
|
|
||||||
|
```text
|
||||||
|
real-a6000-pre-sfm-2026-09-02
|
||||||
|
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||||
|
```
|
||||||
|
|
||||||
|
Candidate replay at the retained checkpoint created no new Candidate work. The proof continued through
|
||||||
|
Matcher, Geometric Verifier v3 and Tracks, then stopped before real Sparse SfM.
|
||||||
|
|
||||||
|
This evidence validates the operational Candidate path on that project. It does not change Candidate
|
||||||
|
Pair v1 identity or the Project DB v8/v9 historical contracts.
|
||||||
|
|
||||||
|
## Relationship to current Project DB
|
||||||
|
|
||||||
|
The Candidate Pair model was introduced in Project DB v8 and its durable Task payload in v9.
|
||||||
|
|
||||||
|
The current schema head is v25:
|
||||||
|
|
||||||
|
```text
|
||||||
|
v22 selected scientific execution foundation
|
||||||
|
v23 generic optical-context overlay
|
||||||
|
v24 raw.develop.batch/1 persistence
|
||||||
|
v25 features.extract.batch/1 persistence
|
||||||
|
```
|
||||||
|
|
||||||
|
Those later additive migrations do not reinterpret Candidate Pair rows.
|
||||||
|
|
||||||
|
## Known limits
|
||||||
|
|
||||||
|
Current limits and non-goals include:
|
||||||
|
|
||||||
|
- retrieval score is not persisted in `candidate_pairs` by design;
|
||||||
|
- Visual Index segment compaction remains separate work;
|
||||||
|
- no generic DAG dependency scheduler is introduced by Candidate Pair;
|
||||||
|
- no Candidate GPU backend is currently validated;
|
||||||
|
- no scratch/spill path is authoritative for Candidate Pair;
|
||||||
|
- arbitrary concurrent pair writers are not provided by the Task owner-publication model.
|
||||||
|
|
||||||
|
The downstream Matcher is implemented; it is not a missing Candidate Pair feature.
|
||||||
|
|
||||||
|
## Pipeline relationship
|
||||||
|
|
||||||
|
```text
|
||||||
|
Feature Store
|
||||||
|
|
|
||||||
|
v
|
||||||
|
Visual Index
|
||||||
|
|
|
||||||
|
v
|
||||||
|
Candidate Pair Generator <- this document
|
||||||
|
|
|
||||||
|
v
|
||||||
|
Matcher
|
||||||
|
|
|
||||||
|
v
|
||||||
|
Geometric Verification
|
||||||
|
|
|
||||||
|
v
|
||||||
|
Tracks
|
||||||
|
```
|
||||||
|
|
||||||
|
Candidate Pair selects plausible image pairs. Matcher computes descriptor-level correspondence evidence.
|
||||||
|
Geometric Verification validates geometry. Track Builder creates multi-view observation tracks. These
|
||||||
|
scientific responsibilities remain separate.
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
```text
|
||||||
|
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||||
|
|
||||||
|
CANDIDATE_PAIR_MODEL=v1
|
||||||
|
CANDIDATE_PAIR_MODEL_STATUS=IMPLEMENTED
|
||||||
|
|
||||||
|
CANDIDATE_PAIR_TASK=candidate_pair.generate/1
|
||||||
|
CANDIDATE_PAIR_TASK_STATUS=IMPLEMENTED
|
||||||
|
|
||||||
|
CANDIDATE_PAIR_PERSISTENCE_VERSION=v8
|
||||||
|
CANDIDATE_PAIR_TASK_PERSISTENCE_VERSION=v9
|
||||||
|
|
||||||
|
CANDIDATE_CURRENT_FIXED_RAM=256_KiB
|
||||||
|
CANDIDATE_CURRENT_PER_ITEM_RAM=8_MiB
|
||||||
|
CANDIDATE_CURRENT_BATCH_RANGE=1..64
|
||||||
|
CANDIDATE_CPU_BATCH_ADAPTATION=COUPLED
|
||||||
|
CANDIDATE_GPU=REJECTED_WITH_MEASURED_REASON
|
||||||
|
CANDIDATE_SCRATCH_CONSUMER=NO
|
||||||
|
|
||||||
|
PER_ITEM_ATOMICITY_REQUIRES_CROSS_ITEM_SERIALISM=NO
|
||||||
|
OWNER_ONLY_PUBLICATION_REQUIRES_SERIAL_PREPARATION=NO
|
||||||
|
|
||||||
|
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||||
|
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||||
|
|
||||||
|
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||||
```
|
```
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue