feat: finalize feature store v1
This commit is contained in:
parent
0f99415317
commit
b12af521e5
32 changed files with 5686 additions and 2973 deletions
|
|
@ -23,10 +23,11 @@ Lardon3D ne vise pas simplement "dossier de photos → objet 3D", mais "ensemble
|
||||||
- **Project** : cycle de vie persistant, identité stable et Project Database ouverte
|
- **Project** : cycle de vie persistant, identité stable et Project Database ouverte
|
||||||
- **Import** : premier task kind de production, exécuté par la file générique en lots bornés et reprenables
|
- **Import** : premier task kind de production, exécuté par la file générique en lots bornés et reprenables
|
||||||
- **ScanSet / Image Catalog v1** : acquisitions, images logiques, provenance et assets SHA-256 persistants et paginés
|
- **ScanSet / Image Catalog v1** : acquisitions, images logiques, provenance et assets SHA-256 persistants et paginés
|
||||||
|
- **Feature Store v1** : extraction ORB réelle, assets binaires versionnés et lecture bornée
|
||||||
- **Image View** : vues triées et filtrées pour la TUI
|
- **Image View** : vues triées et filtrées pour la TUI
|
||||||
- **Task** : moteur de tâches avec pause/reprise, annulation et séquences
|
- **Task** : moteur de tâches avec pause/reprise, annulation et séquences
|
||||||
- **Task Checkpoint v1** : snapshot durable, fichier atomique et reprise sûre
|
- **Task Checkpoint v1** : snapshot durable, fichier atomique et reprise sûre
|
||||||
- **Project Database v4** : tâches, ScanSets, catalogue image, assets et migrations SQLite
|
- **Project Database v5** : tâches, ScanSets, images, Feature Sets/assets et migrations SQLite
|
||||||
- **Task Kind Registry** : identité métier durable et reconstruction runtime explicite
|
- **Task Kind Registry** : identité métier durable et reconstruction runtime explicite
|
||||||
- **Recovery projet** : reprise automatique sélective et bornée des imports récupérables
|
- **Recovery projet** : reprise automatique sélective et bornée des imports récupérables
|
||||||
- **Task Queue** : file FIFO avec sélection adaptative et backpressure
|
- **Task Queue** : file FIFO avec sélection adaptative et backpressure
|
||||||
|
|
@ -102,6 +103,7 @@ Acquisitions
|
||||||
- [Pipeline de reconstruction](docs/architecture/reconstruction_pipeline.md)
|
- [Pipeline de reconstruction](docs/architecture/reconstruction_pipeline.md)
|
||||||
- [Persistance](docs/architecture/persistence.md)
|
- [Persistance](docs/architecture/persistence.md)
|
||||||
- [Base de données projet](docs/architecture/project_database.md)
|
- [Base de données projet](docs/architecture/project_database.md)
|
||||||
|
- [Feature Store](docs/architecture/feature_store.md)
|
||||||
- [Viewer](docs/architecture/viewer.md)
|
- [Viewer](docs/architecture/viewer.md)
|
||||||
- [Revue des fondations](docs/architecture/foundation_review.md)
|
- [Revue des fondations](docs/architecture/foundation_review.md)
|
||||||
|
|
||||||
|
|
|
||||||
117
docs/architecture/feature_store.md
Normal file
117
docs/architecture/feature_store.md
Normal file
|
|
@ -0,0 +1,117 @@
|
||||||
|
# Feature Store v1
|
||||||
|
|
||||||
|
## Rôle et modèle
|
||||||
|
|
||||||
|
**IMPLEMENTED.** Le Feature Store est la mémoire visuelle locale persistante.
|
||||||
|
Un `FeatureSet` logique appartient à une `image_id` et identifie exactement
|
||||||
|
`orb` version 1 plus une configuration canonique. Il référence un
|
||||||
|
`FeatureAsset` physique immutable. Deux images logiques portant le même contenu
|
||||||
|
et la même configuration gardent deux `feature_set_id`, mais partagent le même
|
||||||
|
asset lorsque leurs octets sont identiques. Les futurs matches référenceront
|
||||||
|
`feature_set_id + feature_index`; ces deux valeurs sont immuables après
|
||||||
|
publication.
|
||||||
|
|
||||||
|
`feature_set_id` et `feature_asset_id` sont des identités SQLite
|
||||||
|
`AUTOINCREMENT`. Une identité validée n'est jamais réattribuée. SQLite ne
|
||||||
|
contient que les relations, empreintes, tailles et paramètres ; les tableaux de
|
||||||
|
points et descripteurs restent hors DB.
|
||||||
|
|
||||||
|
## Extracteur et configuration
|
||||||
|
|
||||||
|
L'extracteur production est ORB d'OpenCV 5, derrière une façade C. Aucune
|
||||||
|
exception C++ ni `cv::Mat` ne traverse cette frontière. La registry statique
|
||||||
|
expose `features.extract`, version 1. Une tâche traite exactement une image.
|
||||||
|
|
||||||
|
La configuration v1 contient trois `uint32` : `max_features` (1..8192),
|
||||||
|
`pyramid_levels` (1..16) et `fast_threshold` (1..255). Son fingerprint SHA-256
|
||||||
|
porte sur les 24 octets canoniques `L3DORBP1`, version et trois entiers
|
||||||
|
little-endian. Il ne dépend ni du padding, ni de la locale. Le contrat ORB v1
|
||||||
|
produit 32 octets binaires par point.
|
||||||
|
|
||||||
|
OpenCV n'est pas configuré via un état global par Lardon3D. Le scheduler actuel
|
||||||
|
n'a qu'un worker ; OpenCV peut néanmoins employer son backend parallèle
|
||||||
|
interne. Le contrôle fin de ce parallélisme devra précéder les futurs pools de
|
||||||
|
workers.
|
||||||
|
|
||||||
|
La limite de 100 000 000 pixels est vérifiée après `cv::imread` : l'API utilisée
|
||||||
|
ne fournit pas de sonde de dimensions multi-format fiable sans décodage. Le pic
|
||||||
|
mémoire du décodage peut donc précéder le rejet. Lardon3D ne revendique pas de
|
||||||
|
bornage pré-décodage et n'embarque pas un parseur JPEG/PNG parallèle.
|
||||||
|
|
||||||
|
## Feature File v1
|
||||||
|
|
||||||
|
Le fichier est little-endian et exige IEEE-754 binary32. Sa taille maximale est
|
||||||
|
16 Mio et son nombre maximal de points 8192.
|
||||||
|
|
||||||
|
| Offset | Taille | Champ |
|
||||||
|
|---:|---:|---|
|
||||||
|
| 0 | 8 | magic `L3DFEAT\0` |
|
||||||
|
| 8 | 4 | format version = 1 |
|
||||||
|
| 12 | 4 | header size = 160 |
|
||||||
|
| 16 | 4 | feature count |
|
||||||
|
| 20 | 4 | descriptor dimension = 32 |
|
||||||
|
| 24 | 4 | descriptor type = U8 |
|
||||||
|
| 28 | 4 | keypoint record size = 24 |
|
||||||
|
| 32,36 | 4+4 | largeur, hauteur décodées |
|
||||||
|
| 40,48,56 | 8+8+8 | offsets keypoints/descriptors, taille totale |
|
||||||
|
| 64 | 32 | SHA-256 de l'asset image source |
|
||||||
|
| 96 | 32 | fingerprint paramètres |
|
||||||
|
| 128 | 16 | `orb\0` puis zéros réservés |
|
||||||
|
| 144 | 4 | extractor version = 1 |
|
||||||
|
| 148 | 12 | réservés, zéro obligatoire |
|
||||||
|
|
||||||
|
Chaque keypoint contient six mots de 32 bits : `x`, `y`, `size`, orientation,
|
||||||
|
response en binary32, puis octave signé. `x/y` sont en pixels de l'image telle
|
||||||
|
que décodée par OpenCV, origine en haut à gauche. `size` est le diamètre du
|
||||||
|
voisinage en pixels. L'orientation est en degrés dans `[0,360)`. L'entrée `i`
|
||||||
|
du bloc keypoints correspond exactement aux 32 octets du descriptor `i`.
|
||||||
|
|
||||||
|
Le validateur contrôle magic, version, réservés, type, dimension, bornes,
|
||||||
|
offsets, taille exacte, multiplications, SHA-256 externe et cohérence DB. Une
|
||||||
|
version future est distinguée d'une corruption lorsque le fichier et son record
|
||||||
|
DB sont cohérents. Le reader utilise `pread`, accepte au plus 256 éléments par
|
||||||
|
appel et ne charge jamais le fichier entier.
|
||||||
|
|
||||||
|
Le chemin DB est validé sous sa forme canonique exacte dérivée du SHA-256 avant
|
||||||
|
toute ouverture. Une absence retourne `NOT_FOUND`; troncature, hash divergent,
|
||||||
|
header ou métadonnées DB divergents retournent `CORRUPT`. Une version future
|
||||||
|
retourne distinctement `UNSUPPORTED_VERSION`.
|
||||||
|
|
||||||
|
## Publication et persistance
|
||||||
|
|
||||||
|
Layout : `assets/features/<2 hex>/<sha256 complet lowercase>`. Le SHA-256 porte
|
||||||
|
sur le Feature File complet. Le protocole est : temporaire local, écriture,
|
||||||
|
`fsync`, hash, `link` atomique sans écrasement, validation complète lors d'une
|
||||||
|
adoption concurrente, `fsync` du répertoire, puis transaction SQLite. Un échec
|
||||||
|
du dernier `fsync` est enregistré `PUBLISHED_NOT_DURABLE`. Un échec SQLite après
|
||||||
|
publication laisse un fichier orphelin et aucune ligne logique partielle.
|
||||||
|
Un nouvel essai qui revalide l'asset et réussit le `fsync` promeut explicitement
|
||||||
|
sa durabilité vers `DURABLE` dans la transaction DB.
|
||||||
|
|
||||||
|
Le schéma v5 sépare `feature_assets`, `feature_sets` et
|
||||||
|
`feature_extract_tasks`. L'unicité logique porte sur image, kind, version et
|
||||||
|
fingerprint. Une tâche est persistée avant enqueue, passe par la queue et le
|
||||||
|
Governor (CPU, IO, 576 Mio conservateurs, lot 1), puis checkpointée initialement
|
||||||
|
et terminalement. La reprise recommence l'image entière : il n'existe pas de
|
||||||
|
fausse reprise intra-ORB. La publication est idempotente.
|
||||||
|
|
||||||
|
Pause et annulation sont coopératives avant/après l'appel ORB ; cet appel n'est
|
||||||
|
pas interruptible. L'image gérée est rehashée avant extraction. Une image
|
||||||
|
uniforme produit légitimement un Feature Set READY vide.
|
||||||
|
|
||||||
|
Lardon3D ne garantit pas des octets ORB identiques entre versions d'OpenCV,
|
||||||
|
plateformes ou backends. L'idempotence porte sur l'environnement courant et le
|
||||||
|
contrat `extractor_version`; une évolution qui change durablement la sémantique
|
||||||
|
ou les octets exige d'auditer et, si nécessaire, d'incrémenter cette version.
|
||||||
|
|
||||||
|
## Statut
|
||||||
|
|
||||||
|
**IMPLEMENTED** — ORB réel, format v1, assets content-addressed, DB v5,
|
||||||
|
publication atomique, reader borné, task kind production et reprise automatique.
|
||||||
|
|
||||||
|
**NOT_YET_WIRED** — commande de réconciliation/scrub des orphelins, contrôle fin
|
||||||
|
du backend parallèle OpenCV, orientation EXIF, lancement automatique de
|
||||||
|
l'extraction après import et planification multi-image/DAG.
|
||||||
|
|
||||||
|
**PLANNED** — Visual Index, paires candidates, matching, vérification
|
||||||
|
géométrique, tracks et SfM.
|
||||||
|
|
@ -62,6 +62,12 @@ catalogue, le manifeste ou les images.
|
||||||
|
|
||||||
**Statut :** IMPLEMENTED
|
**Statut :** IMPLEMENTED
|
||||||
|
|
||||||
|
### Feature Store
|
||||||
|
Extraction ORB réelle par tâche persistante, Feature Sets logiques et assets
|
||||||
|
binaires content-addressed lisibles par plages bornées.
|
||||||
|
|
||||||
|
**Statut :** IMPLEMENTED
|
||||||
|
|
||||||
### Task
|
### Task
|
||||||
Moteur de tâches avec états, progression, pause/reprise coopérative,
|
Moteur de tâches avec états, progression, pause/reprise coopérative,
|
||||||
annulation, checkpoints et estimations de ressources.
|
annulation, checkpoints et estimations de ressources.
|
||||||
|
|
|
||||||
|
|
@ -108,7 +108,7 @@ par le `time_t` local avant conversion. Le format reste donc lisible entre
|
||||||
plateformes uniquement pour les valeurs communes à leurs domaines `size_t` et
|
plateformes uniquement pour les valeurs communes à leurs domaines `size_t` et
|
||||||
`time_t`.
|
`time_t`.
|
||||||
|
|
||||||
## Project Database v4
|
## Project Database v5
|
||||||
|
|
||||||
SQLite contient l'état logique interrogable et les références aux fichiers ;
|
SQLite contient l'état logique interrogable et les références aux fichiers ;
|
||||||
les checkpoints et artefacts volumineux restent externes. L'enregistrement du
|
les checkpoints et artefacts volumineux restent externes. L'enregistrement du
|
||||||
|
|
@ -147,8 +147,8 @@ code.
|
||||||
**IMPLEMENTED** — modèle durable, codec v1, lecture validée, publication
|
**IMPLEMENTED** — modèle durable, codec v1, lecture validée, publication
|
||||||
atomique et restauration sûre d'une tâche isolée.
|
atomique et restauration sûre d'une tâche isolée.
|
||||||
|
|
||||||
**IMPLEMENTED** — Project Database v4 pour identité, tâches typées, ScanSets,
|
**IMPLEMENTED** — Project Database v5 pour identité, tâches typées, ScanSets,
|
||||||
images logiques, assets SHA-256, références checkpoint et artefacts génériques.
|
images logiques, Feature Sets/assets SHA-256, checkpoints et artefacts génériques.
|
||||||
|
|
||||||
**IMPLEMENTED** — registry statique bornée et reconstruction explicite avec
|
**IMPLEMENTED** — registry statique bornée et reconstruction explicite avec
|
||||||
ownership du userdata.
|
ownership du userdata.
|
||||||
|
|
@ -205,5 +205,8 @@ retry piloté par l'utilisateur pour les sources indisponibles.
|
||||||
**NOT_YET_WIRED** — migration de la vue TUI en mémoire vers la pagination
|
**NOT_YET_WIRED** — migration de la vue TUI en mémoire vers la pagination
|
||||||
SQLite, scrub des assets et réconciliation globale des orphelins.
|
SQLite, scrub des assets et réconciliation globale des orphelins.
|
||||||
|
|
||||||
**PLANNED** — Feature Store, Visual Index, migrations v5+ et reprise avec
|
**IMPLEMENTED** — Feature Store externe versionné, immutable, borné et relié
|
||||||
|
transactionnellement à ses métadonnées SQLite après publication.
|
||||||
|
|
||||||
|
**PLANNED** — Visual Index, migrations v6+ et reprise avec
|
||||||
dépendances.
|
dépendances.
|
||||||
|
|
|
||||||
|
|
@ -168,9 +168,9 @@ référence vers le fichier checkpoint. Le fichier checkpoint validé reste la
|
||||||
source complète pour `lardon3d_task_restore()` ; la DB seule ne reconstruit
|
source complète pour `lardon3d_task_restore()` ; la DB seule ne reconstruit
|
||||||
jamais une tâche. Un écart ou un fichier invalide interdit la reprise.
|
jamais une tâche. Un écart ou un fichier invalide interdit la reprise.
|
||||||
|
|
||||||
## Schéma v4 implémenté
|
## Schéma v5 implémenté
|
||||||
|
|
||||||
- `metadata(key PRIMARY KEY, value)` contient `schema_version=4` et
|
- `metadata(key PRIMARY KEY, value)` contient `schema_version=5` et
|
||||||
`next_task_id`, prochain ID durable allouable.
|
`next_task_id`, prochain ID durable allouable.
|
||||||
- `project(singleton=1, stable_id UNIQUE, name, created_at, updated_at)` décrit
|
- `project(singleton=1, stable_id UNIQUE, name, created_at, updated_at)` décrit
|
||||||
l'unique identité logique de la DB.
|
l'unique identité logique de la DB.
|
||||||
|
|
@ -209,12 +209,12 @@ SHA-256 et le chemin asset sont déjà indexés par leurs contraintes `UNIQUE`.
|
||||||
|
|
||||||
## Ouverture et migrations
|
## Ouverture et migrations
|
||||||
|
|
||||||
Une DB vide reçoit directement le schéma v4 dans une transaction
|
Une DB vide reçoit directement le schéma v5 dans une transaction
|
||||||
`BEGIN IMMEDIATE`. Une DB v1 reçoit transactionnellement les colonnes nullable
|
`BEGIN IMMEDIATE`. Une DB v1 reçoit transactionnellement les colonnes nullable
|
||||||
`task_kind` et `task_kind_version`, puis les migrations v2→v3. Les anciennes lignes restent
|
`task_kind` et `task_kind_version`, puis les migrations v2→v3. Les anciennes lignes restent
|
||||||
`NULL/NULL`, sans type inventé et sans perte des projets, tâches, checkpoints ou
|
`NULL/NULL`, sans type inventé et sans perte des projets, tâches, checkpoints ou
|
||||||
artefacts. Une interruption ou erreur provoque un rollback complet. Les DB v1,
|
artefacts. Une interruption ou erreur provoque un rollback complet. Les DB v1,
|
||||||
v2 et v3 sont migrées séquentiellement vers v4. Une version future est refusée et une DB contenant
|
v2, v3 et v4 sont migrées séquentiellement vers v5. Une version future est refusée et une DB contenant
|
||||||
des tables sans métadonnée de version est considérée corrompue. La fonction
|
des tables sans métadonnée de version est considérée corrompue. La fonction
|
||||||
interne de migration ne connaît que `0 → 4`, `1 → 2 → 3 → 4`,
|
interne de migration ne connaît que `0 → 4`, `1 → 2 → 3 → 4`,
|
||||||
`2 → 3 → 4` et `3 → 4`.
|
`2 → 3 → 4` et `3 → 4`.
|
||||||
|
|
@ -293,7 +293,7 @@ UPDATE metadata SET value=4
|
||||||
WHERE key='schema_version' AND value=3;
|
WHERE key='schema_version' AND value=3;
|
||||||
```
|
```
|
||||||
|
|
||||||
Configuration v4 : `foreign_keys=ON`, `journal_mode=DELETE`,
|
Configuration v5 : `foreign_keys=ON`, `journal_mode=DELETE`,
|
||||||
`synchronous=FULL`, `busy_timeout=5000`. Le mode DELETE convient au propriétaire
|
`synchronous=FULL`, `busy_timeout=5000`. Le mode DELETE convient au propriétaire
|
||||||
unique actuel, évite les fichiers WAL/SHM durables et conserve la synchronisation
|
unique actuel, évite les fichiers WAL/SHM durables et conserve la synchronisation
|
||||||
forte. Le timeout borne l'attente d'un verrou externe à cinq secondes.
|
forte. Le timeout borne l'attente d'un verrou externe à cinq secondes.
|
||||||
|
|
@ -354,7 +354,7 @@ ouvert.
|
||||||
|
|
||||||
## Statut
|
## Statut
|
||||||
|
|
||||||
**IMPLEMENTED** — SQLite système, schéma v4 et migrations v1→v2→v3→v4, identité
|
**IMPLEMENTED** — SQLite système, schéma v5 et migrations v1→v2→v3→v4→v5, identité
|
||||||
projet, transactions tâche+checkpoint, pagination de reprise et artefacts
|
projet, transactions tâche+checkpoint, pagination de reprise et artefacts
|
||||||
génériques.
|
génériques.
|
||||||
|
|
||||||
|
|
@ -382,7 +382,7 @@ cataloguées », pas « images migrées ».
|
||||||
|
|
||||||
**NOT_YET_WIRED** — autosave à toutes les transitions, retry UI des sources
|
**NOT_YET_WIRED** — autosave à toutes les transitions, retry UI des sources
|
||||||
indisponibles, migration de la TUI legacy et réconciliation des fichiers
|
indisponibles, migration de la TUI legacy et réconciliation des fichiers
|
||||||
orphelins, Feature Store et Visual Index.
|
orphelins et Visual Index. Le Feature Store v1 est implémenté.
|
||||||
|
|
||||||
**PLANNED** — migrations v5+, dépendances d'artefacts, graphe géométrique et
|
**PLANNED** — migrations v5+, dépendances d'artefacts, graphe géométrique et
|
||||||
reconstruction incrémentale.
|
reconstruction incrémentale.
|
||||||
|
|
|
||||||
|
|
@ -42,15 +42,16 @@ content-addressed. Les états Feature/Matching/Reconstruction restent planifiés
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### C. Feature Store
|
### C. Feature Extraction et Feature Store
|
||||||
|
|
||||||
| Aspect | Description |
|
| Aspect | Description |
|
||||||
|--------|-------------|
|
|--------|-------------|
|
||||||
| **Index / métadonnées persistants** | Les descripteurs de features (points clés, descripteurs, orientations) sont persistés sur disque dans un format binaire compact. Le rechargement évite la re-extraction. |
|
| **Extraction** | ORB OpenCV réel, une image par tâche `features.extract`. |
|
||||||
| **Données numériques massives** | Descripteurs floats, coordonnées de keypoints : volumes potentiellement importants. Doivent être stockés de manière séquentielle et indexée. |
|
| **Métadonnées persistantes** | `FeatureSet` logique et `FeatureAsset` physique content-addressed dans ProjectDb v5. |
|
||||||
| **Formats adaptés et bornés** | Format binaire avec en-tête (version, nombre de features, dimensions). Borné par le budget RAM du governor : si le store dépasse la capacité, seuls les N plus récents sont en mémoire. |
|
| **Données numériques massives** | Keypoints et descripteurs U8×32 restent hors SQLite dans Feature File v1. |
|
||||||
|
| **Lecture bornée** | Le reader lit au plus 256 features par plage et ne charge pas tout le fichier. |
|
||||||
|
|
||||||
**Statut :** PLANNED — aucune implémentation existante.
|
**Statut :** IMPLEMENTED v1 — extraction, publication atomique, reprise et reader borné.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -212,7 +213,11 @@ Image Catalog (B) ──► Feature Store (C)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Statut : PLANNED (vision architecturale)
|
## Statut du pipeline
|
||||||
|
|
||||||
|
Import, Image Catalog, Feature Extraction et Feature Store sont
|
||||||
|
**IMPLEMENTED**. Visual Index, Candidate Pair, Matching, Tracks et SfM sont
|
||||||
|
**PLANNED**.
|
||||||
|
|
||||||
Ce document décrit la vision architecturale cible du pipeline de
|
Ce document décrit la vision architecturale cible du pipeline de
|
||||||
reconstruction. Les modules listés ici ne sont pas tous implémentés.
|
reconstruction. Les modules listés ici ne sont pas tous implémentés.
|
||||||
|
|
|
||||||
|
|
@ -73,6 +73,11 @@ Le Resource Governor est l'unique propriétaire des budgets (RAM, GPU, CPU, IO).
|
||||||
l'échantillon peut conserver taille/durée mais n'alimente jamais l'adaptation
|
l'échantillon peut conserver taille/durée mais n'alimente jamais l'adaptation
|
||||||
mémoire.
|
mémoire.
|
||||||
- Pas de communication inter-classes de tâches
|
- Pas de communication inter-classes de tâches
|
||||||
|
- `features.extract` réserve un lot de 1, un thread CPU et un slot I/O, avec
|
||||||
|
64 Mio fixes et 512 Mio par image. Cette estimation conservatrice couvre le
|
||||||
|
chemin actuel sans prétendre mesurer les allocations internes d'OpenCV.
|
||||||
|
`record_batch` couvre la validation source, le décodage, ORB, la publication
|
||||||
|
et la finalisation DB ; `peak_memory_bytes == 0` signifie « mesure inconnue ».
|
||||||
|
|
||||||
## Limites actuelles
|
## Limites actuelles
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -26,7 +26,7 @@ fin de l'exécution. Le constructeur métier n'est jamais appelé sous mutex DB.
|
||||||
|
|
||||||
## Persistance et legacy
|
## Persistance et legacy
|
||||||
|
|
||||||
Le checkpoint générique reste en version 1. Project Database v4 conserve le
|
Le checkpoint générique reste en version 1. Project Database v5 conserve le
|
||||||
kind/version ; les lignes migrées depuis v1 restent `NULL/NULL` et sont classées
|
kind/version ; les lignes migrées depuis v1 restent `NULL/NULL` et sont classées
|
||||||
`LEGACY_UNTYPED`. Un kind inconnu ou une version non supportée reste inspectable
|
`LEGACY_UNTYPED`. Un kind inconnu ou une version non supportée reste inspectable
|
||||||
mais inexécutable. Aucun type n'est inventé et aucun code n'est sélectionné par
|
mais inexécutable. Aucun type n'est inventé et aucun code n'est sélectionné par
|
||||||
|
|
@ -46,5 +46,8 @@ pour restaurer hors mutex DB et transférer chaque tâche acceptée à la queue.
|
||||||
|
|
||||||
**NOT_YET_WIRED** — autosave complet et réconciliation orpheline.
|
**NOT_YET_WIRED** — autosave complet et réconciliation orpheline.
|
||||||
|
|
||||||
**PLANNED** — kinds des tâches ScanSet, Image Catalog, Feature Store, Visual
|
**IMPLEMENTED** — `features.extract` version 1 reconstruit une extraction ORB
|
||||||
|
depuis `image_id` et ses paramètres bornés.
|
||||||
|
|
||||||
|
**PLANNED** — kinds des tâches Visual
|
||||||
Index et reconstruction lorsque ces traitements existeront réellement.
|
Index et reconstruction lorsque ces traitements existeront réellement.
|
||||||
|
|
|
||||||
|
|
@ -106,7 +106,7 @@ localement : son userdata est détruit, sans callback terminal ni écriture
|
||||||
durable d'une fausse annulation. Une annulation explicitement demandée conserve
|
durable d'une fausse annulation. Une annulation explicitement demandée conserve
|
||||||
le contrat de notification terminale.
|
le contrat de notification terminale.
|
||||||
|
|
||||||
La Project Database v4 peut enregistrer transactionnellement un résumé
|
La Project Database v5 peut enregistrer transactionnellement un résumé
|
||||||
`Lardon3DTaskDurableSnapshot` et la référence de son checkpoint. Elle ne stocke
|
`Lardon3DTaskDurableSnapshot` et la référence de son checkpoint. Elle ne stocke
|
||||||
ni estimation sérialisée complète, ni callback, ni réservation, et ne remplace
|
ni estimation sérialisée complète, ni callback, ni réservation, et ne remplace
|
||||||
pas la validation du fichier checkpoint avant `task_restore()`.
|
pas la validation du fichier checkpoint avant `task_restore()`.
|
||||||
|
|
|
||||||
|
|
@ -33,6 +33,11 @@ au ScanSet, résultats du Visual Index et provenance. Une proximité temporelle
|
||||||
pourra servir de signal secondaire ; la proximité dans un dossier et le nom de
|
pourra servir de signal secondaire ; la proximité dans un dossier et le nom de
|
||||||
fichier ne constituent jamais l'identité principale.
|
fichier ne constituent jamais l'identité principale.
|
||||||
|
|
||||||
|
Une observation de feature utilisera la référence durable
|
||||||
|
`feature_set_id + feature_index`. Une fois le Feature Set READY,
|
||||||
|
`feature_index` désigne immuablement la paire keypoint/descripteur de même
|
||||||
|
indice ; aucun chemin source n'entre dans cette identité.
|
||||||
|
|
||||||
## Concepts clés
|
## Concepts clés
|
||||||
|
|
||||||
### Étapes du matching
|
### Étapes du matching
|
||||||
|
|
|
||||||
|
|
@ -26,10 +26,10 @@ Matching & Tracks (appariement détaillé)
|
||||||
|
|
||||||
L'index visuel se situe entre l'extraction de features et la sélection de paires candidates. Il réduit drastiquement l'espace de recherche pour l'appariement.
|
L'index visuel se situe entre l'extraction de features et la sélection de paires candidates. Il réduit drastiquement l'espace de recherche pour l'appariement.
|
||||||
|
|
||||||
Le futur index référencera exclusivement les `image_id` stables du catalogue
|
Le futur index consommera exclusivement un `feature_set_id` et les descripteurs
|
||||||
persistant. Un chemin d'asset ou un nom source n'est ni une identité d'image ni
|
obtenus par plages bornées du Feature Reader. Il ne dépendra ni des chemins
|
||||||
une clé d'index durable. Le partage d'un asset entre deux ScanSets n'empêche pas
|
source, ni d'OpenCV, ni du runtime de la tâche productrice. Un chemin d'asset ou
|
||||||
leurs deux images logiques d'être des observations distinctes.
|
un nom source n'est pas une clé d'index durable.
|
||||||
|
|
||||||
## Concepts clés
|
## Concepts clés
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -57,9 +57,8 @@ git diff --check
|
||||||
allocations :
|
allocations :
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
meson setup build-asan --wipe \
|
CC=clang meson setup build-asan --wipe \
|
||||||
-Db_sanitize=address,undefined \
|
-Db_sanitize=address,undefined
|
||||||
-Db_static=false
|
|
||||||
meson compile -C build-asan -j8
|
meson compile -C build-asan -j8
|
||||||
meson test -C build-asan --print-errorlogs
|
meson test -C build-asan --print-errorlogs
|
||||||
```
|
```
|
||||||
|
|
@ -70,9 +69,9 @@ meson test -C build-asan --print-errorlogs
|
||||||
variables de condition, états partagés) :
|
variables de condition, états partagés) :
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
meson setup build-tsan --wipe \
|
CC=clang meson setup build-tsan --wipe \
|
||||||
-Db_sanitize=thread \
|
-Db_sanitize=thread \
|
||||||
-Db_static=false
|
-Db_lundef=false
|
||||||
meson compile -C build-tsan -j8
|
meson compile -C build-tsan -j8
|
||||||
meson test -C build-tsan --print-errorlogs
|
meson test -C build-tsan --print-errorlogs
|
||||||
```
|
```
|
||||||
|
|
|
||||||
|
|
@ -160,7 +160,7 @@ Pour tout ticket touchant la concurrence, exécuter :
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
# Build TSan
|
# Build TSan
|
||||||
meson setup build-tsan --wipe -Db_sanitize=thread -Db_static=false
|
CC=clang meson setup build-tsan --wipe -Db_sanitize=thread -Db_lundef=false
|
||||||
meson compile -C build-tsan -j8
|
meson compile -C build-tsan -j8
|
||||||
meson test -C build-tsan --print-errorlogs
|
meson test -C build-tsan --print-errorlogs
|
||||||
```
|
```
|
||||||
|
|
|
||||||
|
|
@ -95,12 +95,12 @@ meson test -C build --print-errorlogs
|
||||||
git diff --check
|
git diff --check
|
||||||
|
|
||||||
# 4. Si mémoire/concurrence touchés
|
# 4. Si mémoire/concurrence touchés
|
||||||
meson setup build-asan --wipe -Db_sanitize=address,undefined -Db_static=false
|
CC=clang meson setup build-asan --wipe -Db_sanitize=address,undefined
|
||||||
meson compile -C build-asan -j8
|
meson compile -C build-asan -j8
|
||||||
meson test -C build-asan --print-errorlogs
|
meson test -C build-asan --print-errorlogs
|
||||||
|
|
||||||
# 5. Si concurrence touchée
|
# 5. Si concurrence touchée
|
||||||
meson setup build-tsan --wipe -Db_sanitize=thread -Db_static=false
|
CC=clang meson setup build-tsan --wipe -Db_sanitize=thread -Db_lundef=false
|
||||||
meson compile -C build-tsan -j8
|
meson compile -C build-tsan -j8
|
||||||
meson test -C build-tsan --print-errorlogs
|
meson test -C build-tsan --print-errorlogs
|
||||||
```
|
```
|
||||||
|
|
|
||||||
|
|
@ -29,7 +29,7 @@ Lardon3D suit une feuille de route ordonnée qui privilégie la stabilité et la
|
||||||
|
|
||||||
### Phase 3 : Persistance
|
### Phase 3 : Persistance
|
||||||
- ✅ Fondation versionnée des checkpoints de tâches
|
- ✅ Fondation versionnée des checkpoints de tâches
|
||||||
- ✅ Project Database v4 (tâches, ScanSets, images, assets et migrations)
|
- ✅ Project Database v5 (tâches, ScanSets, images, Feature Sets/assets et migrations)
|
||||||
- ✅ Branchement Project Database au cycle de vie projet et inventaire de reprise
|
- ✅ Branchement Project Database au cycle de vie projet et inventaire de reprise
|
||||||
- ✅ Registry durable des types métier de tâches
|
- ✅ Registry durable des types métier de tâches
|
||||||
- ✅ Premier type métier reconstructible (`import.images`)
|
- ✅ Premier type métier reconstructible (`import.images`)
|
||||||
|
|
@ -37,7 +37,7 @@ Lardon3D suit une feuille de route ordonnée qui privilégie la stabilité et la
|
||||||
- ✅ ScanSet v1 et Image Catalog persistant v1
|
- ✅ ScanSet v1 et Image Catalog persistant v1
|
||||||
|
|
||||||
### Phase 4 : Pipeline avancé
|
### Phase 4 : Pipeline avancé
|
||||||
- 📋 Feature Store
|
- ✅ Feature Store v1 et extraction ORB persistante
|
||||||
- 📋 Visual Index
|
- 📋 Visual Index
|
||||||
- 📋 Candidate Pair Generator
|
- 📋 Candidate Pair Generator
|
||||||
- 📋 Matching et vérification géométrique
|
- 📋 Matching et vérification géométrique
|
||||||
|
|
|
||||||
57
include/lardon3d/feature_extractor.h
Normal file
57
include/lardon3d/feature_extractor.h
Normal file
|
|
@ -0,0 +1,57 @@
|
||||||
|
#ifndef LARDON3D_FEATURE_EXTRACTOR_H
|
||||||
|
#define LARDON3D_FEATURE_EXTRACTOR_H
|
||||||
|
|
||||||
|
#include <stdbool.h>
|
||||||
|
#include <stddef.h>
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
#define LARDON3D_FEATURE_EXTRACTOR_KIND "orb"
|
||||||
|
enum {
|
||||||
|
LARDON3D_FEATURE_EXTRACTOR_VERSION = 1,
|
||||||
|
LARDON3D_FEATURE_DESCRIPTOR_DIMENSION = 32,
|
||||||
|
LARDON3D_FEATURE_MAX_FEATURES = 8192,
|
||||||
|
LARDON3D_FEATURE_MAX_IMAGE_PIXELS = 100000000,
|
||||||
|
};
|
||||||
|
|
||||||
|
typedef struct {
|
||||||
|
uint32_t max_features;
|
||||||
|
uint32_t pyramid_levels;
|
||||||
|
uint32_t fast_threshold;
|
||||||
|
} Lardon3DFeatureExtractorParameters;
|
||||||
|
|
||||||
|
typedef struct {
|
||||||
|
float x;
|
||||||
|
float y;
|
||||||
|
float size;
|
||||||
|
float angle_degrees;
|
||||||
|
float response;
|
||||||
|
int32_t octave;
|
||||||
|
} Lardon3DFeatureKeypoint;
|
||||||
|
|
||||||
|
typedef struct {
|
||||||
|
uint32_t image_width;
|
||||||
|
uint32_t image_height;
|
||||||
|
uint32_t feature_count;
|
||||||
|
Lardon3DFeatureKeypoint *keypoints;
|
||||||
|
unsigned char *descriptors;
|
||||||
|
} Lardon3DExtractedFeatures;
|
||||||
|
|
||||||
|
typedef enum {
|
||||||
|
LARDON3D_FEATURE_EXTRACT_OK = 0,
|
||||||
|
LARDON3D_FEATURE_EXTRACT_INVALID_ARGUMENT,
|
||||||
|
LARDON3D_FEATURE_EXTRACT_IMAGE_NOT_FOUND,
|
||||||
|
LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID,
|
||||||
|
LARDON3D_FEATURE_EXTRACT_OUT_OF_MEMORY,
|
||||||
|
LARDON3D_FEATURE_EXTRACT_ERROR
|
||||||
|
} Lardon3DFeatureExtractResult;
|
||||||
|
|
||||||
|
bool lardon3d_feature_extractor_parameters_valid(
|
||||||
|
const Lardon3DFeatureExtractorParameters *parameters);
|
||||||
|
void lardon3d_feature_extractor_parameter_fingerprint(
|
||||||
|
const Lardon3DFeatureExtractorParameters *parameters, unsigned char fingerprint[32]);
|
||||||
|
Lardon3DFeatureExtractResult
|
||||||
|
lardon3d_feature_extract_orb(const char *path, const Lardon3DFeatureExtractorParameters *parameters,
|
||||||
|
Lardon3DExtractedFeatures *features);
|
||||||
|
void lardon3d_extracted_features_destroy(Lardon3DExtractedFeatures *features);
|
||||||
|
|
||||||
|
#endif
|
||||||
73
include/lardon3d/feature_store.h
Normal file
73
include/lardon3d/feature_store.h
Normal file
|
|
@ -0,0 +1,73 @@
|
||||||
|
#ifndef LARDON3D_FEATURE_STORE_H
|
||||||
|
#define LARDON3D_FEATURE_STORE_H
|
||||||
|
|
||||||
|
#include <stdbool.h>
|
||||||
|
#include <stddef.h>
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
#include <lardon3d/app_state.h>
|
||||||
|
#include <lardon3d/feature_extractor.h>
|
||||||
|
#include <lardon3d/project_db.h>
|
||||||
|
|
||||||
|
enum {
|
||||||
|
LARDON3D_FEATURE_FILE_VERSION = 1,
|
||||||
|
LARDON3D_FEATURE_FILE_HEADER_SIZE = 160,
|
||||||
|
LARDON3D_FEATURE_KEYPOINT_RECORD_SIZE = 24,
|
||||||
|
LARDON3D_FEATURE_READER_RANGE_MAX = 256,
|
||||||
|
LARDON3D_FEATURE_FILE_MAX_SIZE = 16 * 1024 * 1024,
|
||||||
|
};
|
||||||
|
|
||||||
|
typedef enum {
|
||||||
|
LARDON3D_FEATURE_DESCRIPTOR_U8 = 1,
|
||||||
|
LARDON3D_FEATURE_DESCRIPTOR_F32 = 2
|
||||||
|
} Lardon3DFeatureDescriptorType;
|
||||||
|
|
||||||
|
typedef enum {
|
||||||
|
LARDON3D_FEATURE_STORE_OK = 0,
|
||||||
|
LARDON3D_FEATURE_STORE_ALREADY_PRESENT,
|
||||||
|
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT,
|
||||||
|
LARDON3D_FEATURE_STORE_NOT_FOUND,
|
||||||
|
LARDON3D_FEATURE_STORE_INVALID,
|
||||||
|
LARDON3D_FEATURE_STORE_UNSUPPORTED_VERSION,
|
||||||
|
LARDON3D_FEATURE_STORE_CORRUPT,
|
||||||
|
LARDON3D_FEATURE_STORE_IO_ERROR,
|
||||||
|
LARDON3D_FEATURE_STORE_DB_BUSY,
|
||||||
|
LARDON3D_FEATURE_STORE_DB_ERROR,
|
||||||
|
LARDON3D_FEATURE_STORE_PUBLISHED_NOT_DURABLE
|
||||||
|
} Lardon3DFeatureStoreResult;
|
||||||
|
|
||||||
|
typedef struct Lardon3DFeatureReader Lardon3DFeatureReader;
|
||||||
|
|
||||||
|
typedef struct {
|
||||||
|
uint32_t format_version;
|
||||||
|
char extractor_kind[LARDON3D_TASK_KIND_CAPACITY];
|
||||||
|
uint32_t extractor_version;
|
||||||
|
uint32_t feature_count;
|
||||||
|
uint32_t descriptor_dimension;
|
||||||
|
Lardon3DFeatureDescriptorType descriptor_type;
|
||||||
|
uint32_t image_width;
|
||||||
|
uint32_t image_height;
|
||||||
|
unsigned char source_image_sha256[32];
|
||||||
|
unsigned char parameter_fingerprint[32];
|
||||||
|
} Lardon3DFeatureFileMetadata;
|
||||||
|
|
||||||
|
Lardon3DFeatureStoreResult lardon3d_feature_store_publish(
|
||||||
|
Lardon3DAppState *state, uint64_t image_id, uint64_t producer_task_id,
|
||||||
|
const Lardon3DFeatureExtractorParameters *parameters, const Lardon3DExtractedFeatures *features,
|
||||||
|
Lardon3DProjectDbFeatureSet *feature_set);
|
||||||
|
Lardon3DFeatureStoreResult
|
||||||
|
lardon3d_feature_reader_open(const char *project_path,
|
||||||
|
const Lardon3DProjectDbFeatureSet *feature_set,
|
||||||
|
Lardon3DFeatureReader **reader, Lardon3DFeatureFileMetadata *metadata);
|
||||||
|
void lardon3d_feature_reader_close(Lardon3DFeatureReader *reader);
|
||||||
|
Lardon3DFeatureStoreResult lardon3d_feature_reader_keypoints(Lardon3DFeatureReader *reader,
|
||||||
|
uint32_t start,
|
||||||
|
Lardon3DFeatureKeypoint *keypoints,
|
||||||
|
size_t capacity);
|
||||||
|
Lardon3DFeatureStoreResult lardon3d_feature_reader_descriptors(Lardon3DFeatureReader *reader,
|
||||||
|
uint32_t start,
|
||||||
|
unsigned char *descriptors,
|
||||||
|
size_t feature_capacity,
|
||||||
|
size_t descriptor_capacity);
|
||||||
|
|
||||||
|
#endif
|
||||||
24
include/lardon3d/feature_task.h
Normal file
24
include/lardon3d/feature_task.h
Normal file
|
|
@ -0,0 +1,24 @@
|
||||||
|
#ifndef LARDON3D_FEATURE_TASK_H
|
||||||
|
#define LARDON3D_FEATURE_TASK_H
|
||||||
|
|
||||||
|
#include <stdbool.h>
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
#include <lardon3d/app_state.h>
|
||||||
|
#include <lardon3d/feature_extractor.h>
|
||||||
|
#include <lardon3d/task_kind_registry.h>
|
||||||
|
|
||||||
|
#define LARDON3D_FEATURE_EXTRACT_TASK_KIND "features.extract"
|
||||||
|
enum { LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION = 1 };
|
||||||
|
|
||||||
|
Lardon3DTask *
|
||||||
|
lardon3d_project_create_feature_extract_task(Lardon3DAppState *state, uint64_t image_id,
|
||||||
|
const Lardon3DFeatureExtractorParameters *parameters,
|
||||||
|
uint64_t *task_id);
|
||||||
|
bool lardon3d_project_enqueue_feature_extract(Lardon3DAppState *state, uint64_t image_id,
|
||||||
|
const Lardon3DFeatureExtractorParameters *parameters,
|
||||||
|
uint64_t *task_id);
|
||||||
|
bool lardon3d_feature_extract_reconstruct(const Lardon3DTaskDurableSnapshot *snapshot,
|
||||||
|
void *context, Lardon3DTaskKindBinding *binding);
|
||||||
|
|
||||||
|
#endif
|
||||||
|
|
@ -12,61 +12,42 @@
|
||||||
enum { LARDON3D_IMAGE_IMPORT_TASK_KIND_VERSION = 1 };
|
enum { LARDON3D_IMAGE_IMPORT_TASK_KIND_VERSION = 1 };
|
||||||
|
|
||||||
typedef enum {
|
typedef enum {
|
||||||
LARDON3D_IMPORT_TASK_IDLE = 0,
|
LARDON3D_IMPORT_TASK_IDLE = 0,
|
||||||
LARDON3D_IMPORT_TASK_RUNNING,
|
LARDON3D_IMPORT_TASK_RUNNING,
|
||||||
LARDON3D_IMPORT_TASK_SUCCEEDED,
|
LARDON3D_IMPORT_TASK_SUCCEEDED,
|
||||||
LARDON3D_IMPORT_TASK_CANCELLED,
|
LARDON3D_IMPORT_TASK_CANCELLED,
|
||||||
LARDON3D_IMPORT_TASK_FAILED
|
LARDON3D_IMPORT_TASK_FAILED
|
||||||
} Lardon3DImportTaskStatus;
|
} Lardon3DImportTaskStatus;
|
||||||
|
|
||||||
typedef struct {
|
typedef struct {
|
||||||
Lardon3DImportTaskStatus status;
|
Lardon3DImportTaskStatus status;
|
||||||
size_t total;
|
size_t total;
|
||||||
size_t processed;
|
size_t processed;
|
||||||
size_t copied;
|
size_t copied;
|
||||||
size_t already_present;
|
size_t already_present;
|
||||||
size_t ignored;
|
size_t ignored;
|
||||||
char message[256];
|
char message[256];
|
||||||
} Lardon3DImportTaskSnapshot;
|
} Lardon3DImportTaskSnapshot;
|
||||||
|
|
||||||
typedef struct Lardon3DImportTask Lardon3DImportTask;
|
typedef struct Lardon3DImportTask Lardon3DImportTask;
|
||||||
|
|
||||||
typedef struct {
|
typedef Lardon3DTaskReconstructionContext Lardon3DImageImportReconstructionContext;
|
||||||
const char *project_path;
|
|
||||||
Lardon3DProjectDb *project_db;
|
|
||||||
Lardon3DResourceGovernor *resource_governor;
|
|
||||||
} Lardon3DImageImportReconstructionContext;
|
|
||||||
|
|
||||||
Lardon3DTask *lardon3d_project_create_image_import_task(
|
Lardon3DTask *lardon3d_project_create_image_import_task(Lardon3DAppState *state,
|
||||||
Lardon3DAppState *state,
|
uint64_t scanset_id,
|
||||||
uint64_t scanset_id,
|
const char *source_directory,
|
||||||
const char *source_directory,
|
uint64_t *task_id);
|
||||||
uint64_t *task_id
|
bool lardon3d_project_enqueue_image_import(Lardon3DAppState *state, uint64_t scanset_id,
|
||||||
);
|
const char *source_directory, uint64_t *task_id);
|
||||||
bool lardon3d_project_enqueue_image_import(
|
bool lardon3d_image_import_reconstruct(const Lardon3DTaskDurableSnapshot *snapshot, void *context,
|
||||||
Lardon3DAppState *state,
|
Lardon3DTaskKindBinding *binding);
|
||||||
uint64_t scanset_id,
|
|
||||||
const char *source_directory,
|
|
||||||
uint64_t *task_id
|
|
||||||
);
|
|
||||||
bool lardon3d_image_import_reconstruct(
|
|
||||||
const Lardon3DTaskDurableSnapshot *snapshot,
|
|
||||||
void *context,
|
|
||||||
Lardon3DTaskKindBinding *binding
|
|
||||||
);
|
|
||||||
|
|
||||||
/* Compatibilité TUI : handle léger sur une tâche de la queue, sans thread. */
|
/* Compatibilité TUI : handle léger sur une tâche de la queue, sans thread. */
|
||||||
Lardon3DImportTask *lardon3d_import_task_create(void);
|
Lardon3DImportTask *lardon3d_import_task_create(void);
|
||||||
bool lardon3d_import_task_start(
|
bool lardon3d_import_task_start(Lardon3DImportTask *task, Lardon3DAppState *state,
|
||||||
Lardon3DImportTask *task,
|
const char *source_directory);
|
||||||
Lardon3DAppState *state,
|
|
||||||
const char *source_directory
|
|
||||||
);
|
|
||||||
void lardon3d_import_task_request_cancel(Lardon3DImportTask *task);
|
void lardon3d_import_task_request_cancel(Lardon3DImportTask *task);
|
||||||
bool lardon3d_import_task_snapshot(
|
bool lardon3d_import_task_snapshot(Lardon3DImportTask *task, Lardon3DImportTaskSnapshot *snapshot);
|
||||||
Lardon3DImportTask *task,
|
|
||||||
Lardon3DImportTaskSnapshot *snapshot
|
|
||||||
);
|
|
||||||
bool lardon3d_import_task_is_finished(Lardon3DImportTask *task);
|
bool lardon3d_import_task_is_finished(Lardon3DImportTask *task);
|
||||||
bool lardon3d_import_task_join(Lardon3DImportTask *task);
|
bool lardon3d_import_task_join(Lardon3DImportTask *task);
|
||||||
void lardon3d_import_task_destroy(Lardon3DImportTask *task);
|
void lardon3d_import_task_destroy(Lardon3DImportTask *task);
|
||||||
|
|
|
||||||
|
|
@ -8,84 +8,69 @@
|
||||||
#include <lardon3d/task_kind_registry.h>
|
#include <lardon3d/task_kind_registry.h>
|
||||||
|
|
||||||
typedef enum {
|
typedef enum {
|
||||||
LARDON3D_PROJECT_TASK_CHECKPOINT_OK = 0,
|
LARDON3D_PROJECT_TASK_CHECKPOINT_OK = 0,
|
||||||
LARDON3D_PROJECT_TASK_CHECKPOINT_PUBLISHED_NOT_DURABLE,
|
LARDON3D_PROJECT_TASK_CHECKPOINT_PUBLISHED_NOT_DURABLE,
|
||||||
LARDON3D_PROJECT_TASK_CHECKPOINT_NO_PROJECT,
|
LARDON3D_PROJECT_TASK_CHECKPOINT_NO_PROJECT,
|
||||||
LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK,
|
LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK,
|
||||||
LARDON3D_PROJECT_TASK_CHECKPOINT_IO_ERROR,
|
LARDON3D_PROJECT_TASK_CHECKPOINT_IO_ERROR,
|
||||||
LARDON3D_PROJECT_TASK_CHECKPOINT_DB_BUSY,
|
LARDON3D_PROJECT_TASK_CHECKPOINT_DB_BUSY,
|
||||||
LARDON3D_PROJECT_TASK_CHECKPOINT_DB_ERROR
|
LARDON3D_PROJECT_TASK_CHECKPOINT_DB_ERROR
|
||||||
} Lardon3DProjectTaskCheckpointResult;
|
} Lardon3DProjectTaskCheckpointResult;
|
||||||
|
|
||||||
typedef enum {
|
typedef enum {
|
||||||
LARDON3D_PROJECT_RECOVERABLE = 0,
|
LARDON3D_PROJECT_RECOVERABLE = 0,
|
||||||
LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE,
|
LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE,
|
||||||
LARDON3D_PROJECT_RECOVERY_MISSING_CHECKPOINT,
|
LARDON3D_PROJECT_RECOVERY_MISSING_CHECKPOINT,
|
||||||
LARDON3D_PROJECT_RECOVERY_INVALID_CHECKPOINT,
|
LARDON3D_PROJECT_RECOVERY_INVALID_CHECKPOINT,
|
||||||
LARDON3D_PROJECT_RECOVERY_UNSUPPORTED_CHECKPOINT,
|
LARDON3D_PROJECT_RECOVERY_UNSUPPORTED_CHECKPOINT,
|
||||||
LARDON3D_PROJECT_RECOVERY_CHECKPOINT_IO_ERROR,
|
LARDON3D_PROJECT_RECOVERY_CHECKPOINT_IO_ERROR,
|
||||||
LARDON3D_PROJECT_RECOVERY_LEGACY_UNTYPED,
|
LARDON3D_PROJECT_RECOVERY_LEGACY_UNTYPED,
|
||||||
LARDON3D_PROJECT_RECOVERY_UNKNOWN_TASK_KIND,
|
LARDON3D_PROJECT_RECOVERY_UNKNOWN_TASK_KIND,
|
||||||
LARDON3D_PROJECT_RECOVERY_UNSUPPORTED_TASK_KIND_VERSION
|
LARDON3D_PROJECT_RECOVERY_UNSUPPORTED_TASK_KIND_VERSION
|
||||||
} Lardon3DProjectRecoveryStatus;
|
} Lardon3DProjectRecoveryStatus;
|
||||||
|
|
||||||
typedef struct {
|
typedef struct {
|
||||||
uint64_t task_id;
|
uint64_t task_id;
|
||||||
char name[LARDON3D_TASK_NAME_CAPACITY];
|
char name[LARDON3D_TASK_NAME_CAPACITY];
|
||||||
char task_kind[LARDON3D_TASK_KIND_CAPACITY];
|
char task_kind[LARDON3D_TASK_KIND_CAPACITY];
|
||||||
uint32_t task_kind_version;
|
uint32_t task_kind_version;
|
||||||
Lardon3DProjectRecoveryStatus status;
|
Lardon3DProjectRecoveryStatus status;
|
||||||
Lardon3DProjectDbCheckpointDurability durability;
|
Lardon3DProjectDbCheckpointDurability durability;
|
||||||
Lardon3DTaskDurableSnapshot snapshot;
|
Lardon3DTaskDurableSnapshot snapshot;
|
||||||
} Lardon3DProjectRecoveryEntry;
|
} Lardon3DProjectRecoveryEntry;
|
||||||
|
|
||||||
typedef struct {
|
typedef struct {
|
||||||
size_t inspected;
|
size_t inspected;
|
||||||
size_t resumed;
|
size_t resumed;
|
||||||
size_t skipped;
|
size_t skipped;
|
||||||
size_t failed;
|
size_t failed;
|
||||||
size_t published_not_durable;
|
size_t published_not_durable;
|
||||||
bool queue_full;
|
bool queue_full;
|
||||||
} Lardon3DProjectRecoverySummary;
|
} Lardon3DProjectRecoverySummary;
|
||||||
|
|
||||||
bool lardon3d_project_create(
|
bool lardon3d_project_create(Lardon3DAppState *state, const char *name);
|
||||||
Lardon3DAppState *state,
|
|
||||||
const char *name
|
|
||||||
);
|
|
||||||
|
|
||||||
bool lardon3d_project_open(
|
bool lardon3d_project_open(Lardon3DAppState *state, const char *directory_name);
|
||||||
Lardon3DAppState *state,
|
|
||||||
const char *directory_name
|
|
||||||
);
|
|
||||||
|
|
||||||
void lardon3d_project_close(Lardon3DAppState *state);
|
void lardon3d_project_close(Lardon3DAppState *state);
|
||||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_task(
|
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_task(Lardon3DAppState *state,
|
||||||
Lardon3DAppState *state,
|
const Lardon3DTask *task);
|
||||||
const Lardon3DTask *task
|
|
||||||
);
|
|
||||||
Lardon3DProjectTaskCheckpointResult
|
Lardon3DProjectTaskCheckpointResult
|
||||||
lardon3d_project_checkpoint_image_import_task(
|
lardon3d_project_checkpoint_image_import_task(Lardon3DAppState *state, const Lardon3DTask *task,
|
||||||
Lardon3DAppState *state,
|
const char *source_path, uint64_t scanset_id);
|
||||||
const Lardon3DTask *task,
|
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_task(
|
||||||
const char *source_path,
|
Lardon3DAppState *state, const Lardon3DTask *task,
|
||||||
uint64_t scanset_id
|
const Lardon3DProjectDbFeatureExtractTask *parameters);
|
||||||
);
|
Lardon3DProjectDbResult lardon3d_project_list_recoverable(Lardon3DAppState *state,
|
||||||
Lardon3DProjectDbResult lardon3d_project_list_recoverable(
|
const Lardon3DTaskKindRegistry *registry,
|
||||||
Lardon3DAppState *state,
|
uint64_t after_task_id,
|
||||||
const Lardon3DTaskKindRegistry *registry,
|
Lardon3DProjectRecoveryEntry *entries,
|
||||||
uint64_t after_task_id,
|
size_t capacity, size_t *count);
|
||||||
Lardon3DProjectRecoveryEntry *entries,
|
Lardon3DProjectDbResult
|
||||||
size_t capacity,
|
lardon3d_project_resume_recoverable_tasks(Lardon3DAppState *state,
|
||||||
size_t *count
|
const Lardon3DTaskKindRegistry *registry,
|
||||||
);
|
Lardon3DProjectRecoverySummary *summary);
|
||||||
Lardon3DProjectDbResult lardon3d_project_resume_recoverable_tasks(
|
bool lardon3d_project_last_recovery_summary(const Lardon3DAppState *state,
|
||||||
Lardon3DAppState *state,
|
Lardon3DProjectRecoverySummary *summary);
|
||||||
const Lardon3DTaskKindRegistry *registry,
|
|
||||||
Lardon3DProjectRecoverySummary *summary
|
|
||||||
);
|
|
||||||
bool lardon3d_project_last_recovery_summary(
|
|
||||||
const Lardon3DAppState *state,
|
|
||||||
Lardon3DProjectRecoverySummary *summary
|
|
||||||
);
|
|
||||||
|
|
||||||
#endif
|
#endif
|
||||||
|
|
|
||||||
|
|
@ -9,233 +9,259 @@
|
||||||
#include <lardon3d/task.h>
|
#include <lardon3d/task.h>
|
||||||
|
|
||||||
enum {
|
enum {
|
||||||
LARDON3D_PROJECT_DB_SCHEMA_VERSION = 4,
|
LARDON3D_PROJECT_DB_SCHEMA_VERSION = 5,
|
||||||
LARDON3D_PROJECT_DB_ID_CAPACITY = 65,
|
LARDON3D_PROJECT_DB_ID_CAPACITY = 65,
|
||||||
LARDON3D_PROJECT_DB_KIND_CAPACITY = 65,
|
LARDON3D_PROJECT_DB_KIND_CAPACITY = 65,
|
||||||
LARDON3D_PROJECT_DB_PATH_CAPACITY = 4096,
|
LARDON3D_PROJECT_DB_PATH_CAPACITY = 4096,
|
||||||
LARDON3D_PROJECT_DB_ERROR_CAPACITY = 256,
|
LARDON3D_PROJECT_DB_ERROR_CAPACITY = 256,
|
||||||
LARDON3D_PROJECT_DB_RECOVERY_PAGE_MAX = 256,
|
LARDON3D_PROJECT_DB_RECOVERY_PAGE_MAX = 256,
|
||||||
LARDON3D_PROJECT_DB_CATALOG_PAGE_MAX = 256,
|
LARDON3D_PROJECT_DB_CATALOG_PAGE_MAX = 256,
|
||||||
LARDON3D_PROJECT_DB_SCANSET_NAME_CAPACITY = 256,
|
LARDON3D_PROJECT_DB_SCANSET_NAME_CAPACITY = 256,
|
||||||
LARDON3D_PROJECT_DB_IMAGE_NAME_CAPACITY = 256,
|
LARDON3D_PROJECT_DB_IMAGE_NAME_CAPACITY = 256,
|
||||||
LARDON3D_PROJECT_DB_SHA256_SIZE = 32,
|
LARDON3D_PROJECT_DB_SHA256_SIZE = 32,
|
||||||
};
|
};
|
||||||
|
|
||||||
typedef struct Lardon3DProjectDb Lardon3DProjectDb;
|
typedef struct Lardon3DProjectDb Lardon3DProjectDb;
|
||||||
|
|
||||||
typedef enum {
|
typedef enum {
|
||||||
LARDON3D_PROJECT_DB_OK = 0,
|
LARDON3D_PROJECT_DB_OK = 0,
|
||||||
LARDON3D_PROJECT_DB_INVALID_ARGUMENT,
|
LARDON3D_PROJECT_DB_INVALID_ARGUMENT,
|
||||||
LARDON3D_PROJECT_DB_NOT_FOUND,
|
LARDON3D_PROJECT_DB_NOT_FOUND,
|
||||||
LARDON3D_PROJECT_DB_BUSY,
|
LARDON3D_PROJECT_DB_BUSY,
|
||||||
LARDON3D_PROJECT_DB_UNSUPPORTED_SCHEMA,
|
LARDON3D_PROJECT_DB_UNSUPPORTED_SCHEMA,
|
||||||
LARDON3D_PROJECT_DB_CORRUPT,
|
LARDON3D_PROJECT_DB_CORRUPT,
|
||||||
LARDON3D_PROJECT_DB_CONSTRAINT,
|
LARDON3D_PROJECT_DB_CONSTRAINT,
|
||||||
LARDON3D_PROJECT_DB_IO_ERROR
|
LARDON3D_PROJECT_DB_IO_ERROR
|
||||||
} Lardon3DProjectDbResult;
|
} Lardon3DProjectDbResult;
|
||||||
|
|
||||||
typedef enum {
|
typedef enum {
|
||||||
LARDON3D_DB_CHECKPOINT_DURABLE = 0,
|
LARDON3D_DB_CHECKPOINT_DURABLE = 0,
|
||||||
LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE
|
LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE
|
||||||
} Lardon3DProjectDbCheckpointDurability;
|
} Lardon3DProjectDbCheckpointDurability;
|
||||||
|
|
||||||
typedef enum {
|
typedef enum {
|
||||||
LARDON3D_DB_ARTIFACT_STAGED = 0,
|
LARDON3D_DB_ARTIFACT_STAGED = 0,
|
||||||
LARDON3D_DB_ARTIFACT_READY
|
LARDON3D_DB_ARTIFACT_READY
|
||||||
} Lardon3DProjectDbArtifactState;
|
} Lardon3DProjectDbArtifactState;
|
||||||
|
|
||||||
typedef struct {
|
typedef struct {
|
||||||
char stable_id[LARDON3D_PROJECT_DB_ID_CAPACITY];
|
char stable_id[LARDON3D_PROJECT_DB_ID_CAPACITY];
|
||||||
char name[LARDON3D_TASK_NAME_CAPACITY];
|
char name[LARDON3D_TASK_NAME_CAPACITY];
|
||||||
int64_t created_at;
|
int64_t created_at;
|
||||||
int64_t updated_at;
|
int64_t updated_at;
|
||||||
} Lardon3DProjectDbProject;
|
} Lardon3DProjectDbProject;
|
||||||
|
|
||||||
typedef struct {
|
typedef struct {
|
||||||
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
||||||
uint32_t format_version;
|
uint32_t format_version;
|
||||||
Lardon3DProjectDbCheckpointDurability durability;
|
Lardon3DProjectDbCheckpointDurability durability;
|
||||||
int64_t updated_at;
|
int64_t updated_at;
|
||||||
} Lardon3DProjectDbCheckpoint;
|
} Lardon3DProjectDbCheckpoint;
|
||||||
|
|
||||||
typedef struct {
|
typedef struct {
|
||||||
uint64_t task_id;
|
uint64_t task_id;
|
||||||
char name[LARDON3D_TASK_NAME_CAPACITY];
|
char name[LARDON3D_TASK_NAME_CAPACITY];
|
||||||
bool has_task_kind;
|
bool has_task_kind;
|
||||||
char task_kind[LARDON3D_TASK_KIND_CAPACITY];
|
char task_kind[LARDON3D_TASK_KIND_CAPACITY];
|
||||||
uint32_t task_kind_version;
|
uint32_t task_kind_version;
|
||||||
Lardon3DTaskState saved_state;
|
Lardon3DTaskState saved_state;
|
||||||
Lardon3DTaskState recovery_state;
|
Lardon3DTaskState recovery_state;
|
||||||
unsigned int progress;
|
unsigned int progress;
|
||||||
unsigned int sequence_count;
|
unsigned int sequence_count;
|
||||||
struct timespec started_at;
|
struct timespec started_at;
|
||||||
struct timespec finished_at;
|
struct timespec finished_at;
|
||||||
int64_t updated_at;
|
int64_t updated_at;
|
||||||
bool has_checkpoint;
|
bool has_checkpoint;
|
||||||
Lardon3DProjectDbCheckpoint checkpoint;
|
Lardon3DProjectDbCheckpoint checkpoint;
|
||||||
} Lardon3DProjectDbTask;
|
} Lardon3DProjectDbTask;
|
||||||
|
|
||||||
typedef struct {
|
typedef struct {
|
||||||
char artifact_id[LARDON3D_PROJECT_DB_ID_CAPACITY];
|
char artifact_id[LARDON3D_PROJECT_DB_ID_CAPACITY];
|
||||||
char kind[LARDON3D_PROJECT_DB_KIND_CAPACITY];
|
char kind[LARDON3D_PROJECT_DB_KIND_CAPACITY];
|
||||||
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
||||||
Lardon3DProjectDbArtifactState state;
|
Lardon3DProjectDbArtifactState state;
|
||||||
uint64_t size_bytes;
|
uint64_t size_bytes;
|
||||||
bool has_producer_task;
|
bool has_producer_task;
|
||||||
uint64_t producer_task_id;
|
uint64_t producer_task_id;
|
||||||
int64_t created_at;
|
int64_t created_at;
|
||||||
int64_t updated_at;
|
int64_t updated_at;
|
||||||
} Lardon3DProjectDbArtifact;
|
} Lardon3DProjectDbArtifact;
|
||||||
|
|
||||||
typedef struct {
|
typedef struct {
|
||||||
uint64_t task_id;
|
uint64_t task_id;
|
||||||
char source_path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
char source_path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
||||||
uint64_t scanset_id;
|
uint64_t scanset_id;
|
||||||
} Lardon3DProjectDbImageImport;
|
} Lardon3DProjectDbImageImport;
|
||||||
|
|
||||||
typedef struct {
|
typedef struct {
|
||||||
uint64_t scanset_id;
|
uint64_t scanset_id;
|
||||||
char name[LARDON3D_PROJECT_DB_SCANSET_NAME_CAPACITY];
|
char name[LARDON3D_PROJECT_DB_SCANSET_NAME_CAPACITY];
|
||||||
int64_t created_at;
|
int64_t created_at;
|
||||||
int64_t updated_at;
|
int64_t updated_at;
|
||||||
} Lardon3DProjectDbScanSet;
|
} Lardon3DProjectDbScanSet;
|
||||||
|
|
||||||
typedef enum {
|
typedef enum { LARDON3D_DB_IMAGE_ASSET_READY = 1 } Lardon3DProjectDbImageAssetState;
|
||||||
LARDON3D_DB_IMAGE_ASSET_READY = 1
|
|
||||||
} Lardon3DProjectDbImageAssetState;
|
|
||||||
|
|
||||||
typedef struct {
|
typedef struct {
|
||||||
uint64_t asset_id;
|
uint64_t asset_id;
|
||||||
unsigned char sha256[LARDON3D_PROJECT_DB_SHA256_SIZE];
|
unsigned char sha256[LARDON3D_PROJECT_DB_SHA256_SIZE];
|
||||||
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
||||||
uint64_t size_bytes;
|
uint64_t size_bytes;
|
||||||
Lardon3DProjectDbImageAssetState state;
|
Lardon3DProjectDbImageAssetState state;
|
||||||
int64_t created_at;
|
int64_t created_at;
|
||||||
} Lardon3DProjectDbImageAsset;
|
} Lardon3DProjectDbImageAsset;
|
||||||
|
|
||||||
typedef struct {
|
typedef struct {
|
||||||
uint64_t image_id;
|
uint64_t image_id;
|
||||||
uint64_t scanset_id;
|
uint64_t scanset_id;
|
||||||
uint64_t asset_id;
|
uint64_t asset_id;
|
||||||
char original_name[LARDON3D_PROJECT_DB_IMAGE_NAME_CAPACITY];
|
char original_name[LARDON3D_PROJECT_DB_IMAGE_NAME_CAPACITY];
|
||||||
char source_path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
char source_path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
||||||
bool has_producer_task;
|
bool has_producer_task;
|
||||||
uint64_t producer_task_id;
|
uint64_t producer_task_id;
|
||||||
int64_t imported_at;
|
int64_t imported_at;
|
||||||
} Lardon3DProjectDbImage;
|
} Lardon3DProjectDbImage;
|
||||||
|
|
||||||
typedef enum {
|
typedef enum {
|
||||||
LARDON3D_PROJECT_DB_IMAGE_REGISTERED = 0,
|
LARDON3D_PROJECT_DB_IMAGE_REGISTERED = 0,
|
||||||
LARDON3D_PROJECT_DB_IMAGE_ALREADY_PRESENT
|
LARDON3D_PROJECT_DB_IMAGE_ALREADY_PRESENT
|
||||||
} Lardon3DProjectDbImageRegisterStatus;
|
} Lardon3DProjectDbImageRegisterStatus;
|
||||||
|
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_open(
|
typedef enum {
|
||||||
const char *path,
|
LARDON3D_DB_FEATURE_ASSET_DURABLE = 0,
|
||||||
Lardon3DProjectDb **database,
|
LARDON3D_DB_FEATURE_ASSET_PUBLISHED_NOT_DURABLE = 1
|
||||||
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]
|
} Lardon3DProjectDbFeatureDurability;
|
||||||
);
|
|
||||||
|
typedef struct {
|
||||||
|
uint64_t feature_asset_id;
|
||||||
|
unsigned char sha256[32];
|
||||||
|
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
||||||
|
uint64_t size_bytes;
|
||||||
|
Lardon3DProjectDbFeatureDurability durability;
|
||||||
|
int64_t created_at;
|
||||||
|
} Lardon3DProjectDbFeatureAsset;
|
||||||
|
|
||||||
|
typedef struct {
|
||||||
|
uint64_t feature_set_id;
|
||||||
|
uint64_t image_id;
|
||||||
|
uint64_t feature_asset_id;
|
||||||
|
char extractor_kind[LARDON3D_PROJECT_DB_KIND_CAPACITY];
|
||||||
|
uint32_t extractor_version;
|
||||||
|
unsigned char parameter_fingerprint[32];
|
||||||
|
unsigned char source_image_sha256[32];
|
||||||
|
uint32_t feature_count;
|
||||||
|
uint32_t descriptor_type;
|
||||||
|
uint32_t descriptor_dimension;
|
||||||
|
bool has_producer_task;
|
||||||
|
uint64_t producer_task_id;
|
||||||
|
int64_t created_at;
|
||||||
|
Lardon3DProjectDbFeatureAsset asset;
|
||||||
|
} Lardon3DProjectDbFeatureSet;
|
||||||
|
|
||||||
|
typedef struct {
|
||||||
|
uint64_t task_id;
|
||||||
|
uint64_t image_id;
|
||||||
|
char extractor_kind[LARDON3D_PROJECT_DB_KIND_CAPACITY];
|
||||||
|
uint32_t extractor_version;
|
||||||
|
uint32_t max_features;
|
||||||
|
uint32_t pyramid_levels;
|
||||||
|
uint32_t fast_threshold;
|
||||||
|
unsigned char parameter_fingerprint[32];
|
||||||
|
} Lardon3DProjectDbFeatureExtractTask;
|
||||||
|
|
||||||
|
Lardon3DProjectDbResult lardon3d_project_db_open(const char *path, Lardon3DProjectDb **database,
|
||||||
|
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]);
|
||||||
void lardon3d_project_db_close(Lardon3DProjectDb *database);
|
void lardon3d_project_db_close(Lardon3DProjectDb *database);
|
||||||
bool lardon3d_project_db_last_error(
|
bool lardon3d_project_db_last_error(Lardon3DProjectDb *database,
|
||||||
Lardon3DProjectDb *database,
|
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]);
|
||||||
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]
|
Lardon3DProjectDbResult lardon3d_project_db_legacy_catalog_pending(Lardon3DProjectDb *database,
|
||||||
);
|
bool *pending);
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_legacy_catalog_pending(
|
|
||||||
Lardon3DProjectDb *database, bool *pending
|
|
||||||
);
|
|
||||||
unsigned int lardon3d_project_db_schema_version(Lardon3DProjectDb *database);
|
unsigned int lardon3d_project_db_schema_version(Lardon3DProjectDb *database);
|
||||||
|
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_set_project(
|
Lardon3DProjectDbResult lardon3d_project_db_set_project(Lardon3DProjectDb *database,
|
||||||
Lardon3DProjectDb *database,
|
const Lardon3DProjectDbProject *project);
|
||||||
const Lardon3DProjectDbProject *project
|
Lardon3DProjectDbResult lardon3d_project_db_get_project(Lardon3DProjectDb *database,
|
||||||
);
|
Lardon3DProjectDbProject *project);
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_get_project(
|
|
||||||
Lardon3DProjectDb *database,
|
|
||||||
Lardon3DProjectDbProject *project
|
|
||||||
);
|
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_record_task(
|
Lardon3DProjectDbResult lardon3d_project_db_record_task(
|
||||||
Lardon3DProjectDb *database,
|
Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, const char *task_kind,
|
||||||
const Lardon3DTaskDurableSnapshot *snapshot,
|
uint32_t task_kind_version, const Lardon3DProjectDbCheckpoint *checkpoint, int64_t updated_at);
|
||||||
const char *task_kind,
|
|
||||||
uint32_t task_kind_version,
|
|
||||||
const Lardon3DProjectDbCheckpoint *checkpoint,
|
|
||||||
int64_t updated_at
|
|
||||||
);
|
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_record_image_import_task(
|
Lardon3DProjectDbResult lardon3d_project_db_record_image_import_task(
|
||||||
Lardon3DProjectDb *database,
|
Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, const char *task_kind,
|
||||||
const Lardon3DTaskDurableSnapshot *snapshot,
|
uint32_t task_kind_version, const Lardon3DProjectDbCheckpoint *checkpoint,
|
||||||
const char *task_kind,
|
const char *source_path, uint64_t scanset_id, int64_t updated_at);
|
||||||
uint32_t task_kind_version,
|
Lardon3DProjectDbResult lardon3d_project_db_create_scanset(Lardon3DProjectDb *database,
|
||||||
const Lardon3DProjectDbCheckpoint *checkpoint,
|
const char *name,
|
||||||
const char *source_path,
|
Lardon3DProjectDbScanSet *scanset);
|
||||||
uint64_t scanset_id,
|
Lardon3DProjectDbResult lardon3d_project_db_load_scanset(Lardon3DProjectDb *database,
|
||||||
int64_t updated_at
|
uint64_t scanset_id,
|
||||||
);
|
Lardon3DProjectDbScanSet *scanset);
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_create_scanset(
|
Lardon3DProjectDbResult lardon3d_project_db_list_scansets(Lardon3DProjectDb *database,
|
||||||
Lardon3DProjectDb *database, const char *name,
|
uint64_t after_scanset_id,
|
||||||
Lardon3DProjectDbScanSet *scanset
|
Lardon3DProjectDbScanSet *scansets,
|
||||||
);
|
size_t capacity, size_t *count);
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_load_scanset(
|
|
||||||
Lardon3DProjectDb *database, uint64_t scanset_id,
|
|
||||||
Lardon3DProjectDbScanSet *scanset
|
|
||||||
);
|
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_list_scansets(
|
|
||||||
Lardon3DProjectDb *database, uint64_t after_scanset_id,
|
|
||||||
Lardon3DProjectDbScanSet *scansets, size_t capacity, size_t *count
|
|
||||||
);
|
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_register_image(
|
Lardon3DProjectDbResult lardon3d_project_db_register_image(
|
||||||
Lardon3DProjectDb *database, uint64_t scanset_id,
|
Lardon3DProjectDb *database, uint64_t scanset_id,
|
||||||
const unsigned char sha256[LARDON3D_PROJECT_DB_SHA256_SIZE],
|
const unsigned char sha256[LARDON3D_PROJECT_DB_SHA256_SIZE], const char *asset_path,
|
||||||
const char *asset_path, uint64_t size_bytes, const char *original_name,
|
uint64_t size_bytes, const char *original_name, const char *source_path,
|
||||||
const char *source_path, uint64_t producer_task_id, int64_t imported_at,
|
uint64_t producer_task_id, int64_t imported_at, Lardon3DProjectDbImageRegisterStatus *status,
|
||||||
Lardon3DProjectDbImageRegisterStatus *status,
|
Lardon3DProjectDbImage *image);
|
||||||
Lardon3DProjectDbImage *image
|
Lardon3DProjectDbResult lardon3d_project_db_load_image(Lardon3DProjectDb *database,
|
||||||
);
|
uint64_t image_id,
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_load_image(
|
Lardon3DProjectDbImage *image,
|
||||||
Lardon3DProjectDb *database, uint64_t image_id,
|
Lardon3DProjectDbImageAsset *asset);
|
||||||
Lardon3DProjectDbImage *image, Lardon3DProjectDbImageAsset *asset
|
Lardon3DProjectDbResult lardon3d_project_db_list_images(Lardon3DProjectDb *database,
|
||||||
);
|
uint64_t scanset_id,
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_list_images(
|
uint64_t after_image_id,
|
||||||
Lardon3DProjectDb *database, uint64_t scanset_id, uint64_t after_image_id,
|
Lardon3DProjectDbImage *images,
|
||||||
Lardon3DProjectDbImage *images, Lardon3DProjectDbImageAsset *assets,
|
Lardon3DProjectDbImageAsset *assets,
|
||||||
size_t capacity, size_t *count
|
size_t capacity, size_t *count);
|
||||||
);
|
Lardon3DProjectDbResult lardon3d_project_db_count_images(Lardon3DProjectDb *database,
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_count_images(
|
uint64_t scanset_id, uint64_t *count);
|
||||||
Lardon3DProjectDb *database, uint64_t scanset_id, uint64_t *count
|
Lardon3DProjectDbResult
|
||||||
);
|
lardon3d_project_db_load_image_import(Lardon3DProjectDb *database, uint64_t task_id,
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_load_image_import(
|
Lardon3DProjectDbImageImport *parameters);
|
||||||
Lardon3DProjectDb *database,
|
Lardon3DProjectDbResult lardon3d_project_db_allocate_task_id(Lardon3DProjectDb *database,
|
||||||
uint64_t task_id,
|
uint64_t *task_id);
|
||||||
Lardon3DProjectDbImageImport *parameters
|
Lardon3DProjectDbResult lardon3d_project_db_load_task(Lardon3DProjectDb *database, uint64_t task_id,
|
||||||
);
|
Lardon3DProjectDbTask *task);
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_allocate_task_id(
|
Lardon3DProjectDbResult lardon3d_project_db_list_recoverable(Lardon3DProjectDb *database,
|
||||||
Lardon3DProjectDb *database,
|
uint64_t after_task_id,
|
||||||
uint64_t *task_id
|
Lardon3DProjectDbTask *tasks,
|
||||||
);
|
size_t capacity, size_t *count);
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_load_task(
|
Lardon3DProjectDbResult
|
||||||
Lardon3DProjectDb *database,
|
lardon3d_project_db_create_artifact(Lardon3DProjectDb *database,
|
||||||
uint64_t task_id,
|
const Lardon3DProjectDbArtifact *artifact);
|
||||||
Lardon3DProjectDbTask *task
|
Lardon3DProjectDbResult lardon3d_project_db_mark_artifact_ready(Lardon3DProjectDb *database,
|
||||||
);
|
const char *artifact_id,
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_list_recoverable(
|
int64_t updated_at);
|
||||||
Lardon3DProjectDb *database,
|
Lardon3DProjectDbResult lardon3d_project_db_load_artifact(Lardon3DProjectDb *database,
|
||||||
uint64_t after_task_id,
|
const char *artifact_id,
|
||||||
Lardon3DProjectDbTask *tasks,
|
Lardon3DProjectDbArtifact *artifact);
|
||||||
size_t capacity,
|
Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_task(
|
||||||
size_t *count
|
Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, const char *task_kind,
|
||||||
);
|
uint32_t task_kind_version, const Lardon3DProjectDbCheckpoint *checkpoint,
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_create_artifact(
|
const Lardon3DProjectDbFeatureExtractTask *parameters, int64_t updated_at);
|
||||||
Lardon3DProjectDb *database,
|
Lardon3DProjectDbResult
|
||||||
const Lardon3DProjectDbArtifact *artifact
|
lardon3d_project_db_load_feature_extract_task(Lardon3DProjectDb *database, uint64_t task_id,
|
||||||
);
|
Lardon3DProjectDbFeatureExtractTask *parameters);
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_mark_artifact_ready(
|
Lardon3DProjectDbResult lardon3d_project_db_register_feature_set(
|
||||||
Lardon3DProjectDb *database,
|
Lardon3DProjectDb *database, uint64_t image_id, const char *extractor_kind,
|
||||||
const char *artifact_id,
|
uint32_t extractor_version, const unsigned char parameter_fingerprint[32],
|
||||||
int64_t updated_at
|
const unsigned char source_image_sha256[32], uint32_t feature_count, uint32_t descriptor_type,
|
||||||
);
|
uint32_t descriptor_dimension, const unsigned char asset_sha256[32], const char *asset_path,
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_load_artifact(
|
uint64_t asset_size_bytes, Lardon3DProjectDbFeatureDurability durability,
|
||||||
Lardon3DProjectDb *database,
|
uint64_t producer_task_id, int64_t created_at, Lardon3DProjectDbFeatureSet *feature_set);
|
||||||
const char *artifact_id,
|
Lardon3DProjectDbResult
|
||||||
Lardon3DProjectDbArtifact *artifact
|
lardon3d_project_db_find_feature_set(Lardon3DProjectDb *database, uint64_t image_id,
|
||||||
);
|
const char *extractor_kind, uint32_t extractor_version,
|
||||||
|
const unsigned char parameter_fingerprint[32],
|
||||||
|
Lardon3DProjectDbFeatureSet *feature_set);
|
||||||
|
Lardon3DProjectDbResult
|
||||||
|
lardon3d_project_db_load_feature_set(Lardon3DProjectDb *database, uint64_t feature_set_id,
|
||||||
|
Lardon3DProjectDbFeatureSet *feature_set);
|
||||||
|
Lardon3DProjectDbResult
|
||||||
|
lardon3d_project_db_list_feature_sets(Lardon3DProjectDb *database, uint64_t after_feature_set_id,
|
||||||
|
Lardon3DProjectDbFeatureSet *feature_sets, size_t capacity,
|
||||||
|
size_t *count);
|
||||||
|
|
||||||
#endif
|
#endif
|
||||||
|
|
|
||||||
|
|
@ -6,63 +6,59 @@
|
||||||
|
|
||||||
#include <lardon3d/task.h>
|
#include <lardon3d/task.h>
|
||||||
|
|
||||||
|
typedef struct Lardon3DProjectDb Lardon3DProjectDb;
|
||||||
|
typedef struct Lardon3DResourceGovernor Lardon3DResourceGovernor;
|
||||||
|
|
||||||
enum {
|
enum {
|
||||||
LARDON3D_TASK_KIND_REGISTRY_MAX = 64,
|
LARDON3D_TASK_KIND_REGISTRY_MAX = 64,
|
||||||
};
|
};
|
||||||
|
|
||||||
typedef struct {
|
typedef struct {
|
||||||
Lardon3DTaskCallback callback;
|
Lardon3DTaskCallback callback;
|
||||||
void *userdata;
|
void *userdata;
|
||||||
Lardon3DTaskUserdataDestroy userdata_destroy;
|
Lardon3DTaskUserdataDestroy userdata_destroy;
|
||||||
Lardon3DTaskFinishedCallback finished_callback;
|
Lardon3DTaskFinishedCallback finished_callback;
|
||||||
void *finished_userdata;
|
void *finished_userdata;
|
||||||
} Lardon3DTaskKindBinding;
|
} Lardon3DTaskKindBinding;
|
||||||
|
|
||||||
typedef bool (*Lardon3DTaskKindReconstruct)(
|
typedef bool (*Lardon3DTaskKindReconstruct)(const Lardon3DTaskDurableSnapshot *snapshot,
|
||||||
const Lardon3DTaskDurableSnapshot *snapshot,
|
void *context, Lardon3DTaskKindBinding *binding);
|
||||||
void *context,
|
|
||||||
Lardon3DTaskKindBinding *binding
|
|
||||||
);
|
|
||||||
|
|
||||||
typedef struct {
|
typedef struct {
|
||||||
const char *kind;
|
const char *kind;
|
||||||
uint32_t kind_version;
|
uint32_t kind_version;
|
||||||
Lardon3DTaskKindReconstruct reconstruct;
|
Lardon3DTaskKindReconstruct reconstruct;
|
||||||
} Lardon3DTaskKindDescriptor;
|
} Lardon3DTaskKindDescriptor;
|
||||||
|
|
||||||
typedef struct {
|
typedef struct {
|
||||||
const Lardon3DTaskKindDescriptor *descriptors;
|
const Lardon3DTaskKindDescriptor *descriptors;
|
||||||
size_t count;
|
size_t count;
|
||||||
} Lardon3DTaskKindRegistry;
|
} Lardon3DTaskKindRegistry;
|
||||||
|
|
||||||
|
typedef struct {
|
||||||
|
const char *project_path;
|
||||||
|
Lardon3DProjectDb *project_db;
|
||||||
|
Lardon3DResourceGovernor *resource_governor;
|
||||||
|
} Lardon3DTaskReconstructionContext;
|
||||||
|
|
||||||
typedef enum {
|
typedef enum {
|
||||||
LARDON3D_TASK_KIND_OK = 0,
|
LARDON3D_TASK_KIND_OK = 0,
|
||||||
LARDON3D_TASK_KIND_INVALID_ARGUMENT,
|
LARDON3D_TASK_KIND_INVALID_ARGUMENT,
|
||||||
LARDON3D_TASK_KIND_UNKNOWN,
|
LARDON3D_TASK_KIND_UNKNOWN,
|
||||||
LARDON3D_TASK_KIND_UNSUPPORTED_VERSION,
|
LARDON3D_TASK_KIND_UNSUPPORTED_VERSION,
|
||||||
LARDON3D_TASK_KIND_RECONSTRUCTION_FAILED,
|
LARDON3D_TASK_KIND_RECONSTRUCTION_FAILED,
|
||||||
LARDON3D_TASK_KIND_RESTORE_FAILED
|
LARDON3D_TASK_KIND_RESTORE_FAILED
|
||||||
} Lardon3DTaskKindResult;
|
} Lardon3DTaskKindResult;
|
||||||
|
|
||||||
bool lardon3d_task_kind_registry_init(
|
bool lardon3d_task_kind_registry_init(Lardon3DTaskKindRegistry *registry,
|
||||||
Lardon3DTaskKindRegistry *registry,
|
const Lardon3DTaskKindDescriptor *descriptors, size_t count);
|
||||||
const Lardon3DTaskKindDescriptor *descriptors,
|
Lardon3DTaskKindResult
|
||||||
size_t count
|
lardon3d_task_kind_registry_lookup(const Lardon3DTaskKindRegistry *registry, const char *kind,
|
||||||
);
|
uint32_t kind_version,
|
||||||
Lardon3DTaskKindResult lardon3d_task_kind_registry_lookup(
|
const Lardon3DTaskKindDescriptor **descriptor);
|
||||||
const Lardon3DTaskKindRegistry *registry,
|
|
||||||
const char *kind,
|
|
||||||
uint32_t kind_version,
|
|
||||||
const Lardon3DTaskKindDescriptor **descriptor
|
|
||||||
);
|
|
||||||
Lardon3DTaskKindResult lardon3d_task_kind_registry_restore(
|
Lardon3DTaskKindResult lardon3d_task_kind_registry_restore(
|
||||||
const Lardon3DTaskKindRegistry *registry,
|
const Lardon3DTaskKindRegistry *registry, const char *kind, uint32_t kind_version,
|
||||||
const char *kind,
|
const Lardon3DTaskDurableSnapshot *snapshot, void *context, Lardon3DTask **task);
|
||||||
uint32_t kind_version,
|
|
||||||
const Lardon3DTaskDurableSnapshot *snapshot,
|
|
||||||
void *context,
|
|
||||||
Lardon3DTask **task
|
|
||||||
);
|
|
||||||
const Lardon3DTaskKindRegistry *lardon3d_task_kind_registry_production(void);
|
const Lardon3DTaskKindRegistry *lardon3d_task_kind_registry_production(void);
|
||||||
|
|
||||||
#endif
|
#endif
|
||||||
|
|
|
||||||
57
meson.build
57
meson.build
|
|
@ -1,6 +1,6 @@
|
||||||
project(
|
project(
|
||||||
'lardon3d',
|
'lardon3d',
|
||||||
'c',
|
'c', 'cpp',
|
||||||
version: '0.1.0',
|
version: '0.1.0',
|
||||||
license: 'MIT',
|
license: 'MIT',
|
||||||
default_options: [
|
default_options: [
|
||||||
|
|
@ -17,11 +17,13 @@ add_project_arguments(
|
||||||
'-Wshadow',
|
'-Wshadow',
|
||||||
language: 'c',
|
language: 'c',
|
||||||
)
|
)
|
||||||
|
add_project_arguments('-std=c++17', '-Wpedantic', '-Wconversion', '-Wshadow', language: 'cpp')
|
||||||
|
|
||||||
ncursesw = dependency('ncursesw', required: true)
|
ncursesw = dependency('ncursesw', required: true)
|
||||||
threads = dependency('threads')
|
threads = dependency('threads')
|
||||||
sqlite3 = dependency('sqlite3', required: true)
|
sqlite3 = dependency('sqlite3', required: true)
|
||||||
openssl = dependency('openssl', required: true)
|
openssl = dependency('openssl', required: true)
|
||||||
|
opencv = dependency('opencv5', required: true, include_type: 'system', modules: ['opencv_core', 'opencv_imgcodecs', 'opencv_features2d'])
|
||||||
|
|
||||||
executable(
|
executable(
|
||||||
'lardon3d',
|
'lardon3d',
|
||||||
|
|
@ -35,6 +37,9 @@ executable(
|
||||||
'src/import_task.c',
|
'src/import_task.c',
|
||||||
'src/image_catalog.c',
|
'src/image_catalog.c',
|
||||||
'src/image_catalog_persistent.c',
|
'src/image_catalog_persistent.c',
|
||||||
|
'src/feature_extractor_opencv.cpp',
|
||||||
|
'src/feature_store.c',
|
||||||
|
'src/feature_task.c',
|
||||||
'src/image_view.c',
|
'src/image_view.c',
|
||||||
'src/project.c',
|
'src/project.c',
|
||||||
'src/project_db.c',
|
'src/project_db.c',
|
||||||
|
|
@ -48,7 +53,7 @@ executable(
|
||||||
'src/hardware_profile.c',
|
'src/hardware_profile.c',
|
||||||
],
|
],
|
||||||
include_directories: include_directories('include'),
|
include_directories: include_directories('include'),
|
||||||
dependencies: [ncursesw, threads, sqlite3, openssl],
|
dependencies: [ncursesw, threads, sqlite3, openssl, opencv],
|
||||||
)
|
)
|
||||||
|
|
||||||
import_test = executable(
|
import_test = executable(
|
||||||
|
|
@ -82,6 +87,9 @@ import_task_test = executable(
|
||||||
'src/task_checkpoint.c',
|
'src/task_checkpoint.c',
|
||||||
'src/task_kind_registry.c',
|
'src/task_kind_registry.c',
|
||||||
'src/task_kinds.c',
|
'src/task_kinds.c',
|
||||||
|
'src/feature_task.c',
|
||||||
|
'src/feature_store.c',
|
||||||
|
'src/feature_extractor_opencv.cpp',
|
||||||
'src/task_queue.c',
|
'src/task_queue.c',
|
||||||
'src/resource_governor.c',
|
'src/resource_governor.c',
|
||||||
'src/resource_snapshot.c',
|
'src/resource_snapshot.c',
|
||||||
|
|
@ -95,7 +103,7 @@ import_task_test = executable(
|
||||||
'-DLARDON3D_CHECKPOINT_TESTING',
|
'-DLARDON3D_CHECKPOINT_TESTING',
|
||||||
],
|
],
|
||||||
include_directories: include_directories('include'),
|
include_directories: include_directories('include'),
|
||||||
dependencies: [threads, sqlite3, openssl],
|
dependencies: [threads, sqlite3, openssl, opencv],
|
||||||
)
|
)
|
||||||
|
|
||||||
test('import-task', import_task_test, timeout: 30)
|
test('import-task', import_task_test, timeout: 30)
|
||||||
|
|
@ -136,6 +144,44 @@ persistent_image_catalog_test = executable(
|
||||||
|
|
||||||
test('persistent-image-catalog', persistent_image_catalog_test, timeout: 60)
|
test('persistent-image-catalog', persistent_image_catalog_test, timeout: 60)
|
||||||
|
|
||||||
|
feature_store_test = executable(
|
||||||
|
'test-feature-store',
|
||||||
|
sources: [
|
||||||
|
'tests/test_feature_store.c',
|
||||||
|
'src/app_state.c',
|
||||||
|
'src/image_catalog_persistent.c',
|
||||||
|
'src/feature_extractor_opencv.cpp',
|
||||||
|
'src/feature_store.c',
|
||||||
|
'src/project_db.c',
|
||||||
|
'src/task.c',
|
||||||
|
'src/resource_governor.c',
|
||||||
|
'src/resource_snapshot.c',
|
||||||
|
],
|
||||||
|
c_args: ['-DLARDON3D_PROJECT_DB_TESTING', '-DLARDON3D_FEATURE_STORE_TESTING'],
|
||||||
|
include_directories: include_directories('include'),
|
||||||
|
dependencies: [threads, sqlite3, openssl, opencv],
|
||||||
|
)
|
||||||
|
|
||||||
|
test('feature-store', feature_store_test, timeout: 60)
|
||||||
|
|
||||||
|
feature_task_test = executable(
|
||||||
|
'test-feature-task',
|
||||||
|
sources: [
|
||||||
|
'tests/test_feature_task.c', 'src/app_state.c', 'src/project.c',
|
||||||
|
'src/project_db.c', 'src/task.c', 'src/task_checkpoint.c',
|
||||||
|
'src/task_kind_registry.c', 'src/task_kinds.c', 'src/task_queue.c',
|
||||||
|
'src/import.c', 'src/import_task.c', 'src/image_catalog.c',
|
||||||
|
'src/image_catalog_persistent.c', 'src/image_view.c',
|
||||||
|
'src/feature_task.c', 'src/feature_store.c',
|
||||||
|
'src/feature_extractor_opencv.cpp', 'src/resource_governor.c',
|
||||||
|
'src/resource_snapshot.c',
|
||||||
|
],
|
||||||
|
c_args: ['-DLARDON3D_PROJECT_DB_TESTING', '-DLARDON3D_FEATURE_TASK_TESTING'],
|
||||||
|
include_directories: include_directories('include'),
|
||||||
|
dependencies: [threads, sqlite3, openssl, opencv],
|
||||||
|
)
|
||||||
|
test('feature-task', feature_task_test, timeout: 60)
|
||||||
|
|
||||||
image_view_test = executable(
|
image_view_test = executable(
|
||||||
'test-image-view',
|
'test-image-view',
|
||||||
sources: [
|
sources: [
|
||||||
|
|
@ -206,6 +252,9 @@ project_test = executable(
|
||||||
'src/task_checkpoint.c',
|
'src/task_checkpoint.c',
|
||||||
'src/task_kind_registry.c',
|
'src/task_kind_registry.c',
|
||||||
'src/task_kinds.c',
|
'src/task_kinds.c',
|
||||||
|
'src/feature_task.c',
|
||||||
|
'src/feature_store.c',
|
||||||
|
'src/feature_extractor_opencv.cpp',
|
||||||
'src/task_queue.c',
|
'src/task_queue.c',
|
||||||
'src/import.c',
|
'src/import.c',
|
||||||
'src/import_task.c',
|
'src/import_task.c',
|
||||||
|
|
@ -220,7 +269,7 @@ project_test = executable(
|
||||||
'-DLARDON3D_CHECKPOINT_TESTING',
|
'-DLARDON3D_CHECKPOINT_TESTING',
|
||||||
],
|
],
|
||||||
include_directories: include_directories('include'),
|
include_directories: include_directories('include'),
|
||||||
dependencies: [threads, sqlite3, openssl],
|
dependencies: [threads, sqlite3, openssl, opencv],
|
||||||
)
|
)
|
||||||
|
|
||||||
test('project', project_test, timeout: 30)
|
test('project', project_test, timeout: 30)
|
||||||
|
|
|
||||||
142
src/feature_extractor_opencv.cpp
Normal file
142
src/feature_extractor_opencv.cpp
Normal file
|
|
@ -0,0 +1,142 @@
|
||||||
|
#include <cerrno>
|
||||||
|
#include <cmath>
|
||||||
|
#include <cstdlib>
|
||||||
|
#include <cstring>
|
||||||
|
#include <limits>
|
||||||
|
#include <new>
|
||||||
|
#include <sys/stat.h>
|
||||||
|
#include <vector>
|
||||||
|
|
||||||
|
#include <opencv2/features2d.hpp>
|
||||||
|
#include <opencv2/imgcodecs.hpp>
|
||||||
|
#include <openssl/sha.h>
|
||||||
|
|
||||||
|
extern "C" {
|
||||||
|
#include <lardon3d/feature_extractor.h>
|
||||||
|
}
|
||||||
|
|
||||||
|
extern "C" bool
|
||||||
|
lardon3d_feature_extractor_parameters_valid(const Lardon3DFeatureExtractorParameters *parameters) {
|
||||||
|
return parameters && parameters->max_features > 0 &&
|
||||||
|
parameters->max_features <= LARDON3D_FEATURE_MAX_FEATURES &&
|
||||||
|
parameters->pyramid_levels >= 1 && parameters->pyramid_levels <= 16 &&
|
||||||
|
parameters->fast_threshold >= 1 && parameters->fast_threshold <= 255;
|
||||||
|
}
|
||||||
|
|
||||||
|
static void put_u32(unsigned char *output, uint32_t value) {
|
||||||
|
output[0] = static_cast<unsigned char>(value);
|
||||||
|
output[1] = static_cast<unsigned char>(value >> 8);
|
||||||
|
output[2] = static_cast<unsigned char>(value >> 16);
|
||||||
|
output[3] = static_cast<unsigned char>(value >> 24);
|
||||||
|
}
|
||||||
|
|
||||||
|
extern "C" void lardon3d_feature_extractor_parameter_fingerprint(
|
||||||
|
const Lardon3DFeatureExtractorParameters *parameters, unsigned char fingerprint[32]) {
|
||||||
|
if (!fingerprint) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
std::memset(fingerprint, 0, 32);
|
||||||
|
if (!lardon3d_feature_extractor_parameters_valid(parameters)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
unsigned char canonical[24] = {'L', '3', 'D', 'O', 'R', 'B', 'P', '1'};
|
||||||
|
put_u32(canonical + 8, LARDON3D_FEATURE_EXTRACTOR_VERSION);
|
||||||
|
put_u32(canonical + 12, parameters->max_features);
|
||||||
|
put_u32(canonical + 16, parameters->pyramid_levels);
|
||||||
|
put_u32(canonical + 20, parameters->fast_threshold);
|
||||||
|
(void)SHA256(canonical, sizeof(canonical), fingerprint);
|
||||||
|
}
|
||||||
|
|
||||||
|
extern "C" void lardon3d_extracted_features_destroy(Lardon3DExtractedFeatures *features) {
|
||||||
|
if (!features) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
std::free(features->keypoints);
|
||||||
|
std::free(features->descriptors);
|
||||||
|
std::memset(features, 0, sizeof(*features));
|
||||||
|
}
|
||||||
|
|
||||||
|
extern "C" Lardon3DFeatureExtractResult
|
||||||
|
lardon3d_feature_extract_orb(const char *path, const Lardon3DFeatureExtractorParameters *parameters,
|
||||||
|
Lardon3DExtractedFeatures *features) {
|
||||||
|
if (features) {
|
||||||
|
std::memset(features, 0, sizeof(*features));
|
||||||
|
}
|
||||||
|
if (!path || !path[0] || !features || !lardon3d_feature_extractor_parameters_valid(parameters)) {
|
||||||
|
return LARDON3D_FEATURE_EXTRACT_INVALID_ARGUMENT;
|
||||||
|
}
|
||||||
|
struct stat information{};
|
||||||
|
if (lstat(path, &information) != 0) {
|
||||||
|
return errno == ENOENT ? LARDON3D_FEATURE_EXTRACT_IMAGE_NOT_FOUND
|
||||||
|
: LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID;
|
||||||
|
}
|
||||||
|
if (!S_ISREG(information.st_mode) || S_ISLNK(information.st_mode)) {
|
||||||
|
return LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
cv::Mat image = cv::imread(path, cv::IMREAD_GRAYSCALE);
|
||||||
|
if (image.empty()) {
|
||||||
|
return LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID;
|
||||||
|
}
|
||||||
|
if (image.cols <= 0 || image.rows <= 0 ||
|
||||||
|
static_cast<uint64_t>(image.cols) * static_cast<uint64_t>(image.rows) >
|
||||||
|
LARDON3D_FEATURE_MAX_IMAGE_PIXELS) {
|
||||||
|
return LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID;
|
||||||
|
}
|
||||||
|
cv::Ptr<cv::ORB> extractor =
|
||||||
|
cv::ORB::create(static_cast<int>(parameters->max_features), 1.2F,
|
||||||
|
static_cast<int>(parameters->pyramid_levels), 31, 0, 2,
|
||||||
|
cv::ORB::HARRIS_SCORE, 31, static_cast<int>(parameters->fast_threshold));
|
||||||
|
std::vector<cv::KeyPoint> points;
|
||||||
|
cv::Mat descriptors;
|
||||||
|
extractor->detectAndCompute(image, cv::noArray(), points, descriptors);
|
||||||
|
if (points.size() > parameters->max_features ||
|
||||||
|
points.size() > std::numeric_limits<uint32_t>::max() ||
|
||||||
|
(!points.empty() && (descriptors.rows != static_cast<int>(points.size()) ||
|
||||||
|
descriptors.cols != LARDON3D_FEATURE_DESCRIPTOR_DIMENSION ||
|
||||||
|
descriptors.type() != CV_8UC1))) {
|
||||||
|
return LARDON3D_FEATURE_EXTRACT_ERROR;
|
||||||
|
}
|
||||||
|
size_t count = points.size();
|
||||||
|
Lardon3DFeatureKeypoint *keypoints =
|
||||||
|
count == 0 ? nullptr
|
||||||
|
: static_cast<Lardon3DFeatureKeypoint *>(
|
||||||
|
std::calloc(count, sizeof(Lardon3DFeatureKeypoint)));
|
||||||
|
unsigned char *bytes = count == 0 ? nullptr
|
||||||
|
: static_cast<unsigned char *>(std::malloc(
|
||||||
|
count * LARDON3D_FEATURE_DESCRIPTOR_DIMENSION));
|
||||||
|
if (count > 0 && (!keypoints || !bytes)) {
|
||||||
|
std::free(keypoints);
|
||||||
|
std::free(bytes);
|
||||||
|
return LARDON3D_FEATURE_EXTRACT_OUT_OF_MEMORY;
|
||||||
|
}
|
||||||
|
for (size_t index = 0; index < count; ++index) {
|
||||||
|
const cv::KeyPoint &point = points[index];
|
||||||
|
if (!std::isfinite(point.pt.x) || !std::isfinite(point.pt.y) || !std::isfinite(point.size) ||
|
||||||
|
!std::isfinite(point.angle) || !std::isfinite(point.response) || point.pt.x < 0.0F ||
|
||||||
|
point.pt.y < 0.0F || point.pt.x >= static_cast<float>(image.cols) ||
|
||||||
|
point.pt.y >= static_cast<float>(image.rows) || point.size <= 0.0F ||
|
||||||
|
point.angle < 0.0F || point.angle >= 360.0F) {
|
||||||
|
std::free(keypoints);
|
||||||
|
std::free(bytes);
|
||||||
|
return LARDON3D_FEATURE_EXTRACT_ERROR;
|
||||||
|
}
|
||||||
|
keypoints[index] = {point.pt.x, point.pt.y, point.size,
|
||||||
|
point.angle, point.response, point.octave};
|
||||||
|
std::memcpy(bytes + index * LARDON3D_FEATURE_DESCRIPTOR_DIMENSION,
|
||||||
|
descriptors.ptr(static_cast<int>(index)), LARDON3D_FEATURE_DESCRIPTOR_DIMENSION);
|
||||||
|
}
|
||||||
|
features->image_width = static_cast<uint32_t>(image.cols);
|
||||||
|
features->image_height = static_cast<uint32_t>(image.rows);
|
||||||
|
features->feature_count = static_cast<uint32_t>(count);
|
||||||
|
features->keypoints = keypoints;
|
||||||
|
features->descriptors = bytes;
|
||||||
|
return LARDON3D_FEATURE_EXTRACT_OK;
|
||||||
|
} catch (const cv::Exception &) {
|
||||||
|
return LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID;
|
||||||
|
} catch (const std::bad_alloc &) {
|
||||||
|
return LARDON3D_FEATURE_EXTRACT_OUT_OF_MEMORY;
|
||||||
|
} catch (...) {
|
||||||
|
return LARDON3D_FEATURE_EXTRACT_ERROR;
|
||||||
|
}
|
||||||
|
}
|
||||||
575
src/feature_store.c
Normal file
575
src/feature_store.c
Normal file
|
|
@ -0,0 +1,575 @@
|
||||||
|
#include <errno.h>
|
||||||
|
#include <fcntl.h>
|
||||||
|
#include <float.h>
|
||||||
|
#include <math.h>
|
||||||
|
#include <openssl/evp.h>
|
||||||
|
#include <stdbool.h>
|
||||||
|
#include <stdint.h>
|
||||||
|
#include <stdio.h>
|
||||||
|
#include <stdlib.h>
|
||||||
|
#include <string.h>
|
||||||
|
#include <sys/stat.h>
|
||||||
|
#include <unistd.h>
|
||||||
|
|
||||||
|
#include <lardon3d/feature_store.h>
|
||||||
|
|
||||||
|
static const unsigned char feature_magic[8] = {'L', '3', 'D', 'F', 'E', 'A', 'T', 0};
|
||||||
|
_Static_assert(sizeof(float) == 4 && FLT_RADIX == 2 && FLT_MANT_DIG == 24 && FLT_MAX_EXP == 128,
|
||||||
|
"Feature File v1 exige IEEE-754 binary32.");
|
||||||
|
|
||||||
|
struct Lardon3DFeatureReader {
|
||||||
|
int descriptor;
|
||||||
|
uint64_t keypoint_offset;
|
||||||
|
uint64_t descriptor_offset;
|
||||||
|
Lardon3DFeatureFileMetadata metadata;
|
||||||
|
};
|
||||||
|
|
||||||
|
static void put_u32(unsigned char *p, uint32_t v) {
|
||||||
|
p[0] = (unsigned char)v;
|
||||||
|
p[1] = (unsigned char)(v >> 8);
|
||||||
|
p[2] = (unsigned char)(v >> 16);
|
||||||
|
p[3] = (unsigned char)(v >> 24);
|
||||||
|
}
|
||||||
|
static void put_u64(unsigned char *p, uint64_t v) {
|
||||||
|
for (unsigned int i = 0; i < 8; ++i) {
|
||||||
|
p[i] = (unsigned char)(v >> (8U * i));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
static uint32_t get_u32(const unsigned char *p) {
|
||||||
|
return (uint32_t)p[0] | ((uint32_t)p[1] << 8) | ((uint32_t)p[2] << 16) | ((uint32_t)p[3] << 24);
|
||||||
|
}
|
||||||
|
static uint64_t get_u64(const unsigned char *p) {
|
||||||
|
uint64_t v = 0;
|
||||||
|
for (unsigned int i = 0; i < 8; ++i) {
|
||||||
|
v |= (uint64_t)p[i] << (8U * i);
|
||||||
|
}
|
||||||
|
return v;
|
||||||
|
}
|
||||||
|
static void put_float(unsigned char *p, float value) {
|
||||||
|
uint32_t bits = 0;
|
||||||
|
memcpy(&bits, &value, sizeof(bits));
|
||||||
|
put_u32(p, bits);
|
||||||
|
}
|
||||||
|
static float get_float(const unsigned char *p) {
|
||||||
|
uint32_t bits = get_u32(p);
|
||||||
|
float value = 0;
|
||||||
|
memcpy(&value, &bits, sizeof(value));
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool write_all(int fd, const void *data, size_t size) {
|
||||||
|
const unsigned char *bytes = data;
|
||||||
|
size_t used = 0;
|
||||||
|
while (used < size) {
|
||||||
|
ssize_t n = write(fd, bytes + used, size - used);
|
||||||
|
if (n < 0 && errno == EINTR) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (n <= 0) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
used += (size_t)n;
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool read_exact(int fd, void *data, size_t size, off_t offset) {
|
||||||
|
unsigned char *bytes = data;
|
||||||
|
size_t used = 0;
|
||||||
|
while (used < size) {
|
||||||
|
ssize_t n = pread(fd, bytes + used, size - used, offset + (off_t)used);
|
||||||
|
if (n < 0 && errno == EINTR) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (n <= 0) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
used += (size_t)n;
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool sha256_fd(int fd, unsigned char output[32]) {
|
||||||
|
EVP_MD_CTX *context = EVP_MD_CTX_new();
|
||||||
|
if (!context) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
bool ok = EVP_DigestInit_ex(context, EVP_sha256(), NULL) == 1;
|
||||||
|
unsigned char buffer[65536];
|
||||||
|
off_t offset = 0;
|
||||||
|
while (ok) {
|
||||||
|
ssize_t n = pread(fd, buffer, sizeof(buffer), offset);
|
||||||
|
if (n < 0 && errno == EINTR) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (n < 0) {
|
||||||
|
ok = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
if (n == 0) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
ok = EVP_DigestUpdate(context, buffer, (size_t)n) == 1;
|
||||||
|
offset += (off_t)n;
|
||||||
|
}
|
||||||
|
unsigned int length = 0;
|
||||||
|
ok = ok && EVP_DigestFinal_ex(context, output, &length) == 1 && length == 32;
|
||||||
|
EVP_MD_CTX_free(context);
|
||||||
|
return ok;
|
||||||
|
}
|
||||||
|
|
||||||
|
static void hex_sha(const unsigned char hash[32], char text[65]) {
|
||||||
|
static const char digits[] = "0123456789abcdef";
|
||||||
|
for (size_t i = 0; i < 32; ++i) {
|
||||||
|
text[2 * i] = digits[hash[i] >> 4];
|
||||||
|
text[2 * i + 1] = digits[hash[i] & 15];
|
||||||
|
}
|
||||||
|
text[64] = '\0';
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool canonical_feature_path(const unsigned char hash[32], const char *path) {
|
||||||
|
if (!hash || !path) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
char hex[65];
|
||||||
|
hex_sha(hash, hex);
|
||||||
|
char expected[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
||||||
|
int written = snprintf(expected, sizeof(expected), "assets/features/%.2s/%s", hex, hex);
|
||||||
|
return written > 0 && (size_t)written < sizeof(expected) && strcmp(path, expected) == 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool join(char output[PATH_MAX], const char *a, const char *b) {
|
||||||
|
int n = snprintf(output, PATH_MAX, "%s/%s", a, b);
|
||||||
|
return n > 0 && (size_t)n < PATH_MAX;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool ensure_directory(const char *path) {
|
||||||
|
if (mkdir(path, 0755) == 0) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (errno != EEXIST) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
struct stat info;
|
||||||
|
return lstat(path, &info) == 0 && S_ISDIR(info.st_mode) && !S_ISLNK(info.st_mode);
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool sync_directory(const char *path) {
|
||||||
|
int fd = open(path, O_RDONLY | O_DIRECTORY | O_CLOEXEC);
|
||||||
|
if (fd < 0) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
bool ok = fsync(fd) == 0;
|
||||||
|
if (close(fd) != 0) {
|
||||||
|
ok = false;
|
||||||
|
}
|
||||||
|
return ok;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool sync_feature_asset_directory(const char *project_path,
|
||||||
|
const Lardon3DProjectDbFeatureSet *feature_set) {
|
||||||
|
char relative[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
||||||
|
int written = snprintf(relative, sizeof(relative), "%s", feature_set->asset.path);
|
||||||
|
if (written <= 0 || (size_t)written >= sizeof(relative)) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
char *separator = strrchr(relative, '/');
|
||||||
|
if (!separator) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
*separator = '\0';
|
||||||
|
char directory[PATH_MAX];
|
||||||
|
return join(directory, project_path, relative) && sync_directory(directory);
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool valid_features(const Lardon3DExtractedFeatures *features) {
|
||||||
|
if (!features || features->feature_count > LARDON3D_FEATURE_MAX_FEATURES ||
|
||||||
|
features->image_width == 0 || features->image_height == 0 ||
|
||||||
|
(features->feature_count > 0 && (!features->keypoints || !features->descriptors))) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
for (uint32_t i = 0; i < features->feature_count; ++i) {
|
||||||
|
const Lardon3DFeatureKeypoint *p = &features->keypoints[i];
|
||||||
|
if (!isfinite(p->x) || !isfinite(p->y) || !isfinite(p->size) || !isfinite(p->angle_degrees) ||
|
||||||
|
!isfinite(p->response) || p->x < 0 || p->y < 0 || p->x >= (float)features->image_width ||
|
||||||
|
p->y >= (float)features->image_height || p->size <= 0 || p->angle_degrees < 0 ||
|
||||||
|
p->angle_degrees >= 360 || p->octave < 0 || p->octave >= 16) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool write_feature_file(int fd, const Lardon3DProjectDbImageAsset *image_asset,
|
||||||
|
const unsigned char fingerprint[32],
|
||||||
|
const Lardon3DExtractedFeatures *features, uint64_t *file_size) {
|
||||||
|
uint64_t kp_bytes = (uint64_t)features->feature_count * LARDON3D_FEATURE_KEYPOINT_RECORD_SIZE;
|
||||||
|
uint64_t descriptor_bytes =
|
||||||
|
(uint64_t)features->feature_count * LARDON3D_FEATURE_DESCRIPTOR_DIMENSION;
|
||||||
|
uint64_t descriptor_offset = LARDON3D_FEATURE_FILE_HEADER_SIZE + kp_bytes;
|
||||||
|
uint64_t total = descriptor_offset + descriptor_bytes;
|
||||||
|
if (total > LARDON3D_FEATURE_FILE_MAX_SIZE) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
unsigned char header[LARDON3D_FEATURE_FILE_HEADER_SIZE] = {0};
|
||||||
|
memcpy(header, feature_magic, 8);
|
||||||
|
put_u32(header + 8, LARDON3D_FEATURE_FILE_VERSION);
|
||||||
|
put_u32(header + 12, LARDON3D_FEATURE_FILE_HEADER_SIZE);
|
||||||
|
put_u32(header + 16, features->feature_count);
|
||||||
|
put_u32(header + 20, LARDON3D_FEATURE_DESCRIPTOR_DIMENSION);
|
||||||
|
put_u32(header + 24, LARDON3D_FEATURE_DESCRIPTOR_U8);
|
||||||
|
put_u32(header + 28, LARDON3D_FEATURE_KEYPOINT_RECORD_SIZE);
|
||||||
|
put_u32(header + 32, features->image_width);
|
||||||
|
put_u32(header + 36, features->image_height);
|
||||||
|
put_u64(header + 40, LARDON3D_FEATURE_FILE_HEADER_SIZE);
|
||||||
|
put_u64(header + 48, descriptor_offset);
|
||||||
|
put_u64(header + 56, total);
|
||||||
|
memcpy(header + 64, image_asset->sha256, 32);
|
||||||
|
memcpy(header + 96, fingerprint, 32);
|
||||||
|
memcpy(header + 128, LARDON3D_FEATURE_EXTRACTOR_KIND, 4);
|
||||||
|
put_u32(header + 144, LARDON3D_FEATURE_EXTRACTOR_VERSION);
|
||||||
|
if (!write_all(fd, header, sizeof(header))) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
unsigned char record[LARDON3D_FEATURE_KEYPOINT_RECORD_SIZE];
|
||||||
|
for (uint32_t i = 0; i < features->feature_count; ++i) {
|
||||||
|
const Lardon3DFeatureKeypoint *p = &features->keypoints[i];
|
||||||
|
put_float(record, p->x);
|
||||||
|
put_float(record + 4, p->y);
|
||||||
|
put_float(record + 8, p->size);
|
||||||
|
put_float(record + 12, p->angle_degrees);
|
||||||
|
put_float(record + 16, p->response);
|
||||||
|
put_u32(record + 20, (uint32_t)p->octave);
|
||||||
|
if (!write_all(fd, record, sizeof(record))) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (descriptor_bytes > 0 && !write_all(fd, features->descriptors, (size_t)descriptor_bytes)) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
*file_size = total;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
static Lardon3DFeatureStoreResult
|
||||||
|
reuse_existing_feature_set(Lardon3DAppState *state, uint64_t image_id, uint64_t producer_task_id,
|
||||||
|
const unsigned char fingerprint[32],
|
||||||
|
Lardon3DProjectDbFeatureSet *feature_set) {
|
||||||
|
if (feature_set->asset.durability == LARDON3D_DB_FEATURE_ASSET_DURABLE) {
|
||||||
|
return LARDON3D_FEATURE_STORE_ALREADY_PRESENT;
|
||||||
|
}
|
||||||
|
Lardon3DFeatureReader *reader = NULL;
|
||||||
|
Lardon3DFeatureFileMetadata metadata;
|
||||||
|
Lardon3DFeatureStoreResult validation =
|
||||||
|
lardon3d_feature_reader_open(state->project_path, feature_set, &reader, &metadata);
|
||||||
|
lardon3d_feature_reader_close(reader);
|
||||||
|
if (validation != LARDON3D_FEATURE_STORE_OK ||
|
||||||
|
!sync_feature_asset_directory(state->project_path, feature_set)) {
|
||||||
|
return LARDON3D_FEATURE_STORE_PUBLISHED_NOT_DURABLE;
|
||||||
|
}
|
||||||
|
int64_t now = (int64_t)time(NULL);
|
||||||
|
if (now < 0) {
|
||||||
|
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||||
|
}
|
||||||
|
Lardon3DProjectDbResult promoted = lardon3d_project_db_register_feature_set(
|
||||||
|
state->project_db, image_id, LARDON3D_FEATURE_EXTRACTOR_KIND,
|
||||||
|
LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, feature_set->source_image_sha256,
|
||||||
|
feature_set->feature_count, feature_set->descriptor_type, feature_set->descriptor_dimension,
|
||||||
|
feature_set->asset.sha256, feature_set->asset.path, feature_set->asset.size_bytes,
|
||||||
|
LARDON3D_DB_FEATURE_ASSET_DURABLE, producer_task_id, now, feature_set);
|
||||||
|
return promoted == LARDON3D_PROJECT_DB_OK ? LARDON3D_FEATURE_STORE_ALREADY_PRESENT
|
||||||
|
: LARDON3D_FEATURE_STORE_DB_ERROR;
|
||||||
|
}
|
||||||
|
|
||||||
|
static Lardon3DFeatureStoreResult register_published_feature_set(
|
||||||
|
Lardon3DAppState *state, uint64_t image_id, uint64_t producer_task_id,
|
||||||
|
const Lardon3DProjectDbImageAsset *image_asset, const unsigned char fingerprint[32],
|
||||||
|
const Lardon3DExtractedFeatures *features, const unsigned char file_hash[32],
|
||||||
|
const char *relative_path, uint64_t file_size, bool durable,
|
||||||
|
Lardon3DProjectDbFeatureSet *feature_set) {
|
||||||
|
int64_t now = (int64_t)time(NULL);
|
||||||
|
if (now < 0) {
|
||||||
|
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||||
|
}
|
||||||
|
Lardon3DProjectDbResult result = lardon3d_project_db_register_feature_set(
|
||||||
|
state->project_db, image_id, LARDON3D_FEATURE_EXTRACTOR_KIND,
|
||||||
|
LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, image_asset->sha256, features->feature_count,
|
||||||
|
LARDON3D_FEATURE_DESCRIPTOR_U8, LARDON3D_FEATURE_DESCRIPTOR_DIMENSION, file_hash,
|
||||||
|
relative_path, file_size,
|
||||||
|
durable ? LARDON3D_DB_FEATURE_ASSET_DURABLE : LARDON3D_DB_FEATURE_ASSET_PUBLISHED_NOT_DURABLE,
|
||||||
|
producer_task_id, now, feature_set);
|
||||||
|
if (result != LARDON3D_PROJECT_DB_OK) {
|
||||||
|
return result == LARDON3D_PROJECT_DB_BUSY ? LARDON3D_FEATURE_STORE_DB_BUSY
|
||||||
|
: LARDON3D_FEATURE_STORE_DB_ERROR;
|
||||||
|
}
|
||||||
|
return durable ? LARDON3D_FEATURE_STORE_OK : LARDON3D_FEATURE_STORE_PUBLISHED_NOT_DURABLE;
|
||||||
|
}
|
||||||
|
|
||||||
|
Lardon3DFeatureStoreResult lardon3d_feature_store_publish(
|
||||||
|
Lardon3DAppState *state, uint64_t image_id, uint64_t producer_task_id,
|
||||||
|
const Lardon3DFeatureExtractorParameters *parameters, const Lardon3DExtractedFeatures *features,
|
||||||
|
Lardon3DProjectDbFeatureSet *feature_set) {
|
||||||
|
if (feature_set) {
|
||||||
|
memset(feature_set, 0, sizeof(*feature_set));
|
||||||
|
}
|
||||||
|
if (!state || !state->project_loaded || !state->project_db || !feature_set || image_id == 0 ||
|
||||||
|
!lardon3d_feature_extractor_parameters_valid(parameters) || !valid_features(features)) {
|
||||||
|
return LARDON3D_FEATURE_STORE_INVALID_ARGUMENT;
|
||||||
|
}
|
||||||
|
unsigned char fingerprint[32];
|
||||||
|
lardon3d_feature_extractor_parameter_fingerprint(parameters, fingerprint);
|
||||||
|
Lardon3DProjectDbResult found = lardon3d_project_db_find_feature_set(
|
||||||
|
state->project_db, image_id, LARDON3D_FEATURE_EXTRACTOR_KIND,
|
||||||
|
LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, feature_set);
|
||||||
|
if (found == LARDON3D_PROJECT_DB_OK) {
|
||||||
|
return reuse_existing_feature_set(state, image_id, producer_task_id, fingerprint, feature_set);
|
||||||
|
}
|
||||||
|
if (found != LARDON3D_PROJECT_DB_NOT_FOUND) {
|
||||||
|
return found == LARDON3D_PROJECT_DB_BUSY ? LARDON3D_FEATURE_STORE_DB_BUSY
|
||||||
|
: LARDON3D_FEATURE_STORE_DB_ERROR;
|
||||||
|
}
|
||||||
|
Lardon3DProjectDbImage image;
|
||||||
|
Lardon3DProjectDbImageAsset image_asset;
|
||||||
|
if (lardon3d_project_db_load_image(state->project_db, image_id, &image, &image_asset) !=
|
||||||
|
LARDON3D_PROJECT_DB_OK) {
|
||||||
|
return LARDON3D_FEATURE_STORE_NOT_FOUND;
|
||||||
|
}
|
||||||
|
char assets[PATH_MAX], features_dir[PATH_MAX];
|
||||||
|
if (!join(assets, state->project_path, "assets") || !ensure_directory(assets) ||
|
||||||
|
!join(features_dir, assets, "features") || !ensure_directory(features_dir)) {
|
||||||
|
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||||
|
}
|
||||||
|
char temporary[PATH_MAX];
|
||||||
|
int n = snprintf(temporary, sizeof(temporary), "%s/.feature-XXXXXX", features_dir);
|
||||||
|
if (n <= 0 || (size_t)n >= sizeof(temporary)) {
|
||||||
|
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||||
|
}
|
||||||
|
int fd = mkstemp(temporary);
|
||||||
|
if (fd < 0) {
|
||||||
|
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||||
|
}
|
||||||
|
uint64_t file_size = 0;
|
||||||
|
bool written =
|
||||||
|
write_feature_file(fd, &image_asset, fingerprint, features, &file_size) && fsync(fd) == 0;
|
||||||
|
unsigned char file_hash[32];
|
||||||
|
if (written) {
|
||||||
|
written = sha256_fd(fd, file_hash);
|
||||||
|
}
|
||||||
|
if (close(fd) != 0) {
|
||||||
|
written = false;
|
||||||
|
}
|
||||||
|
if (!written) {
|
||||||
|
unlink(temporary);
|
||||||
|
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||||
|
}
|
||||||
|
char hex[65];
|
||||||
|
hex_sha(file_hash, hex);
|
||||||
|
char prefix[PATH_MAX];
|
||||||
|
n = snprintf(prefix, sizeof(prefix), "%s/%.2s", features_dir, hex);
|
||||||
|
if (n <= 0 || (size_t)n >= sizeof(prefix) || !ensure_directory(prefix)) {
|
||||||
|
unlink(temporary);
|
||||||
|
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||||
|
}
|
||||||
|
char relative[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
||||||
|
n = snprintf(relative, sizeof(relative), "assets/features/%.2s/%s", hex, hex);
|
||||||
|
char final[PATH_MAX];
|
||||||
|
if (n <= 0 || (size_t)n >= sizeof(relative) || !join(final, state->project_path, relative)) {
|
||||||
|
unlink(temporary);
|
||||||
|
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||||
|
}
|
||||||
|
if (link(temporary, final) != 0) {
|
||||||
|
if (errno != EEXIST) {
|
||||||
|
unlink(temporary);
|
||||||
|
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||||
|
}
|
||||||
|
int existing = open(final, O_RDONLY | O_NOFOLLOW | O_CLOEXEC);
|
||||||
|
struct stat info;
|
||||||
|
unsigned char existing_hash[32];
|
||||||
|
bool same = existing >= 0 && fstat(existing, &info) == 0 && S_ISREG(info.st_mode) &&
|
||||||
|
info.st_size >= 0 && (uint64_t)info.st_size == file_size &&
|
||||||
|
sha256_fd(existing, existing_hash) && memcmp(existing_hash, file_hash, 32) == 0;
|
||||||
|
if (existing >= 0) {
|
||||||
|
(void)close(existing);
|
||||||
|
}
|
||||||
|
if (!same) {
|
||||||
|
unlink(temporary);
|
||||||
|
return LARDON3D_FEATURE_STORE_CORRUPT;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
unlink(temporary);
|
||||||
|
bool durable = sync_directory(prefix);
|
||||||
|
#ifdef LARDON3D_FEATURE_STORE_TESTING
|
||||||
|
const char *fail_sync = getenv("LARDON3D_TEST_FEATURE_FAIL_DIRECTORY_SYNC");
|
||||||
|
if (fail_sync && strcmp(fail_sync, "1") == 0) {
|
||||||
|
durable = false;
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
return register_published_feature_set(state, image_id, producer_task_id, &image_asset,
|
||||||
|
fingerprint, features, file_hash, relative, file_size,
|
||||||
|
durable, feature_set);
|
||||||
|
}
|
||||||
|
|
||||||
|
static Lardon3DFeatureStoreResult validate_header(const unsigned char h[160], uint64_t actual,
|
||||||
|
Lardon3DFeatureFileMetadata *m, uint64_t *ko,
|
||||||
|
uint64_t *doff) {
|
||||||
|
if (memcmp(h, feature_magic, 8) != 0) {
|
||||||
|
return LARDON3D_FEATURE_STORE_INVALID;
|
||||||
|
}
|
||||||
|
uint32_t version = get_u32(h + 8);
|
||||||
|
if (version != 1) {
|
||||||
|
return version > 1 ? LARDON3D_FEATURE_STORE_UNSUPPORTED_VERSION
|
||||||
|
: LARDON3D_FEATURE_STORE_INVALID;
|
||||||
|
}
|
||||||
|
uint32_t count = get_u32(h + 16), dim = get_u32(h + 20), type = get_u32(h + 24),
|
||||||
|
record = get_u32(h + 28), width = get_u32(h + 32), height = get_u32(h + 36);
|
||||||
|
uint64_t key = get_u64(h + 40), desc = get_u64(h + 48), declared = get_u64(h + 56);
|
||||||
|
uint64_t kb = (uint64_t)count * record, db = (uint64_t)count * dim;
|
||||||
|
if (get_u32(h + 12) != 160 || count > LARDON3D_FEATURE_MAX_FEATURES || dim != 32 ||
|
||||||
|
type != LARDON3D_FEATURE_DESCRIPTOR_U8 || record != 24 || width == 0 || height == 0 ||
|
||||||
|
key != 160 || desc != key + kb || declared != desc + db || declared != actual ||
|
||||||
|
actual > LARDON3D_FEATURE_FILE_MAX_SIZE || memcmp(h + 128, "orb\0", 4) != 0 ||
|
||||||
|
get_u32(h + 144) != 1) {
|
||||||
|
return LARDON3D_FEATURE_STORE_CORRUPT;
|
||||||
|
}
|
||||||
|
for (size_t i = 132; i < 144; ++i) {
|
||||||
|
if (h[i] != 0) {
|
||||||
|
return LARDON3D_FEATURE_STORE_CORRUPT;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (size_t i = 148; i < 160; ++i) {
|
||||||
|
if (h[i] != 0) {
|
||||||
|
return LARDON3D_FEATURE_STORE_CORRUPT;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
*m = (Lardon3DFeatureFileMetadata){.format_version = version,
|
||||||
|
.extractor_version = get_u32(h + 144),
|
||||||
|
.feature_count = count,
|
||||||
|
.descriptor_dimension = dim,
|
||||||
|
.descriptor_type = (Lardon3DFeatureDescriptorType)type,
|
||||||
|
.image_width = width,
|
||||||
|
.image_height = height};
|
||||||
|
memcpy(m->source_image_sha256, h + 64, 32);
|
||||||
|
memcpy(m->parameter_fingerprint, h + 96, 32);
|
||||||
|
memcpy(m->extractor_kind, h + 128, 4);
|
||||||
|
*ko = key;
|
||||||
|
*doff = desc;
|
||||||
|
return LARDON3D_FEATURE_STORE_OK;
|
||||||
|
}
|
||||||
|
|
||||||
|
Lardon3DFeatureStoreResult lardon3d_feature_reader_open(const char *project_path,
|
||||||
|
const Lardon3DProjectDbFeatureSet *set,
|
||||||
|
Lardon3DFeatureReader **out,
|
||||||
|
Lardon3DFeatureFileMetadata *metadata) {
|
||||||
|
if (out) {
|
||||||
|
*out = NULL;
|
||||||
|
}
|
||||||
|
if (!project_path || !set || !out || !metadata ||
|
||||||
|
!canonical_feature_path(set->asset.sha256, set->asset.path)) {
|
||||||
|
return LARDON3D_FEATURE_STORE_INVALID_ARGUMENT;
|
||||||
|
}
|
||||||
|
char path[PATH_MAX];
|
||||||
|
if (!join(path, project_path, set->asset.path)) {
|
||||||
|
return LARDON3D_FEATURE_STORE_INVALID_ARGUMENT;
|
||||||
|
}
|
||||||
|
int fd = open(path, O_RDONLY | O_NOFOLLOW | O_CLOEXEC);
|
||||||
|
if (fd < 0) {
|
||||||
|
return errno == ENOENT ? LARDON3D_FEATURE_STORE_NOT_FOUND : LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||||
|
}
|
||||||
|
struct stat st;
|
||||||
|
if (fstat(fd, &st) != 0 || !S_ISREG(st.st_mode) || st.st_size < 160 ||
|
||||||
|
(uint64_t)st.st_size > LARDON3D_FEATURE_FILE_MAX_SIZE ||
|
||||||
|
(uint64_t)st.st_size != set->asset.size_bytes) {
|
||||||
|
close(fd);
|
||||||
|
return LARDON3D_FEATURE_STORE_CORRUPT;
|
||||||
|
}
|
||||||
|
unsigned char header[160];
|
||||||
|
if (!read_exact(fd, header, sizeof(header), 0)) {
|
||||||
|
close(fd);
|
||||||
|
return LARDON3D_FEATURE_STORE_CORRUPT;
|
||||||
|
}
|
||||||
|
Lardon3DFeatureReader *reader = calloc(1, sizeof(*reader));
|
||||||
|
if (!reader) {
|
||||||
|
close(fd);
|
||||||
|
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||||
|
}
|
||||||
|
Lardon3DFeatureStoreResult result =
|
||||||
|
validate_header(header, (uint64_t)st.st_size, &reader->metadata, &reader->keypoint_offset,
|
||||||
|
&reader->descriptor_offset);
|
||||||
|
if (result != LARDON3D_FEATURE_STORE_OK) {
|
||||||
|
close(fd);
|
||||||
|
free(reader);
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
unsigned char hash[32];
|
||||||
|
if (!sha256_fd(fd, hash) || memcmp(hash, set->asset.sha256, 32) != 0) {
|
||||||
|
close(fd);
|
||||||
|
free(reader);
|
||||||
|
return LARDON3D_FEATURE_STORE_CORRUPT;
|
||||||
|
}
|
||||||
|
if (reader->metadata.feature_count != set->feature_count ||
|
||||||
|
strcmp(reader->metadata.extractor_kind, set->extractor_kind) != 0 ||
|
||||||
|
reader->metadata.extractor_version != set->extractor_version ||
|
||||||
|
reader->metadata.descriptor_dimension != set->descriptor_dimension ||
|
||||||
|
reader->metadata.descriptor_type != (Lardon3DFeatureDescriptorType)set->descriptor_type ||
|
||||||
|
memcmp(reader->metadata.source_image_sha256, set->source_image_sha256, 32) != 0 ||
|
||||||
|
memcmp(reader->metadata.parameter_fingerprint, set->parameter_fingerprint, 32) != 0) {
|
||||||
|
close(fd);
|
||||||
|
free(reader);
|
||||||
|
return LARDON3D_FEATURE_STORE_CORRUPT;
|
||||||
|
}
|
||||||
|
reader->descriptor = fd;
|
||||||
|
*metadata = reader->metadata;
|
||||||
|
*out = reader;
|
||||||
|
return LARDON3D_FEATURE_STORE_OK;
|
||||||
|
}
|
||||||
|
|
||||||
|
void lardon3d_feature_reader_close(Lardon3DFeatureReader *r) {
|
||||||
|
if (!r) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
(void)close(r->descriptor);
|
||||||
|
free(r);
|
||||||
|
}
|
||||||
|
|
||||||
|
Lardon3DFeatureStoreResult lardon3d_feature_reader_keypoints(Lardon3DFeatureReader *r,
|
||||||
|
uint32_t start,
|
||||||
|
Lardon3DFeatureKeypoint *out,
|
||||||
|
size_t capacity) {
|
||||||
|
if (!r || (!out && capacity > 0) || capacity > LARDON3D_FEATURE_READER_RANGE_MAX ||
|
||||||
|
start > r->metadata.feature_count || capacity > (size_t)(r->metadata.feature_count - start)) {
|
||||||
|
return LARDON3D_FEATURE_STORE_INVALID_ARGUMENT;
|
||||||
|
}
|
||||||
|
unsigned char record[24];
|
||||||
|
for (size_t i = 0; i < capacity; ++i) {
|
||||||
|
uint64_t index = (uint64_t)start + i;
|
||||||
|
if (!read_exact(r->descriptor, record, 24, (off_t)(r->keypoint_offset + index * 24))) {
|
||||||
|
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||||
|
}
|
||||||
|
out[i] = (Lardon3DFeatureKeypoint){get_float(record), get_float(record + 4),
|
||||||
|
get_float(record + 8), get_float(record + 12),
|
||||||
|
get_float(record + 16), (int32_t)get_u32(record + 20)};
|
||||||
|
if (!isfinite(out[i].x) || !isfinite(out[i].y) || !isfinite(out[i].size) ||
|
||||||
|
!isfinite(out[i].angle_degrees) || !isfinite(out[i].response)) {
|
||||||
|
return LARDON3D_FEATURE_STORE_CORRUPT;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return LARDON3D_FEATURE_STORE_OK;
|
||||||
|
}
|
||||||
|
|
||||||
|
Lardon3DFeatureStoreResult lardon3d_feature_reader_descriptors(Lardon3DFeatureReader *r,
|
||||||
|
uint32_t start, unsigned char *out,
|
||||||
|
size_t capacity, size_t bytes) {
|
||||||
|
if (!r || (!out && capacity > 0) || capacity > LARDON3D_FEATURE_READER_RANGE_MAX ||
|
||||||
|
start > r->metadata.feature_count || capacity > (size_t)(r->metadata.feature_count - start) ||
|
||||||
|
capacity > SIZE_MAX / r->metadata.descriptor_dimension ||
|
||||||
|
bytes < capacity * r->metadata.descriptor_dimension) {
|
||||||
|
return LARDON3D_FEATURE_STORE_INVALID_ARGUMENT;
|
||||||
|
}
|
||||||
|
size_t amount = capacity * r->metadata.descriptor_dimension;
|
||||||
|
if (amount && !read_exact(r->descriptor, out, amount,
|
||||||
|
(off_t)(r->descriptor_offset +
|
||||||
|
(uint64_t)start * r->metadata.descriptor_dimension))) {
|
||||||
|
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||||
|
}
|
||||||
|
return LARDON3D_FEATURE_STORE_OK;
|
||||||
|
}
|
||||||
325
src/feature_task.c
Normal file
325
src/feature_task.c
Normal file
|
|
@ -0,0 +1,325 @@
|
||||||
|
#include <errno.h>
|
||||||
|
#include <fcntl.h>
|
||||||
|
#include <openssl/evp.h>
|
||||||
|
#include <stdbool.h>
|
||||||
|
#include <stdint.h>
|
||||||
|
#include <stdio.h>
|
||||||
|
#include <stdlib.h>
|
||||||
|
#include <string.h>
|
||||||
|
#include <sys/stat.h>
|
||||||
|
#include <time.h>
|
||||||
|
#include <unistd.h>
|
||||||
|
|
||||||
|
#include <lardon3d/feature_store.h>
|
||||||
|
#include <lardon3d/feature_task.h>
|
||||||
|
#include <lardon3d/project.h>
|
||||||
|
#include <lardon3d/task_queue.h>
|
||||||
|
|
||||||
|
typedef struct {
|
||||||
|
char project_path[PATH_MAX];
|
||||||
|
Lardon3DProjectDb *database;
|
||||||
|
Lardon3DResourceGovernor *governor;
|
||||||
|
Lardon3DProjectDbFeatureExtractTask parameters;
|
||||||
|
} Lardon3DFeatureTaskContext;
|
||||||
|
|
||||||
|
static void destroy_context(void *userdata) { free(userdata); }
|
||||||
|
|
||||||
|
static void runtime_state(const Lardon3DFeatureTaskContext *context, Lardon3DAppState *state) {
|
||||||
|
lardon3d_app_state_init(state);
|
||||||
|
state->project_loaded = true;
|
||||||
|
state->project_db = context->database;
|
||||||
|
state->resource_governor = context->governor;
|
||||||
|
(void)snprintf(state->project_path, sizeof(state->project_path), "%s", context->project_path);
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool file_hash(const char *path, unsigned char output[32]) {
|
||||||
|
int fd = open(path, O_RDONLY | O_NOFOLLOW | O_CLOEXEC);
|
||||||
|
if (fd < 0) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
struct stat st;
|
||||||
|
if (fstat(fd, &st) != 0 || !S_ISREG(st.st_mode)) {
|
||||||
|
close(fd);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
EVP_MD_CTX *ctx = EVP_MD_CTX_new();
|
||||||
|
bool ok = ctx && EVP_DigestInit_ex(ctx, EVP_sha256(), NULL) == 1;
|
||||||
|
unsigned char buffer[65536];
|
||||||
|
while (ok) {
|
||||||
|
ssize_t n = read(fd, buffer, sizeof(buffer));
|
||||||
|
if (n < 0 && errno == EINTR) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (n < 0) {
|
||||||
|
ok = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
if (n == 0) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
ok = EVP_DigestUpdate(ctx, buffer, (size_t)n) == 1;
|
||||||
|
}
|
||||||
|
unsigned int size = 0;
|
||||||
|
ok = ok && EVP_DigestFinal_ex(ctx, output, &size) == 1 && size == 32;
|
||||||
|
if (ctx) {
|
||||||
|
EVP_MD_CTX_free(ctx);
|
||||||
|
}
|
||||||
|
if (close(fd) != 0) {
|
||||||
|
ok = false;
|
||||||
|
}
|
||||||
|
return ok;
|
||||||
|
}
|
||||||
|
|
||||||
|
static uint64_t elapsed_ns(struct timespec a, struct timespec b) {
|
||||||
|
uint64_t s = b.tv_sec >= a.tv_sec ? (uint64_t)(b.tv_sec - a.tv_sec) : 0;
|
||||||
|
long n = b.tv_nsec - a.tv_nsec;
|
||||||
|
if (n < 0 && s) {
|
||||||
|
--s;
|
||||||
|
n += 1000000000L;
|
||||||
|
}
|
||||||
|
return s <= UINT64_MAX / 1000000000ULL ? s * 1000000000ULL + (uint64_t)n : UINT64_MAX;
|
||||||
|
}
|
||||||
|
|
||||||
|
static void finished(const Lardon3DTask *task, void *userdata) {
|
||||||
|
#ifdef LARDON3D_FEATURE_TASK_TESTING
|
||||||
|
const char *skip = getenv("LARDON3D_TEST_FEATURE_SKIP_FINISHED_CHECKPOINT");
|
||||||
|
if (skip && strcmp(skip, "1") == 0) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
Lardon3DFeatureTaskContext *context = userdata;
|
||||||
|
Lardon3DAppState state;
|
||||||
|
runtime_state(context, &state);
|
||||||
|
(void)lardon3d_project_checkpoint_feature_extract_task(&state, task, &context->parameters);
|
||||||
|
}
|
||||||
|
|
||||||
|
static Lardon3DProjectDbResult find_valid_feature_set(const Lardon3DFeatureTaskContext *context,
|
||||||
|
const unsigned char fingerprint[32]) {
|
||||||
|
Lardon3DProjectDbFeatureSet existing;
|
||||||
|
Lardon3DProjectDbResult found = lardon3d_project_db_find_feature_set(
|
||||||
|
context->database, context->parameters.image_id, LARDON3D_FEATURE_EXTRACTOR_KIND,
|
||||||
|
LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, &existing);
|
||||||
|
if (found != LARDON3D_PROJECT_DB_OK) {
|
||||||
|
return found;
|
||||||
|
}
|
||||||
|
Lardon3DFeatureReader *reader = NULL;
|
||||||
|
Lardon3DFeatureFileMetadata metadata;
|
||||||
|
Lardon3DFeatureStoreResult valid =
|
||||||
|
lardon3d_feature_reader_open(context->project_path, &existing, &reader, &metadata);
|
||||||
|
lardon3d_feature_reader_close(reader);
|
||||||
|
return valid == LARDON3D_FEATURE_STORE_OK ? LARDON3D_PROJECT_DB_OK : LARDON3D_PROJECT_DB_CORRUPT;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool load_validated_source(const Lardon3DFeatureTaskContext *context, char path[PATH_MAX]) {
|
||||||
|
Lardon3DProjectDbImage image;
|
||||||
|
Lardon3DProjectDbImageAsset asset;
|
||||||
|
if (lardon3d_project_db_load_image(context->database, context->parameters.image_id, &image,
|
||||||
|
&asset) != LARDON3D_PROJECT_DB_OK) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
int written = snprintf(path, PATH_MAX, "%s/%s", context->project_path, asset.path);
|
||||||
|
if (written <= 0 || (size_t)written >= PATH_MAX) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
unsigned char actual[32];
|
||||||
|
return file_hash(path, actual) && memcmp(actual, asset.sha256, 32) == 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool run(Lardon3DTask *task, void *userdata) {
|
||||||
|
Lardon3DFeatureTaskContext *context = userdata;
|
||||||
|
if (!lardon3d_task_checkpoint(task)) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
struct timespec begin;
|
||||||
|
(void)clock_gettime(CLOCK_MONOTONIC, &begin);
|
||||||
|
unsigned char fingerprint[32];
|
||||||
|
lardon3d_feature_extractor_parameter_fingerprint(
|
||||||
|
&(Lardon3DFeatureExtractorParameters){context->parameters.max_features,
|
||||||
|
context->parameters.pyramid_levels,
|
||||||
|
context->parameters.fast_threshold},
|
||||||
|
fingerprint);
|
||||||
|
Lardon3DProjectDbResult found = find_valid_feature_set(context, fingerprint);
|
||||||
|
if (found == LARDON3D_PROJECT_DB_OK) {
|
||||||
|
return lardon3d_task_set_progress(task, 100, "Features déjà présentes.");
|
||||||
|
}
|
||||||
|
if (found == LARDON3D_PROJECT_DB_CORRUPT) {
|
||||||
|
return lardon3d_task_fail(task, "Feature Set READY absent ou corrompu.");
|
||||||
|
}
|
||||||
|
if (found != LARDON3D_PROJECT_DB_NOT_FOUND) {
|
||||||
|
return lardon3d_task_fail(task, "Recherche Feature Store impossible.");
|
||||||
|
}
|
||||||
|
char path[PATH_MAX];
|
||||||
|
if (!load_validated_source(context, path)) {
|
||||||
|
return lardon3d_task_fail(task, "Asset image absent ou corrompu.");
|
||||||
|
}
|
||||||
|
Lardon3DFeatureExtractorParameters parameters = {context->parameters.max_features,
|
||||||
|
context->parameters.pyramid_levels,
|
||||||
|
context->parameters.fast_threshold};
|
||||||
|
Lardon3DExtractedFeatures features;
|
||||||
|
Lardon3DFeatureExtractResult extracted =
|
||||||
|
lardon3d_feature_extract_orb(path, ¶meters, &features);
|
||||||
|
if (extracted != LARDON3D_FEATURE_EXTRACT_OK) {
|
||||||
|
return lardon3d_task_fail(task, "Décodage ou extraction ORB impossible.");
|
||||||
|
}
|
||||||
|
if (!lardon3d_task_checkpoint(task)) {
|
||||||
|
lardon3d_extracted_features_destroy(&features);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
#ifdef LARDON3D_FEATURE_TASK_TESTING
|
||||||
|
const char *pause = getenv("LARDON3D_TEST_FEATURE_PAUSE_BEFORE_PUBLISH");
|
||||||
|
if (pause && strcmp(pause, "1") == 0) {
|
||||||
|
(void)lardon3d_task_pause(task);
|
||||||
|
if (!lardon3d_task_checkpoint(task)) {
|
||||||
|
lardon3d_extracted_features_destroy(&features);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
Lardon3DAppState state;
|
||||||
|
runtime_state(context, &state);
|
||||||
|
Lardon3DProjectDbFeatureSet set;
|
||||||
|
Lardon3DFeatureStoreResult published = lardon3d_feature_store_publish(
|
||||||
|
&state, context->parameters.image_id, lardon3d_task_id(task), ¶meters, &features, &set);
|
||||||
|
lardon3d_extracted_features_destroy(&features);
|
||||||
|
if (published != LARDON3D_FEATURE_STORE_OK &&
|
||||||
|
published != LARDON3D_FEATURE_STORE_ALREADY_PRESENT &&
|
||||||
|
published != LARDON3D_FEATURE_STORE_PUBLISHED_NOT_DURABLE) {
|
||||||
|
return lardon3d_task_fail(task, "Publication Feature Store impossible.");
|
||||||
|
}
|
||||||
|
struct timespec end;
|
||||||
|
(void)clock_gettime(CLOCK_MONOTONIC, &end);
|
||||||
|
(void)lardon3d_resource_governor_record_batch(context->governor, LARDON3D_RESOURCE_TASK_CPU, 1,
|
||||||
|
elapsed_ns(begin, end), 0);
|
||||||
|
return lardon3d_task_set_progress(task, 100, "Feature Set publié.");
|
||||||
|
}
|
||||||
|
|
||||||
|
static Lardon3DFeatureTaskContext *
|
||||||
|
make_context(const Lardon3DTaskReconstructionContext *runtime,
|
||||||
|
const Lardon3DProjectDbFeatureExtractTask *parameters) {
|
||||||
|
if (!runtime || !runtime->project_path || !runtime->project_db || !runtime->resource_governor ||
|
||||||
|
!parameters) {
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
Lardon3DFeatureTaskContext *context = calloc(1, sizeof(*context));
|
||||||
|
if (!context) {
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
int n =
|
||||||
|
snprintf(context->project_path, sizeof(context->project_path), "%s", runtime->project_path);
|
||||||
|
if (n <= 0 || (size_t)n >= sizeof(context->project_path)) {
|
||||||
|
free(context);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
context->database = runtime->project_db;
|
||||||
|
context->governor = runtime->resource_governor;
|
||||||
|
context->parameters = *parameters;
|
||||||
|
return context;
|
||||||
|
}
|
||||||
|
|
||||||
|
bool lardon3d_feature_extract_reconstruct(const Lardon3DTaskDurableSnapshot *snapshot,
|
||||||
|
void *userdata, Lardon3DTaskKindBinding *binding) {
|
||||||
|
Lardon3DTaskReconstructionContext *runtime = userdata;
|
||||||
|
if (!snapshot || !runtime || !binding) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
Lardon3DProjectDbFeatureExtractTask parameters;
|
||||||
|
if (lardon3d_project_db_load_feature_extract_task(runtime->project_db, snapshot->id,
|
||||||
|
¶meters) != LARDON3D_PROJECT_DB_OK ||
|
||||||
|
strcmp(parameters.extractor_kind, LARDON3D_FEATURE_EXTRACTOR_KIND) != 0 ||
|
||||||
|
parameters.extractor_version != 1) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
unsigned char expected[32];
|
||||||
|
lardon3d_feature_extractor_parameter_fingerprint(
|
||||||
|
&(Lardon3DFeatureExtractorParameters){parameters.max_features, parameters.pyramid_levels,
|
||||||
|
parameters.fast_threshold},
|
||||||
|
expected);
|
||||||
|
if (memcmp(expected, parameters.parameter_fingerprint, 32) != 0) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
Lardon3DFeatureTaskContext *context = make_context(runtime, ¶meters);
|
||||||
|
if (!context) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
*binding = (Lardon3DTaskKindBinding){.callback = run,
|
||||||
|
.userdata = context,
|
||||||
|
.userdata_destroy = destroy_context,
|
||||||
|
.finished_callback = finished,
|
||||||
|
.finished_userdata = context};
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
Lardon3DTask *
|
||||||
|
lardon3d_project_create_feature_extract_task(Lardon3DAppState *state, uint64_t image_id,
|
||||||
|
const Lardon3DFeatureExtractorParameters *parameters,
|
||||||
|
uint64_t *task_id) {
|
||||||
|
if (task_id) {
|
||||||
|
*task_id = 0;
|
||||||
|
}
|
||||||
|
if (!state || !state->project_loaded || !state->project_db || !state->resource_governor ||
|
||||||
|
!parameters || !task_id || !lardon3d_feature_extractor_parameters_valid(parameters)) {
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
Lardon3DProjectDbImage image;
|
||||||
|
Lardon3DProjectDbImageAsset asset;
|
||||||
|
if (lardon3d_project_db_load_image(state->project_db, image_id, &image, &asset) !=
|
||||||
|
LARDON3D_PROJECT_DB_OK) {
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
uint64_t id;
|
||||||
|
if (lardon3d_project_db_allocate_task_id(state->project_db, &id) != LARDON3D_PROJECT_DB_OK) {
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
Lardon3DProjectDbFeatureExtractTask durable = {.task_id = id,
|
||||||
|
.image_id = image_id,
|
||||||
|
.extractor_version = 1,
|
||||||
|
.max_features = parameters->max_features,
|
||||||
|
.pyramid_levels = parameters->pyramid_levels,
|
||||||
|
.fast_threshold = parameters->fast_threshold};
|
||||||
|
snprintf(durable.extractor_kind, sizeof(durable.extractor_kind), "%s",
|
||||||
|
LARDON3D_FEATURE_EXTRACTOR_KIND);
|
||||||
|
lardon3d_feature_extractor_parameter_fingerprint(parameters, durable.parameter_fingerprint);
|
||||||
|
Lardon3DTaskReconstructionContext runtime = {state->project_path, state->project_db,
|
||||||
|
state->resource_governor};
|
||||||
|
Lardon3DFeatureTaskContext *context = make_context(&runtime, &durable);
|
||||||
|
if (!context) {
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
Lardon3DResourceEstimate estimate = {.memory_fixed_bytes = 64ULL * 1024 * 1024,
|
||||||
|
.memory_bytes_per_item = 512ULL * 1024 * 1024,
|
||||||
|
.minimum_batch_size = 1,
|
||||||
|
.maximum_batch_size = 1,
|
||||||
|
.desired_cpu_threads = 1,
|
||||||
|
.desired_io_slots = 1,
|
||||||
|
.task_class = LARDON3D_RESOURCE_TASK_CPU};
|
||||||
|
Lardon3DTask *task = lardon3d_task_create_typed("Extraction de features", &estimate,
|
||||||
|
LARDON3D_FEATURE_EXTRACT_TASK_KIND, 1, run,
|
||||||
|
context, destroy_context);
|
||||||
|
if (!task || !lardon3d_task_assign_id(task, id) ||
|
||||||
|
!lardon3d_task_set_finished_callback(task, finished, context) ||
|
||||||
|
lardon3d_project_checkpoint_feature_extract_task(state, task, &durable) !=
|
||||||
|
LARDON3D_PROJECT_TASK_CHECKPOINT_OK) {
|
||||||
|
lardon3d_task_destroy(task);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
*task_id = id;
|
||||||
|
return task;
|
||||||
|
}
|
||||||
|
|
||||||
|
bool lardon3d_project_enqueue_feature_extract(Lardon3DAppState *state, uint64_t image_id,
|
||||||
|
const Lardon3DFeatureExtractorParameters *parameters,
|
||||||
|
uint64_t *task_id) {
|
||||||
|
if (!state || !state->task_queue) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
Lardon3DTask *task =
|
||||||
|
lardon3d_project_create_feature_extract_task(state, image_id, parameters, task_id);
|
||||||
|
if (!task) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (!lardon3d_task_queue_add(state->task_queue, task, NULL)) {
|
||||||
|
lardon3d_task_destroy(task);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
1934
src/project.c
1934
src/project.c
File diff suppressed because it is too large
Load diff
2999
src/project_db.c
2999
src/project_db.c
File diff suppressed because it is too large
Load diff
|
|
@ -1,17 +1,22 @@
|
||||||
|
#include <lardon3d/feature_task.h>
|
||||||
#include <lardon3d/import_task.h>
|
#include <lardon3d/import_task.h>
|
||||||
#include <lardon3d/task_kind_registry.h>
|
#include <lardon3d/task_kind_registry.h>
|
||||||
|
|
||||||
const Lardon3DTaskKindRegistry *
|
const Lardon3DTaskKindRegistry *lardon3d_task_kind_registry_production(void) {
|
||||||
lardon3d_task_kind_registry_production(void)
|
static const Lardon3DTaskKindDescriptor descriptors[] = {
|
||||||
{
|
{
|
||||||
static const Lardon3DTaskKindDescriptor descriptors[] = {{
|
.kind = LARDON3D_IMAGE_IMPORT_TASK_KIND,
|
||||||
.kind = LARDON3D_IMAGE_IMPORT_TASK_KIND,
|
.kind_version = LARDON3D_IMAGE_IMPORT_TASK_KIND_VERSION,
|
||||||
.kind_version = LARDON3D_IMAGE_IMPORT_TASK_KIND_VERSION,
|
.reconstruct = lardon3d_image_import_reconstruct,
|
||||||
.reconstruct = lardon3d_image_import_reconstruct,
|
},
|
||||||
}};
|
{
|
||||||
static const Lardon3DTaskKindRegistry registry = {
|
.kind = LARDON3D_FEATURE_EXTRACT_TASK_KIND,
|
||||||
.descriptors = descriptors,
|
.kind_version = LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION,
|
||||||
.count = sizeof(descriptors) / sizeof(descriptors[0]),
|
.reconstruct = lardon3d_feature_extract_reconstruct,
|
||||||
};
|
}};
|
||||||
return ®istry;
|
static const Lardon3DTaskKindRegistry registry = {
|
||||||
|
.descriptors = descriptors,
|
||||||
|
.count = sizeof(descriptors) / sizeof(descriptors[0]),
|
||||||
|
};
|
||||||
|
return ®istry;
|
||||||
}
|
}
|
||||||
|
|
|
||||||
432
tests/test_feature_store.c
Normal file
432
tests/test_feature_store.c
Normal file
|
|
@ -0,0 +1,432 @@
|
||||||
|
#include <dirent.h>
|
||||||
|
#include <errno.h>
|
||||||
|
#include <fcntl.h>
|
||||||
|
#include <math.h>
|
||||||
|
#include <pthread.h>
|
||||||
|
#include <stdbool.h>
|
||||||
|
#include <stdint.h>
|
||||||
|
#include <stdio.h>
|
||||||
|
#include <stdlib.h>
|
||||||
|
#include <string.h>
|
||||||
|
#include <sys/stat.h>
|
||||||
|
#include <unistd.h>
|
||||||
|
|
||||||
|
#include <lardon3d/feature_store.h>
|
||||||
|
#include <lardon3d/image_catalog.h>
|
||||||
|
|
||||||
|
Lardon3DProjectDbResult lardon3d_project_db_test_delete_feature_identity(Lardon3DProjectDb *,
|
||||||
|
uint64_t, uint64_t);
|
||||||
|
|
||||||
|
typedef struct {
|
||||||
|
Lardon3DAppState *state;
|
||||||
|
uint64_t image_id;
|
||||||
|
const Lardon3DFeatureExtractorParameters *parameters;
|
||||||
|
const Lardon3DExtractedFeatures *features;
|
||||||
|
Lardon3DFeatureStoreResult result;
|
||||||
|
Lardon3DProjectDbFeatureSet feature_set;
|
||||||
|
} PublishThreadContext;
|
||||||
|
|
||||||
|
static void *publish_thread(void *userdata) {
|
||||||
|
PublishThreadContext *context = userdata;
|
||||||
|
context->result =
|
||||||
|
lardon3d_feature_store_publish(context->state, context->image_id, 0, context->parameters,
|
||||||
|
context->features, &context->feature_set);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
|
#define CHECK(x) \
|
||||||
|
do { \
|
||||||
|
if (!(x)) { \
|
||||||
|
fprintf(stderr, "Échec ligne %d: %s\n", __LINE__, #x); \
|
||||||
|
return false; \
|
||||||
|
} \
|
||||||
|
} while (0)
|
||||||
|
|
||||||
|
static bool join_path(char out[PATH_MAX], const char *a, const char *b) {
|
||||||
|
int n = snprintf(out, PATH_MAX, "%s/%s", a, b);
|
||||||
|
return n > 0 && (size_t)n < PATH_MAX;
|
||||||
|
}
|
||||||
|
static bool remove_tree(const char *path) {
|
||||||
|
struct stat st;
|
||||||
|
if (lstat(path, &st) != 0) {
|
||||||
|
return errno == ENOENT;
|
||||||
|
}
|
||||||
|
if (!S_ISDIR(st.st_mode)) {
|
||||||
|
return unlink(path) == 0;
|
||||||
|
}
|
||||||
|
DIR *d = opendir(path);
|
||||||
|
if (!d) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
bool ok = true;
|
||||||
|
for (struct dirent *e = readdir(d); e; e = readdir(d)) {
|
||||||
|
if (!strcmp(e->d_name, ".") || !strcmp(e->d_name, "..")) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
char child[PATH_MAX];
|
||||||
|
if (!join_path(child, path, e->d_name) || !remove_tree(child)) {
|
||||||
|
ok = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (closedir(d) != 0 || rmdir(path) != 0) {
|
||||||
|
ok = false;
|
||||||
|
}
|
||||||
|
return ok;
|
||||||
|
}
|
||||||
|
static size_t count_files(const char *path) {
|
||||||
|
DIR *d = opendir(path);
|
||||||
|
if (!d) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
size_t count = 0;
|
||||||
|
for (struct dirent *e = readdir(d); e; e = readdir(d)) {
|
||||||
|
if (!strcmp(e->d_name, ".") || !strcmp(e->d_name, "..")) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
char child[PATH_MAX];
|
||||||
|
struct stat s;
|
||||||
|
if (join_path(child, path, e->d_name) && lstat(child, &s) == 0) {
|
||||||
|
if (S_ISDIR(s.st_mode)) {
|
||||||
|
count += count_files(child);
|
||||||
|
} else if (S_ISREG(s.st_mode)) {
|
||||||
|
++count;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
closedir(d);
|
||||||
|
return count;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool write_pgm(const char *path, bool uniform) {
|
||||||
|
FILE *f = fopen(path, "wb");
|
||||||
|
if (!f) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (fprintf(f, "P5\n128 128\n255\n") <= 0) {
|
||||||
|
fclose(f);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
for (unsigned int y = 0; y < 128; ++y) {
|
||||||
|
for (unsigned int x = 0; x < 128; ++x) {
|
||||||
|
unsigned char v = uniform ? 128U : (unsigned char)((((x / 8U) ^ (y / 8U)) & 1U) ? 240U : 15U);
|
||||||
|
if (fwrite(&v, 1, 1, f) != 1) {
|
||||||
|
fclose(f);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return fclose(f) == 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
static uint64_t read_u64_le(const unsigned char *bytes) {
|
||||||
|
uint64_t value = 0;
|
||||||
|
for (unsigned int index = 0; index < 8; ++index) {
|
||||||
|
value |= (uint64_t)bytes[index] << (8U * index);
|
||||||
|
}
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool restore_file(const char *path, const unsigned char *bytes, size_t size) {
|
||||||
|
int descriptor = open(path, O_WRONLY | O_TRUNC | O_CLOEXEC);
|
||||||
|
if (descriptor < 0) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
size_t written = 0;
|
||||||
|
while (written < size) {
|
||||||
|
ssize_t amount = write(descriptor, bytes + written, size - written);
|
||||||
|
if (amount <= 0) {
|
||||||
|
close(descriptor);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
written += (size_t)amount;
|
||||||
|
}
|
||||||
|
return close(descriptor) == 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool expect_reader_result(const char *root, const Lardon3DProjectDbFeatureSet *feature_set,
|
||||||
|
Lardon3DFeatureStoreResult expected) {
|
||||||
|
Lardon3DFeatureReader *reader = NULL;
|
||||||
|
Lardon3DFeatureFileMetadata metadata;
|
||||||
|
Lardon3DFeatureStoreResult result =
|
||||||
|
lardon3d_feature_reader_open(root, feature_set, &reader, &metadata);
|
||||||
|
lardon3d_feature_reader_close(reader);
|
||||||
|
return result == expected;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool run_test(void) {
|
||||||
|
char root[] = "/tmp/lardon3d-feature-store-XXXXXX";
|
||||||
|
CHECK(mkdtemp(root));
|
||||||
|
char dbpath[PATH_MAX], structured[PATH_MAX], uniform[PATH_MAX];
|
||||||
|
CHECK(join_path(dbpath, root, "project.db") && join_path(structured, root, "structured.pgm") &&
|
||||||
|
join_path(uniform, root, "uniform.pgm"));
|
||||||
|
CHECK(write_pgm(structured, false) && write_pgm(uniform, true));
|
||||||
|
Lardon3DProjectDb *db = NULL;
|
||||||
|
char error[256];
|
||||||
|
CHECK(lardon3d_project_db_open(dbpath, &db, error) == LARDON3D_PROJECT_DB_OK);
|
||||||
|
Lardon3DAppState state;
|
||||||
|
lardon3d_app_state_init(&state);
|
||||||
|
state.project_loaded = true;
|
||||||
|
state.project_db = db;
|
||||||
|
CHECK(snprintf(state.project_path, sizeof(state.project_path), "%s", root) > 0);
|
||||||
|
Lardon3DProjectDbScanSet a, b;
|
||||||
|
CHECK(lardon3d_image_catalog_create_scanset(&state, "A", &a) &&
|
||||||
|
lardon3d_image_catalog_create_scanset(&state, "B", &b));
|
||||||
|
Lardon3DProjectDbImage ia, ib, iu;
|
||||||
|
Lardon3DProjectDbImageAsset aa, ab, au;
|
||||||
|
CHECK(lardon3d_image_catalog_import_file(&state, a.scanset_id, structured, 0, &ia, &aa) ==
|
||||||
|
LARDON3D_IMAGE_CATALOG_IMPORTED);
|
||||||
|
CHECK(lardon3d_image_catalog_import_file(&state, b.scanset_id, structured, 0, &ib, &ab) ==
|
||||||
|
LARDON3D_IMAGE_CATALOG_IMPORTED &&
|
||||||
|
aa.asset_id == ab.asset_id);
|
||||||
|
CHECK(lardon3d_image_catalog_import_file(&state, a.scanset_id, uniform, 0, &iu, &au) ==
|
||||||
|
LARDON3D_IMAGE_CATALOG_IMPORTED);
|
||||||
|
Lardon3DFeatureExtractorParameters p = {
|
||||||
|
.max_features = 512, .pyramid_levels = 4, .fast_threshold = 10};
|
||||||
|
CHECK(lardon3d_feature_extractor_parameters_valid(&p));
|
||||||
|
CHECK(!lardon3d_feature_extractor_parameters_valid(
|
||||||
|
&(Lardon3DFeatureExtractorParameters){0, 4, 10}));
|
||||||
|
CHECK(!lardon3d_feature_extractor_parameters_valid(
|
||||||
|
&(Lardon3DFeatureExtractorParameters){8193, 4, 10}));
|
||||||
|
CHECK(!lardon3d_feature_extractor_parameters_valid(
|
||||||
|
&(Lardon3DFeatureExtractorParameters){512, 0, 10}));
|
||||||
|
CHECK(!lardon3d_feature_extractor_parameters_valid(
|
||||||
|
&(Lardon3DFeatureExtractorParameters){512, 17, 10}));
|
||||||
|
CHECK(!lardon3d_feature_extractor_parameters_valid(
|
||||||
|
&(Lardon3DFeatureExtractorParameters){512, 4, 0}));
|
||||||
|
CHECK(!lardon3d_feature_extractor_parameters_valid(
|
||||||
|
&(Lardon3DFeatureExtractorParameters){512, 4, 256}));
|
||||||
|
unsigned char fp[32], fp2[32];
|
||||||
|
lardon3d_feature_extractor_parameter_fingerprint(&p, fp);
|
||||||
|
lardon3d_feature_extractor_parameter_fingerprint(&p, fp2);
|
||||||
|
CHECK(memcmp(fp, fp2, 32) == 0);
|
||||||
|
Lardon3DFeatureExtractorParameters p2 = p;
|
||||||
|
p2.max_features = 256;
|
||||||
|
lardon3d_feature_extractor_parameter_fingerprint(&p2, fp2);
|
||||||
|
CHECK(memcmp(fp, fp2, 32) != 0);
|
||||||
|
Lardon3DExtractedFeatures f;
|
||||||
|
CHECK(lardon3d_feature_extract_orb(structured, &p, &f) == LARDON3D_FEATURE_EXTRACT_OK &&
|
||||||
|
f.feature_count > 0 && f.feature_count <= 512);
|
||||||
|
Lardon3DProjectDbFeatureSet sa, same, sb;
|
||||||
|
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &p, &f, &sa) ==
|
||||||
|
LARDON3D_FEATURE_STORE_OK);
|
||||||
|
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &p, &f, &same) ==
|
||||||
|
LARDON3D_FEATURE_STORE_ALREADY_PRESENT &&
|
||||||
|
same.feature_set_id == sa.feature_set_id);
|
||||||
|
CHECK(lardon3d_feature_store_publish(&state, ib.image_id, 0, &p, &f, &sb) ==
|
||||||
|
LARDON3D_FEATURE_STORE_OK &&
|
||||||
|
sb.feature_set_id != sa.feature_set_id && sb.feature_asset_id == sa.feature_asset_id);
|
||||||
|
Lardon3DFeatureReader *reader = NULL;
|
||||||
|
Lardon3DFeatureFileMetadata metadata;
|
||||||
|
CHECK(lardon3d_feature_reader_open(root, &sa, &reader, &metadata) == LARDON3D_FEATURE_STORE_OK &&
|
||||||
|
metadata.feature_count == f.feature_count);
|
||||||
|
size_t chunk = metadata.feature_count < 8 ? metadata.feature_count : 8;
|
||||||
|
Lardon3DFeatureKeypoint points[8];
|
||||||
|
unsigned char descriptors[8 * 32];
|
||||||
|
CHECK(lardon3d_feature_reader_keypoints(reader, 0, points, chunk) == LARDON3D_FEATURE_STORE_OK);
|
||||||
|
CHECK(lardon3d_feature_reader_descriptors(reader, 0, descriptors, chunk, sizeof(descriptors)) ==
|
||||||
|
LARDON3D_FEATURE_STORE_OK);
|
||||||
|
for (size_t i = 0; i < chunk; ++i) {
|
||||||
|
CHECK(isfinite(points[i].x) && memcmp(descriptors + i * 32, f.descriptors + i * 32, 32) == 0);
|
||||||
|
}
|
||||||
|
CHECK(lardon3d_feature_reader_keypoints(reader, metadata.feature_count, points, 1) ==
|
||||||
|
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT);
|
||||||
|
CHECK(lardon3d_feature_reader_descriptors(reader, UINT32_MAX, descriptors, 1,
|
||||||
|
sizeof(descriptors)) ==
|
||||||
|
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT);
|
||||||
|
CHECK(lardon3d_feature_reader_keypoints(reader, 0, points, 257) ==
|
||||||
|
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT);
|
||||||
|
CHECK(lardon3d_feature_reader_descriptors(reader, 0, descriptors, 257, sizeof(descriptors)) ==
|
||||||
|
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT);
|
||||||
|
lardon3d_feature_reader_close(reader);
|
||||||
|
|
||||||
|
const char *invalid_paths[] = {
|
||||||
|
"/assets/features/00/hash",
|
||||||
|
"../assets/features/00/hash",
|
||||||
|
"assets/features/AA/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
|
||||||
|
"assets/features/aa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
|
||||||
|
"assets/features/0/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
|
||||||
|
"assets/features/aa/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
|
||||||
|
};
|
||||||
|
for (size_t index = 0; index < sizeof(invalid_paths) / sizeof(invalid_paths[0]); ++index) {
|
||||||
|
Lardon3DProjectDbFeatureSet invalid_path_set = sa;
|
||||||
|
CHECK(snprintf(invalid_path_set.asset.path, sizeof(invalid_path_set.asset.path), "%s",
|
||||||
|
invalid_paths[index]) > 0);
|
||||||
|
CHECK(lardon3d_feature_reader_open(root, &invalid_path_set, &reader, &metadata) ==
|
||||||
|
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT);
|
||||||
|
}
|
||||||
|
|
||||||
|
Lardon3DExtractedFeatures invalid_features = f;
|
||||||
|
Lardon3DFeatureKeypoint invalid_point = f.keypoints[0];
|
||||||
|
invalid_point.x = NAN;
|
||||||
|
invalid_features.keypoints = &invalid_point;
|
||||||
|
invalid_features.feature_count = 1;
|
||||||
|
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &p2, &invalid_features, &same) ==
|
||||||
|
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT);
|
||||||
|
|
||||||
|
Lardon3DFeatureExtractorParameters concurrent_parameters = {508, 4, 10};
|
||||||
|
PublishThreadContext concurrent[2] = {
|
||||||
|
{.state = &state,
|
||||||
|
.image_id = ia.image_id,
|
||||||
|
.parameters = &concurrent_parameters,
|
||||||
|
.features = &f},
|
||||||
|
{.state = &state,
|
||||||
|
.image_id = ia.image_id,
|
||||||
|
.parameters = &concurrent_parameters,
|
||||||
|
.features = &f},
|
||||||
|
};
|
||||||
|
pthread_t publish_threads[2];
|
||||||
|
CHECK(pthread_create(&publish_threads[0], NULL, publish_thread, &concurrent[0]) == 0 &&
|
||||||
|
pthread_create(&publish_threads[1], NULL, publish_thread, &concurrent[1]) == 0);
|
||||||
|
CHECK(pthread_join(publish_threads[0], NULL) == 0 && pthread_join(publish_threads[1], NULL) == 0);
|
||||||
|
CHECK((concurrent[0].result == LARDON3D_FEATURE_STORE_OK ||
|
||||||
|
concurrent[0].result == LARDON3D_FEATURE_STORE_ALREADY_PRESENT) &&
|
||||||
|
(concurrent[1].result == LARDON3D_FEATURE_STORE_OK ||
|
||||||
|
concurrent[1].result == LARDON3D_FEATURE_STORE_ALREADY_PRESENT) &&
|
||||||
|
concurrent[0].feature_set.feature_set_id == concurrent[1].feature_set.feature_set_id);
|
||||||
|
Lardon3DExtractedFeatures uf;
|
||||||
|
CHECK(lardon3d_feature_extract_orb(uniform, &p, &uf) == LARDON3D_FEATURE_EXTRACT_OK &&
|
||||||
|
uf.feature_count == 0);
|
||||||
|
Lardon3DProjectDbFeatureSet us;
|
||||||
|
CHECK(lardon3d_feature_store_publish(&state, iu.image_id, 0, &p, &uf, &us) ==
|
||||||
|
LARDON3D_FEATURE_STORE_OK);
|
||||||
|
CHECK(lardon3d_feature_reader_open(root, &us, &reader, &metadata) == LARDON3D_FEATURE_STORE_OK &&
|
||||||
|
metadata.feature_count == 0);
|
||||||
|
lardon3d_feature_reader_close(reader);
|
||||||
|
Lardon3DFeatureExtractorParameters large_parameters = {8192, 4, 10};
|
||||||
|
Lardon3DExtractedFeatures large = {
|
||||||
|
.image_width = 128, .image_height = 128, .feature_count = 8192};
|
||||||
|
large.keypoints = calloc(large.feature_count, sizeof(*large.keypoints));
|
||||||
|
large.descriptors = calloc(large.feature_count, 32);
|
||||||
|
CHECK(large.keypoints && large.descriptors);
|
||||||
|
for (uint32_t i = 0; i < large.feature_count; ++i) {
|
||||||
|
large.keypoints[i] =
|
||||||
|
(Lardon3DFeatureKeypoint){(float)(i % 128), (float)((i / 128) % 64), 1, 0, 1, 0};
|
||||||
|
memset(large.descriptors + (size_t)i * 32, (int)(i & 255), 32);
|
||||||
|
}
|
||||||
|
Lardon3DProjectDbFeatureSet large_set;
|
||||||
|
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &large_parameters, &large,
|
||||||
|
&large_set) == LARDON3D_FEATURE_STORE_OK);
|
||||||
|
CHECK(lardon3d_feature_reader_open(root, &large_set, &reader, &metadata) ==
|
||||||
|
LARDON3D_FEATURE_STORE_OK &&
|
||||||
|
metadata.feature_count == 8192);
|
||||||
|
Lardon3DFeatureKeypoint last;
|
||||||
|
unsigned char last_descriptor[32];
|
||||||
|
CHECK(lardon3d_feature_reader_keypoints(reader, 8191, &last, 1) == LARDON3D_FEATURE_STORE_OK &&
|
||||||
|
lardon3d_feature_reader_descriptors(reader, 8191, last_descriptor, 1,
|
||||||
|
sizeof(last_descriptor)) == LARDON3D_FEATURE_STORE_OK &&
|
||||||
|
last_descriptor[0] == 255);
|
||||||
|
lardon3d_feature_reader_close(reader);
|
||||||
|
lardon3d_extracted_features_destroy(&large);
|
||||||
|
char features_dir[PATH_MAX];
|
||||||
|
CHECK(join_path(features_dir, root, "assets/features"));
|
||||||
|
size_t before = count_files(features_dir);
|
||||||
|
Lardon3DFeatureExtractorParameters failed_parameters = {511, 4, 10};
|
||||||
|
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_FEATURE_REGISTER", "1", 1) == 0);
|
||||||
|
Lardon3DProjectDbFeatureSet failed_set;
|
||||||
|
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &failed_parameters, &f,
|
||||||
|
&failed_set) == LARDON3D_FEATURE_STORE_DB_BUSY);
|
||||||
|
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_FEATURE_REGISTER") == 0 &&
|
||||||
|
count_files(features_dir) == before + 1);
|
||||||
|
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &failed_parameters, &f,
|
||||||
|
&failed_set) == LARDON3D_FEATURE_STORE_OK);
|
||||||
|
Lardon3DFeatureExtractorParameters nondurable_parameters = {510, 4, 10};
|
||||||
|
CHECK(setenv("LARDON3D_TEST_FEATURE_FAIL_DIRECTORY_SYNC", "1", 1) == 0);
|
||||||
|
Lardon3DProjectDbFeatureSet nondurable;
|
||||||
|
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &nondurable_parameters, &f,
|
||||||
|
&nondurable) ==
|
||||||
|
LARDON3D_FEATURE_STORE_PUBLISHED_NOT_DURABLE &&
|
||||||
|
nondurable.asset.durability == LARDON3D_DB_FEATURE_ASSET_PUBLISHED_NOT_DURABLE);
|
||||||
|
CHECK(unsetenv("LARDON3D_TEST_FEATURE_FAIL_DIRECTORY_SYNC") == 0);
|
||||||
|
Lardon3DProjectDbFeatureSet promoted;
|
||||||
|
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &nondurable_parameters, &f,
|
||||||
|
&promoted) == LARDON3D_FEATURE_STORE_ALREADY_PRESENT &&
|
||||||
|
promoted.feature_set_id == nondurable.feature_set_id &&
|
||||||
|
promoted.asset.durability == LARDON3D_DB_FEATURE_ASSET_DURABLE);
|
||||||
|
uint64_t deleted_set = nondurable.feature_set_id, deleted_asset = nondurable.feature_asset_id;
|
||||||
|
CHECK(lardon3d_project_db_test_delete_feature_identity(db, deleted_set, deleted_asset) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
Lardon3DFeatureExtractorParameters replacement_parameters = {509, 4, 10};
|
||||||
|
Lardon3DProjectDbFeatureSet replacement;
|
||||||
|
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &replacement_parameters, &f,
|
||||||
|
&replacement) == LARDON3D_FEATURE_STORE_OK &&
|
||||||
|
replacement.feature_set_id > deleted_set && replacement.feature_asset_id > deleted_asset);
|
||||||
|
Lardon3DProjectDbFeatureSet page[8];
|
||||||
|
size_t count = 0;
|
||||||
|
CHECK(lardon3d_project_db_list_feature_sets(db, 0, page, 1, &count) == LARDON3D_PROJECT_DB_OK &&
|
||||||
|
count == 1);
|
||||||
|
CHECK(lardon3d_project_db_list_feature_sets(db, page[0].feature_set_id, page, 8, &count) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK &&
|
||||||
|
count == 6);
|
||||||
|
for (uint32_t index = 0; index < 2000; ++index) {
|
||||||
|
unsigned char metadata_fingerprint[32] = {0};
|
||||||
|
metadata_fingerprint[0] = (unsigned char)index;
|
||||||
|
metadata_fingerprint[1] = (unsigned char)(index >> 8);
|
||||||
|
metadata_fingerprint[2] = 0xa5;
|
||||||
|
Lardon3DProjectDbFeatureSet metadata_set;
|
||||||
|
CHECK(lardon3d_project_db_register_feature_set(
|
||||||
|
db, ia.image_id, "metadata.fixture", 1, metadata_fingerprint, aa.sha256,
|
||||||
|
sa.feature_count, sa.descriptor_type, sa.descriptor_dimension, sa.asset.sha256,
|
||||||
|
sa.asset.path, sa.asset.size_bytes, sa.asset.durability, 0, index,
|
||||||
|
&metadata_set) == LARDON3D_PROJECT_DB_OK);
|
||||||
|
}
|
||||||
|
Lardon3DProjectDbFeatureSet *metadata_page = calloc(256, sizeof(*metadata_page));
|
||||||
|
CHECK(metadata_page);
|
||||||
|
const size_t page_sizes[] = {1, 8, 64, 256};
|
||||||
|
for (size_t index = 0; index < sizeof(page_sizes) / sizeof(page_sizes[0]); ++index) {
|
||||||
|
CHECK(lardon3d_project_db_list_feature_sets(db, 0, metadata_page, page_sizes[index], &count) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK &&
|
||||||
|
count == page_sizes[index]);
|
||||||
|
}
|
||||||
|
CHECK(lardon3d_project_db_list_feature_sets(db, 0, metadata_page, 257, &count) ==
|
||||||
|
LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||||
|
free(metadata_page);
|
||||||
|
char asset_path[PATH_MAX];
|
||||||
|
CHECK(join_path(asset_path, root, sa.asset.path));
|
||||||
|
int fd = open(asset_path, O_RDONLY | O_CLOEXEC);
|
||||||
|
CHECK(fd >= 0);
|
||||||
|
struct stat asset_info;
|
||||||
|
CHECK(fstat(fd, &asset_info) == 0 && asset_info.st_size >= 160);
|
||||||
|
size_t original_size = (size_t)asset_info.st_size;
|
||||||
|
unsigned char *original = malloc(original_size);
|
||||||
|
CHECK(original && read(fd, original, original_size) == (ssize_t)original_size && close(fd) == 0);
|
||||||
|
|
||||||
|
uint64_t descriptor_offset = read_u64_le(original + 48);
|
||||||
|
const off_t truncated_sizes[] = {
|
||||||
|
100,
|
||||||
|
(off_t)descriptor_offset - 1,
|
||||||
|
(off_t)descriptor_offset,
|
||||||
|
(off_t)original_size - 1,
|
||||||
|
};
|
||||||
|
for (size_t index = 0; index < sizeof(truncated_sizes) / sizeof(truncated_sizes[0]); ++index) {
|
||||||
|
CHECK(truncate(asset_path, truncated_sizes[index]) == 0 &&
|
||||||
|
expect_reader_result(root, &sa, LARDON3D_FEATURE_STORE_CORRUPT) &&
|
||||||
|
restore_file(asset_path, original, original_size));
|
||||||
|
}
|
||||||
|
|
||||||
|
const size_t corrupt_offsets[] = {12, 16, 20, 24, 28, 40, 48, 56, 148, 144, 128};
|
||||||
|
for (size_t index = 0; index < sizeof(corrupt_offsets) / sizeof(corrupt_offsets[0]); ++index) {
|
||||||
|
unsigned char mutated = (unsigned char)(original[corrupt_offsets[index]] ^ 0x5aU);
|
||||||
|
fd = open(asset_path, O_WRONLY | O_CLOEXEC);
|
||||||
|
CHECK(fd >= 0 && pwrite(fd, &mutated, 1, (off_t)corrupt_offsets[index]) == 1 &&
|
||||||
|
close(fd) == 0 && expect_reader_result(root, &sa, LARDON3D_FEATURE_STORE_CORRUPT) &&
|
||||||
|
restore_file(asset_path, original, original_size));
|
||||||
|
}
|
||||||
|
|
||||||
|
unsigned char future_version[4] = {2, 0, 0, 0};
|
||||||
|
fd = open(asset_path, O_WRONLY | O_CLOEXEC);
|
||||||
|
CHECK(fd >= 0 && pwrite(fd, future_version, sizeof(future_version), 8) == 4 && close(fd) == 0 &&
|
||||||
|
expect_reader_result(root, &sa, LARDON3D_FEATURE_STORE_UNSUPPORTED_VERSION) &&
|
||||||
|
restore_file(asset_path, original, original_size));
|
||||||
|
|
||||||
|
fd = open(asset_path, O_WRONLY | O_TRUNC | O_CLOEXEC);
|
||||||
|
CHECK(fd >= 0 && write(fd, "bad", 3) == 3 && close(fd) == 0 &&
|
||||||
|
expect_reader_result(root, &sa, LARDON3D_FEATURE_STORE_CORRUPT));
|
||||||
|
free(original);
|
||||||
|
lardon3d_extracted_features_destroy(&f);
|
||||||
|
lardon3d_extracted_features_destroy(&uf);
|
||||||
|
lardon3d_project_db_close(db);
|
||||||
|
CHECK(remove_tree(root));
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
int main(void) { return run_test() ? 0 : 1; }
|
||||||
188
tests/test_feature_task.c
Normal file
188
tests/test_feature_task.c
Normal file
|
|
@ -0,0 +1,188 @@
|
||||||
|
#include <dirent.h>
|
||||||
|
#include <errno.h>
|
||||||
|
#include <sched.h>
|
||||||
|
#include <stdbool.h>
|
||||||
|
#include <stdint.h>
|
||||||
|
#include <stdio.h>
|
||||||
|
#include <stdlib.h>
|
||||||
|
#include <string.h>
|
||||||
|
#include <sys/stat.h>
|
||||||
|
#include <unistd.h>
|
||||||
|
|
||||||
|
#include <lardon3d/feature_store.h>
|
||||||
|
#include <lardon3d/feature_task.h>
|
||||||
|
#include <lardon3d/image_catalog.h>
|
||||||
|
#include <lardon3d/project.h>
|
||||||
|
#include <lardon3d/task_queue.h>
|
||||||
|
|
||||||
|
#define CHECK(x) \
|
||||||
|
do { \
|
||||||
|
if (!(x)) { \
|
||||||
|
fprintf(stderr, "Échec ligne %d: %s\n", __LINE__, #x); \
|
||||||
|
return false; \
|
||||||
|
} \
|
||||||
|
} while (0)
|
||||||
|
static bool join_path(char out[PATH_MAX], const char *a, const char *b) {
|
||||||
|
int n = snprintf(out, PATH_MAX, "%s/%s", a, b);
|
||||||
|
return n > 0 && (size_t)n < PATH_MAX;
|
||||||
|
}
|
||||||
|
static bool remove_tree(const char *p) {
|
||||||
|
struct stat s;
|
||||||
|
if (lstat(p, &s) != 0) {
|
||||||
|
return errno == ENOENT;
|
||||||
|
}
|
||||||
|
if (!S_ISDIR(s.st_mode)) {
|
||||||
|
return unlink(p) == 0;
|
||||||
|
}
|
||||||
|
DIR *d = opendir(p);
|
||||||
|
if (!d) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
bool ok = true;
|
||||||
|
for (struct dirent *e = readdir(d); e; e = readdir(d)) {
|
||||||
|
if (!strcmp(e->d_name, ".") || !strcmp(e->d_name, "..")) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
char c[PATH_MAX];
|
||||||
|
if (!join_path(c, p, e->d_name) || !remove_tree(c)) {
|
||||||
|
ok = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (closedir(d) || rmdir(p)) {
|
||||||
|
ok = false;
|
||||||
|
}
|
||||||
|
return ok;
|
||||||
|
}
|
||||||
|
static bool write_pgm(const char *p) {
|
||||||
|
FILE *f = fopen(p, "wb");
|
||||||
|
if (!f) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
fprintf(f, "P5\n192 192\n255\n");
|
||||||
|
for (unsigned y = 0; y < 192; y++) {
|
||||||
|
for (unsigned x = 0; x < 192; x++) {
|
||||||
|
unsigned char v = (unsigned char)((((x / 6) ^ (y / 6)) & 1) ? 245 : 10);
|
||||||
|
if (fwrite(&v, 1, 1, f) != 1) {
|
||||||
|
fclose(f);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return fclose(f) == 0;
|
||||||
|
}
|
||||||
|
static bool runtime(Lardon3DAppState *s) {
|
||||||
|
s->hardware_profile = (Lardon3DHardwareProfile){.logical_cpu_count = 64,
|
||||||
|
.page_size_bytes = 4096,
|
||||||
|
.memory_total_bytes = UINT64_MAX,
|
||||||
|
.cpu_architecture = "test"};
|
||||||
|
Lardon3DResourcePolicy p = {
|
||||||
|
.maximum_cpu_load_ratio = 1, .maximum_io_pressure_avg10 = 100, .io_slot_capacity = 2};
|
||||||
|
s->resource_governor = lardon3d_resource_governor_create(&s->hardware_profile, &p);
|
||||||
|
s->task_queue = s->resource_governor ? lardon3d_task_queue_create(s->resource_governor, 4) : NULL;
|
||||||
|
return s->task_queue != NULL;
|
||||||
|
}
|
||||||
|
static bool wait_state(Lardon3DTaskQueue *q, uint64_t id, Lardon3DTaskState wanted,
|
||||||
|
Lardon3DTaskSnapshot *out) {
|
||||||
|
for (size_t i = 0; i < 2000000; i++) {
|
||||||
|
if (lardon3d_task_queue_get(q, id, out) && out->state == wanted) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
sched_yield();
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool run_test(void) {
|
||||||
|
char root[] = "/tmp/lardon3d-feature-task-XXXXXX";
|
||||||
|
CHECK(mkdtemp(root) && setenv("LARDON3D_PROJECTS_ROOT", root, 1) == 0);
|
||||||
|
char source[PATH_MAX];
|
||||||
|
CHECK(join_path(source, root, "source.pgm") && write_pgm(source));
|
||||||
|
Lardon3DAppState state;
|
||||||
|
lardon3d_app_state_init(&state);
|
||||||
|
CHECK(runtime(&state) && lardon3d_project_create(&state, "Features"));
|
||||||
|
Lardon3DProjectDbScanSet scanset;
|
||||||
|
CHECK(lardon3d_image_catalog_create_scanset(&state, "A", &scanset));
|
||||||
|
Lardon3DProjectDbImage image;
|
||||||
|
Lardon3DProjectDbImageAsset asset;
|
||||||
|
CHECK(lardon3d_image_catalog_import_file(&state, scanset.scanset_id, source, 0, &image, &asset) ==
|
||||||
|
LARDON3D_IMAGE_CATALOG_IMPORTED);
|
||||||
|
Lardon3DFeatureExtractorParameters parameters = {512, 4, 10};
|
||||||
|
CHECK(setenv("LARDON3D_TEST_FEATURE_PAUSE_BEFORE_PUBLISH", "1", 1) == 0 &&
|
||||||
|
setenv("LARDON3D_TEST_FEATURE_SKIP_FINISHED_CHECKPOINT", "1", 1) == 0);
|
||||||
|
uint64_t task_id = 0;
|
||||||
|
CHECK(lardon3d_project_enqueue_feature_extract(&state, image.image_id, ¶meters, &task_id));
|
||||||
|
Lardon3DTaskSnapshot snapshot;
|
||||||
|
CHECK(wait_state(state.task_queue, task_id, TASK_PAUSED, &snapshot));
|
||||||
|
lardon3d_task_queue_destroy(state.task_queue);
|
||||||
|
state.task_queue = NULL;
|
||||||
|
lardon3d_project_close(&state);
|
||||||
|
lardon3d_resource_governor_destroy(state.resource_governor);
|
||||||
|
state.resource_governor = NULL;
|
||||||
|
CHECK(unsetenv("LARDON3D_TEST_FEATURE_PAUSE_BEFORE_PUBLISH") == 0 &&
|
||||||
|
unsetenv("LARDON3D_TEST_FEATURE_SKIP_FINISHED_CHECKPOINT") == 0);
|
||||||
|
lardon3d_app_state_init(&state);
|
||||||
|
CHECK(runtime(&state) && lardon3d_project_open(&state, "Features"));
|
||||||
|
Lardon3DProjectRecoverySummary summary;
|
||||||
|
CHECK(lardon3d_project_last_recovery_summary(&state, &summary) && summary.resumed == 1);
|
||||||
|
CHECK(wait_state(state.task_queue, task_id, TASK_COMPLETED, &snapshot) &&
|
||||||
|
snapshot.progress == 100);
|
||||||
|
unsigned char fp[32];
|
||||||
|
lardon3d_feature_extractor_parameter_fingerprint(¶meters, fp);
|
||||||
|
Lardon3DProjectDbFeatureSet set;
|
||||||
|
CHECK(lardon3d_project_db_find_feature_set(state.project_db, image.image_id,
|
||||||
|
LARDON3D_FEATURE_EXTRACTOR_KIND, 1, fp,
|
||||||
|
&set) == LARDON3D_PROJECT_DB_OK);
|
||||||
|
lardon3d_task_queue_destroy(state.task_queue);
|
||||||
|
state.task_queue = NULL;
|
||||||
|
lardon3d_project_close(&state);
|
||||||
|
lardon3d_resource_governor_destroy(state.resource_governor);
|
||||||
|
state.resource_governor = NULL;
|
||||||
|
lardon3d_app_state_init(&state);
|
||||||
|
CHECK(runtime(&state) && lardon3d_project_open(&state, "Features"));
|
||||||
|
CHECK(lardon3d_project_last_recovery_summary(&state, &summary) && summary.resumed == 0);
|
||||||
|
CHECK(lardon3d_project_db_load_feature_set(state.project_db, set.feature_set_id, &set) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(unlink(source) == 0);
|
||||||
|
Lardon3DFeatureReader *reader = NULL;
|
||||||
|
Lardon3DFeatureFileMetadata metadata;
|
||||||
|
CHECK(lardon3d_feature_reader_open(state.project_path, &set, &reader, &metadata) ==
|
||||||
|
LARDON3D_FEATURE_STORE_OK);
|
||||||
|
lardon3d_feature_reader_close(reader);
|
||||||
|
|
||||||
|
CHECK(write_pgm(source));
|
||||||
|
Lardon3DProjectDbImage missing_image;
|
||||||
|
Lardon3DProjectDbImageAsset missing_asset;
|
||||||
|
CHECK(lardon3d_image_catalog_import_file(&state, scanset.scanset_id, source, 0, &missing_image,
|
||||||
|
&missing_asset) ==
|
||||||
|
LARDON3D_IMAGE_CATALOG_ALREADY_PRESENT);
|
||||||
|
char managed_path[PATH_MAX];
|
||||||
|
CHECK(join_path(managed_path, state.project_path, missing_asset.path));
|
||||||
|
CHECK(unlink(managed_path) == 0);
|
||||||
|
uint64_t missing_task_id = 0;
|
||||||
|
Lardon3DFeatureExtractorParameters missing_parameters = {511, 4, 10};
|
||||||
|
CHECK(lardon3d_project_enqueue_feature_extract(&state, missing_image.image_id,
|
||||||
|
&missing_parameters, &missing_task_id));
|
||||||
|
CHECK(wait_state(state.task_queue, missing_task_id, TASK_FAILED, &snapshot));
|
||||||
|
|
||||||
|
char invalid_source[PATH_MAX];
|
||||||
|
CHECK(join_path(invalid_source, root, "invalid.pgm"));
|
||||||
|
FILE *invalid = fopen(invalid_source, "wb");
|
||||||
|
CHECK(invalid && fwrite("not an image", 1, 12, invalid) == 12 && fclose(invalid) == 0);
|
||||||
|
Lardon3DProjectDbImage invalid_image;
|
||||||
|
Lardon3DProjectDbImageAsset invalid_asset;
|
||||||
|
CHECK(lardon3d_image_catalog_import_file(&state, scanset.scanset_id, invalid_source, 0,
|
||||||
|
&invalid_image,
|
||||||
|
&invalid_asset) == LARDON3D_IMAGE_CATALOG_IMPORTED);
|
||||||
|
uint64_t invalid_task_id = 0;
|
||||||
|
Lardon3DFeatureExtractorParameters invalid_parameters = {510, 4, 10};
|
||||||
|
CHECK(lardon3d_project_enqueue_feature_extract(&state, invalid_image.image_id,
|
||||||
|
&invalid_parameters, &invalid_task_id));
|
||||||
|
CHECK(wait_state(state.task_queue, invalid_task_id, TASK_FAILED, &snapshot));
|
||||||
|
lardon3d_task_queue_destroy(state.task_queue);
|
||||||
|
state.task_queue = NULL;
|
||||||
|
lardon3d_project_close(&state);
|
||||||
|
lardon3d_resource_governor_destroy(state.resource_governor);
|
||||||
|
CHECK(remove_tree(root));
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
int main(void) { return run_test() ? 0 : 1; }
|
||||||
|
|
@ -11,432 +11,561 @@
|
||||||
|
|
||||||
#include <lardon3d/project_db.h>
|
#include <lardon3d/project_db.h>
|
||||||
|
|
||||||
#define CHECK(condition) do { if (!(condition)) { \
|
#define CHECK(condition) \
|
||||||
(void)fprintf(stderr, "Échec ligne %d : %s\n", __LINE__, #condition); return false; \
|
do { \
|
||||||
} } while (0)
|
if (!(condition)) { \
|
||||||
|
(void)fprintf(stderr, "Échec ligne %d : %s\n", __LINE__, #condition); \
|
||||||
|
return false; \
|
||||||
|
} \
|
||||||
|
} while (0)
|
||||||
|
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_test_orphan_checkpoint(Lardon3DProjectDb *database);
|
Lardon3DProjectDbResult lardon3d_project_db_test_orphan_checkpoint(Lardon3DProjectDb *database);
|
||||||
Lardon3DProjectDbResult lardon3d_project_db_test_delete_catalog_identity(
|
Lardon3DProjectDbResult
|
||||||
Lardon3DProjectDb *database, uint64_t scanset_id, uint64_t image_id,
|
lardon3d_project_db_test_delete_catalog_identity(Lardon3DProjectDb *database, uint64_t scanset_id,
|
||||||
uint64_t asset_id);
|
uint64_t image_id, uint64_t asset_id);
|
||||||
|
|
||||||
static void
|
static void asset_path_for_hash(const unsigned char hash[LARDON3D_PROJECT_DB_SHA256_SIZE],
|
||||||
asset_path_for_hash(const unsigned char hash[LARDON3D_PROJECT_DB_SHA256_SIZE],
|
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY]) {
|
||||||
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY])
|
static const char digits[] = "0123456789abcdef";
|
||||||
{
|
char hex[65];
|
||||||
static const char digits[] = "0123456789abcdef";
|
for (size_t index = 0; index < sizeof(hex) / 2; ++index) {
|
||||||
char hex[65];
|
hex[index * 2] = digits[hash[index] >> 4];
|
||||||
for (size_t index = 0; index < sizeof(hex) / 2; ++index) {
|
hex[index * 2 + 1] = digits[hash[index] & 15U];
|
||||||
hex[index * 2] = digits[hash[index] >> 4];
|
}
|
||||||
hex[index * 2 + 1] = digits[hash[index] & 15U];
|
hex[64] = '\0';
|
||||||
}
|
(void)snprintf(path, LARDON3D_PROJECT_DB_PATH_CAPACITY, "assets/images/%c%c/%s", hex[0], hex[1],
|
||||||
hex[64] = '\0';
|
hex);
|
||||||
(void)snprintf(path, LARDON3D_PROJECT_DB_PATH_CAPACITY,
|
|
||||||
"assets/images/%c%c/%s", hex[0], hex[1], hex);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
static Lardon3DTaskDurableSnapshot
|
static Lardon3DTaskDurableSnapshot task_snapshot(uint64_t id, Lardon3DTaskState saved) {
|
||||||
task_snapshot(uint64_t id, Lardon3DTaskState saved)
|
Lardon3DTaskDurableSnapshot snapshot = {
|
||||||
{
|
.id = id,
|
||||||
Lardon3DTaskDurableSnapshot snapshot = {
|
.estimate = {.minimum_batch_size = 1, .maximum_batch_size = 1, .desired_cpu_threads = 1},
|
||||||
.id = id,
|
.progress = saved == TASK_COMPLETED ? 100 : 25,
|
||||||
.estimate = {.minimum_batch_size = 1, .maximum_batch_size = 1, .desired_cpu_threads = 1},
|
.saved_state = saved,
|
||||||
.progress = saved == TASK_COMPLETED ? 100 : 25,
|
.recovery_state = saved == TASK_RUNNING || saved == TASK_PAUSED ? TASK_PENDING : saved,
|
||||||
.saved_state = saved,
|
.started_at = {.tv_sec = 10, .tv_nsec = 20},
|
||||||
.recovery_state = saved == TASK_RUNNING || saved == TASK_PAUSED ? TASK_PENDING : saved,
|
.finished_at = {.tv_sec = 30, .tv_nsec = 40},
|
||||||
.started_at = {.tv_sec = 10, .tv_nsec = 20},
|
.sequence_count = 2,
|
||||||
.finished_at = {.tv_sec = 30, .tv_nsec = 40},
|
};
|
||||||
.sequence_count = 2,
|
(void)snprintf(snapshot.name, sizeof(snapshot.name), "Tâche %llu", (unsigned long long)id);
|
||||||
};
|
return snapshot;
|
||||||
(void)snprintf(snapshot.name, sizeof(snapshot.name), "Tâche %llu", (unsigned long long)id);
|
|
||||||
return snapshot;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
typedef struct { Lardon3DProjectDb *database; bool success; } ThreadContext;
|
typedef struct {
|
||||||
typedef struct { Lardon3DProjectDb *database; uint64_t id; bool success; } IdThreadContext;
|
Lardon3DProjectDb *database;
|
||||||
|
bool success;
|
||||||
|
} ThreadContext;
|
||||||
|
typedef struct {
|
||||||
|
Lardon3DProjectDb *database;
|
||||||
|
uint64_t id;
|
||||||
|
bool success;
|
||||||
|
} IdThreadContext;
|
||||||
|
|
||||||
static void *
|
static void *read_thread(void *userdata) {
|
||||||
read_thread(void *userdata)
|
ThreadContext *context = userdata;
|
||||||
{
|
context->success = true;
|
||||||
ThreadContext *context = userdata;
|
for (size_t index = 0; index < 100; ++index) {
|
||||||
context->success = true;
|
|
||||||
for (size_t index = 0; index < 100; ++index) {
|
|
||||||
Lardon3DProjectDbTask task;
|
|
||||||
if (lardon3d_project_db_load_task(context->database, 1, &task) != LARDON3D_PROJECT_DB_OK) {
|
|
||||||
context->success = false;
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
static void *
|
|
||||||
allocate_id_thread(void *userdata)
|
|
||||||
{
|
|
||||||
IdThreadContext *context = userdata;
|
|
||||||
context->success = lardon3d_project_db_allocate_task_id(
|
|
||||||
context->database, &context->id) == LARDON3D_PROJECT_DB_OK;
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
static bool
|
|
||||||
create_future_database(const char *path)
|
|
||||||
{
|
|
||||||
sqlite3 *connection = NULL;
|
|
||||||
if (sqlite3_open(path, &connection) != SQLITE_OK) return false;
|
|
||||||
bool ok = sqlite3_exec(connection, "CREATE TABLE metadata(key TEXT PRIMARY KEY,value INTEGER NOT NULL);"
|
|
||||||
"INSERT INTO metadata VALUES('schema_version',5);", NULL, NULL, NULL) == SQLITE_OK;
|
|
||||||
return sqlite3_close(connection) == SQLITE_OK && ok;
|
|
||||||
}
|
|
||||||
|
|
||||||
static bool
|
|
||||||
create_v1_database(const char *path)
|
|
||||||
{
|
|
||||||
static const char sql[] =
|
|
||||||
"PRAGMA foreign_keys=ON;"
|
|
||||||
"CREATE TABLE metadata(key TEXT PRIMARY KEY,value INTEGER NOT NULL);"
|
|
||||||
"INSERT INTO metadata VALUES('schema_version',1);"
|
|
||||||
"CREATE TABLE project(singleton INTEGER PRIMARY KEY CHECK(singleton=1),stable_id TEXT NOT NULL UNIQUE,name TEXT NOT NULL,created_at INTEGER NOT NULL,updated_at INTEGER NOT NULL);"
|
|
||||||
"CREATE TABLE tasks(task_id INTEGER PRIMARY KEY CHECK(task_id>0),name TEXT NOT NULL,saved_state INTEGER NOT NULL CHECK(saved_state BETWEEN 0 AND 5),recovery_state INTEGER NOT NULL CHECK(recovery_state BETWEEN 0 AND 5),progress INTEGER NOT NULL CHECK(progress BETWEEN 0 AND 100),sequence_count INTEGER NOT NULL CHECK(sequence_count>=0),started_sec INTEGER NOT NULL,started_nsec INTEGER NOT NULL CHECK(started_nsec BETWEEN 0 AND 999999999),finished_sec INTEGER NOT NULL,finished_nsec INTEGER NOT NULL CHECK(finished_nsec BETWEEN 0 AND 999999999),updated_at INTEGER NOT NULL);"
|
|
||||||
"CREATE INDEX tasks_recovery_state_idx ON tasks(recovery_state,task_id);"
|
|
||||||
"CREATE TABLE checkpoints(task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE CASCADE,path TEXT NOT NULL,format_version INTEGER NOT NULL CHECK(format_version>0),durability INTEGER NOT NULL CHECK(durability BETWEEN 0 AND 1),updated_at INTEGER NOT NULL);"
|
|
||||||
"CREATE TABLE artifacts(artifact_id TEXT PRIMARY KEY,kind TEXT NOT NULL,path TEXT NOT NULL,state INTEGER NOT NULL CHECK(state BETWEEN 0 AND 1),size_bytes INTEGER NOT NULL CHECK(size_bytes>=0),producer_task_id INTEGER REFERENCES tasks(task_id),created_at INTEGER NOT NULL,updated_at INTEGER NOT NULL);"
|
|
||||||
"CREATE INDEX artifacts_state_idx ON artifacts(state,artifact_id);"
|
|
||||||
"CREATE INDEX artifacts_producer_idx ON artifacts(producer_task_id);"
|
|
||||||
"INSERT INTO project VALUES(1,'legacy-project','Legacy',1,1);"
|
|
||||||
"INSERT INTO tasks VALUES(9,'Legacy task',1,0,12,3,1,0,0,0,2);"
|
|
||||||
"INSERT INTO checkpoints VALUES(9,'legacy.chk',1,0,2);"
|
|
||||||
"INSERT INTO artifacts VALUES('legacy-artifact','legacy','legacy.bin',0,0,9,2,2);";
|
|
||||||
sqlite3 *connection = NULL;
|
|
||||||
if (sqlite3_open(path, &connection) != SQLITE_OK) return false;
|
|
||||||
bool ok = sqlite3_exec(connection, sql, NULL, NULL, NULL) == SQLITE_OK;
|
|
||||||
return sqlite3_close(connection) == SQLITE_OK && ok;
|
|
||||||
}
|
|
||||||
|
|
||||||
static bool
|
|
||||||
create_v2_database(const char *path)
|
|
||||||
{
|
|
||||||
if (!create_v1_database(path)) return false;
|
|
||||||
sqlite3 *connection = NULL;
|
|
||||||
if (sqlite3_open(path, &connection) != SQLITE_OK) return false;
|
|
||||||
bool ok = sqlite3_exec(connection,
|
|
||||||
"ALTER TABLE tasks ADD COLUMN task_kind TEXT;"
|
|
||||||
"ALTER TABLE tasks ADD COLUMN task_kind_version INTEGER CHECK(task_kind_version IS NULL OR task_kind_version>0);"
|
|
||||||
"UPDATE tasks SET task_kind='test.work',task_kind_version=1 WHERE task_id=9;"
|
|
||||||
"UPDATE metadata SET value=2 WHERE key='schema_version'",
|
|
||||||
NULL, NULL, NULL) == SQLITE_OK;
|
|
||||||
return sqlite3_close(connection) == SQLITE_OK && ok;
|
|
||||||
}
|
|
||||||
|
|
||||||
static bool
|
|
||||||
create_v3_database(const char *path)
|
|
||||||
{
|
|
||||||
if (!create_v2_database(path)) return false;
|
|
||||||
sqlite3 *connection = NULL;
|
|
||||||
if (sqlite3_open(path, &connection) != SQLITE_OK) return false;
|
|
||||||
bool ok = sqlite3_exec(connection,
|
|
||||||
"CREATE TABLE image_import_tasks(task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE CASCADE,source_path TEXT NOT NULL);"
|
|
||||||
"INSERT INTO image_import_tasks VALUES(9,'/legacy/source');"
|
|
||||||
"INSERT INTO metadata(key,value) VALUES('next_task_id',10);"
|
|
||||||
"UPDATE metadata SET value=3 WHERE key='schema_version'",
|
|
||||||
NULL, NULL, NULL) == SQLITE_OK;
|
|
||||||
return sqlite3_close(connection) == SQLITE_OK && ok;
|
|
||||||
}
|
|
||||||
|
|
||||||
static bool
|
|
||||||
query_integer(const char *path, const char *sql, sqlite3_int64 expected)
|
|
||||||
{
|
|
||||||
sqlite3 *connection = NULL;
|
|
||||||
sqlite3_stmt *statement = NULL;
|
|
||||||
if (sqlite3_open_v2(path, &connection, SQLITE_OPEN_READONLY, NULL) != SQLITE_OK
|
|
||||||
|| sqlite3_prepare_v2(connection, sql, -1, &statement, NULL) != SQLITE_OK) {
|
|
||||||
if (statement) (void)sqlite3_finalize(statement);
|
|
||||||
if (connection) (void)sqlite3_close(connection);
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
bool matches = sqlite3_step(statement) == SQLITE_ROW
|
|
||||||
&& sqlite3_column_int64(statement, 0) == expected
|
|
||||||
&& sqlite3_step(statement) == SQLITE_DONE;
|
|
||||||
return sqlite3_finalize(statement) == SQLITE_OK
|
|
||||||
&& sqlite3_close(connection) == SQLITE_OK && matches;
|
|
||||||
}
|
|
||||||
|
|
||||||
static bool
|
|
||||||
run_test(void)
|
|
||||||
{
|
|
||||||
char directory[] = "/tmp/lardon3d-project-db-XXXXXX";
|
|
||||||
CHECK(mkdtemp(directory));
|
|
||||||
char database_path[512], artifact_path[512], future_path[512], corrupt_path[512];
|
|
||||||
char legacy_path[512], failed_migration_path[512], v2_path[512];
|
|
||||||
char failed_v3_migration_path[512], v3_path[512], failed_v4_path[512];
|
|
||||||
CHECK(snprintf(database_path, sizeof(database_path), "%s/project.db", directory) > 0);
|
|
||||||
CHECK(snprintf(artifact_path, sizeof(artifact_path), "%s/artifact.bin", directory) > 0);
|
|
||||||
CHECK(snprintf(future_path, sizeof(future_path), "%s/future.db", directory) > 0);
|
|
||||||
CHECK(snprintf(corrupt_path, sizeof(corrupt_path), "%s/corrupt.db", directory) > 0);
|
|
||||||
CHECK(snprintf(legacy_path, sizeof(legacy_path), "%s/legacy.db", directory) > 0);
|
|
||||||
CHECK(snprintf(failed_migration_path, sizeof(failed_migration_path), "%s/failed-migration.db", directory) > 0);
|
|
||||||
CHECK(snprintf(v2_path, sizeof(v2_path), "%s/v2.db", directory) > 0);
|
|
||||||
CHECK(snprintf(failed_v3_migration_path, sizeof(failed_v3_migration_path),
|
|
||||||
"%s/failed-v3-migration.db", directory) > 0);
|
|
||||||
CHECK(snprintf(v3_path, sizeof(v3_path), "%s/v3.db", directory) > 0);
|
|
||||||
CHECK(snprintf(failed_v4_path, sizeof(failed_v4_path),
|
|
||||||
"%s/failed-v4-migration.db", directory) > 0);
|
|
||||||
|
|
||||||
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY];
|
|
||||||
Lardon3DProjectDb *database = NULL;
|
|
||||||
CHECK(lardon3d_project_db_open(database_path, &database, error) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
CHECK(database && lardon3d_project_db_schema_version(database) == 4);
|
|
||||||
bool legacy_pending = true;
|
|
||||||
CHECK(lardon3d_project_db_legacy_catalog_pending(database, &legacy_pending)
|
|
||||||
== LARDON3D_PROJECT_DB_OK && !legacy_pending);
|
|
||||||
|
|
||||||
Lardon3DProjectDbScanSet deleted_scanset;
|
|
||||||
CHECK(lardon3d_project_db_create_scanset(database, "Deleted identity",
|
|
||||||
&deleted_scanset) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
unsigned char first_hash[LARDON3D_PROJECT_DB_SHA256_SIZE] = {1};
|
|
||||||
char first_asset_path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
|
||||||
asset_path_for_hash(first_hash, first_asset_path);
|
|
||||||
Lardon3DProjectDbImageRegisterStatus identity_status;
|
|
||||||
Lardon3DProjectDbImage deleted_image;
|
|
||||||
CHECK(lardon3d_project_db_register_image(database,
|
|
||||||
deleted_scanset.scanset_id, first_hash, first_asset_path, 1,
|
|
||||||
"deleted.jpg", "/source/deleted.jpg", 0, 1, &identity_status,
|
|
||||||
&deleted_image) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
CHECK(lardon3d_project_db_test_delete_catalog_identity(database,
|
|
||||||
deleted_scanset.scanset_id, deleted_image.image_id,
|
|
||||||
deleted_image.asset_id) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
Lardon3DProjectDbScanSet replacement_scanset;
|
|
||||||
CHECK(lardon3d_project_db_create_scanset(database, "Replacement identity",
|
|
||||||
&replacement_scanset) == LARDON3D_PROJECT_DB_OK
|
|
||||||
&& replacement_scanset.scanset_id > deleted_scanset.scanset_id);
|
|
||||||
unsigned char second_hash[LARDON3D_PROJECT_DB_SHA256_SIZE] = {2};
|
|
||||||
char second_asset_path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
|
||||||
asset_path_for_hash(second_hash, second_asset_path);
|
|
||||||
Lardon3DProjectDbImage replacement_image;
|
|
||||||
CHECK(lardon3d_project_db_register_image(database,
|
|
||||||
replacement_scanset.scanset_id, second_hash, second_asset_path, 1,
|
|
||||||
"replacement.jpg", "/source/replacement.jpg", 0, 2,
|
|
||||||
&identity_status, &replacement_image) == LARDON3D_PROJECT_DB_OK
|
|
||||||
&& replacement_image.image_id > deleted_image.image_id
|
|
||||||
&& replacement_image.asset_id > deleted_image.asset_id);
|
|
||||||
|
|
||||||
Lardon3DProjectDbProject project = {.created_at = 100, .updated_at = 100};
|
|
||||||
(void)snprintf(project.stable_id, sizeof(project.stable_id), "project-0001");
|
|
||||||
(void)snprintf(project.name, sizeof(project.name), "Projet test");
|
|
||||||
CHECK(lardon3d_project_db_set_project(database, &project) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
Lardon3DProjectDbProject loaded_project;
|
|
||||||
CHECK(lardon3d_project_db_get_project(database, &loaded_project) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
CHECK(strcmp(loaded_project.stable_id, project.stable_id) == 0);
|
|
||||||
project.updated_at = 101; (void)snprintf(project.name, sizeof(project.name), "Projet renommé");
|
|
||||||
CHECK(lardon3d_project_db_set_project(database, &project) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
Lardon3DProjectDbProject conflict = project; (void)snprintf(conflict.stable_id, sizeof(conflict.stable_id), "other");
|
|
||||||
CHECK(lardon3d_project_db_set_project(database, &conflict) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
|
||||||
char copied_error[LARDON3D_PROJECT_DB_ERROR_CAPACITY];
|
|
||||||
CHECK(lardon3d_project_db_last_error(database, copied_error));
|
|
||||||
CHECK(copied_error[0] != '\0');
|
|
||||||
project = loaded_project; memset(project.name, 'x', sizeof(project.name));
|
|
||||||
CHECK(lardon3d_project_db_set_project(database, &project) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
|
||||||
|
|
||||||
Lardon3DTaskDurableSnapshot running = task_snapshot(1, TASK_RUNNING);
|
|
||||||
Lardon3DProjectDbCheckpoint checkpoint = {.format_version = 1, .durability = LARDON3D_DB_CHECKPOINT_DURABLE, .updated_at = 200};
|
|
||||||
(void)snprintf(checkpoint.path, sizeof(checkpoint.path), "%s/checkpoints/task-1.chk", directory);
|
|
||||||
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 200) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
running.progress = 30; running.sequence_count = 3;
|
|
||||||
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 201) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
Lardon3DProjectDbScanSet import_scanset;
|
|
||||||
CHECK(lardon3d_project_db_create_scanset(database, "Legacy import",
|
|
||||||
&import_scanset) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
Lardon3DProjectDbTask task;
|
Lardon3DProjectDbTask task;
|
||||||
CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK);
|
if (lardon3d_project_db_load_task(context->database, 1, &task) != LARDON3D_PROJECT_DB_OK) {
|
||||||
CHECK(task.recovery_state == TASK_PENDING && task.progress == 30 && task.sequence_count == 3);
|
context->success = false;
|
||||||
CHECK(task.has_task_kind && strcmp(task.task_kind, "test.work") == 0
|
break;
|
||||||
&& task.task_kind_version == 1);
|
}
|
||||||
CHECK(task.has_checkpoint && strcmp(task.checkpoint.path, checkpoint.path) == 0);
|
}
|
||||||
checkpoint.durability = LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE;
|
return NULL;
|
||||||
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 201) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
CHECK(task.checkpoint.durability == LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE);
|
|
||||||
CHECK(lardon3d_project_db_record_task(database, &running, "test.other", 1,
|
|
||||||
&checkpoint, 202) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
|
||||||
CHECK(lardon3d_project_db_record_task(database, &running, "Test.invalid", 1,
|
|
||||||
&checkpoint, 202) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
|
||||||
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 0,
|
|
||||||
&checkpoint, 202) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
|
||||||
CHECK(lardon3d_project_db_record_image_import_task(database, &running,
|
|
||||||
"import.images", 1, &checkpoint, "/tmp/source-a",
|
|
||||||
import_scanset.scanset_id, 202)
|
|
||||||
== LARDON3D_PROJECT_DB_CONSTRAINT);
|
|
||||||
|
|
||||||
Lardon3DTaskDurableSnapshot completed = task_snapshot(2, TASK_COMPLETED);
|
|
||||||
CHECK(lardon3d_project_db_record_task(database, &completed, "test.work", 1, NULL, 202) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
Lardon3DTaskDurableSnapshot no_checkpoint = task_snapshot(4, TASK_PENDING);
|
|
||||||
CHECK(lardon3d_project_db_record_task(database, &no_checkpoint, NULL, 0, NULL, 202) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
Lardon3DProjectDbTask page[1]; size_t count = 0;
|
|
||||||
CHECK(lardon3d_project_db_list_recoverable(database, 0, page, 1, &count) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
CHECK(count == 1 && page[0].task_id == 1);
|
|
||||||
CHECK(lardon3d_project_db_list_recoverable(database, 1, page, 1, &count) == LARDON3D_PROJECT_DB_OK && count == 0);
|
|
||||||
CHECK(lardon3d_project_db_list_recoverable(database, 0, page, LARDON3D_PROJECT_DB_RECOVERY_PAGE_MAX + 1, &count) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
|
||||||
|
|
||||||
Lardon3DTaskDurableSnapshot image_import = task_snapshot(5, TASK_PENDING);
|
|
||||||
CHECK(lardon3d_project_db_record_image_import_task(database, &image_import,
|
|
||||||
"import.images", 1, &checkpoint, "/tmp/source-a",
|
|
||||||
import_scanset.scanset_id, 202)
|
|
||||||
== LARDON3D_PROJECT_DB_OK);
|
|
||||||
Lardon3DProjectDbImageImport import_parameters;
|
|
||||||
CHECK(lardon3d_project_db_load_image_import(database, 5,
|
|
||||||
&import_parameters) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
CHECK(import_parameters.task_id == 5
|
|
||||||
&& strcmp(import_parameters.source_path, "/tmp/source-a") == 0
|
|
||||||
&& import_parameters.scanset_id == import_scanset.scanset_id);
|
|
||||||
CHECK(lardon3d_project_db_record_image_import_task(database, &image_import,
|
|
||||||
"import.images", 1, &checkpoint, "/tmp/source-b",
|
|
||||||
import_scanset.scanset_id, 203)
|
|
||||||
== LARDON3D_PROJECT_DB_CONSTRAINT);
|
|
||||||
|
|
||||||
Lardon3DTaskDurableSnapshot rollback_task = task_snapshot(3, TASK_PENDING);
|
|
||||||
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_CHECKPOINT", "1", 1) == 0);
|
|
||||||
CHECK(lardon3d_project_db_record_task(database, &rollback_task, "test.work", 1, &checkpoint, 203) == LARDON3D_PROJECT_DB_IO_ERROR);
|
|
||||||
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_CHECKPOINT") == 0);
|
|
||||||
CHECK(lardon3d_project_db_load_task(database, 3, &task) == LARDON3D_PROJECT_DB_NOT_FOUND);
|
|
||||||
CHECK(lardon3d_project_db_test_orphan_checkpoint(database) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
|
||||||
|
|
||||||
Lardon3DProjectDbArtifact artifact = {.state = LARDON3D_DB_ARTIFACT_STAGED, .has_producer_task = true, .producer_task_id = 1, .created_at = 300, .updated_at = 300};
|
|
||||||
(void)snprintf(artifact.artifact_id, sizeof(artifact.artifact_id), "artifact-1");
|
|
||||||
(void)snprintf(artifact.kind, sizeof(artifact.kind), "generic-test");
|
|
||||||
(void)snprintf(artifact.path, sizeof(artifact.path), "%s", artifact_path);
|
|
||||||
CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
Lardon3DProjectDbArtifact loaded_artifact;
|
|
||||||
CHECK(lardon3d_project_db_load_artifact(database, artifact.artifact_id, &loaded_artifact) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
CHECK(loaded_artifact.state == LARDON3D_DB_ARTIFACT_STAGED);
|
|
||||||
CHECK(lardon3d_project_db_mark_artifact_ready(database, artifact.artifact_id, 301) == LARDON3D_PROJECT_DB_IO_ERROR);
|
|
||||||
int descriptor = open(artifact_path, O_WRONLY | O_CREAT | O_TRUNC, 0600); CHECK(descriptor >= 0);
|
|
||||||
const char payload[] = "validated"; CHECK(write(descriptor, payload, sizeof(payload)) == (ssize_t)sizeof(payload)); CHECK(close(descriptor) == 0);
|
|
||||||
CHECK(lardon3d_project_db_mark_artifact_ready(database, artifact.artifact_id, 301) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
CHECK(lardon3d_project_db_load_artifact(database, artifact.artifact_id, &loaded_artifact) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
CHECK(loaded_artifact.state == LARDON3D_DB_ARTIFACT_READY && loaded_artifact.size_bytes == sizeof(payload));
|
|
||||||
CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
|
||||||
artifact.producer_task_id = 999;
|
|
||||||
(void)snprintf(artifact.artifact_id, sizeof(artifact.artifact_id), "orphan-artifact");
|
|
||||||
CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
|
||||||
artifact = loaded_artifact; artifact.state = LARDON3D_DB_ARTIFACT_STAGED; artifact.size_bytes = 0;
|
|
||||||
memset(artifact.path, 'x', sizeof(artifact.path));
|
|
||||||
CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
|
||||||
|
|
||||||
ThreadContext contexts[2] = {{.database = database}, {.database = database}}; pthread_t threads[2];
|
|
||||||
CHECK(pthread_create(&threads[0], NULL, read_thread, &contexts[0]) == 0);
|
|
||||||
CHECK(pthread_create(&threads[1], NULL, read_thread, &contexts[1]) == 0);
|
|
||||||
CHECK(pthread_join(threads[0], NULL) == 0 && pthread_join(threads[1], NULL) == 0);
|
|
||||||
CHECK(contexts[0].success && contexts[1].success);
|
|
||||||
IdThreadContext id_contexts[2] = {{.database = database}, {.database = database}};
|
|
||||||
CHECK(pthread_create(&threads[0], NULL, allocate_id_thread,
|
|
||||||
&id_contexts[0]) == 0);
|
|
||||||
CHECK(pthread_create(&threads[1], NULL, allocate_id_thread,
|
|
||||||
&id_contexts[1]) == 0);
|
|
||||||
CHECK(pthread_join(threads[0], NULL) == 0
|
|
||||||
&& pthread_join(threads[1], NULL) == 0);
|
|
||||||
CHECK(id_contexts[0].success && id_contexts[1].success
|
|
||||||
&& id_contexts[0].id != id_contexts[1].id
|
|
||||||
&& id_contexts[0].id > 5 && id_contexts[1].id > 5);
|
|
||||||
|
|
||||||
char too_long[LARDON3D_PROJECT_DB_PATH_CAPACITY + 1]; memset(too_long, 'x', sizeof(too_long)); too_long[sizeof(too_long) - 1] = '\0';
|
|
||||||
CHECK(lardon3d_project_db_open(too_long, &database, error) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
|
||||||
lardon3d_project_db_close(contexts[0].database); database = NULL;
|
|
||||||
CHECK(query_integer(database_path, "SELECT value FROM metadata WHERE key='schema_version'", 4));
|
|
||||||
CHECK(query_integer(database_path, "SELECT count(*) FROM tasks WHERE task_id=1", 1));
|
|
||||||
CHECK(lardon3d_project_db_open(database_path, &database, error) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
CHECK(lardon3d_project_db_load_artifact(database, "artifact-1", &loaded_artifact) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
lardon3d_project_db_close(database);
|
|
||||||
|
|
||||||
CHECK(create_future_database(future_path));
|
|
||||||
CHECK(lardon3d_project_db_open(future_path, &database, error) == LARDON3D_PROJECT_DB_UNSUPPORTED_SCHEMA);
|
|
||||||
descriptor = open(corrupt_path, O_WRONLY | O_CREAT | O_TRUNC, 0600); CHECK(descriptor >= 0);
|
|
||||||
CHECK(write(descriptor, "not sqlite", 10) == 10 && close(descriptor) == 0);
|
|
||||||
CHECK(lardon3d_project_db_open(corrupt_path, &database, error) == LARDON3D_PROJECT_DB_CORRUPT);
|
|
||||||
|
|
||||||
CHECK(create_v1_database(legacy_path));
|
|
||||||
CHECK(lardon3d_project_db_open(legacy_path, &database, error) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
CHECK(lardon3d_project_db_schema_version(database) == 4);
|
|
||||||
CHECK(lardon3d_project_db_get_project(database, &loaded_project) == LARDON3D_PROJECT_DB_OK
|
|
||||||
&& strcmp(loaded_project.stable_id, "legacy-project") == 0);
|
|
||||||
CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
CHECK(!task.has_task_kind && task.has_checkpoint
|
|
||||||
&& strcmp(task.checkpoint.path, "legacy.chk") == 0);
|
|
||||||
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact",
|
|
||||||
&loaded_artifact) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
lardon3d_project_db_close(database); database = NULL;
|
|
||||||
CHECK(query_integer(legacy_path, "SELECT value FROM metadata WHERE key='schema_version'", 4));
|
|
||||||
|
|
||||||
CHECK(create_v1_database(failed_migration_path));
|
|
||||||
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V2", "1", 1) == 0);
|
|
||||||
CHECK(lardon3d_project_db_open(failed_migration_path, &database, error)
|
|
||||||
== LARDON3D_PROJECT_DB_IO_ERROR);
|
|
||||||
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V2") == 0);
|
|
||||||
CHECK(query_integer(failed_migration_path,
|
|
||||||
"SELECT value FROM metadata WHERE key='schema_version'", 1));
|
|
||||||
CHECK(lardon3d_project_db_open(failed_migration_path, &database, error)
|
|
||||||
== LARDON3D_PROJECT_DB_OK);
|
|
||||||
CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK
|
|
||||||
&& !task.has_task_kind);
|
|
||||||
lardon3d_project_db_close(database); database = NULL;
|
|
||||||
|
|
||||||
CHECK(create_v2_database(v2_path));
|
|
||||||
CHECK(lardon3d_project_db_open(v2_path, &database, error)
|
|
||||||
== LARDON3D_PROJECT_DB_OK);
|
|
||||||
CHECK(lardon3d_project_db_load_task(database, 9, &task)
|
|
||||||
== LARDON3D_PROJECT_DB_OK && task.has_task_kind
|
|
||||||
&& strcmp(task.task_kind, "test.work") == 0);
|
|
||||||
CHECK(task.has_checkpoint
|
|
||||||
&& strcmp(task.checkpoint.path, "legacy.chk") == 0);
|
|
||||||
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact",
|
|
||||||
&loaded_artifact) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
lardon3d_project_db_close(database); database = NULL;
|
|
||||||
CHECK(query_integer(v2_path,
|
|
||||||
"SELECT value FROM metadata WHERE key='schema_version'", 4));
|
|
||||||
|
|
||||||
CHECK(create_v2_database(failed_v3_migration_path));
|
|
||||||
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V3", "1", 1) == 0);
|
|
||||||
CHECK(lardon3d_project_db_open(failed_v3_migration_path, &database, error)
|
|
||||||
== LARDON3D_PROJECT_DB_IO_ERROR);
|
|
||||||
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V3") == 0);
|
|
||||||
CHECK(query_integer(failed_v3_migration_path,
|
|
||||||
"SELECT value FROM metadata WHERE key='schema_version'", 2));
|
|
||||||
CHECK(query_integer(failed_v3_migration_path,
|
|
||||||
"SELECT count(*) FROM sqlite_master WHERE type='table' AND name='image_import_tasks'", 0));
|
|
||||||
|
|
||||||
CHECK(create_v3_database(v3_path));
|
|
||||||
CHECK(lardon3d_project_db_open(v3_path, &database, error)
|
|
||||||
== LARDON3D_PROJECT_DB_OK);
|
|
||||||
CHECK(lardon3d_project_db_load_image_import(database, 9,
|
|
||||||
&import_parameters) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
CHECK(strcmp(import_parameters.source_path, "/legacy/source") == 0
|
|
||||||
&& import_parameters.scanset_id > 0);
|
|
||||||
CHECK(lardon3d_project_db_legacy_catalog_pending(database, &legacy_pending)
|
|
||||||
== LARDON3D_PROJECT_DB_OK && legacy_pending);
|
|
||||||
CHECK(lardon3d_project_db_load_task(database, 9, &task)
|
|
||||||
== LARDON3D_PROJECT_DB_OK && task.has_task_kind);
|
|
||||||
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact",
|
|
||||||
&loaded_artifact) == LARDON3D_PROJECT_DB_OK);
|
|
||||||
lardon3d_project_db_close(database); database = NULL;
|
|
||||||
CHECK(query_integer(v3_path,
|
|
||||||
"SELECT value FROM metadata WHERE key='schema_version'", 4));
|
|
||||||
|
|
||||||
CHECK(create_v3_database(failed_v4_path));
|
|
||||||
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V4", "1", 1) == 0);
|
|
||||||
CHECK(lardon3d_project_db_open(failed_v4_path, &database, error)
|
|
||||||
== LARDON3D_PROJECT_DB_IO_ERROR);
|
|
||||||
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V4") == 0);
|
|
||||||
CHECK(query_integer(failed_v4_path,
|
|
||||||
"SELECT value FROM metadata WHERE key='schema_version'", 3));
|
|
||||||
CHECK(query_integer(failed_v4_path,
|
|
||||||
"SELECT count(*) FROM sqlite_master WHERE type='table' AND name='scansets'", 0));
|
|
||||||
|
|
||||||
CHECK(unlink(artifact_path) == 0); CHECK(unlink(database_path) == 0); CHECK(unlink(future_path) == 0); CHECK(unlink(corrupt_path) == 0);
|
|
||||||
CHECK(unlink(legacy_path) == 0); CHECK(unlink(failed_migration_path) == 0);
|
|
||||||
CHECK(unlink(v2_path) == 0); CHECK(unlink(failed_v3_migration_path) == 0);
|
|
||||||
CHECK(unlink(v3_path) == 0); CHECK(unlink(failed_v4_path) == 0);
|
|
||||||
CHECK(rmdir(directory) == 0);
|
|
||||||
return true;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
int main(void) { return run_test() ? EXIT_SUCCESS : EXIT_FAILURE; }
|
static void *allocate_id_thread(void *userdata) {
|
||||||
|
IdThreadContext *context = userdata;
|
||||||
|
context->success = lardon3d_project_db_allocate_task_id(context->database, &context->id) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK;
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool create_future_database(const char *path) {
|
||||||
|
sqlite3 *connection = NULL;
|
||||||
|
if (sqlite3_open(path, &connection) != SQLITE_OK) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
bool ok = sqlite3_exec(connection,
|
||||||
|
"CREATE TABLE metadata(key TEXT PRIMARY KEY,value INTEGER NOT NULL);"
|
||||||
|
"INSERT INTO metadata VALUES('schema_version',6);",
|
||||||
|
NULL, NULL, NULL) == SQLITE_OK;
|
||||||
|
return sqlite3_close(connection) == SQLITE_OK && ok;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool create_v1_database(const char *path) {
|
||||||
|
static const char sql[] =
|
||||||
|
"PRAGMA foreign_keys=ON;"
|
||||||
|
"CREATE TABLE metadata(key TEXT PRIMARY KEY,value INTEGER NOT NULL);"
|
||||||
|
"INSERT INTO metadata VALUES('schema_version',1);"
|
||||||
|
"CREATE TABLE project(singleton INTEGER PRIMARY KEY CHECK(singleton=1),stable_id TEXT NOT "
|
||||||
|
"NULL UNIQUE,name TEXT NOT NULL,created_at INTEGER NOT NULL,updated_at INTEGER NOT NULL);"
|
||||||
|
"CREATE TABLE tasks(task_id INTEGER PRIMARY KEY CHECK(task_id>0),name TEXT NOT "
|
||||||
|
"NULL,saved_state INTEGER NOT NULL CHECK(saved_state BETWEEN 0 AND 5),recovery_state "
|
||||||
|
"INTEGER NOT NULL CHECK(recovery_state BETWEEN 0 AND 5),progress INTEGER NOT NULL "
|
||||||
|
"CHECK(progress BETWEEN 0 AND 100),sequence_count INTEGER NOT NULL "
|
||||||
|
"CHECK(sequence_count>=0),started_sec INTEGER NOT NULL,started_nsec INTEGER NOT NULL "
|
||||||
|
"CHECK(started_nsec BETWEEN 0 AND 999999999),finished_sec INTEGER NOT NULL,finished_nsec "
|
||||||
|
"INTEGER NOT NULL CHECK(finished_nsec BETWEEN 0 AND 999999999),updated_at INTEGER NOT "
|
||||||
|
"NULL);"
|
||||||
|
"CREATE INDEX tasks_recovery_state_idx ON tasks(recovery_state,task_id);"
|
||||||
|
"CREATE TABLE checkpoints(task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE "
|
||||||
|
"CASCADE,path TEXT NOT NULL,format_version INTEGER NOT NULL "
|
||||||
|
"CHECK(format_version>0),durability INTEGER NOT NULL CHECK(durability BETWEEN 0 AND "
|
||||||
|
"1),updated_at INTEGER NOT NULL);"
|
||||||
|
"CREATE TABLE artifacts(artifact_id TEXT PRIMARY KEY,kind TEXT NOT NULL,path TEXT NOT "
|
||||||
|
"NULL,state INTEGER NOT NULL CHECK(state BETWEEN 0 AND 1),size_bytes INTEGER NOT NULL "
|
||||||
|
"CHECK(size_bytes>=0),producer_task_id INTEGER REFERENCES tasks(task_id),created_at "
|
||||||
|
"INTEGER NOT NULL,updated_at INTEGER NOT NULL);"
|
||||||
|
"CREATE INDEX artifacts_state_idx ON artifacts(state,artifact_id);"
|
||||||
|
"CREATE INDEX artifacts_producer_idx ON artifacts(producer_task_id);"
|
||||||
|
"INSERT INTO project VALUES(1,'legacy-project','Legacy',1,1);"
|
||||||
|
"INSERT INTO tasks VALUES(9,'Legacy task',1,0,12,3,1,0,0,0,2);"
|
||||||
|
"INSERT INTO checkpoints VALUES(9,'legacy.chk',1,0,2);"
|
||||||
|
"INSERT INTO artifacts VALUES('legacy-artifact','legacy','legacy.bin',0,0,9,2,2);";
|
||||||
|
sqlite3 *connection = NULL;
|
||||||
|
if (sqlite3_open(path, &connection) != SQLITE_OK) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
bool ok = sqlite3_exec(connection, sql, NULL, NULL, NULL) == SQLITE_OK;
|
||||||
|
return sqlite3_close(connection) == SQLITE_OK && ok;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool create_v2_database(const char *path) {
|
||||||
|
if (!create_v1_database(path)) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
sqlite3 *connection = NULL;
|
||||||
|
if (sqlite3_open(path, &connection) != SQLITE_OK) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
bool ok =
|
||||||
|
sqlite3_exec(connection,
|
||||||
|
"ALTER TABLE tasks ADD COLUMN task_kind TEXT;"
|
||||||
|
"ALTER TABLE tasks ADD COLUMN task_kind_version INTEGER "
|
||||||
|
"CHECK(task_kind_version IS NULL OR task_kind_version>0);"
|
||||||
|
"UPDATE tasks SET task_kind='test.work',task_kind_version=1 WHERE task_id=9;"
|
||||||
|
"UPDATE metadata SET value=2 WHERE key='schema_version'",
|
||||||
|
NULL, NULL, NULL) == SQLITE_OK;
|
||||||
|
return sqlite3_close(connection) == SQLITE_OK && ok;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool create_v3_database(const char *path) {
|
||||||
|
if (!create_v2_database(path)) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
sqlite3 *connection = NULL;
|
||||||
|
if (sqlite3_open(path, &connection) != SQLITE_OK) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
bool ok = sqlite3_exec(connection,
|
||||||
|
"CREATE TABLE image_import_tasks(task_id INTEGER PRIMARY KEY REFERENCES "
|
||||||
|
"tasks(task_id) ON DELETE CASCADE,source_path TEXT NOT NULL);"
|
||||||
|
"INSERT INTO image_import_tasks VALUES(9,'/legacy/source');"
|
||||||
|
"INSERT INTO metadata(key,value) VALUES('next_task_id',10);"
|
||||||
|
"UPDATE metadata SET value=3 WHERE key='schema_version'",
|
||||||
|
NULL, NULL, NULL) == SQLITE_OK;
|
||||||
|
return sqlite3_close(connection) == SQLITE_OK && ok;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool create_v4_database(const char *path) {
|
||||||
|
if (!create_v3_database(path)) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
sqlite3 *connection = NULL;
|
||||||
|
if (sqlite3_open(path, &connection) != SQLITE_OK) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
const char *sql =
|
||||||
|
"CREATE TABLE scansets(scanset_id INTEGER PRIMARY KEY AUTOINCREMENT "
|
||||||
|
"CHECK(scanset_id>0),name TEXT NOT NULL,created_at INTEGER NOT NULL,updated_at INTEGER NOT "
|
||||||
|
"NULL);"
|
||||||
|
"CREATE TABLE image_assets(asset_id INTEGER PRIMARY KEY AUTOINCREMENT "
|
||||||
|
"CHECK(asset_id>0),sha256 BLOB NOT NULL UNIQUE,path TEXT NOT NULL UNIQUE,size_bytes "
|
||||||
|
"INTEGER NOT NULL,state INTEGER NOT NULL,created_at INTEGER NOT NULL);"
|
||||||
|
"CREATE TABLE images(image_id INTEGER PRIMARY KEY AUTOINCREMENT "
|
||||||
|
"CHECK(image_id>0),scanset_id INTEGER NOT NULL REFERENCES scansets(scanset_id),asset_id "
|
||||||
|
"INTEGER NOT NULL REFERENCES image_assets(asset_id),original_name TEXT NOT "
|
||||||
|
"NULL,source_path TEXT NOT NULL,producer_task_id INTEGER REFERENCES "
|
||||||
|
"tasks(task_id),imported_at INTEGER NOT NULL,UNIQUE(scanset_id,asset_id));"
|
||||||
|
"CREATE INDEX images_scanset_idx ON images(scanset_id,image_id);"
|
||||||
|
"CREATE INDEX images_producer_idx ON images(producer_task_id,image_id);"
|
||||||
|
"ALTER TABLE image_import_tasks ADD COLUMN scanset_id INTEGER REFERENCES "
|
||||||
|
"scansets(scanset_id);"
|
||||||
|
"INSERT INTO scansets(name,created_at,updated_at) VALUES('Legacy',0,0);"
|
||||||
|
"UPDATE image_import_tasks SET scanset_id=1;"
|
||||||
|
"INSERT INTO metadata VALUES('legacy_image_catalog_pending',1);"
|
||||||
|
"UPDATE metadata SET value=4 WHERE key='schema_version';";
|
||||||
|
bool ok = sqlite3_exec(connection, sql, NULL, NULL, NULL) == SQLITE_OK;
|
||||||
|
return sqlite3_close(connection) == SQLITE_OK && ok;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool query_integer(const char *path, const char *sql, sqlite3_int64 expected) {
|
||||||
|
sqlite3 *connection = NULL;
|
||||||
|
sqlite3_stmt *statement = NULL;
|
||||||
|
if (sqlite3_open_v2(path, &connection, SQLITE_OPEN_READONLY, NULL) != SQLITE_OK ||
|
||||||
|
sqlite3_prepare_v2(connection, sql, -1, &statement, NULL) != SQLITE_OK) {
|
||||||
|
if (statement) {
|
||||||
|
(void)sqlite3_finalize(statement);
|
||||||
|
}
|
||||||
|
if (connection) {
|
||||||
|
(void)sqlite3_close(connection);
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
bool matches = sqlite3_step(statement) == SQLITE_ROW &&
|
||||||
|
sqlite3_column_int64(statement, 0) == expected &&
|
||||||
|
sqlite3_step(statement) == SQLITE_DONE;
|
||||||
|
return sqlite3_finalize(statement) == SQLITE_OK && sqlite3_close(connection) == SQLITE_OK &&
|
||||||
|
matches;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool run_test(void) {
|
||||||
|
char directory[] = "/tmp/lardon3d-project-db-XXXXXX";
|
||||||
|
CHECK(mkdtemp(directory));
|
||||||
|
char database_path[512], artifact_path[512], future_path[512], corrupt_path[512];
|
||||||
|
char legacy_path[512], failed_migration_path[512], v2_path[512];
|
||||||
|
char failed_v3_migration_path[512], v3_path[512], failed_v4_path[512];
|
||||||
|
char v4_path[512], failed_v5_path[512];
|
||||||
|
CHECK(snprintf(database_path, sizeof(database_path), "%s/project.db", directory) > 0);
|
||||||
|
CHECK(snprintf(artifact_path, sizeof(artifact_path), "%s/artifact.bin", directory) > 0);
|
||||||
|
CHECK(snprintf(future_path, sizeof(future_path), "%s/future.db", directory) > 0);
|
||||||
|
CHECK(snprintf(corrupt_path, sizeof(corrupt_path), "%s/corrupt.db", directory) > 0);
|
||||||
|
CHECK(snprintf(legacy_path, sizeof(legacy_path), "%s/legacy.db", directory) > 0);
|
||||||
|
CHECK(snprintf(failed_migration_path, sizeof(failed_migration_path), "%s/failed-migration.db",
|
||||||
|
directory) > 0);
|
||||||
|
CHECK(snprintf(v2_path, sizeof(v2_path), "%s/v2.db", directory) > 0);
|
||||||
|
CHECK(snprintf(failed_v3_migration_path, sizeof(failed_v3_migration_path),
|
||||||
|
"%s/failed-v3-migration.db", directory) > 0);
|
||||||
|
CHECK(snprintf(v3_path, sizeof(v3_path), "%s/v3.db", directory) > 0);
|
||||||
|
CHECK(snprintf(failed_v4_path, sizeof(failed_v4_path), "%s/failed-v4-migration.db", directory) >
|
||||||
|
0);
|
||||||
|
CHECK(snprintf(v4_path, sizeof(v4_path), "%s/v4.db", directory) > 0);
|
||||||
|
CHECK(snprintf(failed_v5_path, sizeof(failed_v5_path), "%s/failed-v5-migration.db", directory) >
|
||||||
|
0);
|
||||||
|
|
||||||
|
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY];
|
||||||
|
Lardon3DProjectDb *database = NULL;
|
||||||
|
CHECK(lardon3d_project_db_open(database_path, &database, error) == LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(database && lardon3d_project_db_schema_version(database) == 5);
|
||||||
|
bool legacy_pending = true;
|
||||||
|
CHECK(lardon3d_project_db_legacy_catalog_pending(database, &legacy_pending) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK &&
|
||||||
|
!legacy_pending);
|
||||||
|
|
||||||
|
Lardon3DProjectDbScanSet deleted_scanset;
|
||||||
|
CHECK(lardon3d_project_db_create_scanset(database, "Deleted identity", &deleted_scanset) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
unsigned char first_hash[LARDON3D_PROJECT_DB_SHA256_SIZE] = {1};
|
||||||
|
char first_asset_path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
||||||
|
asset_path_for_hash(first_hash, first_asset_path);
|
||||||
|
Lardon3DProjectDbImageRegisterStatus identity_status;
|
||||||
|
Lardon3DProjectDbImage deleted_image;
|
||||||
|
CHECK(lardon3d_project_db_register_image(database, deleted_scanset.scanset_id, first_hash,
|
||||||
|
first_asset_path, 1, "deleted.jpg",
|
||||||
|
"/source/deleted.jpg", 0, 1, &identity_status,
|
||||||
|
&deleted_image) == LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(lardon3d_project_db_test_delete_catalog_identity(
|
||||||
|
database, deleted_scanset.scanset_id, deleted_image.image_id, deleted_image.asset_id) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
Lardon3DProjectDbScanSet replacement_scanset;
|
||||||
|
CHECK(lardon3d_project_db_create_scanset(database, "Replacement identity",
|
||||||
|
&replacement_scanset) == LARDON3D_PROJECT_DB_OK &&
|
||||||
|
replacement_scanset.scanset_id > deleted_scanset.scanset_id);
|
||||||
|
unsigned char second_hash[LARDON3D_PROJECT_DB_SHA256_SIZE] = {2};
|
||||||
|
char second_asset_path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
||||||
|
asset_path_for_hash(second_hash, second_asset_path);
|
||||||
|
Lardon3DProjectDbImage replacement_image;
|
||||||
|
CHECK(lardon3d_project_db_register_image(database, replacement_scanset.scanset_id, second_hash,
|
||||||
|
second_asset_path, 1, "replacement.jpg",
|
||||||
|
"/source/replacement.jpg", 0, 2, &identity_status,
|
||||||
|
&replacement_image) == LARDON3D_PROJECT_DB_OK &&
|
||||||
|
replacement_image.image_id > deleted_image.image_id &&
|
||||||
|
replacement_image.asset_id > deleted_image.asset_id);
|
||||||
|
|
||||||
|
Lardon3DProjectDbProject project = {.created_at = 100, .updated_at = 100};
|
||||||
|
(void)snprintf(project.stable_id, sizeof(project.stable_id), "project-0001");
|
||||||
|
(void)snprintf(project.name, sizeof(project.name), "Projet test");
|
||||||
|
CHECK(lardon3d_project_db_set_project(database, &project) == LARDON3D_PROJECT_DB_OK);
|
||||||
|
Lardon3DProjectDbProject loaded_project;
|
||||||
|
CHECK(lardon3d_project_db_get_project(database, &loaded_project) == LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(strcmp(loaded_project.stable_id, project.stable_id) == 0);
|
||||||
|
project.updated_at = 101;
|
||||||
|
(void)snprintf(project.name, sizeof(project.name), "Projet renommé");
|
||||||
|
CHECK(lardon3d_project_db_set_project(database, &project) == LARDON3D_PROJECT_DB_OK);
|
||||||
|
Lardon3DProjectDbProject conflict = project;
|
||||||
|
(void)snprintf(conflict.stable_id, sizeof(conflict.stable_id), "other");
|
||||||
|
CHECK(lardon3d_project_db_set_project(database, &conflict) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||||
|
char copied_error[LARDON3D_PROJECT_DB_ERROR_CAPACITY];
|
||||||
|
CHECK(lardon3d_project_db_last_error(database, copied_error));
|
||||||
|
CHECK(copied_error[0] != '\0');
|
||||||
|
project = loaded_project;
|
||||||
|
memset(project.name, 'x', sizeof(project.name));
|
||||||
|
CHECK(lardon3d_project_db_set_project(database, &project) ==
|
||||||
|
LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||||
|
|
||||||
|
Lardon3DTaskDurableSnapshot running = task_snapshot(1, TASK_RUNNING);
|
||||||
|
Lardon3DProjectDbCheckpoint checkpoint = {
|
||||||
|
.format_version = 1, .durability = LARDON3D_DB_CHECKPOINT_DURABLE, .updated_at = 200};
|
||||||
|
(void)snprintf(checkpoint.path, sizeof(checkpoint.path), "%s/checkpoints/task-1.chk", directory);
|
||||||
|
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 200) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
running.progress = 30;
|
||||||
|
running.sequence_count = 3;
|
||||||
|
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 201) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
Lardon3DProjectDbScanSet import_scanset;
|
||||||
|
CHECK(lardon3d_project_db_create_scanset(database, "Legacy import", &import_scanset) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
Lardon3DProjectDbTask task;
|
||||||
|
CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(task.recovery_state == TASK_PENDING && task.progress == 30 && task.sequence_count == 3);
|
||||||
|
CHECK(task.has_task_kind && strcmp(task.task_kind, "test.work") == 0 &&
|
||||||
|
task.task_kind_version == 1);
|
||||||
|
CHECK(task.has_checkpoint && strcmp(task.checkpoint.path, checkpoint.path) == 0);
|
||||||
|
checkpoint.durability = LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE;
|
||||||
|
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 201) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(task.checkpoint.durability == LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE);
|
||||||
|
CHECK(lardon3d_project_db_record_task(database, &running, "test.other", 1, &checkpoint, 202) ==
|
||||||
|
LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||||
|
CHECK(lardon3d_project_db_record_task(database, &running, "Test.invalid", 1, &checkpoint, 202) ==
|
||||||
|
LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||||
|
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 0, &checkpoint, 202) ==
|
||||||
|
LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||||
|
CHECK(lardon3d_project_db_record_image_import_task(
|
||||||
|
database, &running, "import.images", 1, &checkpoint, "/tmp/source-a",
|
||||||
|
import_scanset.scanset_id, 202) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||||
|
|
||||||
|
Lardon3DTaskDurableSnapshot completed = task_snapshot(2, TASK_COMPLETED);
|
||||||
|
CHECK(lardon3d_project_db_record_task(database, &completed, "test.work", 1, NULL, 202) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
Lardon3DTaskDurableSnapshot no_checkpoint = task_snapshot(4, TASK_PENDING);
|
||||||
|
CHECK(lardon3d_project_db_record_task(database, &no_checkpoint, NULL, 0, NULL, 202) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
Lardon3DProjectDbTask page[1];
|
||||||
|
size_t count = 0;
|
||||||
|
CHECK(lardon3d_project_db_list_recoverable(database, 0, page, 1, &count) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(count == 1 && page[0].task_id == 1);
|
||||||
|
CHECK(lardon3d_project_db_list_recoverable(database, 1, page, 1, &count) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK &&
|
||||||
|
count == 0);
|
||||||
|
CHECK(lardon3d_project_db_list_recoverable(database, 0, page,
|
||||||
|
LARDON3D_PROJECT_DB_RECOVERY_PAGE_MAX + 1,
|
||||||
|
&count) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||||
|
|
||||||
|
Lardon3DTaskDurableSnapshot image_import = task_snapshot(5, TASK_PENDING);
|
||||||
|
CHECK(lardon3d_project_db_record_image_import_task(
|
||||||
|
database, &image_import, "import.images", 1, &checkpoint, "/tmp/source-a",
|
||||||
|
import_scanset.scanset_id, 202) == LARDON3D_PROJECT_DB_OK);
|
||||||
|
Lardon3DProjectDbImageImport import_parameters;
|
||||||
|
CHECK(lardon3d_project_db_load_image_import(database, 5, &import_parameters) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(import_parameters.task_id == 5 &&
|
||||||
|
strcmp(import_parameters.source_path, "/tmp/source-a") == 0 &&
|
||||||
|
import_parameters.scanset_id == import_scanset.scanset_id);
|
||||||
|
CHECK(lardon3d_project_db_record_image_import_task(
|
||||||
|
database, &image_import, "import.images", 1, &checkpoint, "/tmp/source-b",
|
||||||
|
import_scanset.scanset_id, 203) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||||
|
|
||||||
|
Lardon3DTaskDurableSnapshot rollback_task = task_snapshot(3, TASK_PENDING);
|
||||||
|
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_CHECKPOINT", "1", 1) == 0);
|
||||||
|
CHECK(lardon3d_project_db_record_task(database, &rollback_task, "test.work", 1, &checkpoint,
|
||||||
|
203) == LARDON3D_PROJECT_DB_IO_ERROR);
|
||||||
|
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_CHECKPOINT") == 0);
|
||||||
|
CHECK(lardon3d_project_db_load_task(database, 3, &task) == LARDON3D_PROJECT_DB_NOT_FOUND);
|
||||||
|
CHECK(lardon3d_project_db_test_orphan_checkpoint(database) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||||
|
|
||||||
|
Lardon3DProjectDbArtifact artifact = {.state = LARDON3D_DB_ARTIFACT_STAGED,
|
||||||
|
.has_producer_task = true,
|
||||||
|
.producer_task_id = 1,
|
||||||
|
.created_at = 300,
|
||||||
|
.updated_at = 300};
|
||||||
|
(void)snprintf(artifact.artifact_id, sizeof(artifact.artifact_id), "artifact-1");
|
||||||
|
(void)snprintf(artifact.kind, sizeof(artifact.kind), "generic-test");
|
||||||
|
(void)snprintf(artifact.path, sizeof(artifact.path), "%s", artifact_path);
|
||||||
|
CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_OK);
|
||||||
|
Lardon3DProjectDbArtifact loaded_artifact;
|
||||||
|
CHECK(lardon3d_project_db_load_artifact(database, artifact.artifact_id, &loaded_artifact) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(loaded_artifact.state == LARDON3D_DB_ARTIFACT_STAGED);
|
||||||
|
CHECK(lardon3d_project_db_mark_artifact_ready(database, artifact.artifact_id, 301) ==
|
||||||
|
LARDON3D_PROJECT_DB_IO_ERROR);
|
||||||
|
int descriptor = open(artifact_path, O_WRONLY | O_CREAT | O_TRUNC, 0600);
|
||||||
|
CHECK(descriptor >= 0);
|
||||||
|
const char payload[] = "validated";
|
||||||
|
CHECK(write(descriptor, payload, sizeof(payload)) == (ssize_t)sizeof(payload));
|
||||||
|
CHECK(close(descriptor) == 0);
|
||||||
|
CHECK(lardon3d_project_db_mark_artifact_ready(database, artifact.artifact_id, 301) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(lardon3d_project_db_load_artifact(database, artifact.artifact_id, &loaded_artifact) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(loaded_artifact.state == LARDON3D_DB_ARTIFACT_READY &&
|
||||||
|
loaded_artifact.size_bytes == sizeof(payload));
|
||||||
|
CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||||
|
artifact.producer_task_id = 999;
|
||||||
|
(void)snprintf(artifact.artifact_id, sizeof(artifact.artifact_id), "orphan-artifact");
|
||||||
|
CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||||
|
artifact = loaded_artifact;
|
||||||
|
artifact.state = LARDON3D_DB_ARTIFACT_STAGED;
|
||||||
|
artifact.size_bytes = 0;
|
||||||
|
memset(artifact.path, 'x', sizeof(artifact.path));
|
||||||
|
CHECK(lardon3d_project_db_create_artifact(database, &artifact) ==
|
||||||
|
LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||||
|
|
||||||
|
ThreadContext contexts[2] = {{.database = database}, {.database = database}};
|
||||||
|
pthread_t threads[2];
|
||||||
|
CHECK(pthread_create(&threads[0], NULL, read_thread, &contexts[0]) == 0);
|
||||||
|
CHECK(pthread_create(&threads[1], NULL, read_thread, &contexts[1]) == 0);
|
||||||
|
CHECK(pthread_join(threads[0], NULL) == 0 && pthread_join(threads[1], NULL) == 0);
|
||||||
|
CHECK(contexts[0].success && contexts[1].success);
|
||||||
|
IdThreadContext id_contexts[2] = {{.database = database}, {.database = database}};
|
||||||
|
CHECK(pthread_create(&threads[0], NULL, allocate_id_thread, &id_contexts[0]) == 0);
|
||||||
|
CHECK(pthread_create(&threads[1], NULL, allocate_id_thread, &id_contexts[1]) == 0);
|
||||||
|
CHECK(pthread_join(threads[0], NULL) == 0 && pthread_join(threads[1], NULL) == 0);
|
||||||
|
CHECK(id_contexts[0].success && id_contexts[1].success &&
|
||||||
|
id_contexts[0].id != id_contexts[1].id && id_contexts[0].id > 5 && id_contexts[1].id > 5);
|
||||||
|
|
||||||
|
char too_long[LARDON3D_PROJECT_DB_PATH_CAPACITY + 1];
|
||||||
|
memset(too_long, 'x', sizeof(too_long));
|
||||||
|
too_long[sizeof(too_long) - 1] = '\0';
|
||||||
|
CHECK(lardon3d_project_db_open(too_long, &database, error) ==
|
||||||
|
LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||||
|
lardon3d_project_db_close(contexts[0].database);
|
||||||
|
database = NULL;
|
||||||
|
CHECK(query_integer(database_path, "SELECT value FROM metadata WHERE key='schema_version'", 5));
|
||||||
|
CHECK(query_integer(database_path, "SELECT count(*) FROM tasks WHERE task_id=1", 1));
|
||||||
|
CHECK(lardon3d_project_db_open(database_path, &database, error) == LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(lardon3d_project_db_load_artifact(database, "artifact-1", &loaded_artifact) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
lardon3d_project_db_close(database);
|
||||||
|
|
||||||
|
CHECK(create_future_database(future_path));
|
||||||
|
CHECK(lardon3d_project_db_open(future_path, &database, error) ==
|
||||||
|
LARDON3D_PROJECT_DB_UNSUPPORTED_SCHEMA);
|
||||||
|
descriptor = open(corrupt_path, O_WRONLY | O_CREAT | O_TRUNC, 0600);
|
||||||
|
CHECK(descriptor >= 0);
|
||||||
|
CHECK(write(descriptor, "not sqlite", 10) == 10 && close(descriptor) == 0);
|
||||||
|
CHECK(lardon3d_project_db_open(corrupt_path, &database, error) == LARDON3D_PROJECT_DB_CORRUPT);
|
||||||
|
|
||||||
|
CHECK(create_v1_database(legacy_path));
|
||||||
|
CHECK(lardon3d_project_db_open(legacy_path, &database, error) == LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(lardon3d_project_db_schema_version(database) == 5);
|
||||||
|
CHECK(lardon3d_project_db_get_project(database, &loaded_project) == LARDON3D_PROJECT_DB_OK &&
|
||||||
|
strcmp(loaded_project.stable_id, "legacy-project") == 0);
|
||||||
|
CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(!task.has_task_kind && task.has_checkpoint &&
|
||||||
|
strcmp(task.checkpoint.path, "legacy.chk") == 0);
|
||||||
|
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact", &loaded_artifact) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
lardon3d_project_db_close(database);
|
||||||
|
database = NULL;
|
||||||
|
CHECK(query_integer(legacy_path, "SELECT value FROM metadata WHERE key='schema_version'", 5));
|
||||||
|
|
||||||
|
CHECK(create_v1_database(failed_migration_path));
|
||||||
|
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V2", "1", 1) == 0);
|
||||||
|
CHECK(lardon3d_project_db_open(failed_migration_path, &database, error) ==
|
||||||
|
LARDON3D_PROJECT_DB_IO_ERROR);
|
||||||
|
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V2") == 0);
|
||||||
|
CHECK(query_integer(failed_migration_path,
|
||||||
|
"SELECT value FROM metadata WHERE key='schema_version'", 1));
|
||||||
|
CHECK(lardon3d_project_db_open(failed_migration_path, &database, error) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK &&
|
||||||
|
!task.has_task_kind);
|
||||||
|
lardon3d_project_db_close(database);
|
||||||
|
database = NULL;
|
||||||
|
|
||||||
|
CHECK(create_v2_database(v2_path));
|
||||||
|
CHECK(lardon3d_project_db_open(v2_path, &database, error) == LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK &&
|
||||||
|
task.has_task_kind && strcmp(task.task_kind, "test.work") == 0);
|
||||||
|
CHECK(task.has_checkpoint && strcmp(task.checkpoint.path, "legacy.chk") == 0);
|
||||||
|
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact", &loaded_artifact) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
lardon3d_project_db_close(database);
|
||||||
|
database = NULL;
|
||||||
|
CHECK(query_integer(v2_path, "SELECT value FROM metadata WHERE key='schema_version'", 5));
|
||||||
|
|
||||||
|
CHECK(create_v2_database(failed_v3_migration_path));
|
||||||
|
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V3", "1", 1) == 0);
|
||||||
|
CHECK(lardon3d_project_db_open(failed_v3_migration_path, &database, error) ==
|
||||||
|
LARDON3D_PROJECT_DB_IO_ERROR);
|
||||||
|
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V3") == 0);
|
||||||
|
CHECK(query_integer(failed_v3_migration_path,
|
||||||
|
"SELECT value FROM metadata WHERE key='schema_version'", 2));
|
||||||
|
CHECK(query_integer(
|
||||||
|
failed_v3_migration_path,
|
||||||
|
"SELECT count(*) FROM sqlite_master WHERE type='table' AND name='image_import_tasks'", 0));
|
||||||
|
|
||||||
|
CHECK(create_v3_database(v3_path));
|
||||||
|
CHECK(lardon3d_project_db_open(v3_path, &database, error) == LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(lardon3d_project_db_load_image_import(database, 9, &import_parameters) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(strcmp(import_parameters.source_path, "/legacy/source") == 0 &&
|
||||||
|
import_parameters.scanset_id > 0);
|
||||||
|
CHECK(lardon3d_project_db_legacy_catalog_pending(database, &legacy_pending) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK &&
|
||||||
|
legacy_pending);
|
||||||
|
CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK &&
|
||||||
|
task.has_task_kind);
|
||||||
|
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact", &loaded_artifact) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
lardon3d_project_db_close(database);
|
||||||
|
database = NULL;
|
||||||
|
CHECK(query_integer(v3_path, "SELECT value FROM metadata WHERE key='schema_version'", 5));
|
||||||
|
|
||||||
|
CHECK(create_v3_database(failed_v4_path));
|
||||||
|
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V4", "1", 1) == 0);
|
||||||
|
CHECK(lardon3d_project_db_open(failed_v4_path, &database, error) == LARDON3D_PROJECT_DB_IO_ERROR);
|
||||||
|
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V4") == 0);
|
||||||
|
CHECK(query_integer(failed_v4_path, "SELECT value FROM metadata WHERE key='schema_version'", 3));
|
||||||
|
CHECK(query_integer(failed_v4_path,
|
||||||
|
"SELECT count(*) FROM sqlite_master WHERE type='table' AND name='scansets'",
|
||||||
|
0));
|
||||||
|
|
||||||
|
CHECK(create_v4_database(v4_path));
|
||||||
|
Lardon3DProjectDbResult v4_result = lardon3d_project_db_open(v4_path, &database, error);
|
||||||
|
if (v4_result != LARDON3D_PROJECT_DB_OK) {
|
||||||
|
fprintf(stderr, "Migration v4 (%d): %s\n", (int)v4_result, error);
|
||||||
|
}
|
||||||
|
CHECK(v4_result == LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(lardon3d_project_db_schema_version(database) == 5);
|
||||||
|
CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK);
|
||||||
|
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact", &loaded_artifact) ==
|
||||||
|
LARDON3D_PROJECT_DB_OK);
|
||||||
|
lardon3d_project_db_close(database);
|
||||||
|
database = NULL;
|
||||||
|
|
||||||
|
CHECK(create_v4_database(failed_v5_path));
|
||||||
|
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V5", "1", 1) == 0);
|
||||||
|
CHECK(lardon3d_project_db_open(failed_v5_path, &database, error) == LARDON3D_PROJECT_DB_IO_ERROR);
|
||||||
|
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V5") == 0);
|
||||||
|
CHECK(query_integer(failed_v5_path, "SELECT value FROM metadata WHERE key='schema_version'", 4));
|
||||||
|
CHECK(query_integer(
|
||||||
|
failed_v5_path,
|
||||||
|
"SELECT count(*) FROM sqlite_master WHERE type='table' AND name='feature_sets'", 0));
|
||||||
|
|
||||||
|
CHECK(unlink(artifact_path) == 0);
|
||||||
|
CHECK(unlink(database_path) == 0);
|
||||||
|
CHECK(unlink(future_path) == 0);
|
||||||
|
CHECK(unlink(corrupt_path) == 0);
|
||||||
|
CHECK(unlink(legacy_path) == 0);
|
||||||
|
CHECK(unlink(failed_migration_path) == 0);
|
||||||
|
CHECK(unlink(v2_path) == 0);
|
||||||
|
CHECK(unlink(failed_v3_migration_path) == 0);
|
||||||
|
CHECK(unlink(v3_path) == 0);
|
||||||
|
CHECK(unlink(failed_v4_path) == 0);
|
||||||
|
CHECK(unlink(v4_path) == 0);
|
||||||
|
CHECK(unlink(failed_v5_path) == 0);
|
||||||
|
CHECK(rmdir(directory) == 0);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
int main(void) {
|
||||||
|
return run_test() ? EXIT_SUCCESS : EXIT_FAILURE;
|
||||||
|
}
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue