feat: finalize feature store v1
This commit is contained in:
parent
0f99415317
commit
b12af521e5
32 changed files with 5686 additions and 2973 deletions
|
|
@ -23,10 +23,11 @@ Lardon3D ne vise pas simplement "dossier de photos → objet 3D", mais "ensemble
|
|||
- **Project** : cycle de vie persistant, identité stable et Project Database ouverte
|
||||
- **Import** : premier task kind de production, exécuté par la file générique en lots bornés et reprenables
|
||||
- **ScanSet / Image Catalog v1** : acquisitions, images logiques, provenance et assets SHA-256 persistants et paginés
|
||||
- **Feature Store v1** : extraction ORB réelle, assets binaires versionnés et lecture bornée
|
||||
- **Image View** : vues triées et filtrées pour la TUI
|
||||
- **Task** : moteur de tâches avec pause/reprise, annulation et séquences
|
||||
- **Task Checkpoint v1** : snapshot durable, fichier atomique et reprise sûre
|
||||
- **Project Database v4** : tâches, ScanSets, catalogue image, assets et migrations SQLite
|
||||
- **Project Database v5** : tâches, ScanSets, images, Feature Sets/assets et migrations SQLite
|
||||
- **Task Kind Registry** : identité métier durable et reconstruction runtime explicite
|
||||
- **Recovery projet** : reprise automatique sélective et bornée des imports récupérables
|
||||
- **Task Queue** : file FIFO avec sélection adaptative et backpressure
|
||||
|
|
@ -102,6 +103,7 @@ Acquisitions
|
|||
- [Pipeline de reconstruction](docs/architecture/reconstruction_pipeline.md)
|
||||
- [Persistance](docs/architecture/persistence.md)
|
||||
- [Base de données projet](docs/architecture/project_database.md)
|
||||
- [Feature Store](docs/architecture/feature_store.md)
|
||||
- [Viewer](docs/architecture/viewer.md)
|
||||
- [Revue des fondations](docs/architecture/foundation_review.md)
|
||||
|
||||
|
|
|
|||
117
docs/architecture/feature_store.md
Normal file
117
docs/architecture/feature_store.md
Normal file
|
|
@ -0,0 +1,117 @@
|
|||
# Feature Store v1
|
||||
|
||||
## Rôle et modèle
|
||||
|
||||
**IMPLEMENTED.** Le Feature Store est la mémoire visuelle locale persistante.
|
||||
Un `FeatureSet` logique appartient à une `image_id` et identifie exactement
|
||||
`orb` version 1 plus une configuration canonique. Il référence un
|
||||
`FeatureAsset` physique immutable. Deux images logiques portant le même contenu
|
||||
et la même configuration gardent deux `feature_set_id`, mais partagent le même
|
||||
asset lorsque leurs octets sont identiques. Les futurs matches référenceront
|
||||
`feature_set_id + feature_index`; ces deux valeurs sont immuables après
|
||||
publication.
|
||||
|
||||
`feature_set_id` et `feature_asset_id` sont des identités SQLite
|
||||
`AUTOINCREMENT`. Une identité validée n'est jamais réattribuée. SQLite ne
|
||||
contient que les relations, empreintes, tailles et paramètres ; les tableaux de
|
||||
points et descripteurs restent hors DB.
|
||||
|
||||
## Extracteur et configuration
|
||||
|
||||
L'extracteur production est ORB d'OpenCV 5, derrière une façade C. Aucune
|
||||
exception C++ ni `cv::Mat` ne traverse cette frontière. La registry statique
|
||||
expose `features.extract`, version 1. Une tâche traite exactement une image.
|
||||
|
||||
La configuration v1 contient trois `uint32` : `max_features` (1..8192),
|
||||
`pyramid_levels` (1..16) et `fast_threshold` (1..255). Son fingerprint SHA-256
|
||||
porte sur les 24 octets canoniques `L3DORBP1`, version et trois entiers
|
||||
little-endian. Il ne dépend ni du padding, ni de la locale. Le contrat ORB v1
|
||||
produit 32 octets binaires par point.
|
||||
|
||||
OpenCV n'est pas configuré via un état global par Lardon3D. Le scheduler actuel
|
||||
n'a qu'un worker ; OpenCV peut néanmoins employer son backend parallèle
|
||||
interne. Le contrôle fin de ce parallélisme devra précéder les futurs pools de
|
||||
workers.
|
||||
|
||||
La limite de 100 000 000 pixels est vérifiée après `cv::imread` : l'API utilisée
|
||||
ne fournit pas de sonde de dimensions multi-format fiable sans décodage. Le pic
|
||||
mémoire du décodage peut donc précéder le rejet. Lardon3D ne revendique pas de
|
||||
bornage pré-décodage et n'embarque pas un parseur JPEG/PNG parallèle.
|
||||
|
||||
## Feature File v1
|
||||
|
||||
Le fichier est little-endian et exige IEEE-754 binary32. Sa taille maximale est
|
||||
16 Mio et son nombre maximal de points 8192.
|
||||
|
||||
| Offset | Taille | Champ |
|
||||
|---:|---:|---|
|
||||
| 0 | 8 | magic `L3DFEAT\0` |
|
||||
| 8 | 4 | format version = 1 |
|
||||
| 12 | 4 | header size = 160 |
|
||||
| 16 | 4 | feature count |
|
||||
| 20 | 4 | descriptor dimension = 32 |
|
||||
| 24 | 4 | descriptor type = U8 |
|
||||
| 28 | 4 | keypoint record size = 24 |
|
||||
| 32,36 | 4+4 | largeur, hauteur décodées |
|
||||
| 40,48,56 | 8+8+8 | offsets keypoints/descriptors, taille totale |
|
||||
| 64 | 32 | SHA-256 de l'asset image source |
|
||||
| 96 | 32 | fingerprint paramètres |
|
||||
| 128 | 16 | `orb\0` puis zéros réservés |
|
||||
| 144 | 4 | extractor version = 1 |
|
||||
| 148 | 12 | réservés, zéro obligatoire |
|
||||
|
||||
Chaque keypoint contient six mots de 32 bits : `x`, `y`, `size`, orientation,
|
||||
response en binary32, puis octave signé. `x/y` sont en pixels de l'image telle
|
||||
que décodée par OpenCV, origine en haut à gauche. `size` est le diamètre du
|
||||
voisinage en pixels. L'orientation est en degrés dans `[0,360)`. L'entrée `i`
|
||||
du bloc keypoints correspond exactement aux 32 octets du descriptor `i`.
|
||||
|
||||
Le validateur contrôle magic, version, réservés, type, dimension, bornes,
|
||||
offsets, taille exacte, multiplications, SHA-256 externe et cohérence DB. Une
|
||||
version future est distinguée d'une corruption lorsque le fichier et son record
|
||||
DB sont cohérents. Le reader utilise `pread`, accepte au plus 256 éléments par
|
||||
appel et ne charge jamais le fichier entier.
|
||||
|
||||
Le chemin DB est validé sous sa forme canonique exacte dérivée du SHA-256 avant
|
||||
toute ouverture. Une absence retourne `NOT_FOUND`; troncature, hash divergent,
|
||||
header ou métadonnées DB divergents retournent `CORRUPT`. Une version future
|
||||
retourne distinctement `UNSUPPORTED_VERSION`.
|
||||
|
||||
## Publication et persistance
|
||||
|
||||
Layout : `assets/features/<2 hex>/<sha256 complet lowercase>`. Le SHA-256 porte
|
||||
sur le Feature File complet. Le protocole est : temporaire local, écriture,
|
||||
`fsync`, hash, `link` atomique sans écrasement, validation complète lors d'une
|
||||
adoption concurrente, `fsync` du répertoire, puis transaction SQLite. Un échec
|
||||
du dernier `fsync` est enregistré `PUBLISHED_NOT_DURABLE`. Un échec SQLite après
|
||||
publication laisse un fichier orphelin et aucune ligne logique partielle.
|
||||
Un nouvel essai qui revalide l'asset et réussit le `fsync` promeut explicitement
|
||||
sa durabilité vers `DURABLE` dans la transaction DB.
|
||||
|
||||
Le schéma v5 sépare `feature_assets`, `feature_sets` et
|
||||
`feature_extract_tasks`. L'unicité logique porte sur image, kind, version et
|
||||
fingerprint. Une tâche est persistée avant enqueue, passe par la queue et le
|
||||
Governor (CPU, IO, 576 Mio conservateurs, lot 1), puis checkpointée initialement
|
||||
et terminalement. La reprise recommence l'image entière : il n'existe pas de
|
||||
fausse reprise intra-ORB. La publication est idempotente.
|
||||
|
||||
Pause et annulation sont coopératives avant/après l'appel ORB ; cet appel n'est
|
||||
pas interruptible. L'image gérée est rehashée avant extraction. Une image
|
||||
uniforme produit légitimement un Feature Set READY vide.
|
||||
|
||||
Lardon3D ne garantit pas des octets ORB identiques entre versions d'OpenCV,
|
||||
plateformes ou backends. L'idempotence porte sur l'environnement courant et le
|
||||
contrat `extractor_version`; une évolution qui change durablement la sémantique
|
||||
ou les octets exige d'auditer et, si nécessaire, d'incrémenter cette version.
|
||||
|
||||
## Statut
|
||||
|
||||
**IMPLEMENTED** — ORB réel, format v1, assets content-addressed, DB v5,
|
||||
publication atomique, reader borné, task kind production et reprise automatique.
|
||||
|
||||
**NOT_YET_WIRED** — commande de réconciliation/scrub des orphelins, contrôle fin
|
||||
du backend parallèle OpenCV, orientation EXIF, lancement automatique de
|
||||
l'extraction après import et planification multi-image/DAG.
|
||||
|
||||
**PLANNED** — Visual Index, paires candidates, matching, vérification
|
||||
géométrique, tracks et SfM.
|
||||
|
|
@ -62,6 +62,12 @@ catalogue, le manifeste ou les images.
|
|||
|
||||
**Statut :** IMPLEMENTED
|
||||
|
||||
### Feature Store
|
||||
Extraction ORB réelle par tâche persistante, Feature Sets logiques et assets
|
||||
binaires content-addressed lisibles par plages bornées.
|
||||
|
||||
**Statut :** IMPLEMENTED
|
||||
|
||||
### Task
|
||||
Moteur de tâches avec états, progression, pause/reprise coopérative,
|
||||
annulation, checkpoints et estimations de ressources.
|
||||
|
|
|
|||
|
|
@ -108,7 +108,7 @@ par le `time_t` local avant conversion. Le format reste donc lisible entre
|
|||
plateformes uniquement pour les valeurs communes à leurs domaines `size_t` et
|
||||
`time_t`.
|
||||
|
||||
## Project Database v4
|
||||
## Project Database v5
|
||||
|
||||
SQLite contient l'état logique interrogable et les références aux fichiers ;
|
||||
les checkpoints et artefacts volumineux restent externes. L'enregistrement du
|
||||
|
|
@ -147,8 +147,8 @@ code.
|
|||
**IMPLEMENTED** — modèle durable, codec v1, lecture validée, publication
|
||||
atomique et restauration sûre d'une tâche isolée.
|
||||
|
||||
**IMPLEMENTED** — Project Database v4 pour identité, tâches typées, ScanSets,
|
||||
images logiques, assets SHA-256, références checkpoint et artefacts génériques.
|
||||
**IMPLEMENTED** — Project Database v5 pour identité, tâches typées, ScanSets,
|
||||
images logiques, Feature Sets/assets SHA-256, checkpoints et artefacts génériques.
|
||||
|
||||
**IMPLEMENTED** — registry statique bornée et reconstruction explicite avec
|
||||
ownership du userdata.
|
||||
|
|
@ -205,5 +205,8 @@ retry piloté par l'utilisateur pour les sources indisponibles.
|
|||
**NOT_YET_WIRED** — migration de la vue TUI en mémoire vers la pagination
|
||||
SQLite, scrub des assets et réconciliation globale des orphelins.
|
||||
|
||||
**PLANNED** — Feature Store, Visual Index, migrations v5+ et reprise avec
|
||||
**IMPLEMENTED** — Feature Store externe versionné, immutable, borné et relié
|
||||
transactionnellement à ses métadonnées SQLite après publication.
|
||||
|
||||
**PLANNED** — Visual Index, migrations v6+ et reprise avec
|
||||
dépendances.
|
||||
|
|
|
|||
|
|
@ -168,9 +168,9 @@ référence vers le fichier checkpoint. Le fichier checkpoint validé reste la
|
|||
source complète pour `lardon3d_task_restore()` ; la DB seule ne reconstruit
|
||||
jamais une tâche. Un écart ou un fichier invalide interdit la reprise.
|
||||
|
||||
## Schéma v4 implémenté
|
||||
## Schéma v5 implémenté
|
||||
|
||||
- `metadata(key PRIMARY KEY, value)` contient `schema_version=4` et
|
||||
- `metadata(key PRIMARY KEY, value)` contient `schema_version=5` et
|
||||
`next_task_id`, prochain ID durable allouable.
|
||||
- `project(singleton=1, stable_id UNIQUE, name, created_at, updated_at)` décrit
|
||||
l'unique identité logique de la DB.
|
||||
|
|
@ -209,12 +209,12 @@ SHA-256 et le chemin asset sont déjà indexés par leurs contraintes `UNIQUE`.
|
|||
|
||||
## Ouverture et migrations
|
||||
|
||||
Une DB vide reçoit directement le schéma v4 dans une transaction
|
||||
Une DB vide reçoit directement le schéma v5 dans une transaction
|
||||
`BEGIN IMMEDIATE`. Une DB v1 reçoit transactionnellement les colonnes nullable
|
||||
`task_kind` et `task_kind_version`, puis les migrations v2→v3. Les anciennes lignes restent
|
||||
`NULL/NULL`, sans type inventé et sans perte des projets, tâches, checkpoints ou
|
||||
artefacts. Une interruption ou erreur provoque un rollback complet. Les DB v1,
|
||||
v2 et v3 sont migrées séquentiellement vers v4. Une version future est refusée et une DB contenant
|
||||
v2, v3 et v4 sont migrées séquentiellement vers v5. Une version future est refusée et une DB contenant
|
||||
des tables sans métadonnée de version est considérée corrompue. La fonction
|
||||
interne de migration ne connaît que `0 → 4`, `1 → 2 → 3 → 4`,
|
||||
`2 → 3 → 4` et `3 → 4`.
|
||||
|
|
@ -293,7 +293,7 @@ UPDATE metadata SET value=4
|
|||
WHERE key='schema_version' AND value=3;
|
||||
```
|
||||
|
||||
Configuration v4 : `foreign_keys=ON`, `journal_mode=DELETE`,
|
||||
Configuration v5 : `foreign_keys=ON`, `journal_mode=DELETE`,
|
||||
`synchronous=FULL`, `busy_timeout=5000`. Le mode DELETE convient au propriétaire
|
||||
unique actuel, évite les fichiers WAL/SHM durables et conserve la synchronisation
|
||||
forte. Le timeout borne l'attente d'un verrou externe à cinq secondes.
|
||||
|
|
@ -354,7 +354,7 @@ ouvert.
|
|||
|
||||
## Statut
|
||||
|
||||
**IMPLEMENTED** — SQLite système, schéma v4 et migrations v1→v2→v3→v4, identité
|
||||
**IMPLEMENTED** — SQLite système, schéma v5 et migrations v1→v2→v3→v4→v5, identité
|
||||
projet, transactions tâche+checkpoint, pagination de reprise et artefacts
|
||||
génériques.
|
||||
|
||||
|
|
@ -382,7 +382,7 @@ cataloguées », pas « images migrées ».
|
|||
|
||||
**NOT_YET_WIRED** — autosave à toutes les transitions, retry UI des sources
|
||||
indisponibles, migration de la TUI legacy et réconciliation des fichiers
|
||||
orphelins, Feature Store et Visual Index.
|
||||
orphelins et Visual Index. Le Feature Store v1 est implémenté.
|
||||
|
||||
**PLANNED** — migrations v5+, dépendances d'artefacts, graphe géométrique et
|
||||
reconstruction incrémentale.
|
||||
|
|
|
|||
|
|
@ -42,15 +42,16 @@ content-addressed. Les états Feature/Matching/Reconstruction restent planifiés
|
|||
|
||||
---
|
||||
|
||||
### C. Feature Store
|
||||
### C. Feature Extraction et Feature Store
|
||||
|
||||
| Aspect | Description |
|
||||
|--------|-------------|
|
||||
| **Index / métadonnées persistants** | Les descripteurs de features (points clés, descripteurs, orientations) sont persistés sur disque dans un format binaire compact. Le rechargement évite la re-extraction. |
|
||||
| **Données numériques massives** | Descripteurs floats, coordonnées de keypoints : volumes potentiellement importants. Doivent être stockés de manière séquentielle et indexée. |
|
||||
| **Formats adaptés et bornés** | Format binaire avec en-tête (version, nombre de features, dimensions). Borné par le budget RAM du governor : si le store dépasse la capacité, seuls les N plus récents sont en mémoire. |
|
||||
| **Extraction** | ORB OpenCV réel, une image par tâche `features.extract`. |
|
||||
| **Métadonnées persistantes** | `FeatureSet` logique et `FeatureAsset` physique content-addressed dans ProjectDb v5. |
|
||||
| **Données numériques massives** | Keypoints et descripteurs U8×32 restent hors SQLite dans Feature File v1. |
|
||||
| **Lecture bornée** | Le reader lit au plus 256 features par plage et ne charge pas tout le fichier. |
|
||||
|
||||
**Statut :** PLANNED — aucune implémentation existante.
|
||||
**Statut :** IMPLEMENTED v1 — extraction, publication atomique, reprise et reader borné.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -212,7 +213,11 @@ Image Catalog (B) ──► Feature Store (C)
|
|||
|
||||
---
|
||||
|
||||
## Statut : PLANNED (vision architecturale)
|
||||
## Statut du pipeline
|
||||
|
||||
Import, Image Catalog, Feature Extraction et Feature Store sont
|
||||
**IMPLEMENTED**. Visual Index, Candidate Pair, Matching, Tracks et SfM sont
|
||||
**PLANNED**.
|
||||
|
||||
Ce document décrit la vision architecturale cible du pipeline de
|
||||
reconstruction. Les modules listés ici ne sont pas tous implémentés.
|
||||
|
|
|
|||
|
|
@ -73,6 +73,11 @@ Le Resource Governor est l'unique propriétaire des budgets (RAM, GPU, CPU, IO).
|
|||
l'échantillon peut conserver taille/durée mais n'alimente jamais l'adaptation
|
||||
mémoire.
|
||||
- Pas de communication inter-classes de tâches
|
||||
- `features.extract` réserve un lot de 1, un thread CPU et un slot I/O, avec
|
||||
64 Mio fixes et 512 Mio par image. Cette estimation conservatrice couvre le
|
||||
chemin actuel sans prétendre mesurer les allocations internes d'OpenCV.
|
||||
`record_batch` couvre la validation source, le décodage, ORB, la publication
|
||||
et la finalisation DB ; `peak_memory_bytes == 0` signifie « mesure inconnue ».
|
||||
|
||||
## Limites actuelles
|
||||
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ fin de l'exécution. Le constructeur métier n'est jamais appelé sous mutex DB.
|
|||
|
||||
## Persistance et legacy
|
||||
|
||||
Le checkpoint générique reste en version 1. Project Database v4 conserve le
|
||||
Le checkpoint générique reste en version 1. Project Database v5 conserve le
|
||||
kind/version ; les lignes migrées depuis v1 restent `NULL/NULL` et sont classées
|
||||
`LEGACY_UNTYPED`. Un kind inconnu ou une version non supportée reste inspectable
|
||||
mais inexécutable. Aucun type n'est inventé et aucun code n'est sélectionné par
|
||||
|
|
@ -46,5 +46,8 @@ pour restaurer hors mutex DB et transférer chaque tâche acceptée à la queue.
|
|||
|
||||
**NOT_YET_WIRED** — autosave complet et réconciliation orpheline.
|
||||
|
||||
**PLANNED** — kinds des tâches ScanSet, Image Catalog, Feature Store, Visual
|
||||
**IMPLEMENTED** — `features.extract` version 1 reconstruit une extraction ORB
|
||||
depuis `image_id` et ses paramètres bornés.
|
||||
|
||||
**PLANNED** — kinds des tâches Visual
|
||||
Index et reconstruction lorsque ces traitements existeront réellement.
|
||||
|
|
|
|||
|
|
@ -106,7 +106,7 @@ localement : son userdata est détruit, sans callback terminal ni écriture
|
|||
durable d'une fausse annulation. Une annulation explicitement demandée conserve
|
||||
le contrat de notification terminale.
|
||||
|
||||
La Project Database v4 peut enregistrer transactionnellement un résumé
|
||||
La Project Database v5 peut enregistrer transactionnellement un résumé
|
||||
`Lardon3DTaskDurableSnapshot` et la référence de son checkpoint. Elle ne stocke
|
||||
ni estimation sérialisée complète, ni callback, ni réservation, et ne remplace
|
||||
pas la validation du fichier checkpoint avant `task_restore()`.
|
||||
|
|
|
|||
|
|
@ -33,6 +33,11 @@ au ScanSet, résultats du Visual Index et provenance. Une proximité temporelle
|
|||
pourra servir de signal secondaire ; la proximité dans un dossier et le nom de
|
||||
fichier ne constituent jamais l'identité principale.
|
||||
|
||||
Une observation de feature utilisera la référence durable
|
||||
`feature_set_id + feature_index`. Une fois le Feature Set READY,
|
||||
`feature_index` désigne immuablement la paire keypoint/descripteur de même
|
||||
indice ; aucun chemin source n'entre dans cette identité.
|
||||
|
||||
## Concepts clés
|
||||
|
||||
### Étapes du matching
|
||||
|
|
|
|||
|
|
@ -26,10 +26,10 @@ Matching & Tracks (appariement détaillé)
|
|||
|
||||
L'index visuel se situe entre l'extraction de features et la sélection de paires candidates. Il réduit drastiquement l'espace de recherche pour l'appariement.
|
||||
|
||||
Le futur index référencera exclusivement les `image_id` stables du catalogue
|
||||
persistant. Un chemin d'asset ou un nom source n'est ni une identité d'image ni
|
||||
une clé d'index durable. Le partage d'un asset entre deux ScanSets n'empêche pas
|
||||
leurs deux images logiques d'être des observations distinctes.
|
||||
Le futur index consommera exclusivement un `feature_set_id` et les descripteurs
|
||||
obtenus par plages bornées du Feature Reader. Il ne dépendra ni des chemins
|
||||
source, ni d'OpenCV, ni du runtime de la tâche productrice. Un chemin d'asset ou
|
||||
un nom source n'est pas une clé d'index durable.
|
||||
|
||||
## Concepts clés
|
||||
|
||||
|
|
|
|||
|
|
@ -57,9 +57,8 @@ git diff --check
|
|||
allocations :
|
||||
|
||||
```sh
|
||||
meson setup build-asan --wipe \
|
||||
-Db_sanitize=address,undefined \
|
||||
-Db_static=false
|
||||
CC=clang meson setup build-asan --wipe \
|
||||
-Db_sanitize=address,undefined
|
||||
meson compile -C build-asan -j8
|
||||
meson test -C build-asan --print-errorlogs
|
||||
```
|
||||
|
|
@ -70,9 +69,9 @@ meson test -C build-asan --print-errorlogs
|
|||
variables de condition, états partagés) :
|
||||
|
||||
```sh
|
||||
meson setup build-tsan --wipe \
|
||||
CC=clang meson setup build-tsan --wipe \
|
||||
-Db_sanitize=thread \
|
||||
-Db_static=false
|
||||
-Db_lundef=false
|
||||
meson compile -C build-tsan -j8
|
||||
meson test -C build-tsan --print-errorlogs
|
||||
```
|
||||
|
|
|
|||
|
|
@ -160,7 +160,7 @@ Pour tout ticket touchant la concurrence, exécuter :
|
|||
|
||||
```sh
|
||||
# Build TSan
|
||||
meson setup build-tsan --wipe -Db_sanitize=thread -Db_static=false
|
||||
CC=clang meson setup build-tsan --wipe -Db_sanitize=thread -Db_lundef=false
|
||||
meson compile -C build-tsan -j8
|
||||
meson test -C build-tsan --print-errorlogs
|
||||
```
|
||||
|
|
|
|||
|
|
@ -95,12 +95,12 @@ meson test -C build --print-errorlogs
|
|||
git diff --check
|
||||
|
||||
# 4. Si mémoire/concurrence touchés
|
||||
meson setup build-asan --wipe -Db_sanitize=address,undefined -Db_static=false
|
||||
CC=clang meson setup build-asan --wipe -Db_sanitize=address,undefined
|
||||
meson compile -C build-asan -j8
|
||||
meson test -C build-asan --print-errorlogs
|
||||
|
||||
# 5. Si concurrence touchée
|
||||
meson setup build-tsan --wipe -Db_sanitize=thread -Db_static=false
|
||||
CC=clang meson setup build-tsan --wipe -Db_sanitize=thread -Db_lundef=false
|
||||
meson compile -C build-tsan -j8
|
||||
meson test -C build-tsan --print-errorlogs
|
||||
```
|
||||
|
|
|
|||
|
|
@ -29,7 +29,7 @@ Lardon3D suit une feuille de route ordonnée qui privilégie la stabilité et la
|
|||
|
||||
### Phase 3 : Persistance
|
||||
- ✅ Fondation versionnée des checkpoints de tâches
|
||||
- ✅ Project Database v4 (tâches, ScanSets, images, assets et migrations)
|
||||
- ✅ Project Database v5 (tâches, ScanSets, images, Feature Sets/assets et migrations)
|
||||
- ✅ Branchement Project Database au cycle de vie projet et inventaire de reprise
|
||||
- ✅ Registry durable des types métier de tâches
|
||||
- ✅ Premier type métier reconstructible (`import.images`)
|
||||
|
|
@ -37,7 +37,7 @@ Lardon3D suit une feuille de route ordonnée qui privilégie la stabilité et la
|
|||
- ✅ ScanSet v1 et Image Catalog persistant v1
|
||||
|
||||
### Phase 4 : Pipeline avancé
|
||||
- 📋 Feature Store
|
||||
- ✅ Feature Store v1 et extraction ORB persistante
|
||||
- 📋 Visual Index
|
||||
- 📋 Candidate Pair Generator
|
||||
- 📋 Matching et vérification géométrique
|
||||
|
|
|
|||
57
include/lardon3d/feature_extractor.h
Normal file
57
include/lardon3d/feature_extractor.h
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
#ifndef LARDON3D_FEATURE_EXTRACTOR_H
|
||||
#define LARDON3D_FEATURE_EXTRACTOR_H
|
||||
|
||||
#include <stdbool.h>
|
||||
#include <stddef.h>
|
||||
#include <stdint.h>
|
||||
|
||||
#define LARDON3D_FEATURE_EXTRACTOR_KIND "orb"
|
||||
enum {
|
||||
LARDON3D_FEATURE_EXTRACTOR_VERSION = 1,
|
||||
LARDON3D_FEATURE_DESCRIPTOR_DIMENSION = 32,
|
||||
LARDON3D_FEATURE_MAX_FEATURES = 8192,
|
||||
LARDON3D_FEATURE_MAX_IMAGE_PIXELS = 100000000,
|
||||
};
|
||||
|
||||
typedef struct {
|
||||
uint32_t max_features;
|
||||
uint32_t pyramid_levels;
|
||||
uint32_t fast_threshold;
|
||||
} Lardon3DFeatureExtractorParameters;
|
||||
|
||||
typedef struct {
|
||||
float x;
|
||||
float y;
|
||||
float size;
|
||||
float angle_degrees;
|
||||
float response;
|
||||
int32_t octave;
|
||||
} Lardon3DFeatureKeypoint;
|
||||
|
||||
typedef struct {
|
||||
uint32_t image_width;
|
||||
uint32_t image_height;
|
||||
uint32_t feature_count;
|
||||
Lardon3DFeatureKeypoint *keypoints;
|
||||
unsigned char *descriptors;
|
||||
} Lardon3DExtractedFeatures;
|
||||
|
||||
typedef enum {
|
||||
LARDON3D_FEATURE_EXTRACT_OK = 0,
|
||||
LARDON3D_FEATURE_EXTRACT_INVALID_ARGUMENT,
|
||||
LARDON3D_FEATURE_EXTRACT_IMAGE_NOT_FOUND,
|
||||
LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID,
|
||||
LARDON3D_FEATURE_EXTRACT_OUT_OF_MEMORY,
|
||||
LARDON3D_FEATURE_EXTRACT_ERROR
|
||||
} Lardon3DFeatureExtractResult;
|
||||
|
||||
bool lardon3d_feature_extractor_parameters_valid(
|
||||
const Lardon3DFeatureExtractorParameters *parameters);
|
||||
void lardon3d_feature_extractor_parameter_fingerprint(
|
||||
const Lardon3DFeatureExtractorParameters *parameters, unsigned char fingerprint[32]);
|
||||
Lardon3DFeatureExtractResult
|
||||
lardon3d_feature_extract_orb(const char *path, const Lardon3DFeatureExtractorParameters *parameters,
|
||||
Lardon3DExtractedFeatures *features);
|
||||
void lardon3d_extracted_features_destroy(Lardon3DExtractedFeatures *features);
|
||||
|
||||
#endif
|
||||
73
include/lardon3d/feature_store.h
Normal file
73
include/lardon3d/feature_store.h
Normal file
|
|
@ -0,0 +1,73 @@
|
|||
#ifndef LARDON3D_FEATURE_STORE_H
|
||||
#define LARDON3D_FEATURE_STORE_H
|
||||
|
||||
#include <stdbool.h>
|
||||
#include <stddef.h>
|
||||
#include <stdint.h>
|
||||
|
||||
#include <lardon3d/app_state.h>
|
||||
#include <lardon3d/feature_extractor.h>
|
||||
#include <lardon3d/project_db.h>
|
||||
|
||||
enum {
|
||||
LARDON3D_FEATURE_FILE_VERSION = 1,
|
||||
LARDON3D_FEATURE_FILE_HEADER_SIZE = 160,
|
||||
LARDON3D_FEATURE_KEYPOINT_RECORD_SIZE = 24,
|
||||
LARDON3D_FEATURE_READER_RANGE_MAX = 256,
|
||||
LARDON3D_FEATURE_FILE_MAX_SIZE = 16 * 1024 * 1024,
|
||||
};
|
||||
|
||||
typedef enum {
|
||||
LARDON3D_FEATURE_DESCRIPTOR_U8 = 1,
|
||||
LARDON3D_FEATURE_DESCRIPTOR_F32 = 2
|
||||
} Lardon3DFeatureDescriptorType;
|
||||
|
||||
typedef enum {
|
||||
LARDON3D_FEATURE_STORE_OK = 0,
|
||||
LARDON3D_FEATURE_STORE_ALREADY_PRESENT,
|
||||
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT,
|
||||
LARDON3D_FEATURE_STORE_NOT_FOUND,
|
||||
LARDON3D_FEATURE_STORE_INVALID,
|
||||
LARDON3D_FEATURE_STORE_UNSUPPORTED_VERSION,
|
||||
LARDON3D_FEATURE_STORE_CORRUPT,
|
||||
LARDON3D_FEATURE_STORE_IO_ERROR,
|
||||
LARDON3D_FEATURE_STORE_DB_BUSY,
|
||||
LARDON3D_FEATURE_STORE_DB_ERROR,
|
||||
LARDON3D_FEATURE_STORE_PUBLISHED_NOT_DURABLE
|
||||
} Lardon3DFeatureStoreResult;
|
||||
|
||||
typedef struct Lardon3DFeatureReader Lardon3DFeatureReader;
|
||||
|
||||
typedef struct {
|
||||
uint32_t format_version;
|
||||
char extractor_kind[LARDON3D_TASK_KIND_CAPACITY];
|
||||
uint32_t extractor_version;
|
||||
uint32_t feature_count;
|
||||
uint32_t descriptor_dimension;
|
||||
Lardon3DFeatureDescriptorType descriptor_type;
|
||||
uint32_t image_width;
|
||||
uint32_t image_height;
|
||||
unsigned char source_image_sha256[32];
|
||||
unsigned char parameter_fingerprint[32];
|
||||
} Lardon3DFeatureFileMetadata;
|
||||
|
||||
Lardon3DFeatureStoreResult lardon3d_feature_store_publish(
|
||||
Lardon3DAppState *state, uint64_t image_id, uint64_t producer_task_id,
|
||||
const Lardon3DFeatureExtractorParameters *parameters, const Lardon3DExtractedFeatures *features,
|
||||
Lardon3DProjectDbFeatureSet *feature_set);
|
||||
Lardon3DFeatureStoreResult
|
||||
lardon3d_feature_reader_open(const char *project_path,
|
||||
const Lardon3DProjectDbFeatureSet *feature_set,
|
||||
Lardon3DFeatureReader **reader, Lardon3DFeatureFileMetadata *metadata);
|
||||
void lardon3d_feature_reader_close(Lardon3DFeatureReader *reader);
|
||||
Lardon3DFeatureStoreResult lardon3d_feature_reader_keypoints(Lardon3DFeatureReader *reader,
|
||||
uint32_t start,
|
||||
Lardon3DFeatureKeypoint *keypoints,
|
||||
size_t capacity);
|
||||
Lardon3DFeatureStoreResult lardon3d_feature_reader_descriptors(Lardon3DFeatureReader *reader,
|
||||
uint32_t start,
|
||||
unsigned char *descriptors,
|
||||
size_t feature_capacity,
|
||||
size_t descriptor_capacity);
|
||||
|
||||
#endif
|
||||
24
include/lardon3d/feature_task.h
Normal file
24
include/lardon3d/feature_task.h
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
#ifndef LARDON3D_FEATURE_TASK_H
|
||||
#define LARDON3D_FEATURE_TASK_H
|
||||
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
|
||||
#include <lardon3d/app_state.h>
|
||||
#include <lardon3d/feature_extractor.h>
|
||||
#include <lardon3d/task_kind_registry.h>
|
||||
|
||||
#define LARDON3D_FEATURE_EXTRACT_TASK_KIND "features.extract"
|
||||
enum { LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION = 1 };
|
||||
|
||||
Lardon3DTask *
|
||||
lardon3d_project_create_feature_extract_task(Lardon3DAppState *state, uint64_t image_id,
|
||||
const Lardon3DFeatureExtractorParameters *parameters,
|
||||
uint64_t *task_id);
|
||||
bool lardon3d_project_enqueue_feature_extract(Lardon3DAppState *state, uint64_t image_id,
|
||||
const Lardon3DFeatureExtractorParameters *parameters,
|
||||
uint64_t *task_id);
|
||||
bool lardon3d_feature_extract_reconstruct(const Lardon3DTaskDurableSnapshot *snapshot,
|
||||
void *context, Lardon3DTaskKindBinding *binding);
|
||||
|
||||
#endif
|
||||
|
|
@ -31,42 +31,23 @@ typedef struct {
|
|||
|
||||
typedef struct Lardon3DImportTask Lardon3DImportTask;
|
||||
|
||||
typedef struct {
|
||||
const char *project_path;
|
||||
Lardon3DProjectDb *project_db;
|
||||
Lardon3DResourceGovernor *resource_governor;
|
||||
} Lardon3DImageImportReconstructionContext;
|
||||
typedef Lardon3DTaskReconstructionContext Lardon3DImageImportReconstructionContext;
|
||||
|
||||
Lardon3DTask *lardon3d_project_create_image_import_task(
|
||||
Lardon3DAppState *state,
|
||||
Lardon3DTask *lardon3d_project_create_image_import_task(Lardon3DAppState *state,
|
||||
uint64_t scanset_id,
|
||||
const char *source_directory,
|
||||
uint64_t *task_id
|
||||
);
|
||||
bool lardon3d_project_enqueue_image_import(
|
||||
Lardon3DAppState *state,
|
||||
uint64_t scanset_id,
|
||||
const char *source_directory,
|
||||
uint64_t *task_id
|
||||
);
|
||||
bool lardon3d_image_import_reconstruct(
|
||||
const Lardon3DTaskDurableSnapshot *snapshot,
|
||||
void *context,
|
||||
Lardon3DTaskKindBinding *binding
|
||||
);
|
||||
uint64_t *task_id);
|
||||
bool lardon3d_project_enqueue_image_import(Lardon3DAppState *state, uint64_t scanset_id,
|
||||
const char *source_directory, uint64_t *task_id);
|
||||
bool lardon3d_image_import_reconstruct(const Lardon3DTaskDurableSnapshot *snapshot, void *context,
|
||||
Lardon3DTaskKindBinding *binding);
|
||||
|
||||
/* Compatibilité TUI : handle léger sur une tâche de la queue, sans thread. */
|
||||
Lardon3DImportTask *lardon3d_import_task_create(void);
|
||||
bool lardon3d_import_task_start(
|
||||
Lardon3DImportTask *task,
|
||||
Lardon3DAppState *state,
|
||||
const char *source_directory
|
||||
);
|
||||
bool lardon3d_import_task_start(Lardon3DImportTask *task, Lardon3DAppState *state,
|
||||
const char *source_directory);
|
||||
void lardon3d_import_task_request_cancel(Lardon3DImportTask *task);
|
||||
bool lardon3d_import_task_snapshot(
|
||||
Lardon3DImportTask *task,
|
||||
Lardon3DImportTaskSnapshot *snapshot
|
||||
);
|
||||
bool lardon3d_import_task_snapshot(Lardon3DImportTask *task, Lardon3DImportTaskSnapshot *snapshot);
|
||||
bool lardon3d_import_task_is_finished(Lardon3DImportTask *task);
|
||||
bool lardon3d_import_task_join(Lardon3DImportTask *task);
|
||||
void lardon3d_import_task_destroy(Lardon3DImportTask *task);
|
||||
|
|
|
|||
|
|
@ -48,44 +48,29 @@ typedef struct {
|
|||
bool queue_full;
|
||||
} Lardon3DProjectRecoverySummary;
|
||||
|
||||
bool lardon3d_project_create(
|
||||
Lardon3DAppState *state,
|
||||
const char *name
|
||||
);
|
||||
bool lardon3d_project_create(Lardon3DAppState *state, const char *name);
|
||||
|
||||
bool lardon3d_project_open(
|
||||
Lardon3DAppState *state,
|
||||
const char *directory_name
|
||||
);
|
||||
bool lardon3d_project_open(Lardon3DAppState *state, const char *directory_name);
|
||||
|
||||
void lardon3d_project_close(Lardon3DAppState *state);
|
||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_task(
|
||||
Lardon3DAppState *state,
|
||||
const Lardon3DTask *task
|
||||
);
|
||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_task(Lardon3DAppState *state,
|
||||
const Lardon3DTask *task);
|
||||
Lardon3DProjectTaskCheckpointResult
|
||||
lardon3d_project_checkpoint_image_import_task(
|
||||
Lardon3DAppState *state,
|
||||
const Lardon3DTask *task,
|
||||
const char *source_path,
|
||||
uint64_t scanset_id
|
||||
);
|
||||
Lardon3DProjectDbResult lardon3d_project_list_recoverable(
|
||||
Lardon3DAppState *state,
|
||||
lardon3d_project_checkpoint_image_import_task(Lardon3DAppState *state, const Lardon3DTask *task,
|
||||
const char *source_path, uint64_t scanset_id);
|
||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_task(
|
||||
Lardon3DAppState *state, const Lardon3DTask *task,
|
||||
const Lardon3DProjectDbFeatureExtractTask *parameters);
|
||||
Lardon3DProjectDbResult lardon3d_project_list_recoverable(Lardon3DAppState *state,
|
||||
const Lardon3DTaskKindRegistry *registry,
|
||||
uint64_t after_task_id,
|
||||
Lardon3DProjectRecoveryEntry *entries,
|
||||
size_t capacity,
|
||||
size_t *count
|
||||
);
|
||||
Lardon3DProjectDbResult lardon3d_project_resume_recoverable_tasks(
|
||||
Lardon3DAppState *state,
|
||||
size_t capacity, size_t *count);
|
||||
Lardon3DProjectDbResult
|
||||
lardon3d_project_resume_recoverable_tasks(Lardon3DAppState *state,
|
||||
const Lardon3DTaskKindRegistry *registry,
|
||||
Lardon3DProjectRecoverySummary *summary
|
||||
);
|
||||
bool lardon3d_project_last_recovery_summary(
|
||||
const Lardon3DAppState *state,
|
||||
Lardon3DProjectRecoverySummary *summary
|
||||
);
|
||||
Lardon3DProjectRecoverySummary *summary);
|
||||
bool lardon3d_project_last_recovery_summary(const Lardon3DAppState *state,
|
||||
Lardon3DProjectRecoverySummary *summary);
|
||||
|
||||
#endif
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
#include <lardon3d/task.h>
|
||||
|
||||
enum {
|
||||
LARDON3D_PROJECT_DB_SCHEMA_VERSION = 4,
|
||||
LARDON3D_PROJECT_DB_SCHEMA_VERSION = 5,
|
||||
LARDON3D_PROJECT_DB_ID_CAPACITY = 65,
|
||||
LARDON3D_PROJECT_DB_KIND_CAPACITY = 65,
|
||||
LARDON3D_PROJECT_DB_PATH_CAPACITY = 4096,
|
||||
|
|
@ -100,9 +100,7 @@ typedef struct {
|
|||
int64_t updated_at;
|
||||
} Lardon3DProjectDbScanSet;
|
||||
|
||||
typedef enum {
|
||||
LARDON3D_DB_IMAGE_ASSET_READY = 1
|
||||
} Lardon3DProjectDbImageAssetState;
|
||||
typedef enum { LARDON3D_DB_IMAGE_ASSET_READY = 1 } Lardon3DProjectDbImageAssetState;
|
||||
|
||||
typedef struct {
|
||||
uint64_t asset_id;
|
||||
|
|
@ -129,113 +127,141 @@ typedef enum {
|
|||
LARDON3D_PROJECT_DB_IMAGE_ALREADY_PRESENT
|
||||
} Lardon3DProjectDbImageRegisterStatus;
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_db_open(
|
||||
const char *path,
|
||||
Lardon3DProjectDb **database,
|
||||
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]
|
||||
);
|
||||
typedef enum {
|
||||
LARDON3D_DB_FEATURE_ASSET_DURABLE = 0,
|
||||
LARDON3D_DB_FEATURE_ASSET_PUBLISHED_NOT_DURABLE = 1
|
||||
} Lardon3DProjectDbFeatureDurability;
|
||||
|
||||
typedef struct {
|
||||
uint64_t feature_asset_id;
|
||||
unsigned char sha256[32];
|
||||
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
||||
uint64_t size_bytes;
|
||||
Lardon3DProjectDbFeatureDurability durability;
|
||||
int64_t created_at;
|
||||
} Lardon3DProjectDbFeatureAsset;
|
||||
|
||||
typedef struct {
|
||||
uint64_t feature_set_id;
|
||||
uint64_t image_id;
|
||||
uint64_t feature_asset_id;
|
||||
char extractor_kind[LARDON3D_PROJECT_DB_KIND_CAPACITY];
|
||||
uint32_t extractor_version;
|
||||
unsigned char parameter_fingerprint[32];
|
||||
unsigned char source_image_sha256[32];
|
||||
uint32_t feature_count;
|
||||
uint32_t descriptor_type;
|
||||
uint32_t descriptor_dimension;
|
||||
bool has_producer_task;
|
||||
uint64_t producer_task_id;
|
||||
int64_t created_at;
|
||||
Lardon3DProjectDbFeatureAsset asset;
|
||||
} Lardon3DProjectDbFeatureSet;
|
||||
|
||||
typedef struct {
|
||||
uint64_t task_id;
|
||||
uint64_t image_id;
|
||||
char extractor_kind[LARDON3D_PROJECT_DB_KIND_CAPACITY];
|
||||
uint32_t extractor_version;
|
||||
uint32_t max_features;
|
||||
uint32_t pyramid_levels;
|
||||
uint32_t fast_threshold;
|
||||
unsigned char parameter_fingerprint[32];
|
||||
} Lardon3DProjectDbFeatureExtractTask;
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_db_open(const char *path, Lardon3DProjectDb **database,
|
||||
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]);
|
||||
void lardon3d_project_db_close(Lardon3DProjectDb *database);
|
||||
bool lardon3d_project_db_last_error(
|
||||
Lardon3DProjectDb *database,
|
||||
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]
|
||||
);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_legacy_catalog_pending(
|
||||
Lardon3DProjectDb *database, bool *pending
|
||||
);
|
||||
bool lardon3d_project_db_last_error(Lardon3DProjectDb *database,
|
||||
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_legacy_catalog_pending(Lardon3DProjectDb *database,
|
||||
bool *pending);
|
||||
unsigned int lardon3d_project_db_schema_version(Lardon3DProjectDb *database);
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_db_set_project(
|
||||
Lardon3DProjectDb *database,
|
||||
const Lardon3DProjectDbProject *project
|
||||
);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_get_project(
|
||||
Lardon3DProjectDb *database,
|
||||
Lardon3DProjectDbProject *project
|
||||
);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_set_project(Lardon3DProjectDb *database,
|
||||
const Lardon3DProjectDbProject *project);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_get_project(Lardon3DProjectDb *database,
|
||||
Lardon3DProjectDbProject *project);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_record_task(
|
||||
Lardon3DProjectDb *database,
|
||||
const Lardon3DTaskDurableSnapshot *snapshot,
|
||||
const char *task_kind,
|
||||
uint32_t task_kind_version,
|
||||
const Lardon3DProjectDbCheckpoint *checkpoint,
|
||||
int64_t updated_at
|
||||
);
|
||||
Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, const char *task_kind,
|
||||
uint32_t task_kind_version, const Lardon3DProjectDbCheckpoint *checkpoint, int64_t updated_at);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_record_image_import_task(
|
||||
Lardon3DProjectDb *database,
|
||||
const Lardon3DTaskDurableSnapshot *snapshot,
|
||||
const char *task_kind,
|
||||
uint32_t task_kind_version,
|
||||
const Lardon3DProjectDbCheckpoint *checkpoint,
|
||||
const char *source_path,
|
||||
Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, const char *task_kind,
|
||||
uint32_t task_kind_version, const Lardon3DProjectDbCheckpoint *checkpoint,
|
||||
const char *source_path, uint64_t scanset_id, int64_t updated_at);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_create_scanset(Lardon3DProjectDb *database,
|
||||
const char *name,
|
||||
Lardon3DProjectDbScanSet *scanset);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_load_scanset(Lardon3DProjectDb *database,
|
||||
uint64_t scanset_id,
|
||||
int64_t updated_at
|
||||
);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_create_scanset(
|
||||
Lardon3DProjectDb *database, const char *name,
|
||||
Lardon3DProjectDbScanSet *scanset
|
||||
);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_load_scanset(
|
||||
Lardon3DProjectDb *database, uint64_t scanset_id,
|
||||
Lardon3DProjectDbScanSet *scanset
|
||||
);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_list_scansets(
|
||||
Lardon3DProjectDb *database, uint64_t after_scanset_id,
|
||||
Lardon3DProjectDbScanSet *scansets, size_t capacity, size_t *count
|
||||
);
|
||||
Lardon3DProjectDbScanSet *scanset);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_list_scansets(Lardon3DProjectDb *database,
|
||||
uint64_t after_scanset_id,
|
||||
Lardon3DProjectDbScanSet *scansets,
|
||||
size_t capacity, size_t *count);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_register_image(
|
||||
Lardon3DProjectDb *database, uint64_t scanset_id,
|
||||
const unsigned char sha256[LARDON3D_PROJECT_DB_SHA256_SIZE],
|
||||
const char *asset_path, uint64_t size_bytes, const char *original_name,
|
||||
const char *source_path, uint64_t producer_task_id, int64_t imported_at,
|
||||
Lardon3DProjectDbImageRegisterStatus *status,
|
||||
Lardon3DProjectDbImage *image
|
||||
);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_load_image(
|
||||
Lardon3DProjectDb *database, uint64_t image_id,
|
||||
Lardon3DProjectDbImage *image, Lardon3DProjectDbImageAsset *asset
|
||||
);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_list_images(
|
||||
Lardon3DProjectDb *database, uint64_t scanset_id, uint64_t after_image_id,
|
||||
Lardon3DProjectDbImage *images, Lardon3DProjectDbImageAsset *assets,
|
||||
size_t capacity, size_t *count
|
||||
);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_count_images(
|
||||
Lardon3DProjectDb *database, uint64_t scanset_id, uint64_t *count
|
||||
);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_load_image_import(
|
||||
Lardon3DProjectDb *database,
|
||||
uint64_t task_id,
|
||||
Lardon3DProjectDbImageImport *parameters
|
||||
);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_allocate_task_id(
|
||||
Lardon3DProjectDb *database,
|
||||
uint64_t *task_id
|
||||
);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_load_task(
|
||||
Lardon3DProjectDb *database,
|
||||
uint64_t task_id,
|
||||
Lardon3DProjectDbTask *task
|
||||
);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_list_recoverable(
|
||||
Lardon3DProjectDb *database,
|
||||
const unsigned char sha256[LARDON3D_PROJECT_DB_SHA256_SIZE], const char *asset_path,
|
||||
uint64_t size_bytes, const char *original_name, const char *source_path,
|
||||
uint64_t producer_task_id, int64_t imported_at, Lardon3DProjectDbImageRegisterStatus *status,
|
||||
Lardon3DProjectDbImage *image);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_load_image(Lardon3DProjectDb *database,
|
||||
uint64_t image_id,
|
||||
Lardon3DProjectDbImage *image,
|
||||
Lardon3DProjectDbImageAsset *asset);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_list_images(Lardon3DProjectDb *database,
|
||||
uint64_t scanset_id,
|
||||
uint64_t after_image_id,
|
||||
Lardon3DProjectDbImage *images,
|
||||
Lardon3DProjectDbImageAsset *assets,
|
||||
size_t capacity, size_t *count);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_count_images(Lardon3DProjectDb *database,
|
||||
uint64_t scanset_id, uint64_t *count);
|
||||
Lardon3DProjectDbResult
|
||||
lardon3d_project_db_load_image_import(Lardon3DProjectDb *database, uint64_t task_id,
|
||||
Lardon3DProjectDbImageImport *parameters);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_allocate_task_id(Lardon3DProjectDb *database,
|
||||
uint64_t *task_id);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_load_task(Lardon3DProjectDb *database, uint64_t task_id,
|
||||
Lardon3DProjectDbTask *task);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_list_recoverable(Lardon3DProjectDb *database,
|
||||
uint64_t after_task_id,
|
||||
Lardon3DProjectDbTask *tasks,
|
||||
size_t capacity,
|
||||
size_t *count
|
||||
);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_create_artifact(
|
||||
Lardon3DProjectDb *database,
|
||||
const Lardon3DProjectDbArtifact *artifact
|
||||
);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_mark_artifact_ready(
|
||||
Lardon3DProjectDb *database,
|
||||
size_t capacity, size_t *count);
|
||||
Lardon3DProjectDbResult
|
||||
lardon3d_project_db_create_artifact(Lardon3DProjectDb *database,
|
||||
const Lardon3DProjectDbArtifact *artifact);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_mark_artifact_ready(Lardon3DProjectDb *database,
|
||||
const char *artifact_id,
|
||||
int64_t updated_at
|
||||
);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_load_artifact(
|
||||
Lardon3DProjectDb *database,
|
||||
int64_t updated_at);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_load_artifact(Lardon3DProjectDb *database,
|
||||
const char *artifact_id,
|
||||
Lardon3DProjectDbArtifact *artifact
|
||||
);
|
||||
Lardon3DProjectDbArtifact *artifact);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_task(
|
||||
Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, const char *task_kind,
|
||||
uint32_t task_kind_version, const Lardon3DProjectDbCheckpoint *checkpoint,
|
||||
const Lardon3DProjectDbFeatureExtractTask *parameters, int64_t updated_at);
|
||||
Lardon3DProjectDbResult
|
||||
lardon3d_project_db_load_feature_extract_task(Lardon3DProjectDb *database, uint64_t task_id,
|
||||
Lardon3DProjectDbFeatureExtractTask *parameters);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_register_feature_set(
|
||||
Lardon3DProjectDb *database, uint64_t image_id, const char *extractor_kind,
|
||||
uint32_t extractor_version, const unsigned char parameter_fingerprint[32],
|
||||
const unsigned char source_image_sha256[32], uint32_t feature_count, uint32_t descriptor_type,
|
||||
uint32_t descriptor_dimension, const unsigned char asset_sha256[32], const char *asset_path,
|
||||
uint64_t asset_size_bytes, Lardon3DProjectDbFeatureDurability durability,
|
||||
uint64_t producer_task_id, int64_t created_at, Lardon3DProjectDbFeatureSet *feature_set);
|
||||
Lardon3DProjectDbResult
|
||||
lardon3d_project_db_find_feature_set(Lardon3DProjectDb *database, uint64_t image_id,
|
||||
const char *extractor_kind, uint32_t extractor_version,
|
||||
const unsigned char parameter_fingerprint[32],
|
||||
Lardon3DProjectDbFeatureSet *feature_set);
|
||||
Lardon3DProjectDbResult
|
||||
lardon3d_project_db_load_feature_set(Lardon3DProjectDb *database, uint64_t feature_set_id,
|
||||
Lardon3DProjectDbFeatureSet *feature_set);
|
||||
Lardon3DProjectDbResult
|
||||
lardon3d_project_db_list_feature_sets(Lardon3DProjectDb *database, uint64_t after_feature_set_id,
|
||||
Lardon3DProjectDbFeatureSet *feature_sets, size_t capacity,
|
||||
size_t *count);
|
||||
|
||||
#endif
|
||||
|
|
|
|||
|
|
@ -6,6 +6,9 @@
|
|||
|
||||
#include <lardon3d/task.h>
|
||||
|
||||
typedef struct Lardon3DProjectDb Lardon3DProjectDb;
|
||||
typedef struct Lardon3DResourceGovernor Lardon3DResourceGovernor;
|
||||
|
||||
enum {
|
||||
LARDON3D_TASK_KIND_REGISTRY_MAX = 64,
|
||||
};
|
||||
|
|
@ -18,11 +21,8 @@ typedef struct {
|
|||
void *finished_userdata;
|
||||
} Lardon3DTaskKindBinding;
|
||||
|
||||
typedef bool (*Lardon3DTaskKindReconstruct)(
|
||||
const Lardon3DTaskDurableSnapshot *snapshot,
|
||||
void *context,
|
||||
Lardon3DTaskKindBinding *binding
|
||||
);
|
||||
typedef bool (*Lardon3DTaskKindReconstruct)(const Lardon3DTaskDurableSnapshot *snapshot,
|
||||
void *context, Lardon3DTaskKindBinding *binding);
|
||||
|
||||
typedef struct {
|
||||
const char *kind;
|
||||
|
|
@ -35,6 +35,12 @@ typedef struct {
|
|||
size_t count;
|
||||
} Lardon3DTaskKindRegistry;
|
||||
|
||||
typedef struct {
|
||||
const char *project_path;
|
||||
Lardon3DProjectDb *project_db;
|
||||
Lardon3DResourceGovernor *resource_governor;
|
||||
} Lardon3DTaskReconstructionContext;
|
||||
|
||||
typedef enum {
|
||||
LARDON3D_TASK_KIND_OK = 0,
|
||||
LARDON3D_TASK_KIND_INVALID_ARGUMENT,
|
||||
|
|
@ -44,25 +50,15 @@ typedef enum {
|
|||
LARDON3D_TASK_KIND_RESTORE_FAILED
|
||||
} Lardon3DTaskKindResult;
|
||||
|
||||
bool lardon3d_task_kind_registry_init(
|
||||
Lardon3DTaskKindRegistry *registry,
|
||||
const Lardon3DTaskKindDescriptor *descriptors,
|
||||
size_t count
|
||||
);
|
||||
Lardon3DTaskKindResult lardon3d_task_kind_registry_lookup(
|
||||
const Lardon3DTaskKindRegistry *registry,
|
||||
const char *kind,
|
||||
bool lardon3d_task_kind_registry_init(Lardon3DTaskKindRegistry *registry,
|
||||
const Lardon3DTaskKindDescriptor *descriptors, size_t count);
|
||||
Lardon3DTaskKindResult
|
||||
lardon3d_task_kind_registry_lookup(const Lardon3DTaskKindRegistry *registry, const char *kind,
|
||||
uint32_t kind_version,
|
||||
const Lardon3DTaskKindDescriptor **descriptor
|
||||
);
|
||||
const Lardon3DTaskKindDescriptor **descriptor);
|
||||
Lardon3DTaskKindResult lardon3d_task_kind_registry_restore(
|
||||
const Lardon3DTaskKindRegistry *registry,
|
||||
const char *kind,
|
||||
uint32_t kind_version,
|
||||
const Lardon3DTaskDurableSnapshot *snapshot,
|
||||
void *context,
|
||||
Lardon3DTask **task
|
||||
);
|
||||
const Lardon3DTaskKindRegistry *registry, const char *kind, uint32_t kind_version,
|
||||
const Lardon3DTaskDurableSnapshot *snapshot, void *context, Lardon3DTask **task);
|
||||
const Lardon3DTaskKindRegistry *lardon3d_task_kind_registry_production(void);
|
||||
|
||||
#endif
|
||||
|
|
|
|||
57
meson.build
57
meson.build
|
|
@ -1,6 +1,6 @@
|
|||
project(
|
||||
'lardon3d',
|
||||
'c',
|
||||
'c', 'cpp',
|
||||
version: '0.1.0',
|
||||
license: 'MIT',
|
||||
default_options: [
|
||||
|
|
@ -17,11 +17,13 @@ add_project_arguments(
|
|||
'-Wshadow',
|
||||
language: 'c',
|
||||
)
|
||||
add_project_arguments('-std=c++17', '-Wpedantic', '-Wconversion', '-Wshadow', language: 'cpp')
|
||||
|
||||
ncursesw = dependency('ncursesw', required: true)
|
||||
threads = dependency('threads')
|
||||
sqlite3 = dependency('sqlite3', required: true)
|
||||
openssl = dependency('openssl', required: true)
|
||||
opencv = dependency('opencv5', required: true, include_type: 'system', modules: ['opencv_core', 'opencv_imgcodecs', 'opencv_features2d'])
|
||||
|
||||
executable(
|
||||
'lardon3d',
|
||||
|
|
@ -35,6 +37,9 @@ executable(
|
|||
'src/import_task.c',
|
||||
'src/image_catalog.c',
|
||||
'src/image_catalog_persistent.c',
|
||||
'src/feature_extractor_opencv.cpp',
|
||||
'src/feature_store.c',
|
||||
'src/feature_task.c',
|
||||
'src/image_view.c',
|
||||
'src/project.c',
|
||||
'src/project_db.c',
|
||||
|
|
@ -48,7 +53,7 @@ executable(
|
|||
'src/hardware_profile.c',
|
||||
],
|
||||
include_directories: include_directories('include'),
|
||||
dependencies: [ncursesw, threads, sqlite3, openssl],
|
||||
dependencies: [ncursesw, threads, sqlite3, openssl, opencv],
|
||||
)
|
||||
|
||||
import_test = executable(
|
||||
|
|
@ -82,6 +87,9 @@ import_task_test = executable(
|
|||
'src/task_checkpoint.c',
|
||||
'src/task_kind_registry.c',
|
||||
'src/task_kinds.c',
|
||||
'src/feature_task.c',
|
||||
'src/feature_store.c',
|
||||
'src/feature_extractor_opencv.cpp',
|
||||
'src/task_queue.c',
|
||||
'src/resource_governor.c',
|
||||
'src/resource_snapshot.c',
|
||||
|
|
@ -95,7 +103,7 @@ import_task_test = executable(
|
|||
'-DLARDON3D_CHECKPOINT_TESTING',
|
||||
],
|
||||
include_directories: include_directories('include'),
|
||||
dependencies: [threads, sqlite3, openssl],
|
||||
dependencies: [threads, sqlite3, openssl, opencv],
|
||||
)
|
||||
|
||||
test('import-task', import_task_test, timeout: 30)
|
||||
|
|
@ -136,6 +144,44 @@ persistent_image_catalog_test = executable(
|
|||
|
||||
test('persistent-image-catalog', persistent_image_catalog_test, timeout: 60)
|
||||
|
||||
feature_store_test = executable(
|
||||
'test-feature-store',
|
||||
sources: [
|
||||
'tests/test_feature_store.c',
|
||||
'src/app_state.c',
|
||||
'src/image_catalog_persistent.c',
|
||||
'src/feature_extractor_opencv.cpp',
|
||||
'src/feature_store.c',
|
||||
'src/project_db.c',
|
||||
'src/task.c',
|
||||
'src/resource_governor.c',
|
||||
'src/resource_snapshot.c',
|
||||
],
|
||||
c_args: ['-DLARDON3D_PROJECT_DB_TESTING', '-DLARDON3D_FEATURE_STORE_TESTING'],
|
||||
include_directories: include_directories('include'),
|
||||
dependencies: [threads, sqlite3, openssl, opencv],
|
||||
)
|
||||
|
||||
test('feature-store', feature_store_test, timeout: 60)
|
||||
|
||||
feature_task_test = executable(
|
||||
'test-feature-task',
|
||||
sources: [
|
||||
'tests/test_feature_task.c', 'src/app_state.c', 'src/project.c',
|
||||
'src/project_db.c', 'src/task.c', 'src/task_checkpoint.c',
|
||||
'src/task_kind_registry.c', 'src/task_kinds.c', 'src/task_queue.c',
|
||||
'src/import.c', 'src/import_task.c', 'src/image_catalog.c',
|
||||
'src/image_catalog_persistent.c', 'src/image_view.c',
|
||||
'src/feature_task.c', 'src/feature_store.c',
|
||||
'src/feature_extractor_opencv.cpp', 'src/resource_governor.c',
|
||||
'src/resource_snapshot.c',
|
||||
],
|
||||
c_args: ['-DLARDON3D_PROJECT_DB_TESTING', '-DLARDON3D_FEATURE_TASK_TESTING'],
|
||||
include_directories: include_directories('include'),
|
||||
dependencies: [threads, sqlite3, openssl, opencv],
|
||||
)
|
||||
test('feature-task', feature_task_test, timeout: 60)
|
||||
|
||||
image_view_test = executable(
|
||||
'test-image-view',
|
||||
sources: [
|
||||
|
|
@ -206,6 +252,9 @@ project_test = executable(
|
|||
'src/task_checkpoint.c',
|
||||
'src/task_kind_registry.c',
|
||||
'src/task_kinds.c',
|
||||
'src/feature_task.c',
|
||||
'src/feature_store.c',
|
||||
'src/feature_extractor_opencv.cpp',
|
||||
'src/task_queue.c',
|
||||
'src/import.c',
|
||||
'src/import_task.c',
|
||||
|
|
@ -220,7 +269,7 @@ project_test = executable(
|
|||
'-DLARDON3D_CHECKPOINT_TESTING',
|
||||
],
|
||||
include_directories: include_directories('include'),
|
||||
dependencies: [threads, sqlite3, openssl],
|
||||
dependencies: [threads, sqlite3, openssl, opencv],
|
||||
)
|
||||
|
||||
test('project', project_test, timeout: 30)
|
||||
|
|
|
|||
142
src/feature_extractor_opencv.cpp
Normal file
142
src/feature_extractor_opencv.cpp
Normal file
|
|
@ -0,0 +1,142 @@
|
|||
#include <cerrno>
|
||||
#include <cmath>
|
||||
#include <cstdlib>
|
||||
#include <cstring>
|
||||
#include <limits>
|
||||
#include <new>
|
||||
#include <sys/stat.h>
|
||||
#include <vector>
|
||||
|
||||
#include <opencv2/features2d.hpp>
|
||||
#include <opencv2/imgcodecs.hpp>
|
||||
#include <openssl/sha.h>
|
||||
|
||||
extern "C" {
|
||||
#include <lardon3d/feature_extractor.h>
|
||||
}
|
||||
|
||||
extern "C" bool
|
||||
lardon3d_feature_extractor_parameters_valid(const Lardon3DFeatureExtractorParameters *parameters) {
|
||||
return parameters && parameters->max_features > 0 &&
|
||||
parameters->max_features <= LARDON3D_FEATURE_MAX_FEATURES &&
|
||||
parameters->pyramid_levels >= 1 && parameters->pyramid_levels <= 16 &&
|
||||
parameters->fast_threshold >= 1 && parameters->fast_threshold <= 255;
|
||||
}
|
||||
|
||||
static void put_u32(unsigned char *output, uint32_t value) {
|
||||
output[0] = static_cast<unsigned char>(value);
|
||||
output[1] = static_cast<unsigned char>(value >> 8);
|
||||
output[2] = static_cast<unsigned char>(value >> 16);
|
||||
output[3] = static_cast<unsigned char>(value >> 24);
|
||||
}
|
||||
|
||||
extern "C" void lardon3d_feature_extractor_parameter_fingerprint(
|
||||
const Lardon3DFeatureExtractorParameters *parameters, unsigned char fingerprint[32]) {
|
||||
if (!fingerprint) {
|
||||
return;
|
||||
}
|
||||
std::memset(fingerprint, 0, 32);
|
||||
if (!lardon3d_feature_extractor_parameters_valid(parameters)) {
|
||||
return;
|
||||
}
|
||||
unsigned char canonical[24] = {'L', '3', 'D', 'O', 'R', 'B', 'P', '1'};
|
||||
put_u32(canonical + 8, LARDON3D_FEATURE_EXTRACTOR_VERSION);
|
||||
put_u32(canonical + 12, parameters->max_features);
|
||||
put_u32(canonical + 16, parameters->pyramid_levels);
|
||||
put_u32(canonical + 20, parameters->fast_threshold);
|
||||
(void)SHA256(canonical, sizeof(canonical), fingerprint);
|
||||
}
|
||||
|
||||
extern "C" void lardon3d_extracted_features_destroy(Lardon3DExtractedFeatures *features) {
|
||||
if (!features) {
|
||||
return;
|
||||
}
|
||||
std::free(features->keypoints);
|
||||
std::free(features->descriptors);
|
||||
std::memset(features, 0, sizeof(*features));
|
||||
}
|
||||
|
||||
extern "C" Lardon3DFeatureExtractResult
|
||||
lardon3d_feature_extract_orb(const char *path, const Lardon3DFeatureExtractorParameters *parameters,
|
||||
Lardon3DExtractedFeatures *features) {
|
||||
if (features) {
|
||||
std::memset(features, 0, sizeof(*features));
|
||||
}
|
||||
if (!path || !path[0] || !features || !lardon3d_feature_extractor_parameters_valid(parameters)) {
|
||||
return LARDON3D_FEATURE_EXTRACT_INVALID_ARGUMENT;
|
||||
}
|
||||
struct stat information{};
|
||||
if (lstat(path, &information) != 0) {
|
||||
return errno == ENOENT ? LARDON3D_FEATURE_EXTRACT_IMAGE_NOT_FOUND
|
||||
: LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID;
|
||||
}
|
||||
if (!S_ISREG(information.st_mode) || S_ISLNK(information.st_mode)) {
|
||||
return LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID;
|
||||
}
|
||||
try {
|
||||
cv::Mat image = cv::imread(path, cv::IMREAD_GRAYSCALE);
|
||||
if (image.empty()) {
|
||||
return LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID;
|
||||
}
|
||||
if (image.cols <= 0 || image.rows <= 0 ||
|
||||
static_cast<uint64_t>(image.cols) * static_cast<uint64_t>(image.rows) >
|
||||
LARDON3D_FEATURE_MAX_IMAGE_PIXELS) {
|
||||
return LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID;
|
||||
}
|
||||
cv::Ptr<cv::ORB> extractor =
|
||||
cv::ORB::create(static_cast<int>(parameters->max_features), 1.2F,
|
||||
static_cast<int>(parameters->pyramid_levels), 31, 0, 2,
|
||||
cv::ORB::HARRIS_SCORE, 31, static_cast<int>(parameters->fast_threshold));
|
||||
std::vector<cv::KeyPoint> points;
|
||||
cv::Mat descriptors;
|
||||
extractor->detectAndCompute(image, cv::noArray(), points, descriptors);
|
||||
if (points.size() > parameters->max_features ||
|
||||
points.size() > std::numeric_limits<uint32_t>::max() ||
|
||||
(!points.empty() && (descriptors.rows != static_cast<int>(points.size()) ||
|
||||
descriptors.cols != LARDON3D_FEATURE_DESCRIPTOR_DIMENSION ||
|
||||
descriptors.type() != CV_8UC1))) {
|
||||
return LARDON3D_FEATURE_EXTRACT_ERROR;
|
||||
}
|
||||
size_t count = points.size();
|
||||
Lardon3DFeatureKeypoint *keypoints =
|
||||
count == 0 ? nullptr
|
||||
: static_cast<Lardon3DFeatureKeypoint *>(
|
||||
std::calloc(count, sizeof(Lardon3DFeatureKeypoint)));
|
||||
unsigned char *bytes = count == 0 ? nullptr
|
||||
: static_cast<unsigned char *>(std::malloc(
|
||||
count * LARDON3D_FEATURE_DESCRIPTOR_DIMENSION));
|
||||
if (count > 0 && (!keypoints || !bytes)) {
|
||||
std::free(keypoints);
|
||||
std::free(bytes);
|
||||
return LARDON3D_FEATURE_EXTRACT_OUT_OF_MEMORY;
|
||||
}
|
||||
for (size_t index = 0; index < count; ++index) {
|
||||
const cv::KeyPoint &point = points[index];
|
||||
if (!std::isfinite(point.pt.x) || !std::isfinite(point.pt.y) || !std::isfinite(point.size) ||
|
||||
!std::isfinite(point.angle) || !std::isfinite(point.response) || point.pt.x < 0.0F ||
|
||||
point.pt.y < 0.0F || point.pt.x >= static_cast<float>(image.cols) ||
|
||||
point.pt.y >= static_cast<float>(image.rows) || point.size <= 0.0F ||
|
||||
point.angle < 0.0F || point.angle >= 360.0F) {
|
||||
std::free(keypoints);
|
||||
std::free(bytes);
|
||||
return LARDON3D_FEATURE_EXTRACT_ERROR;
|
||||
}
|
||||
keypoints[index] = {point.pt.x, point.pt.y, point.size,
|
||||
point.angle, point.response, point.octave};
|
||||
std::memcpy(bytes + index * LARDON3D_FEATURE_DESCRIPTOR_DIMENSION,
|
||||
descriptors.ptr(static_cast<int>(index)), LARDON3D_FEATURE_DESCRIPTOR_DIMENSION);
|
||||
}
|
||||
features->image_width = static_cast<uint32_t>(image.cols);
|
||||
features->image_height = static_cast<uint32_t>(image.rows);
|
||||
features->feature_count = static_cast<uint32_t>(count);
|
||||
features->keypoints = keypoints;
|
||||
features->descriptors = bytes;
|
||||
return LARDON3D_FEATURE_EXTRACT_OK;
|
||||
} catch (const cv::Exception &) {
|
||||
return LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID;
|
||||
} catch (const std::bad_alloc &) {
|
||||
return LARDON3D_FEATURE_EXTRACT_OUT_OF_MEMORY;
|
||||
} catch (...) {
|
||||
return LARDON3D_FEATURE_EXTRACT_ERROR;
|
||||
}
|
||||
}
|
||||
575
src/feature_store.c
Normal file
575
src/feature_store.c
Normal file
|
|
@ -0,0 +1,575 @@
|
|||
#include <errno.h>
|
||||
#include <fcntl.h>
|
||||
#include <float.h>
|
||||
#include <math.h>
|
||||
#include <openssl/evp.h>
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/stat.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <lardon3d/feature_store.h>
|
||||
|
||||
static const unsigned char feature_magic[8] = {'L', '3', 'D', 'F', 'E', 'A', 'T', 0};
|
||||
_Static_assert(sizeof(float) == 4 && FLT_RADIX == 2 && FLT_MANT_DIG == 24 && FLT_MAX_EXP == 128,
|
||||
"Feature File v1 exige IEEE-754 binary32.");
|
||||
|
||||
struct Lardon3DFeatureReader {
|
||||
int descriptor;
|
||||
uint64_t keypoint_offset;
|
||||
uint64_t descriptor_offset;
|
||||
Lardon3DFeatureFileMetadata metadata;
|
||||
};
|
||||
|
||||
static void put_u32(unsigned char *p, uint32_t v) {
|
||||
p[0] = (unsigned char)v;
|
||||
p[1] = (unsigned char)(v >> 8);
|
||||
p[2] = (unsigned char)(v >> 16);
|
||||
p[3] = (unsigned char)(v >> 24);
|
||||
}
|
||||
static void put_u64(unsigned char *p, uint64_t v) {
|
||||
for (unsigned int i = 0; i < 8; ++i) {
|
||||
p[i] = (unsigned char)(v >> (8U * i));
|
||||
}
|
||||
}
|
||||
static uint32_t get_u32(const unsigned char *p) {
|
||||
return (uint32_t)p[0] | ((uint32_t)p[1] << 8) | ((uint32_t)p[2] << 16) | ((uint32_t)p[3] << 24);
|
||||
}
|
||||
static uint64_t get_u64(const unsigned char *p) {
|
||||
uint64_t v = 0;
|
||||
for (unsigned int i = 0; i < 8; ++i) {
|
||||
v |= (uint64_t)p[i] << (8U * i);
|
||||
}
|
||||
return v;
|
||||
}
|
||||
static void put_float(unsigned char *p, float value) {
|
||||
uint32_t bits = 0;
|
||||
memcpy(&bits, &value, sizeof(bits));
|
||||
put_u32(p, bits);
|
||||
}
|
||||
static float get_float(const unsigned char *p) {
|
||||
uint32_t bits = get_u32(p);
|
||||
float value = 0;
|
||||
memcpy(&value, &bits, sizeof(value));
|
||||
return value;
|
||||
}
|
||||
|
||||
static bool write_all(int fd, const void *data, size_t size) {
|
||||
const unsigned char *bytes = data;
|
||||
size_t used = 0;
|
||||
while (used < size) {
|
||||
ssize_t n = write(fd, bytes + used, size - used);
|
||||
if (n < 0 && errno == EINTR) {
|
||||
continue;
|
||||
}
|
||||
if (n <= 0) {
|
||||
return false;
|
||||
}
|
||||
used += (size_t)n;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
static bool read_exact(int fd, void *data, size_t size, off_t offset) {
|
||||
unsigned char *bytes = data;
|
||||
size_t used = 0;
|
||||
while (used < size) {
|
||||
ssize_t n = pread(fd, bytes + used, size - used, offset + (off_t)used);
|
||||
if (n < 0 && errno == EINTR) {
|
||||
continue;
|
||||
}
|
||||
if (n <= 0) {
|
||||
return false;
|
||||
}
|
||||
used += (size_t)n;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
static bool sha256_fd(int fd, unsigned char output[32]) {
|
||||
EVP_MD_CTX *context = EVP_MD_CTX_new();
|
||||
if (!context) {
|
||||
return false;
|
||||
}
|
||||
bool ok = EVP_DigestInit_ex(context, EVP_sha256(), NULL) == 1;
|
||||
unsigned char buffer[65536];
|
||||
off_t offset = 0;
|
||||
while (ok) {
|
||||
ssize_t n = pread(fd, buffer, sizeof(buffer), offset);
|
||||
if (n < 0 && errno == EINTR) {
|
||||
continue;
|
||||
}
|
||||
if (n < 0) {
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
if (n == 0) {
|
||||
break;
|
||||
}
|
||||
ok = EVP_DigestUpdate(context, buffer, (size_t)n) == 1;
|
||||
offset += (off_t)n;
|
||||
}
|
||||
unsigned int length = 0;
|
||||
ok = ok && EVP_DigestFinal_ex(context, output, &length) == 1 && length == 32;
|
||||
EVP_MD_CTX_free(context);
|
||||
return ok;
|
||||
}
|
||||
|
||||
static void hex_sha(const unsigned char hash[32], char text[65]) {
|
||||
static const char digits[] = "0123456789abcdef";
|
||||
for (size_t i = 0; i < 32; ++i) {
|
||||
text[2 * i] = digits[hash[i] >> 4];
|
||||
text[2 * i + 1] = digits[hash[i] & 15];
|
||||
}
|
||||
text[64] = '\0';
|
||||
}
|
||||
|
||||
static bool canonical_feature_path(const unsigned char hash[32], const char *path) {
|
||||
if (!hash || !path) {
|
||||
return false;
|
||||
}
|
||||
char hex[65];
|
||||
hex_sha(hash, hex);
|
||||
char expected[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
||||
int written = snprintf(expected, sizeof(expected), "assets/features/%.2s/%s", hex, hex);
|
||||
return written > 0 && (size_t)written < sizeof(expected) && strcmp(path, expected) == 0;
|
||||
}
|
||||
|
||||
static bool join(char output[PATH_MAX], const char *a, const char *b) {
|
||||
int n = snprintf(output, PATH_MAX, "%s/%s", a, b);
|
||||
return n > 0 && (size_t)n < PATH_MAX;
|
||||
}
|
||||
|
||||
static bool ensure_directory(const char *path) {
|
||||
if (mkdir(path, 0755) == 0) {
|
||||
return true;
|
||||
}
|
||||
if (errno != EEXIST) {
|
||||
return false;
|
||||
}
|
||||
struct stat info;
|
||||
return lstat(path, &info) == 0 && S_ISDIR(info.st_mode) && !S_ISLNK(info.st_mode);
|
||||
}
|
||||
|
||||
static bool sync_directory(const char *path) {
|
||||
int fd = open(path, O_RDONLY | O_DIRECTORY | O_CLOEXEC);
|
||||
if (fd < 0) {
|
||||
return false;
|
||||
}
|
||||
bool ok = fsync(fd) == 0;
|
||||
if (close(fd) != 0) {
|
||||
ok = false;
|
||||
}
|
||||
return ok;
|
||||
}
|
||||
|
||||
static bool sync_feature_asset_directory(const char *project_path,
|
||||
const Lardon3DProjectDbFeatureSet *feature_set) {
|
||||
char relative[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
||||
int written = snprintf(relative, sizeof(relative), "%s", feature_set->asset.path);
|
||||
if (written <= 0 || (size_t)written >= sizeof(relative)) {
|
||||
return false;
|
||||
}
|
||||
char *separator = strrchr(relative, '/');
|
||||
if (!separator) {
|
||||
return false;
|
||||
}
|
||||
*separator = '\0';
|
||||
char directory[PATH_MAX];
|
||||
return join(directory, project_path, relative) && sync_directory(directory);
|
||||
}
|
||||
|
||||
static bool valid_features(const Lardon3DExtractedFeatures *features) {
|
||||
if (!features || features->feature_count > LARDON3D_FEATURE_MAX_FEATURES ||
|
||||
features->image_width == 0 || features->image_height == 0 ||
|
||||
(features->feature_count > 0 && (!features->keypoints || !features->descriptors))) {
|
||||
return false;
|
||||
}
|
||||
for (uint32_t i = 0; i < features->feature_count; ++i) {
|
||||
const Lardon3DFeatureKeypoint *p = &features->keypoints[i];
|
||||
if (!isfinite(p->x) || !isfinite(p->y) || !isfinite(p->size) || !isfinite(p->angle_degrees) ||
|
||||
!isfinite(p->response) || p->x < 0 || p->y < 0 || p->x >= (float)features->image_width ||
|
||||
p->y >= (float)features->image_height || p->size <= 0 || p->angle_degrees < 0 ||
|
||||
p->angle_degrees >= 360 || p->octave < 0 || p->octave >= 16) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
static bool write_feature_file(int fd, const Lardon3DProjectDbImageAsset *image_asset,
|
||||
const unsigned char fingerprint[32],
|
||||
const Lardon3DExtractedFeatures *features, uint64_t *file_size) {
|
||||
uint64_t kp_bytes = (uint64_t)features->feature_count * LARDON3D_FEATURE_KEYPOINT_RECORD_SIZE;
|
||||
uint64_t descriptor_bytes =
|
||||
(uint64_t)features->feature_count * LARDON3D_FEATURE_DESCRIPTOR_DIMENSION;
|
||||
uint64_t descriptor_offset = LARDON3D_FEATURE_FILE_HEADER_SIZE + kp_bytes;
|
||||
uint64_t total = descriptor_offset + descriptor_bytes;
|
||||
if (total > LARDON3D_FEATURE_FILE_MAX_SIZE) {
|
||||
return false;
|
||||
}
|
||||
unsigned char header[LARDON3D_FEATURE_FILE_HEADER_SIZE] = {0};
|
||||
memcpy(header, feature_magic, 8);
|
||||
put_u32(header + 8, LARDON3D_FEATURE_FILE_VERSION);
|
||||
put_u32(header + 12, LARDON3D_FEATURE_FILE_HEADER_SIZE);
|
||||
put_u32(header + 16, features->feature_count);
|
||||
put_u32(header + 20, LARDON3D_FEATURE_DESCRIPTOR_DIMENSION);
|
||||
put_u32(header + 24, LARDON3D_FEATURE_DESCRIPTOR_U8);
|
||||
put_u32(header + 28, LARDON3D_FEATURE_KEYPOINT_RECORD_SIZE);
|
||||
put_u32(header + 32, features->image_width);
|
||||
put_u32(header + 36, features->image_height);
|
||||
put_u64(header + 40, LARDON3D_FEATURE_FILE_HEADER_SIZE);
|
||||
put_u64(header + 48, descriptor_offset);
|
||||
put_u64(header + 56, total);
|
||||
memcpy(header + 64, image_asset->sha256, 32);
|
||||
memcpy(header + 96, fingerprint, 32);
|
||||
memcpy(header + 128, LARDON3D_FEATURE_EXTRACTOR_KIND, 4);
|
||||
put_u32(header + 144, LARDON3D_FEATURE_EXTRACTOR_VERSION);
|
||||
if (!write_all(fd, header, sizeof(header))) {
|
||||
return false;
|
||||
}
|
||||
unsigned char record[LARDON3D_FEATURE_KEYPOINT_RECORD_SIZE];
|
||||
for (uint32_t i = 0; i < features->feature_count; ++i) {
|
||||
const Lardon3DFeatureKeypoint *p = &features->keypoints[i];
|
||||
put_float(record, p->x);
|
||||
put_float(record + 4, p->y);
|
||||
put_float(record + 8, p->size);
|
||||
put_float(record + 12, p->angle_degrees);
|
||||
put_float(record + 16, p->response);
|
||||
put_u32(record + 20, (uint32_t)p->octave);
|
||||
if (!write_all(fd, record, sizeof(record))) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
if (descriptor_bytes > 0 && !write_all(fd, features->descriptors, (size_t)descriptor_bytes)) {
|
||||
return false;
|
||||
}
|
||||
*file_size = total;
|
||||
return true;
|
||||
}
|
||||
|
||||
static Lardon3DFeatureStoreResult
|
||||
reuse_existing_feature_set(Lardon3DAppState *state, uint64_t image_id, uint64_t producer_task_id,
|
||||
const unsigned char fingerprint[32],
|
||||
Lardon3DProjectDbFeatureSet *feature_set) {
|
||||
if (feature_set->asset.durability == LARDON3D_DB_FEATURE_ASSET_DURABLE) {
|
||||
return LARDON3D_FEATURE_STORE_ALREADY_PRESENT;
|
||||
}
|
||||
Lardon3DFeatureReader *reader = NULL;
|
||||
Lardon3DFeatureFileMetadata metadata;
|
||||
Lardon3DFeatureStoreResult validation =
|
||||
lardon3d_feature_reader_open(state->project_path, feature_set, &reader, &metadata);
|
||||
lardon3d_feature_reader_close(reader);
|
||||
if (validation != LARDON3D_FEATURE_STORE_OK ||
|
||||
!sync_feature_asset_directory(state->project_path, feature_set)) {
|
||||
return LARDON3D_FEATURE_STORE_PUBLISHED_NOT_DURABLE;
|
||||
}
|
||||
int64_t now = (int64_t)time(NULL);
|
||||
if (now < 0) {
|
||||
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||
}
|
||||
Lardon3DProjectDbResult promoted = lardon3d_project_db_register_feature_set(
|
||||
state->project_db, image_id, LARDON3D_FEATURE_EXTRACTOR_KIND,
|
||||
LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, feature_set->source_image_sha256,
|
||||
feature_set->feature_count, feature_set->descriptor_type, feature_set->descriptor_dimension,
|
||||
feature_set->asset.sha256, feature_set->asset.path, feature_set->asset.size_bytes,
|
||||
LARDON3D_DB_FEATURE_ASSET_DURABLE, producer_task_id, now, feature_set);
|
||||
return promoted == LARDON3D_PROJECT_DB_OK ? LARDON3D_FEATURE_STORE_ALREADY_PRESENT
|
||||
: LARDON3D_FEATURE_STORE_DB_ERROR;
|
||||
}
|
||||
|
||||
static Lardon3DFeatureStoreResult register_published_feature_set(
|
||||
Lardon3DAppState *state, uint64_t image_id, uint64_t producer_task_id,
|
||||
const Lardon3DProjectDbImageAsset *image_asset, const unsigned char fingerprint[32],
|
||||
const Lardon3DExtractedFeatures *features, const unsigned char file_hash[32],
|
||||
const char *relative_path, uint64_t file_size, bool durable,
|
||||
Lardon3DProjectDbFeatureSet *feature_set) {
|
||||
int64_t now = (int64_t)time(NULL);
|
||||
if (now < 0) {
|
||||
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||
}
|
||||
Lardon3DProjectDbResult result = lardon3d_project_db_register_feature_set(
|
||||
state->project_db, image_id, LARDON3D_FEATURE_EXTRACTOR_KIND,
|
||||
LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, image_asset->sha256, features->feature_count,
|
||||
LARDON3D_FEATURE_DESCRIPTOR_U8, LARDON3D_FEATURE_DESCRIPTOR_DIMENSION, file_hash,
|
||||
relative_path, file_size,
|
||||
durable ? LARDON3D_DB_FEATURE_ASSET_DURABLE : LARDON3D_DB_FEATURE_ASSET_PUBLISHED_NOT_DURABLE,
|
||||
producer_task_id, now, feature_set);
|
||||
if (result != LARDON3D_PROJECT_DB_OK) {
|
||||
return result == LARDON3D_PROJECT_DB_BUSY ? LARDON3D_FEATURE_STORE_DB_BUSY
|
||||
: LARDON3D_FEATURE_STORE_DB_ERROR;
|
||||
}
|
||||
return durable ? LARDON3D_FEATURE_STORE_OK : LARDON3D_FEATURE_STORE_PUBLISHED_NOT_DURABLE;
|
||||
}
|
||||
|
||||
Lardon3DFeatureStoreResult lardon3d_feature_store_publish(
|
||||
Lardon3DAppState *state, uint64_t image_id, uint64_t producer_task_id,
|
||||
const Lardon3DFeatureExtractorParameters *parameters, const Lardon3DExtractedFeatures *features,
|
||||
Lardon3DProjectDbFeatureSet *feature_set) {
|
||||
if (feature_set) {
|
||||
memset(feature_set, 0, sizeof(*feature_set));
|
||||
}
|
||||
if (!state || !state->project_loaded || !state->project_db || !feature_set || image_id == 0 ||
|
||||
!lardon3d_feature_extractor_parameters_valid(parameters) || !valid_features(features)) {
|
||||
return LARDON3D_FEATURE_STORE_INVALID_ARGUMENT;
|
||||
}
|
||||
unsigned char fingerprint[32];
|
||||
lardon3d_feature_extractor_parameter_fingerprint(parameters, fingerprint);
|
||||
Lardon3DProjectDbResult found = lardon3d_project_db_find_feature_set(
|
||||
state->project_db, image_id, LARDON3D_FEATURE_EXTRACTOR_KIND,
|
||||
LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, feature_set);
|
||||
if (found == LARDON3D_PROJECT_DB_OK) {
|
||||
return reuse_existing_feature_set(state, image_id, producer_task_id, fingerprint, feature_set);
|
||||
}
|
||||
if (found != LARDON3D_PROJECT_DB_NOT_FOUND) {
|
||||
return found == LARDON3D_PROJECT_DB_BUSY ? LARDON3D_FEATURE_STORE_DB_BUSY
|
||||
: LARDON3D_FEATURE_STORE_DB_ERROR;
|
||||
}
|
||||
Lardon3DProjectDbImage image;
|
||||
Lardon3DProjectDbImageAsset image_asset;
|
||||
if (lardon3d_project_db_load_image(state->project_db, image_id, &image, &image_asset) !=
|
||||
LARDON3D_PROJECT_DB_OK) {
|
||||
return LARDON3D_FEATURE_STORE_NOT_FOUND;
|
||||
}
|
||||
char assets[PATH_MAX], features_dir[PATH_MAX];
|
||||
if (!join(assets, state->project_path, "assets") || !ensure_directory(assets) ||
|
||||
!join(features_dir, assets, "features") || !ensure_directory(features_dir)) {
|
||||
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||
}
|
||||
char temporary[PATH_MAX];
|
||||
int n = snprintf(temporary, sizeof(temporary), "%s/.feature-XXXXXX", features_dir);
|
||||
if (n <= 0 || (size_t)n >= sizeof(temporary)) {
|
||||
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||
}
|
||||
int fd = mkstemp(temporary);
|
||||
if (fd < 0) {
|
||||
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||
}
|
||||
uint64_t file_size = 0;
|
||||
bool written =
|
||||
write_feature_file(fd, &image_asset, fingerprint, features, &file_size) && fsync(fd) == 0;
|
||||
unsigned char file_hash[32];
|
||||
if (written) {
|
||||
written = sha256_fd(fd, file_hash);
|
||||
}
|
||||
if (close(fd) != 0) {
|
||||
written = false;
|
||||
}
|
||||
if (!written) {
|
||||
unlink(temporary);
|
||||
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||
}
|
||||
char hex[65];
|
||||
hex_sha(file_hash, hex);
|
||||
char prefix[PATH_MAX];
|
||||
n = snprintf(prefix, sizeof(prefix), "%s/%.2s", features_dir, hex);
|
||||
if (n <= 0 || (size_t)n >= sizeof(prefix) || !ensure_directory(prefix)) {
|
||||
unlink(temporary);
|
||||
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||
}
|
||||
char relative[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
||||
n = snprintf(relative, sizeof(relative), "assets/features/%.2s/%s", hex, hex);
|
||||
char final[PATH_MAX];
|
||||
if (n <= 0 || (size_t)n >= sizeof(relative) || !join(final, state->project_path, relative)) {
|
||||
unlink(temporary);
|
||||
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||
}
|
||||
if (link(temporary, final) != 0) {
|
||||
if (errno != EEXIST) {
|
||||
unlink(temporary);
|
||||
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||
}
|
||||
int existing = open(final, O_RDONLY | O_NOFOLLOW | O_CLOEXEC);
|
||||
struct stat info;
|
||||
unsigned char existing_hash[32];
|
||||
bool same = existing >= 0 && fstat(existing, &info) == 0 && S_ISREG(info.st_mode) &&
|
||||
info.st_size >= 0 && (uint64_t)info.st_size == file_size &&
|
||||
sha256_fd(existing, existing_hash) && memcmp(existing_hash, file_hash, 32) == 0;
|
||||
if (existing >= 0) {
|
||||
(void)close(existing);
|
||||
}
|
||||
if (!same) {
|
||||
unlink(temporary);
|
||||
return LARDON3D_FEATURE_STORE_CORRUPT;
|
||||
}
|
||||
}
|
||||
unlink(temporary);
|
||||
bool durable = sync_directory(prefix);
|
||||
#ifdef LARDON3D_FEATURE_STORE_TESTING
|
||||
const char *fail_sync = getenv("LARDON3D_TEST_FEATURE_FAIL_DIRECTORY_SYNC");
|
||||
if (fail_sync && strcmp(fail_sync, "1") == 0) {
|
||||
durable = false;
|
||||
}
|
||||
#endif
|
||||
return register_published_feature_set(state, image_id, producer_task_id, &image_asset,
|
||||
fingerprint, features, file_hash, relative, file_size,
|
||||
durable, feature_set);
|
||||
}
|
||||
|
||||
static Lardon3DFeatureStoreResult validate_header(const unsigned char h[160], uint64_t actual,
|
||||
Lardon3DFeatureFileMetadata *m, uint64_t *ko,
|
||||
uint64_t *doff) {
|
||||
if (memcmp(h, feature_magic, 8) != 0) {
|
||||
return LARDON3D_FEATURE_STORE_INVALID;
|
||||
}
|
||||
uint32_t version = get_u32(h + 8);
|
||||
if (version != 1) {
|
||||
return version > 1 ? LARDON3D_FEATURE_STORE_UNSUPPORTED_VERSION
|
||||
: LARDON3D_FEATURE_STORE_INVALID;
|
||||
}
|
||||
uint32_t count = get_u32(h + 16), dim = get_u32(h + 20), type = get_u32(h + 24),
|
||||
record = get_u32(h + 28), width = get_u32(h + 32), height = get_u32(h + 36);
|
||||
uint64_t key = get_u64(h + 40), desc = get_u64(h + 48), declared = get_u64(h + 56);
|
||||
uint64_t kb = (uint64_t)count * record, db = (uint64_t)count * dim;
|
||||
if (get_u32(h + 12) != 160 || count > LARDON3D_FEATURE_MAX_FEATURES || dim != 32 ||
|
||||
type != LARDON3D_FEATURE_DESCRIPTOR_U8 || record != 24 || width == 0 || height == 0 ||
|
||||
key != 160 || desc != key + kb || declared != desc + db || declared != actual ||
|
||||
actual > LARDON3D_FEATURE_FILE_MAX_SIZE || memcmp(h + 128, "orb\0", 4) != 0 ||
|
||||
get_u32(h + 144) != 1) {
|
||||
return LARDON3D_FEATURE_STORE_CORRUPT;
|
||||
}
|
||||
for (size_t i = 132; i < 144; ++i) {
|
||||
if (h[i] != 0) {
|
||||
return LARDON3D_FEATURE_STORE_CORRUPT;
|
||||
}
|
||||
}
|
||||
for (size_t i = 148; i < 160; ++i) {
|
||||
if (h[i] != 0) {
|
||||
return LARDON3D_FEATURE_STORE_CORRUPT;
|
||||
}
|
||||
}
|
||||
*m = (Lardon3DFeatureFileMetadata){.format_version = version,
|
||||
.extractor_version = get_u32(h + 144),
|
||||
.feature_count = count,
|
||||
.descriptor_dimension = dim,
|
||||
.descriptor_type = (Lardon3DFeatureDescriptorType)type,
|
||||
.image_width = width,
|
||||
.image_height = height};
|
||||
memcpy(m->source_image_sha256, h + 64, 32);
|
||||
memcpy(m->parameter_fingerprint, h + 96, 32);
|
||||
memcpy(m->extractor_kind, h + 128, 4);
|
||||
*ko = key;
|
||||
*doff = desc;
|
||||
return LARDON3D_FEATURE_STORE_OK;
|
||||
}
|
||||
|
||||
Lardon3DFeatureStoreResult lardon3d_feature_reader_open(const char *project_path,
|
||||
const Lardon3DProjectDbFeatureSet *set,
|
||||
Lardon3DFeatureReader **out,
|
||||
Lardon3DFeatureFileMetadata *metadata) {
|
||||
if (out) {
|
||||
*out = NULL;
|
||||
}
|
||||
if (!project_path || !set || !out || !metadata ||
|
||||
!canonical_feature_path(set->asset.sha256, set->asset.path)) {
|
||||
return LARDON3D_FEATURE_STORE_INVALID_ARGUMENT;
|
||||
}
|
||||
char path[PATH_MAX];
|
||||
if (!join(path, project_path, set->asset.path)) {
|
||||
return LARDON3D_FEATURE_STORE_INVALID_ARGUMENT;
|
||||
}
|
||||
int fd = open(path, O_RDONLY | O_NOFOLLOW | O_CLOEXEC);
|
||||
if (fd < 0) {
|
||||
return errno == ENOENT ? LARDON3D_FEATURE_STORE_NOT_FOUND : LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||
}
|
||||
struct stat st;
|
||||
if (fstat(fd, &st) != 0 || !S_ISREG(st.st_mode) || st.st_size < 160 ||
|
||||
(uint64_t)st.st_size > LARDON3D_FEATURE_FILE_MAX_SIZE ||
|
||||
(uint64_t)st.st_size != set->asset.size_bytes) {
|
||||
close(fd);
|
||||
return LARDON3D_FEATURE_STORE_CORRUPT;
|
||||
}
|
||||
unsigned char header[160];
|
||||
if (!read_exact(fd, header, sizeof(header), 0)) {
|
||||
close(fd);
|
||||
return LARDON3D_FEATURE_STORE_CORRUPT;
|
||||
}
|
||||
Lardon3DFeatureReader *reader = calloc(1, sizeof(*reader));
|
||||
if (!reader) {
|
||||
close(fd);
|
||||
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||
}
|
||||
Lardon3DFeatureStoreResult result =
|
||||
validate_header(header, (uint64_t)st.st_size, &reader->metadata, &reader->keypoint_offset,
|
||||
&reader->descriptor_offset);
|
||||
if (result != LARDON3D_FEATURE_STORE_OK) {
|
||||
close(fd);
|
||||
free(reader);
|
||||
return result;
|
||||
}
|
||||
unsigned char hash[32];
|
||||
if (!sha256_fd(fd, hash) || memcmp(hash, set->asset.sha256, 32) != 0) {
|
||||
close(fd);
|
||||
free(reader);
|
||||
return LARDON3D_FEATURE_STORE_CORRUPT;
|
||||
}
|
||||
if (reader->metadata.feature_count != set->feature_count ||
|
||||
strcmp(reader->metadata.extractor_kind, set->extractor_kind) != 0 ||
|
||||
reader->metadata.extractor_version != set->extractor_version ||
|
||||
reader->metadata.descriptor_dimension != set->descriptor_dimension ||
|
||||
reader->metadata.descriptor_type != (Lardon3DFeatureDescriptorType)set->descriptor_type ||
|
||||
memcmp(reader->metadata.source_image_sha256, set->source_image_sha256, 32) != 0 ||
|
||||
memcmp(reader->metadata.parameter_fingerprint, set->parameter_fingerprint, 32) != 0) {
|
||||
close(fd);
|
||||
free(reader);
|
||||
return LARDON3D_FEATURE_STORE_CORRUPT;
|
||||
}
|
||||
reader->descriptor = fd;
|
||||
*metadata = reader->metadata;
|
||||
*out = reader;
|
||||
return LARDON3D_FEATURE_STORE_OK;
|
||||
}
|
||||
|
||||
void lardon3d_feature_reader_close(Lardon3DFeatureReader *r) {
|
||||
if (!r) {
|
||||
return;
|
||||
}
|
||||
(void)close(r->descriptor);
|
||||
free(r);
|
||||
}
|
||||
|
||||
Lardon3DFeatureStoreResult lardon3d_feature_reader_keypoints(Lardon3DFeatureReader *r,
|
||||
uint32_t start,
|
||||
Lardon3DFeatureKeypoint *out,
|
||||
size_t capacity) {
|
||||
if (!r || (!out && capacity > 0) || capacity > LARDON3D_FEATURE_READER_RANGE_MAX ||
|
||||
start > r->metadata.feature_count || capacity > (size_t)(r->metadata.feature_count - start)) {
|
||||
return LARDON3D_FEATURE_STORE_INVALID_ARGUMENT;
|
||||
}
|
||||
unsigned char record[24];
|
||||
for (size_t i = 0; i < capacity; ++i) {
|
||||
uint64_t index = (uint64_t)start + i;
|
||||
if (!read_exact(r->descriptor, record, 24, (off_t)(r->keypoint_offset + index * 24))) {
|
||||
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||
}
|
||||
out[i] = (Lardon3DFeatureKeypoint){get_float(record), get_float(record + 4),
|
||||
get_float(record + 8), get_float(record + 12),
|
||||
get_float(record + 16), (int32_t)get_u32(record + 20)};
|
||||
if (!isfinite(out[i].x) || !isfinite(out[i].y) || !isfinite(out[i].size) ||
|
||||
!isfinite(out[i].angle_degrees) || !isfinite(out[i].response)) {
|
||||
return LARDON3D_FEATURE_STORE_CORRUPT;
|
||||
}
|
||||
}
|
||||
return LARDON3D_FEATURE_STORE_OK;
|
||||
}
|
||||
|
||||
Lardon3DFeatureStoreResult lardon3d_feature_reader_descriptors(Lardon3DFeatureReader *r,
|
||||
uint32_t start, unsigned char *out,
|
||||
size_t capacity, size_t bytes) {
|
||||
if (!r || (!out && capacity > 0) || capacity > LARDON3D_FEATURE_READER_RANGE_MAX ||
|
||||
start > r->metadata.feature_count || capacity > (size_t)(r->metadata.feature_count - start) ||
|
||||
capacity > SIZE_MAX / r->metadata.descriptor_dimension ||
|
||||
bytes < capacity * r->metadata.descriptor_dimension) {
|
||||
return LARDON3D_FEATURE_STORE_INVALID_ARGUMENT;
|
||||
}
|
||||
size_t amount = capacity * r->metadata.descriptor_dimension;
|
||||
if (amount && !read_exact(r->descriptor, out, amount,
|
||||
(off_t)(r->descriptor_offset +
|
||||
(uint64_t)start * r->metadata.descriptor_dimension))) {
|
||||
return LARDON3D_FEATURE_STORE_IO_ERROR;
|
||||
}
|
||||
return LARDON3D_FEATURE_STORE_OK;
|
||||
}
|
||||
325
src/feature_task.c
Normal file
325
src/feature_task.c
Normal file
|
|
@ -0,0 +1,325 @@
|
|||
#include <errno.h>
|
||||
#include <fcntl.h>
|
||||
#include <openssl/evp.h>
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/stat.h>
|
||||
#include <time.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <lardon3d/feature_store.h>
|
||||
#include <lardon3d/feature_task.h>
|
||||
#include <lardon3d/project.h>
|
||||
#include <lardon3d/task_queue.h>
|
||||
|
||||
typedef struct {
|
||||
char project_path[PATH_MAX];
|
||||
Lardon3DProjectDb *database;
|
||||
Lardon3DResourceGovernor *governor;
|
||||
Lardon3DProjectDbFeatureExtractTask parameters;
|
||||
} Lardon3DFeatureTaskContext;
|
||||
|
||||
static void destroy_context(void *userdata) { free(userdata); }
|
||||
|
||||
static void runtime_state(const Lardon3DFeatureTaskContext *context, Lardon3DAppState *state) {
|
||||
lardon3d_app_state_init(state);
|
||||
state->project_loaded = true;
|
||||
state->project_db = context->database;
|
||||
state->resource_governor = context->governor;
|
||||
(void)snprintf(state->project_path, sizeof(state->project_path), "%s", context->project_path);
|
||||
}
|
||||
|
||||
static bool file_hash(const char *path, unsigned char output[32]) {
|
||||
int fd = open(path, O_RDONLY | O_NOFOLLOW | O_CLOEXEC);
|
||||
if (fd < 0) {
|
||||
return false;
|
||||
}
|
||||
struct stat st;
|
||||
if (fstat(fd, &st) != 0 || !S_ISREG(st.st_mode)) {
|
||||
close(fd);
|
||||
return false;
|
||||
}
|
||||
EVP_MD_CTX *ctx = EVP_MD_CTX_new();
|
||||
bool ok = ctx && EVP_DigestInit_ex(ctx, EVP_sha256(), NULL) == 1;
|
||||
unsigned char buffer[65536];
|
||||
while (ok) {
|
||||
ssize_t n = read(fd, buffer, sizeof(buffer));
|
||||
if (n < 0 && errno == EINTR) {
|
||||
continue;
|
||||
}
|
||||
if (n < 0) {
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
if (n == 0) {
|
||||
break;
|
||||
}
|
||||
ok = EVP_DigestUpdate(ctx, buffer, (size_t)n) == 1;
|
||||
}
|
||||
unsigned int size = 0;
|
||||
ok = ok && EVP_DigestFinal_ex(ctx, output, &size) == 1 && size == 32;
|
||||
if (ctx) {
|
||||
EVP_MD_CTX_free(ctx);
|
||||
}
|
||||
if (close(fd) != 0) {
|
||||
ok = false;
|
||||
}
|
||||
return ok;
|
||||
}
|
||||
|
||||
static uint64_t elapsed_ns(struct timespec a, struct timespec b) {
|
||||
uint64_t s = b.tv_sec >= a.tv_sec ? (uint64_t)(b.tv_sec - a.tv_sec) : 0;
|
||||
long n = b.tv_nsec - a.tv_nsec;
|
||||
if (n < 0 && s) {
|
||||
--s;
|
||||
n += 1000000000L;
|
||||
}
|
||||
return s <= UINT64_MAX / 1000000000ULL ? s * 1000000000ULL + (uint64_t)n : UINT64_MAX;
|
||||
}
|
||||
|
||||
static void finished(const Lardon3DTask *task, void *userdata) {
|
||||
#ifdef LARDON3D_FEATURE_TASK_TESTING
|
||||
const char *skip = getenv("LARDON3D_TEST_FEATURE_SKIP_FINISHED_CHECKPOINT");
|
||||
if (skip && strcmp(skip, "1") == 0) {
|
||||
return;
|
||||
}
|
||||
#endif
|
||||
Lardon3DFeatureTaskContext *context = userdata;
|
||||
Lardon3DAppState state;
|
||||
runtime_state(context, &state);
|
||||
(void)lardon3d_project_checkpoint_feature_extract_task(&state, task, &context->parameters);
|
||||
}
|
||||
|
||||
static Lardon3DProjectDbResult find_valid_feature_set(const Lardon3DFeatureTaskContext *context,
|
||||
const unsigned char fingerprint[32]) {
|
||||
Lardon3DProjectDbFeatureSet existing;
|
||||
Lardon3DProjectDbResult found = lardon3d_project_db_find_feature_set(
|
||||
context->database, context->parameters.image_id, LARDON3D_FEATURE_EXTRACTOR_KIND,
|
||||
LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, &existing);
|
||||
if (found != LARDON3D_PROJECT_DB_OK) {
|
||||
return found;
|
||||
}
|
||||
Lardon3DFeatureReader *reader = NULL;
|
||||
Lardon3DFeatureFileMetadata metadata;
|
||||
Lardon3DFeatureStoreResult valid =
|
||||
lardon3d_feature_reader_open(context->project_path, &existing, &reader, &metadata);
|
||||
lardon3d_feature_reader_close(reader);
|
||||
return valid == LARDON3D_FEATURE_STORE_OK ? LARDON3D_PROJECT_DB_OK : LARDON3D_PROJECT_DB_CORRUPT;
|
||||
}
|
||||
|
||||
static bool load_validated_source(const Lardon3DFeatureTaskContext *context, char path[PATH_MAX]) {
|
||||
Lardon3DProjectDbImage image;
|
||||
Lardon3DProjectDbImageAsset asset;
|
||||
if (lardon3d_project_db_load_image(context->database, context->parameters.image_id, &image,
|
||||
&asset) != LARDON3D_PROJECT_DB_OK) {
|
||||
return false;
|
||||
}
|
||||
int written = snprintf(path, PATH_MAX, "%s/%s", context->project_path, asset.path);
|
||||
if (written <= 0 || (size_t)written >= PATH_MAX) {
|
||||
return false;
|
||||
}
|
||||
unsigned char actual[32];
|
||||
return file_hash(path, actual) && memcmp(actual, asset.sha256, 32) == 0;
|
||||
}
|
||||
|
||||
static bool run(Lardon3DTask *task, void *userdata) {
|
||||
Lardon3DFeatureTaskContext *context = userdata;
|
||||
if (!lardon3d_task_checkpoint(task)) {
|
||||
return false;
|
||||
}
|
||||
struct timespec begin;
|
||||
(void)clock_gettime(CLOCK_MONOTONIC, &begin);
|
||||
unsigned char fingerprint[32];
|
||||
lardon3d_feature_extractor_parameter_fingerprint(
|
||||
&(Lardon3DFeatureExtractorParameters){context->parameters.max_features,
|
||||
context->parameters.pyramid_levels,
|
||||
context->parameters.fast_threshold},
|
||||
fingerprint);
|
||||
Lardon3DProjectDbResult found = find_valid_feature_set(context, fingerprint);
|
||||
if (found == LARDON3D_PROJECT_DB_OK) {
|
||||
return lardon3d_task_set_progress(task, 100, "Features déjà présentes.");
|
||||
}
|
||||
if (found == LARDON3D_PROJECT_DB_CORRUPT) {
|
||||
return lardon3d_task_fail(task, "Feature Set READY absent ou corrompu.");
|
||||
}
|
||||
if (found != LARDON3D_PROJECT_DB_NOT_FOUND) {
|
||||
return lardon3d_task_fail(task, "Recherche Feature Store impossible.");
|
||||
}
|
||||
char path[PATH_MAX];
|
||||
if (!load_validated_source(context, path)) {
|
||||
return lardon3d_task_fail(task, "Asset image absent ou corrompu.");
|
||||
}
|
||||
Lardon3DFeatureExtractorParameters parameters = {context->parameters.max_features,
|
||||
context->parameters.pyramid_levels,
|
||||
context->parameters.fast_threshold};
|
||||
Lardon3DExtractedFeatures features;
|
||||
Lardon3DFeatureExtractResult extracted =
|
||||
lardon3d_feature_extract_orb(path, ¶meters, &features);
|
||||
if (extracted != LARDON3D_FEATURE_EXTRACT_OK) {
|
||||
return lardon3d_task_fail(task, "Décodage ou extraction ORB impossible.");
|
||||
}
|
||||
if (!lardon3d_task_checkpoint(task)) {
|
||||
lardon3d_extracted_features_destroy(&features);
|
||||
return false;
|
||||
}
|
||||
#ifdef LARDON3D_FEATURE_TASK_TESTING
|
||||
const char *pause = getenv("LARDON3D_TEST_FEATURE_PAUSE_BEFORE_PUBLISH");
|
||||
if (pause && strcmp(pause, "1") == 0) {
|
||||
(void)lardon3d_task_pause(task);
|
||||
if (!lardon3d_task_checkpoint(task)) {
|
||||
lardon3d_extracted_features_destroy(&features);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
#endif
|
||||
Lardon3DAppState state;
|
||||
runtime_state(context, &state);
|
||||
Lardon3DProjectDbFeatureSet set;
|
||||
Lardon3DFeatureStoreResult published = lardon3d_feature_store_publish(
|
||||
&state, context->parameters.image_id, lardon3d_task_id(task), ¶meters, &features, &set);
|
||||
lardon3d_extracted_features_destroy(&features);
|
||||
if (published != LARDON3D_FEATURE_STORE_OK &&
|
||||
published != LARDON3D_FEATURE_STORE_ALREADY_PRESENT &&
|
||||
published != LARDON3D_FEATURE_STORE_PUBLISHED_NOT_DURABLE) {
|
||||
return lardon3d_task_fail(task, "Publication Feature Store impossible.");
|
||||
}
|
||||
struct timespec end;
|
||||
(void)clock_gettime(CLOCK_MONOTONIC, &end);
|
||||
(void)lardon3d_resource_governor_record_batch(context->governor, LARDON3D_RESOURCE_TASK_CPU, 1,
|
||||
elapsed_ns(begin, end), 0);
|
||||
return lardon3d_task_set_progress(task, 100, "Feature Set publié.");
|
||||
}
|
||||
|
||||
static Lardon3DFeatureTaskContext *
|
||||
make_context(const Lardon3DTaskReconstructionContext *runtime,
|
||||
const Lardon3DProjectDbFeatureExtractTask *parameters) {
|
||||
if (!runtime || !runtime->project_path || !runtime->project_db || !runtime->resource_governor ||
|
||||
!parameters) {
|
||||
return NULL;
|
||||
}
|
||||
Lardon3DFeatureTaskContext *context = calloc(1, sizeof(*context));
|
||||
if (!context) {
|
||||
return NULL;
|
||||
}
|
||||
int n =
|
||||
snprintf(context->project_path, sizeof(context->project_path), "%s", runtime->project_path);
|
||||
if (n <= 0 || (size_t)n >= sizeof(context->project_path)) {
|
||||
free(context);
|
||||
return NULL;
|
||||
}
|
||||
context->database = runtime->project_db;
|
||||
context->governor = runtime->resource_governor;
|
||||
context->parameters = *parameters;
|
||||
return context;
|
||||
}
|
||||
|
||||
bool lardon3d_feature_extract_reconstruct(const Lardon3DTaskDurableSnapshot *snapshot,
|
||||
void *userdata, Lardon3DTaskKindBinding *binding) {
|
||||
Lardon3DTaskReconstructionContext *runtime = userdata;
|
||||
if (!snapshot || !runtime || !binding) {
|
||||
return false;
|
||||
}
|
||||
Lardon3DProjectDbFeatureExtractTask parameters;
|
||||
if (lardon3d_project_db_load_feature_extract_task(runtime->project_db, snapshot->id,
|
||||
¶meters) != LARDON3D_PROJECT_DB_OK ||
|
||||
strcmp(parameters.extractor_kind, LARDON3D_FEATURE_EXTRACTOR_KIND) != 0 ||
|
||||
parameters.extractor_version != 1) {
|
||||
return false;
|
||||
}
|
||||
unsigned char expected[32];
|
||||
lardon3d_feature_extractor_parameter_fingerprint(
|
||||
&(Lardon3DFeatureExtractorParameters){parameters.max_features, parameters.pyramid_levels,
|
||||
parameters.fast_threshold},
|
||||
expected);
|
||||
if (memcmp(expected, parameters.parameter_fingerprint, 32) != 0) {
|
||||
return false;
|
||||
}
|
||||
Lardon3DFeatureTaskContext *context = make_context(runtime, ¶meters);
|
||||
if (!context) {
|
||||
return false;
|
||||
}
|
||||
*binding = (Lardon3DTaskKindBinding){.callback = run,
|
||||
.userdata = context,
|
||||
.userdata_destroy = destroy_context,
|
||||
.finished_callback = finished,
|
||||
.finished_userdata = context};
|
||||
return true;
|
||||
}
|
||||
|
||||
Lardon3DTask *
|
||||
lardon3d_project_create_feature_extract_task(Lardon3DAppState *state, uint64_t image_id,
|
||||
const Lardon3DFeatureExtractorParameters *parameters,
|
||||
uint64_t *task_id) {
|
||||
if (task_id) {
|
||||
*task_id = 0;
|
||||
}
|
||||
if (!state || !state->project_loaded || !state->project_db || !state->resource_governor ||
|
||||
!parameters || !task_id || !lardon3d_feature_extractor_parameters_valid(parameters)) {
|
||||
return NULL;
|
||||
}
|
||||
Lardon3DProjectDbImage image;
|
||||
Lardon3DProjectDbImageAsset asset;
|
||||
if (lardon3d_project_db_load_image(state->project_db, image_id, &image, &asset) !=
|
||||
LARDON3D_PROJECT_DB_OK) {
|
||||
return NULL;
|
||||
}
|
||||
uint64_t id;
|
||||
if (lardon3d_project_db_allocate_task_id(state->project_db, &id) != LARDON3D_PROJECT_DB_OK) {
|
||||
return NULL;
|
||||
}
|
||||
Lardon3DProjectDbFeatureExtractTask durable = {.task_id = id,
|
||||
.image_id = image_id,
|
||||
.extractor_version = 1,
|
||||
.max_features = parameters->max_features,
|
||||
.pyramid_levels = parameters->pyramid_levels,
|
||||
.fast_threshold = parameters->fast_threshold};
|
||||
snprintf(durable.extractor_kind, sizeof(durable.extractor_kind), "%s",
|
||||
LARDON3D_FEATURE_EXTRACTOR_KIND);
|
||||
lardon3d_feature_extractor_parameter_fingerprint(parameters, durable.parameter_fingerprint);
|
||||
Lardon3DTaskReconstructionContext runtime = {state->project_path, state->project_db,
|
||||
state->resource_governor};
|
||||
Lardon3DFeatureTaskContext *context = make_context(&runtime, &durable);
|
||||
if (!context) {
|
||||
return NULL;
|
||||
}
|
||||
Lardon3DResourceEstimate estimate = {.memory_fixed_bytes = 64ULL * 1024 * 1024,
|
||||
.memory_bytes_per_item = 512ULL * 1024 * 1024,
|
||||
.minimum_batch_size = 1,
|
||||
.maximum_batch_size = 1,
|
||||
.desired_cpu_threads = 1,
|
||||
.desired_io_slots = 1,
|
||||
.task_class = LARDON3D_RESOURCE_TASK_CPU};
|
||||
Lardon3DTask *task = lardon3d_task_create_typed("Extraction de features", &estimate,
|
||||
LARDON3D_FEATURE_EXTRACT_TASK_KIND, 1, run,
|
||||
context, destroy_context);
|
||||
if (!task || !lardon3d_task_assign_id(task, id) ||
|
||||
!lardon3d_task_set_finished_callback(task, finished, context) ||
|
||||
lardon3d_project_checkpoint_feature_extract_task(state, task, &durable) !=
|
||||
LARDON3D_PROJECT_TASK_CHECKPOINT_OK) {
|
||||
lardon3d_task_destroy(task);
|
||||
return NULL;
|
||||
}
|
||||
*task_id = id;
|
||||
return task;
|
||||
}
|
||||
|
||||
bool lardon3d_project_enqueue_feature_extract(Lardon3DAppState *state, uint64_t image_id,
|
||||
const Lardon3DFeatureExtractorParameters *parameters,
|
||||
uint64_t *task_id) {
|
||||
if (!state || !state->task_queue) {
|
||||
return false;
|
||||
}
|
||||
Lardon3DTask *task =
|
||||
lardon3d_project_create_feature_extract_task(state, image_id, parameters, task_id);
|
||||
if (!task) {
|
||||
return false;
|
||||
}
|
||||
if (!lardon3d_task_queue_add(state->task_queue, task, NULL)) {
|
||||
lardon3d_task_destroy(task);
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
586
src/project.c
586
src/project.c
|
|
@ -11,10 +11,10 @@
|
|||
#include <time.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <lardon3d/project.h>
|
||||
#include <lardon3d/image_catalog.h>
|
||||
#include <lardon3d/image_view.h>
|
||||
#include <lardon3d/import_task.h>
|
||||
#include <lardon3d/project.h>
|
||||
#include <lardon3d/project_db.h>
|
||||
#include <lardon3d/task_checkpoint.h>
|
||||
#include <lardon3d/task_queue.h>
|
||||
|
|
@ -35,69 +35,41 @@ typedef struct {
|
|||
unsigned int version;
|
||||
} ProjectMetadata;
|
||||
|
||||
static void
|
||||
set_status(Lardon3DAppState *state, const char *message)
|
||||
{
|
||||
(void)snprintf(
|
||||
state->status_message,
|
||||
sizeof(state->status_message),
|
||||
"%s",
|
||||
message
|
||||
);
|
||||
static void set_status(Lardon3DAppState *state, const char *message) {
|
||||
(void)snprintf(state->status_message, sizeof(state->status_message), "%s", message);
|
||||
}
|
||||
|
||||
static void
|
||||
store_recovery_summary(Lardon3DAppState *state,
|
||||
const Lardon3DProjectRecoverySummary *summary)
|
||||
{
|
||||
static void store_recovery_summary(Lardon3DAppState *state,
|
||||
const Lardon3DProjectRecoverySummary *summary) {
|
||||
state->recovery_inspected = summary ? summary->inspected : 0;
|
||||
state->recovery_resumed = summary ? summary->resumed : 0;
|
||||
state->recovery_skipped = summary ? summary->skipped : 0;
|
||||
state->recovery_failed = summary ? summary->failed : 0;
|
||||
state->recovery_published_not_durable = summary
|
||||
? summary->published_not_durable : 0;
|
||||
state->recovery_published_not_durable = summary ? summary->published_not_durable : 0;
|
||||
state->recovery_queue_full = summary && summary->queue_full;
|
||||
}
|
||||
|
||||
static void
|
||||
set_database_status(
|
||||
Lardon3DAppState *state,
|
||||
Lardon3DProjectDb *database,
|
||||
const char *open_error,
|
||||
const char *fallback
|
||||
)
|
||||
{
|
||||
static void set_database_status(Lardon3DAppState *state, Lardon3DProjectDb *database,
|
||||
const char *open_error, const char *fallback) {
|
||||
char detail[LARDON3D_PROJECT_DB_ERROR_CAPACITY] = "";
|
||||
if (database) {
|
||||
(void)lardon3d_project_db_last_error(database, detail);
|
||||
} else if (open_error) {
|
||||
(void)snprintf(detail, sizeof(detail), "%s", open_error);
|
||||
}
|
||||
(void)snprintf(
|
||||
state->status_message,
|
||||
sizeof(state->status_message),
|
||||
"Erreur project.db : %.220s",
|
||||
detail[0] ? detail : fallback
|
||||
);
|
||||
(void)snprintf(state->status_message, sizeof(state->status_message), "Erreur project.db : %.220s",
|
||||
detail[0] ? detail : fallback);
|
||||
}
|
||||
|
||||
static void
|
||||
clear_catalog(Lardon3DAppState *state)
|
||||
{
|
||||
static void clear_catalog(Lardon3DAppState *state) {
|
||||
lardon3d_image_view_destroy(state->image_view);
|
||||
state->image_view = NULL;
|
||||
lardon3d_image_catalog_destroy(state->image_catalog);
|
||||
state->image_catalog = NULL;
|
||||
}
|
||||
|
||||
static bool
|
||||
normalize_name(
|
||||
Lardon3DAppState *state,
|
||||
const char *input,
|
||||
char *output,
|
||||
size_t output_size
|
||||
)
|
||||
{
|
||||
static bool normalize_name(Lardon3DAppState *state, const char *input, char *output,
|
||||
size_t output_size) {
|
||||
if (!input) {
|
||||
set_status(state, "Erreur : le nom du projet est vide.");
|
||||
return false;
|
||||
|
|
@ -122,15 +94,13 @@ normalize_name(
|
|||
set_status(state, "Erreur : le nom du projet est trop long.");
|
||||
return false;
|
||||
}
|
||||
if ((length == 1 && start[0] == '.')
|
||||
|| (length == 2 && start[0] == '.' && start[1] == '.')) {
|
||||
if ((length == 1 && start[0] == '.') || (length == 2 && start[0] == '.' && start[1] == '.')) {
|
||||
set_status(state, "Erreur : nom de projet interdit.");
|
||||
return false;
|
||||
}
|
||||
|
||||
for (const char *character = start; character < end; ++character) {
|
||||
if (*character == '/' || *character == '\\'
|
||||
|| iscntrl((unsigned char)*character)) {
|
||||
if (*character == '/' || *character == '\\' || iscntrl((unsigned char)*character)) {
|
||||
set_status(state, "Erreur : nom de projet interdit.");
|
||||
return false;
|
||||
}
|
||||
|
|
@ -141,28 +111,17 @@ normalize_name(
|
|||
return true;
|
||||
}
|
||||
|
||||
static bool
|
||||
copy_path(char *destination, size_t size, const char *source)
|
||||
{
|
||||
static bool copy_path(char *destination, size_t size, const char *source) {
|
||||
int written = snprintf(destination, size, "%s", source);
|
||||
return written >= 0 && (size_t)written < size;
|
||||
}
|
||||
|
||||
static bool
|
||||
join_path(
|
||||
char *destination,
|
||||
size_t size,
|
||||
const char *parent,
|
||||
const char *child
|
||||
)
|
||||
{
|
||||
static bool join_path(char *destination, size_t size, const char *parent, const char *child) {
|
||||
int written = snprintf(destination, size, "%s/%s", parent, child);
|
||||
return written >= 0 && (size_t)written < size;
|
||||
}
|
||||
|
||||
static bool
|
||||
generate_stable_id(char output[LARDON3D_PROJECT_DB_ID_CAPACITY])
|
||||
{
|
||||
static bool generate_stable_id(char output[LARDON3D_PROJECT_DB_ID_CAPACITY]) {
|
||||
unsigned char bytes[16];
|
||||
int descriptor = open("/dev/urandom", O_RDONLY | O_CLOEXEC);
|
||||
if (descriptor < 0) {
|
||||
|
|
@ -189,9 +148,7 @@ generate_stable_id(char output[LARDON3D_PROJECT_DB_ID_CAPACITY])
|
|||
return true;
|
||||
}
|
||||
|
||||
static bool
|
||||
resolve_projects_root(Lardon3DAppState *state, char root[PATH_MAX])
|
||||
{
|
||||
static bool resolve_projects_root(Lardon3DAppState *state, char root[PATH_MAX]) {
|
||||
const char *configured = getenv("LARDON3D_PROJECTS_ROOT");
|
||||
if (configured && configured[0]) {
|
||||
if (configured[0] != '/' || !copy_path(root, PATH_MAX, configured)) {
|
||||
|
|
@ -205,12 +162,7 @@ resolve_projects_root(Lardon3DAppState *state, char root[PATH_MAX])
|
|||
return false;
|
||||
}
|
||||
|
||||
int written = snprintf(
|
||||
root,
|
||||
PATH_MAX,
|
||||
"%s/Documents/Lardon/Projets3D",
|
||||
home
|
||||
);
|
||||
int written = snprintf(root, PATH_MAX, "%s/Documents/Lardon/Projets3D", home);
|
||||
if (written < 0 || (size_t)written >= PATH_MAX) {
|
||||
set_status(state, "Erreur : chemin racine trop long.");
|
||||
return false;
|
||||
|
|
@ -224,22 +176,15 @@ resolve_projects_root(Lardon3DAppState *state, char root[PATH_MAX])
|
|||
return true;
|
||||
}
|
||||
|
||||
static void
|
||||
cleanup_directories(CreatedDirectories *created)
|
||||
{
|
||||
static void cleanup_directories(CreatedDirectories *created) {
|
||||
while (created->count > 0) {
|
||||
--created->count;
|
||||
(void)rmdir(created->paths[created->count]);
|
||||
}
|
||||
}
|
||||
|
||||
static bool
|
||||
ensure_directory(
|
||||
Lardon3DAppState *state,
|
||||
const char *path,
|
||||
CreatedDirectories *created
|
||||
)
|
||||
{
|
||||
static bool ensure_directory(Lardon3DAppState *state, const char *path,
|
||||
CreatedDirectories *created) {
|
||||
struct stat info;
|
||||
if (lstat(path, &info) == 0) {
|
||||
if (!S_ISDIR(info.st_mode)) {
|
||||
|
|
@ -256,11 +201,7 @@ ensure_directory(
|
|||
set_status(state, "Erreur : impossible de créer un dossier.");
|
||||
return false;
|
||||
}
|
||||
if (!copy_path(
|
||||
created->paths[created->count],
|
||||
sizeof(created->paths[created->count]),
|
||||
path
|
||||
)) {
|
||||
if (!copy_path(created->paths[created->count], sizeof(created->paths[created->count]), path)) {
|
||||
(void)rmdir(path);
|
||||
set_status(state, "Erreur : chemin de dossier trop long.");
|
||||
return false;
|
||||
|
|
@ -269,13 +210,8 @@ ensure_directory(
|
|||
return true;
|
||||
}
|
||||
|
||||
static bool
|
||||
ensure_directory_tree(
|
||||
Lardon3DAppState *state,
|
||||
const char *path,
|
||||
CreatedDirectories *created
|
||||
)
|
||||
{
|
||||
static bool ensure_directory_tree(Lardon3DAppState *state, const char *path,
|
||||
CreatedDirectories *created) {
|
||||
char partial[PATH_MAX];
|
||||
if (!copy_path(partial, sizeof(partial), path)) {
|
||||
set_status(state, "Erreur : chemin racine trop long.");
|
||||
|
|
@ -296,23 +232,12 @@ ensure_directory_tree(
|
|||
return ensure_directory(state, partial, created);
|
||||
}
|
||||
|
||||
static bool
|
||||
write_project_ini(
|
||||
Lardon3DAppState *state,
|
||||
const char *project_path,
|
||||
const char *project_name,
|
||||
const char *stable_id
|
||||
)
|
||||
{
|
||||
static bool write_project_ini(Lardon3DAppState *state, const char *project_path,
|
||||
const char *project_name, const char *stable_id) {
|
||||
char temporary_path[PATH_MAX];
|
||||
char final_path[PATH_MAX];
|
||||
if (!join_path(final_path, sizeof(final_path), project_path, "project.ini")
|
||||
|| !join_path(
|
||||
temporary_path,
|
||||
sizeof(temporary_path),
|
||||
project_path,
|
||||
".project.ini.tmp.XXXXXX"
|
||||
)) {
|
||||
if (!join_path(final_path, sizeof(final_path), project_path, "project.ini") ||
|
||||
!join_path(temporary_path, sizeof(temporary_path), project_path, ".project.ini.tmp.XXXXXX")) {
|
||||
set_status(state, "Erreur : chemin de project.ini trop long.");
|
||||
return false;
|
||||
}
|
||||
|
|
@ -331,12 +256,8 @@ write_project_ini(
|
|||
return false;
|
||||
}
|
||||
|
||||
bool success = fprintf(
|
||||
file,
|
||||
"[project]\nname=%s\nstable_id=%s\nversion=2\n",
|
||||
project_name,
|
||||
stable_id
|
||||
) >= 0;
|
||||
bool success =
|
||||
fprintf(file, "[project]\nname=%s\nstable_id=%s\nversion=2\n", project_name, stable_id) >= 0;
|
||||
if (success) {
|
||||
success = fflush(file) == 0;
|
||||
}
|
||||
|
|
@ -356,9 +277,7 @@ write_project_ini(
|
|||
return success;
|
||||
}
|
||||
|
||||
bool
|
||||
lardon3d_project_create(Lardon3DAppState *state, const char *name)
|
||||
{
|
||||
bool lardon3d_project_create(Lardon3DAppState *state, const char *name) {
|
||||
if (!state) {
|
||||
return false;
|
||||
}
|
||||
|
|
@ -371,12 +290,7 @@ lardon3d_project_create(Lardon3DAppState *state, const char *name)
|
|||
char root[PATH_MAX];
|
||||
char project_path[PATH_MAX];
|
||||
char stable_id[LARDON3D_PROJECT_DB_ID_CAPACITY];
|
||||
if (!normalize_name(
|
||||
state,
|
||||
name,
|
||||
normalized_name,
|
||||
sizeof(normalized_name)
|
||||
)) {
|
||||
if (!normalize_name(state, name, normalized_name, sizeof(normalized_name))) {
|
||||
return false;
|
||||
}
|
||||
if (!resolve_projects_root(state, root)) {
|
||||
|
|
@ -386,12 +300,7 @@ lardon3d_project_create(Lardon3DAppState *state, const char *name)
|
|||
set_status(state, "Erreur : impossible de créer l'identité du projet.");
|
||||
return false;
|
||||
}
|
||||
if (!join_path(
|
||||
project_path,
|
||||
sizeof(project_path),
|
||||
root,
|
||||
normalized_name
|
||||
)) {
|
||||
if (!join_path(project_path, sizeof(project_path), root, normalized_name)) {
|
||||
set_status(state, "Erreur : chemin du projet trop long.");
|
||||
return false;
|
||||
}
|
||||
|
|
@ -414,23 +323,11 @@ lardon3d_project_create(Lardon3DAppState *state, const char *name)
|
|||
}
|
||||
|
||||
const char *subdirectories[] = {
|
||||
"images",
|
||||
"reconstruction",
|
||||
"exports",
|
||||
"logs",
|
||||
".lardon3d",
|
||||
".lardon3d/checkpoints",
|
||||
"images", "reconstruction", "exports", "logs", ".lardon3d", ".lardon3d/checkpoints",
|
||||
};
|
||||
for (size_t index = 0;
|
||||
index < sizeof(subdirectories) / sizeof(subdirectories[0]);
|
||||
++index) {
|
||||
for (size_t index = 0; index < sizeof(subdirectories) / sizeof(subdirectories[0]); ++index) {
|
||||
char path[PATH_MAX];
|
||||
if (!join_path(
|
||||
path,
|
||||
sizeof(path),
|
||||
project_path,
|
||||
subdirectories[index]
|
||||
)) {
|
||||
if (!join_path(path, sizeof(path), project_path, subdirectories[index])) {
|
||||
set_status(state, "Erreur : chemin de dossier trop long.");
|
||||
cleanup_directories(&created);
|
||||
return false;
|
||||
|
|
@ -448,22 +345,18 @@ lardon3d_project_create(Lardon3DAppState *state, const char *name)
|
|||
|
||||
char database_path[PATH_MAX];
|
||||
char ini_path[PATH_MAX];
|
||||
if (!join_path(database_path, sizeof(database_path), project_path, "project.db")
|
||||
|| !join_path(ini_path, sizeof(ini_path), project_path, "project.ini")) {
|
||||
if (!join_path(database_path, sizeof(database_path), project_path, "project.db") ||
|
||||
!join_path(ini_path, sizeof(ini_path), project_path, "project.ini")) {
|
||||
cleanup_directories(&created);
|
||||
set_status(state, "Erreur : chemin de base projet trop long.");
|
||||
return false;
|
||||
}
|
||||
Lardon3DProjectDb *database = NULL;
|
||||
char database_error[LARDON3D_PROJECT_DB_ERROR_CAPACITY];
|
||||
Lardon3DProjectDbResult database_result = lardon3d_project_db_open(
|
||||
database_path,
|
||||
&database,
|
||||
database_error
|
||||
);
|
||||
Lardon3DProjectDbResult database_result =
|
||||
lardon3d_project_db_open(database_path, &database, database_error);
|
||||
struct timespec now;
|
||||
if (database_result == LARDON3D_PROJECT_DB_OK
|
||||
&& clock_gettime(CLOCK_REALTIME, &now) == 0) {
|
||||
if (database_result == LARDON3D_PROJECT_DB_OK && clock_gettime(CLOCK_REALTIME, &now) == 0) {
|
||||
Lardon3DProjectDbProject project = {
|
||||
.created_at = now.tv_sec,
|
||||
.updated_at = now.tv_sec,
|
||||
|
|
@ -475,12 +368,7 @@ lardon3d_project_create(Lardon3DAppState *state, const char *name)
|
|||
database_result = LARDON3D_PROJECT_DB_IO_ERROR;
|
||||
}
|
||||
if (database_result != LARDON3D_PROJECT_DB_OK) {
|
||||
set_database_status(
|
||||
state,
|
||||
database,
|
||||
database_error,
|
||||
"initialisation impossible"
|
||||
);
|
||||
set_database_status(state, database, database_error, "initialisation impossible");
|
||||
lardon3d_project_db_close(database);
|
||||
(void)unlink(database_path);
|
||||
(void)unlink(ini_path);
|
||||
|
|
@ -492,38 +380,15 @@ lardon3d_project_create(Lardon3DAppState *state, const char *name)
|
|||
state->project_loaded = true;
|
||||
store_recovery_summary(state, NULL);
|
||||
state->project_db = database;
|
||||
(void)copy_path(
|
||||
state->project_name,
|
||||
sizeof(state->project_name),
|
||||
normalized_name
|
||||
);
|
||||
(void)copy_path(
|
||||
state->project_path,
|
||||
sizeof(state->project_path),
|
||||
project_path
|
||||
);
|
||||
(void)snprintf(
|
||||
state->project_stable_id,
|
||||
sizeof(state->project_stable_id),
|
||||
"%s",
|
||||
stable_id
|
||||
);
|
||||
(void)snprintf(
|
||||
state->status_message,
|
||||
sizeof(state->status_message),
|
||||
"Projet créé : %s",
|
||||
state->project_name
|
||||
);
|
||||
(void)copy_path(state->project_name, sizeof(state->project_name), normalized_name);
|
||||
(void)copy_path(state->project_path, sizeof(state->project_path), project_path);
|
||||
(void)snprintf(state->project_stable_id, sizeof(state->project_stable_id), "%s", stable_id);
|
||||
(void)snprintf(state->status_message, sizeof(state->status_message), "Projet créé : %s",
|
||||
state->project_name);
|
||||
return true;
|
||||
}
|
||||
|
||||
static bool
|
||||
read_project_ini(
|
||||
Lardon3DAppState *state,
|
||||
const char *path,
|
||||
ProjectMetadata *metadata
|
||||
)
|
||||
{
|
||||
static bool read_project_ini(Lardon3DAppState *state, const char *path, ProjectMetadata *metadata) {
|
||||
int descriptor = open(path, O_RDONLY | O_NOFOLLOW);
|
||||
if (descriptor < 0) {
|
||||
set_status(state, "Erreur : project.ini est absent ou inaccessible.");
|
||||
|
|
@ -570,37 +435,24 @@ read_project_ini(
|
|||
} else if (line[0] == '[') {
|
||||
in_project_section = false;
|
||||
} else if (in_project_section && strncmp(line, "name=", 5) == 0) {
|
||||
if (name_found || !normalize_name(
|
||||
state,
|
||||
line + 5,
|
||||
metadata->name,
|
||||
sizeof(state->project_name)
|
||||
)) {
|
||||
if (name_found ||
|
||||
!normalize_name(state, line + 5, metadata->name, sizeof(state->project_name))) {
|
||||
valid = false;
|
||||
}
|
||||
name_found = true;
|
||||
} else if (in_project_section && strncmp(line, "stable_id=", 10) == 0) {
|
||||
size_t id_length = strnlen(
|
||||
line + 10,
|
||||
sizeof(metadata->stable_id)
|
||||
);
|
||||
size_t id_length = strnlen(line + 10, sizeof(metadata->stable_id));
|
||||
if (stable_id_found || id_length != 32) {
|
||||
valid = false;
|
||||
} else {
|
||||
for (size_t index = 0; index < id_length; ++index) {
|
||||
char character = line[10 + index];
|
||||
if (!((character >= '0' && character <= '9')
|
||||
|| (character >= 'a' && character <= 'f'))) {
|
||||
if (!((character >= '0' && character <= '9') || (character >= 'a' && character <= 'f'))) {
|
||||
valid = false;
|
||||
}
|
||||
}
|
||||
if (valid) {
|
||||
(void)snprintf(
|
||||
metadata->stable_id,
|
||||
sizeof(metadata->stable_id),
|
||||
"%s",
|
||||
line + 10
|
||||
);
|
||||
(void)snprintf(metadata->stable_id, sizeof(metadata->stable_id), "%s", line + 10);
|
||||
}
|
||||
}
|
||||
stable_id_found = true;
|
||||
|
|
@ -624,21 +476,15 @@ read_project_ini(
|
|||
if (fclose(file) != 0) {
|
||||
valid = false;
|
||||
}
|
||||
if (!valid || !section_found || !name_found || !version_found
|
||||
|| (metadata->version == 2 && !stable_id_found)
|
||||
|| (metadata->version == 1 && stable_id_found)) {
|
||||
if (!valid || !section_found || !name_found || !version_found ||
|
||||
(metadata->version == 2 && !stable_id_found) || (metadata->version == 1 && stable_id_found)) {
|
||||
set_status(state, "Erreur : project.ini invalide.");
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
bool
|
||||
lardon3d_project_open(
|
||||
Lardon3DAppState *state,
|
||||
const char *directory_name
|
||||
)
|
||||
{
|
||||
bool lardon3d_project_open(Lardon3DAppState *state, const char *directory_name) {
|
||||
if (!state) {
|
||||
return false;
|
||||
}
|
||||
|
|
@ -652,25 +498,15 @@ lardon3d_project_open(
|
|||
char project_path[PATH_MAX];
|
||||
char ini_path[PATH_MAX];
|
||||
char database_path[PATH_MAX];
|
||||
if (!normalize_name(
|
||||
state,
|
||||
directory_name,
|
||||
normalized_directory,
|
||||
sizeof(normalized_directory)
|
||||
)) {
|
||||
if (!normalize_name(state, directory_name, normalized_directory, sizeof(normalized_directory))) {
|
||||
return false;
|
||||
}
|
||||
if (!resolve_projects_root(state, root)) {
|
||||
return false;
|
||||
}
|
||||
if (!join_path(
|
||||
project_path,
|
||||
sizeof(project_path),
|
||||
root,
|
||||
normalized_directory
|
||||
)
|
||||
|| !join_path(ini_path, sizeof(ini_path), project_path, "project.ini")
|
||||
|| !join_path(database_path, sizeof(database_path), project_path, "project.db")) {
|
||||
if (!join_path(project_path, sizeof(project_path), root, normalized_directory) ||
|
||||
!join_path(ini_path, sizeof(ini_path), project_path, "project.ini") ||
|
||||
!join_path(database_path, sizeof(database_path), project_path, "project.db")) {
|
||||
set_status(state, "Erreur : chemin du projet trop long.");
|
||||
return false;
|
||||
}
|
||||
|
|
@ -701,29 +537,21 @@ lardon3d_project_open(
|
|||
CreatedDirectories created = {0};
|
||||
char internal_path[PATH_MAX];
|
||||
char checkpoint_path[PATH_MAX];
|
||||
if (!join_path(internal_path, sizeof(internal_path), project_path, ".lardon3d")
|
||||
|| !join_path(checkpoint_path, sizeof(checkpoint_path), internal_path, "checkpoints")
|
||||
|| !ensure_directory(state, internal_path, &created)
|
||||
|| !ensure_directory(state, checkpoint_path, &created)) {
|
||||
if (!join_path(internal_path, sizeof(internal_path), project_path, ".lardon3d") ||
|
||||
!join_path(checkpoint_path, sizeof(checkpoint_path), internal_path, "checkpoints") ||
|
||||
!ensure_directory(state, internal_path, &created) ||
|
||||
!ensure_directory(state, checkpoint_path, &created)) {
|
||||
cleanup_directories(&created);
|
||||
return false;
|
||||
}
|
||||
|
||||
Lardon3DProjectDb *database = NULL;
|
||||
char database_error[LARDON3D_PROJECT_DB_ERROR_CAPACITY];
|
||||
Lardon3DProjectDbResult database_result = lardon3d_project_db_open(
|
||||
database_path,
|
||||
&database,
|
||||
database_error
|
||||
);
|
||||
Lardon3DProjectDbResult database_result =
|
||||
lardon3d_project_db_open(database_path, &database, database_error);
|
||||
if (database_result != LARDON3D_PROJECT_DB_OK) {
|
||||
cleanup_directories(&created);
|
||||
set_database_status(
|
||||
state,
|
||||
NULL,
|
||||
database_error,
|
||||
"ouverture impossible"
|
||||
);
|
||||
set_database_status(state, NULL, database_error, "ouverture impossible");
|
||||
return false;
|
||||
}
|
||||
|
||||
|
|
@ -734,16 +562,10 @@ lardon3d_project_open(
|
|||
if (clock_gettime(CLOCK_REALTIME, &now) != 0) {
|
||||
database_result = LARDON3D_PROJECT_DB_IO_ERROR;
|
||||
} else if (database_result == LARDON3D_PROJECT_DB_OK) {
|
||||
if (metadata.stable_id[0]
|
||||
&& strcmp(metadata.stable_id, db_project.stable_id) != 0) {
|
||||
if (metadata.stable_id[0] && strcmp(metadata.stable_id, db_project.stable_id) != 0) {
|
||||
database_result = LARDON3D_PROJECT_DB_CONSTRAINT;
|
||||
} else if (!metadata.stable_id[0]) {
|
||||
(void)snprintf(
|
||||
metadata.stable_id,
|
||||
sizeof(metadata.stable_id),
|
||||
"%s",
|
||||
db_project.stable_id
|
||||
);
|
||||
(void)snprintf(metadata.stable_id, sizeof(metadata.stable_id), "%s", db_project.stable_id);
|
||||
rewrite_metadata = true;
|
||||
}
|
||||
if (database_result == LARDON3D_PROJECT_DB_OK) {
|
||||
|
|
@ -766,26 +588,22 @@ lardon3d_project_open(
|
|||
.created_at = now.tv_sec,
|
||||
.updated_at = now.tv_sec,
|
||||
};
|
||||
(void)snprintf(new_project.stable_id, sizeof(new_project.stable_id), "%s", metadata.stable_id);
|
||||
(void)snprintf(new_project.stable_id, sizeof(new_project.stable_id), "%s",
|
||||
metadata.stable_id);
|
||||
(void)snprintf(new_project.name, sizeof(new_project.name), "%s", metadata.name);
|
||||
database_result = lardon3d_project_db_set_project(database, &new_project);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (database_result == LARDON3D_PROJECT_DB_OK && rewrite_metadata
|
||||
&& !write_project_ini(state, project_path, metadata.name, metadata.stable_id)) {
|
||||
if (database_result == LARDON3D_PROJECT_DB_OK && rewrite_metadata &&
|
||||
!write_project_ini(state, project_path, metadata.name, metadata.stable_id)) {
|
||||
database_result = LARDON3D_PROJECT_DB_IO_ERROR;
|
||||
}
|
||||
if (database_result != LARDON3D_PROJECT_DB_OK) {
|
||||
if (database_result == LARDON3D_PROJECT_DB_CONSTRAINT) {
|
||||
set_status(state, "Erreur : identité project.ini/project.db incohérente.");
|
||||
} else {
|
||||
set_database_status(
|
||||
state,
|
||||
database,
|
||||
database_error,
|
||||
"initialisation impossible"
|
||||
);
|
||||
set_database_status(state, database, database_error, "initialisation impossible");
|
||||
}
|
||||
lardon3d_project_db_close(database);
|
||||
if (!database_existed) {
|
||||
|
|
@ -798,45 +616,27 @@ lardon3d_project_open(
|
|||
clear_catalog(state);
|
||||
state->project_loaded = true;
|
||||
state->project_db = database;
|
||||
(void)copy_path(
|
||||
state->project_name,
|
||||
sizeof(state->project_name),
|
||||
metadata.name
|
||||
);
|
||||
(void)snprintf(
|
||||
state->project_stable_id,
|
||||
sizeof(state->project_stable_id),
|
||||
"%s",
|
||||
metadata.stable_id
|
||||
);
|
||||
(void)copy_path(
|
||||
state->project_path,
|
||||
sizeof(state->project_path),
|
||||
project_path
|
||||
);
|
||||
(void)copy_path(state->project_name, sizeof(state->project_name), metadata.name);
|
||||
(void)snprintf(state->project_stable_id, sizeof(state->project_stable_id), "%s",
|
||||
metadata.stable_id);
|
||||
(void)copy_path(state->project_path, sizeof(state->project_path), project_path);
|
||||
store_recovery_summary(state, NULL);
|
||||
if (state->task_queue && state->resource_governor) {
|
||||
Lardon3DProjectRecoverySummary summary;
|
||||
(void)lardon3d_project_resume_recoverable_tasks(state,
|
||||
lardon3d_task_kind_registry_production(), &summary);
|
||||
(void)lardon3d_project_resume_recoverable_tasks(state, lardon3d_task_kind_registry_production(),
|
||||
&summary);
|
||||
(void)snprintf(state->status_message, sizeof(state->status_message),
|
||||
"Projet ouvert — %zu tâche(s) reprise(s), %zu ignorée(s), %zu en échec%s.",
|
||||
summary.resumed, summary.skipped, summary.failed,
|
||||
summary.queue_full ? ", fenêtre de reprise saturée" : "");
|
||||
} else {
|
||||
(void)snprintf(
|
||||
state->status_message,
|
||||
sizeof(state->status_message),
|
||||
"Projet ouvert : %s",
|
||||
state->project_name
|
||||
);
|
||||
(void)snprintf(state->status_message, sizeof(state->status_message), "Projet ouvert : %s",
|
||||
state->project_name);
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
void
|
||||
lardon3d_project_close(Lardon3DAppState *state)
|
||||
{
|
||||
void lardon3d_project_close(Lardon3DAppState *state) {
|
||||
if (!state) {
|
||||
return;
|
||||
}
|
||||
|
|
@ -857,33 +657,19 @@ lardon3d_project_close(Lardon3DAppState *state)
|
|||
set_status(state, "Projet fermé.");
|
||||
}
|
||||
|
||||
static bool
|
||||
checkpoint_paths(
|
||||
const Lardon3DAppState *state,
|
||||
uint64_t task_id,
|
||||
static bool checkpoint_paths(const Lardon3DAppState *state, uint64_t task_id,
|
||||
char relative[LARDON3D_PROJECT_DB_PATH_CAPACITY],
|
||||
char absolute[PATH_MAX]
|
||||
)
|
||||
{
|
||||
int written = snprintf(
|
||||
relative,
|
||||
LARDON3D_PROJECT_DB_PATH_CAPACITY,
|
||||
".lardon3d/checkpoints/%" PRIu64 ".chk",
|
||||
task_id
|
||||
);
|
||||
return written > 0
|
||||
&& (size_t)written < LARDON3D_PROJECT_DB_PATH_CAPACITY
|
||||
&& join_path(absolute, PATH_MAX, state->project_path, relative);
|
||||
char absolute[PATH_MAX]) {
|
||||
int written = snprintf(relative, LARDON3D_PROJECT_DB_PATH_CAPACITY,
|
||||
".lardon3d/checkpoints/%" PRIu64 ".chk", task_id);
|
||||
return written > 0 && (size_t)written < LARDON3D_PROJECT_DB_PATH_CAPACITY &&
|
||||
join_path(absolute, PATH_MAX, state->project_path, relative);
|
||||
}
|
||||
|
||||
static Lardon3DProjectTaskCheckpointResult
|
||||
checkpoint_task_internal(
|
||||
Lardon3DAppState *state,
|
||||
const Lardon3DTask *task,
|
||||
const char *image_import_source,
|
||||
uint64_t image_import_scanset_id
|
||||
)
|
||||
{
|
||||
checkpoint_task_internal(Lardon3DAppState *state, const Lardon3DTask *task,
|
||||
const char *image_import_source, uint64_t image_import_scanset_id,
|
||||
const Lardon3DProjectDbFeatureExtractTask *feature_parameters) {
|
||||
if (!state || !state->project_loaded || !state->project_db) {
|
||||
return LARDON3D_PROJECT_TASK_CHECKPOINT_NO_PROJECT;
|
||||
}
|
||||
|
|
@ -901,15 +687,12 @@ checkpoint_task_internal(
|
|||
if (!checkpoint_paths(state, snapshot.id, relative, absolute)) {
|
||||
return LARDON3D_PROJECT_TASK_CHECKPOINT_IO_ERROR;
|
||||
}
|
||||
Lardon3DTaskCheckpointResult saved = lardon3d_task_checkpoint_save(
|
||||
absolute,
|
||||
&snapshot
|
||||
);
|
||||
Lardon3DTaskCheckpointResult saved = lardon3d_task_checkpoint_save(absolute, &snapshot);
|
||||
if (saved == LARDON3D_TASK_CHECKPOINT_INVALID) {
|
||||
return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
|
||||
}
|
||||
if (saved != LARDON3D_TASK_CHECKPOINT_OK
|
||||
&& saved != LARDON3D_TASK_CHECKPOINT_PUBLISHED_NOT_DURABLE) {
|
||||
if (saved != LARDON3D_TASK_CHECKPOINT_OK &&
|
||||
saved != LARDON3D_TASK_CHECKPOINT_PUBLISHED_NOT_DURABLE) {
|
||||
return LARDON3D_PROJECT_TASK_CHECKPOINT_IO_ERROR;
|
||||
}
|
||||
struct timespec now;
|
||||
|
|
@ -924,14 +707,17 @@ checkpoint_task_internal(
|
|||
.updated_at = now.tv_sec,
|
||||
};
|
||||
(void)snprintf(checkpoint.path, sizeof(checkpoint.path), "%s", relative);
|
||||
Lardon3DProjectDbResult recorded = image_import_source
|
||||
Lardon3DProjectDbResult recorded =
|
||||
image_import_source
|
||||
? lardon3d_project_db_record_image_import_task(
|
||||
state->project_db, &snapshot, task_kind, task_kind_version,
|
||||
&checkpoint, image_import_source, image_import_scanset_id,
|
||||
now.tv_sec)
|
||||
: lardon3d_project_db_record_task(
|
||||
state->project_db, &snapshot, task_kind, task_kind_version,
|
||||
&checkpoint, now.tv_sec);
|
||||
state->project_db, &snapshot, task_kind, task_kind_version, &checkpoint,
|
||||
image_import_source, image_import_scanset_id, now.tv_sec)
|
||||
: feature_parameters
|
||||
? lardon3d_project_db_record_feature_extract_task(state->project_db, &snapshot, task_kind,
|
||||
task_kind_version, &checkpoint,
|
||||
feature_parameters, now.tv_sec)
|
||||
: lardon3d_project_db_record_task(state->project_db, &snapshot, task_kind,
|
||||
task_kind_version, &checkpoint, now.tv_sec);
|
||||
if (recorded == LARDON3D_PROJECT_DB_BUSY) {
|
||||
return LARDON3D_PROJECT_TASK_CHECKPOINT_DB_BUSY;
|
||||
}
|
||||
|
|
@ -943,59 +729,49 @@ checkpoint_task_internal(
|
|||
: LARDON3D_PROJECT_TASK_CHECKPOINT_PUBLISHED_NOT_DURABLE;
|
||||
}
|
||||
|
||||
Lardon3DProjectTaskCheckpointResult
|
||||
lardon3d_project_checkpoint_task(
|
||||
Lardon3DAppState *state,
|
||||
const Lardon3DTask *task
|
||||
)
|
||||
{
|
||||
return checkpoint_task_internal(state, task, NULL, 0);
|
||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_task(Lardon3DAppState *state,
|
||||
const Lardon3DTask *task) {
|
||||
return checkpoint_task_internal(state, task, NULL, 0, NULL);
|
||||
}
|
||||
|
||||
Lardon3DProjectTaskCheckpointResult
|
||||
lardon3d_project_checkpoint_image_import_task(
|
||||
Lardon3DAppState *state,
|
||||
const Lardon3DTask *task,
|
||||
const char *source_path,
|
||||
uint64_t scanset_id
|
||||
)
|
||||
{
|
||||
lardon3d_project_checkpoint_image_import_task(Lardon3DAppState *state, const Lardon3DTask *task,
|
||||
const char *source_path, uint64_t scanset_id) {
|
||||
if (!source_path || !source_path[0] || scanset_id == 0) {
|
||||
return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
|
||||
}
|
||||
return checkpoint_task_internal(state, task, source_path, scanset_id);
|
||||
return checkpoint_task_internal(state, task, source_path, scanset_id, NULL);
|
||||
}
|
||||
|
||||
static bool
|
||||
coherent_recovery(
|
||||
const Lardon3DProjectDbTask *database_task,
|
||||
const Lardon3DTaskDurableSnapshot *snapshot
|
||||
)
|
||||
{
|
||||
return snapshot->id == database_task->task_id
|
||||
&& strcmp(snapshot->name, database_task->name) == 0
|
||||
&& snapshot->saved_state == database_task->saved_state
|
||||
&& snapshot->recovery_state == database_task->recovery_state
|
||||
&& snapshot->progress == database_task->progress
|
||||
&& snapshot->sequence_count == database_task->sequence_count;
|
||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_task(
|
||||
Lardon3DAppState *state, const Lardon3DTask *task,
|
||||
const Lardon3DProjectDbFeatureExtractTask *parameters) {
|
||||
if (!parameters) {
|
||||
return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
|
||||
}
|
||||
return checkpoint_task_internal(state, task, NULL, 0, parameters);
|
||||
}
|
||||
|
||||
Lardon3DProjectDbResult
|
||||
lardon3d_project_list_recoverable(
|
||||
Lardon3DAppState *state,
|
||||
static bool coherent_recovery(const Lardon3DProjectDbTask *database_task,
|
||||
const Lardon3DTaskDurableSnapshot *snapshot) {
|
||||
return snapshot->id == database_task->task_id &&
|
||||
strcmp(snapshot->name, database_task->name) == 0 &&
|
||||
snapshot->saved_state == database_task->saved_state &&
|
||||
snapshot->recovery_state == database_task->recovery_state &&
|
||||
snapshot->progress == database_task->progress &&
|
||||
snapshot->sequence_count == database_task->sequence_count;
|
||||
}
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_list_recoverable(Lardon3DAppState *state,
|
||||
const Lardon3DTaskKindRegistry *registry,
|
||||
uint64_t after_task_id,
|
||||
Lardon3DProjectRecoveryEntry *entries,
|
||||
size_t capacity,
|
||||
size_t *count
|
||||
)
|
||||
{
|
||||
size_t capacity, size_t *count) {
|
||||
if (count) {
|
||||
*count = 0;
|
||||
}
|
||||
if (!state || !state->project_loaded || !state->project_db || !registry || !entries
|
||||
|| !count || capacity == 0
|
||||
|| capacity > LARDON3D_PROJECT_DB_RECOVERY_PAGE_MAX) {
|
||||
if (!state || !state->project_loaded || !state->project_db || !registry || !entries || !count ||
|
||||
capacity == 0 || capacity > LARDON3D_PROJECT_DB_RECOVERY_PAGE_MAX) {
|
||||
return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
|
||||
}
|
||||
enum { RECOVERY_CHUNK = 8 };
|
||||
|
|
@ -1008,12 +784,7 @@ lardon3d_project_list_recoverable(
|
|||
}
|
||||
size_t task_count = 0;
|
||||
Lardon3DProjectDbResult result = lardon3d_project_db_list_recoverable(
|
||||
state->project_db,
|
||||
cursor,
|
||||
tasks,
|
||||
requested,
|
||||
&task_count
|
||||
);
|
||||
state->project_db, cursor, tasks, requested, &task_count);
|
||||
if (result != LARDON3D_PROJECT_DB_OK) {
|
||||
return result;
|
||||
}
|
||||
|
|
@ -1024,16 +795,15 @@ lardon3d_project_list_recoverable(
|
|||
entry->durability = tasks[index].checkpoint.durability;
|
||||
(void)snprintf(entry->name, sizeof(entry->name), "%s", tasks[index].name);
|
||||
if (tasks[index].has_task_kind) {
|
||||
(void)snprintf(entry->task_kind, sizeof(entry->task_kind), "%s",
|
||||
tasks[index].task_kind);
|
||||
(void)snprintf(entry->task_kind, sizeof(entry->task_kind), "%s", tasks[index].task_kind);
|
||||
entry->task_kind_version = tasks[index].task_kind_version;
|
||||
}
|
||||
char relative[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
||||
char absolute[PATH_MAX];
|
||||
const Lardon3DTaskKindDescriptor *descriptor = NULL;
|
||||
Lardon3DTaskKindResult kind_result = tasks[index].has_task_kind
|
||||
? lardon3d_task_kind_registry_lookup(
|
||||
registry, tasks[index].task_kind,
|
||||
Lardon3DTaskKindResult kind_result =
|
||||
tasks[index].has_task_kind
|
||||
? lardon3d_task_kind_registry_lookup(registry, tasks[index].task_kind,
|
||||
tasks[index].task_kind_version, &descriptor)
|
||||
: LARDON3D_TASK_KIND_UNKNOWN;
|
||||
if (!tasks[index].has_task_kind) {
|
||||
|
|
@ -1044,31 +814,27 @@ lardon3d_project_list_recoverable(
|
|||
entry->status = LARDON3D_PROJECT_RECOVERY_UNSUPPORTED_TASK_KIND_VERSION;
|
||||
} else if (kind_result != LARDON3D_TASK_KIND_OK) {
|
||||
entry->status = LARDON3D_PROJECT_RECOVERY_UNKNOWN_TASK_KIND;
|
||||
} else if (!checkpoint_paths(state, entry->task_id, relative, absolute)
|
||||
|| strcmp(relative, tasks[index].checkpoint.path) != 0) {
|
||||
} else if (!checkpoint_paths(state, entry->task_id, relative, absolute) ||
|
||||
strcmp(relative, tasks[index].checkpoint.path) != 0) {
|
||||
entry->status = LARDON3D_PROJECT_RECOVERY_INVALID_CHECKPOINT;
|
||||
} else {
|
||||
uint32_t version = 0;
|
||||
Lardon3DTaskCheckpointResult loaded = lardon3d_task_checkpoint_load(
|
||||
absolute,
|
||||
&entry->snapshot,
|
||||
&version
|
||||
);
|
||||
Lardon3DTaskCheckpointResult loaded =
|
||||
lardon3d_task_checkpoint_load(absolute, &entry->snapshot, &version);
|
||||
if (loaded == LARDON3D_TASK_CHECKPOINT_NOT_FOUND) {
|
||||
entry->status = LARDON3D_PROJECT_RECOVERY_MISSING_CHECKPOINT;
|
||||
} else if (loaded == LARDON3D_TASK_CHECKPOINT_UNSUPPORTED_VERSION) {
|
||||
entry->status = LARDON3D_PROJECT_RECOVERY_UNSUPPORTED_CHECKPOINT;
|
||||
} else if (loaded == LARDON3D_TASK_CHECKPOINT_INVALID) {
|
||||
entry->status = LARDON3D_PROJECT_RECOVERY_INVALID_CHECKPOINT;
|
||||
} else if (loaded != LARDON3D_TASK_CHECKPOINT_OK
|
||||
|| version != tasks[index].checkpoint.format_version
|
||||
|| !coherent_recovery(&tasks[index], &entry->snapshot)) {
|
||||
} else if (loaded != LARDON3D_TASK_CHECKPOINT_OK ||
|
||||
version != tasks[index].checkpoint.format_version ||
|
||||
!coherent_recovery(&tasks[index], &entry->snapshot)) {
|
||||
entry->status = loaded == LARDON3D_TASK_CHECKPOINT_IO_ERROR
|
||||
? LARDON3D_PROJECT_RECOVERY_CHECKPOINT_IO_ERROR
|
||||
: LARDON3D_PROJECT_RECOVERY_INVALID_CHECKPOINT;
|
||||
} else {
|
||||
entry->status = entry->durability
|
||||
== LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE
|
||||
entry->status = entry->durability == LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE
|
||||
? LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE
|
||||
: LARDON3D_PROJECT_RECOVERABLE;
|
||||
}
|
||||
|
|
@ -1084,16 +850,14 @@ lardon3d_project_list_recoverable(
|
|||
}
|
||||
|
||||
Lardon3DProjectDbResult
|
||||
lardon3d_project_resume_recoverable_tasks(
|
||||
Lardon3DAppState *state,
|
||||
lardon3d_project_resume_recoverable_tasks(Lardon3DAppState *state,
|
||||
const Lardon3DTaskKindRegistry *registry,
|
||||
Lardon3DProjectRecoverySummary *summary
|
||||
)
|
||||
{
|
||||
if (summary) memset(summary, 0, sizeof(*summary));
|
||||
if (!state || !state->project_loaded || !state->project_db
|
||||
|| !state->task_queue || !state->resource_governor || !registry
|
||||
|| !summary) {
|
||||
Lardon3DProjectRecoverySummary *summary) {
|
||||
if (summary) {
|
||||
memset(summary, 0, sizeof(*summary));
|
||||
}
|
||||
if (!state || !state->project_loaded || !state->project_db || !state->task_queue ||
|
||||
!state->resource_governor || !registry || !summary) {
|
||||
return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
|
||||
}
|
||||
enum { RECOVERY_PAGE_SIZE = 8 };
|
||||
|
|
@ -1103,8 +867,8 @@ lardon3d_project_resume_recoverable_tasks(
|
|||
do {
|
||||
Lardon3DProjectRecoveryEntry entries[RECOVERY_PAGE_SIZE];
|
||||
size_t count = 0;
|
||||
result = lardon3d_project_list_recoverable(state, registry, cursor,
|
||||
entries, RECOVERY_PAGE_SIZE, &count);
|
||||
result = lardon3d_project_list_recoverable(state, registry, cursor, entries, RECOVERY_PAGE_SIZE,
|
||||
&count);
|
||||
if (result != LARDON3D_PROJECT_DB_OK) {
|
||||
++summary->failed;
|
||||
break;
|
||||
|
|
@ -1113,20 +877,17 @@ lardon3d_project_resume_recoverable_tasks(
|
|||
Lardon3DProjectRecoveryEntry *entry = &entries[index];
|
||||
cursor = entry->task_id;
|
||||
++summary->inspected;
|
||||
bool recoverable = entry->status == LARDON3D_PROJECT_RECOVERABLE
|
||||
|| entry->status
|
||||
== LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE;
|
||||
bool recoverable = entry->status == LARDON3D_PROJECT_RECOVERABLE ||
|
||||
entry->status == LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE;
|
||||
if (!recoverable || entry->snapshot.recovery_state != TASK_PENDING) {
|
||||
++summary->skipped;
|
||||
continue;
|
||||
}
|
||||
if (entry->status
|
||||
== LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE) {
|
||||
if (entry->status == LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE) {
|
||||
++summary->published_not_durable;
|
||||
}
|
||||
Lardon3DTaskSnapshot existing;
|
||||
if (lardon3d_task_queue_get(state->task_queue, entry->task_id,
|
||||
&existing)) {
|
||||
if (lardon3d_task_queue_get(state->task_queue, entry->task_id, &existing)) {
|
||||
++summary->skipped;
|
||||
continue;
|
||||
}
|
||||
|
|
@ -1136,10 +897,8 @@ lardon3d_project_resume_recoverable_tasks(
|
|||
.resource_governor = state->resource_governor,
|
||||
};
|
||||
Lardon3DTask *task = NULL;
|
||||
Lardon3DTaskKindResult restored =
|
||||
lardon3d_task_kind_registry_restore(registry,
|
||||
entry->task_kind, entry->task_kind_version,
|
||||
&entry->snapshot, &context, &task);
|
||||
Lardon3DTaskKindResult restored = lardon3d_task_kind_registry_restore(
|
||||
registry, entry->task_kind, entry->task_kind_version, &entry->snapshot, &context, &task);
|
||||
if (restored != LARDON3D_TASK_KIND_OK || !task) {
|
||||
++summary->failed;
|
||||
continue;
|
||||
|
|
@ -1158,27 +917,26 @@ lardon3d_project_resume_recoverable_tasks(
|
|||
} else {
|
||||
++summary->failed;
|
||||
}
|
||||
if (added == LARDON3D_TASK_QUEUE_ADD_FULL
|
||||
|| added == LARDON3D_TASK_QUEUE_ADD_STOPPING) {
|
||||
if (added == LARDON3D_TASK_QUEUE_ADD_FULL || added == LARDON3D_TASK_QUEUE_ADD_STOPPING) {
|
||||
summary->queue_full = added == LARDON3D_TASK_QUEUE_ADD_FULL;
|
||||
stop = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (stop || count < RECOVERY_PAGE_SIZE) break;
|
||||
if (stop || count < RECOVERY_PAGE_SIZE) {
|
||||
break;
|
||||
}
|
||||
} while (cursor > 0);
|
||||
store_recovery_summary(state, summary);
|
||||
return result;
|
||||
}
|
||||
|
||||
bool
|
||||
lardon3d_project_last_recovery_summary(
|
||||
const Lardon3DAppState *state,
|
||||
Lardon3DProjectRecoverySummary *summary
|
||||
)
|
||||
{
|
||||
if (!state || !summary) return false;
|
||||
*summary = (Lardon3DProjectRecoverySummary) {
|
||||
bool lardon3d_project_last_recovery_summary(const Lardon3DAppState *state,
|
||||
Lardon3DProjectRecoverySummary *summary) {
|
||||
if (!state || !summary) {
|
||||
return false;
|
||||
}
|
||||
*summary = (Lardon3DProjectRecoverySummary){
|
||||
.inspected = state->recovery_inspected,
|
||||
.resumed = state->recovery_resumed,
|
||||
.skipped = state->recovery_skipped,
|
||||
|
|
|
|||
1935
src/project_db.c
1935
src/project_db.c
File diff suppressed because it is too large
Load diff
|
|
@ -1,13 +1,18 @@
|
|||
#include <lardon3d/feature_task.h>
|
||||
#include <lardon3d/import_task.h>
|
||||
#include <lardon3d/task_kind_registry.h>
|
||||
|
||||
const Lardon3DTaskKindRegistry *
|
||||
lardon3d_task_kind_registry_production(void)
|
||||
{
|
||||
static const Lardon3DTaskKindDescriptor descriptors[] = {{
|
||||
const Lardon3DTaskKindRegistry *lardon3d_task_kind_registry_production(void) {
|
||||
static const Lardon3DTaskKindDescriptor descriptors[] = {
|
||||
{
|
||||
.kind = LARDON3D_IMAGE_IMPORT_TASK_KIND,
|
||||
.kind_version = LARDON3D_IMAGE_IMPORT_TASK_KIND_VERSION,
|
||||
.reconstruct = lardon3d_image_import_reconstruct,
|
||||
},
|
||||
{
|
||||
.kind = LARDON3D_FEATURE_EXTRACT_TASK_KIND,
|
||||
.kind_version = LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION,
|
||||
.reconstruct = lardon3d_feature_extract_reconstruct,
|
||||
}};
|
||||
static const Lardon3DTaskKindRegistry registry = {
|
||||
.descriptors = descriptors,
|
||||
|
|
|
|||
432
tests/test_feature_store.c
Normal file
432
tests/test_feature_store.c
Normal file
|
|
@ -0,0 +1,432 @@
|
|||
#include <dirent.h>
|
||||
#include <errno.h>
|
||||
#include <fcntl.h>
|
||||
#include <math.h>
|
||||
#include <pthread.h>
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/stat.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <lardon3d/feature_store.h>
|
||||
#include <lardon3d/image_catalog.h>
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_db_test_delete_feature_identity(Lardon3DProjectDb *,
|
||||
uint64_t, uint64_t);
|
||||
|
||||
typedef struct {
|
||||
Lardon3DAppState *state;
|
||||
uint64_t image_id;
|
||||
const Lardon3DFeatureExtractorParameters *parameters;
|
||||
const Lardon3DExtractedFeatures *features;
|
||||
Lardon3DFeatureStoreResult result;
|
||||
Lardon3DProjectDbFeatureSet feature_set;
|
||||
} PublishThreadContext;
|
||||
|
||||
static void *publish_thread(void *userdata) {
|
||||
PublishThreadContext *context = userdata;
|
||||
context->result =
|
||||
lardon3d_feature_store_publish(context->state, context->image_id, 0, context->parameters,
|
||||
context->features, &context->feature_set);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
#define CHECK(x) \
|
||||
do { \
|
||||
if (!(x)) { \
|
||||
fprintf(stderr, "Échec ligne %d: %s\n", __LINE__, #x); \
|
||||
return false; \
|
||||
} \
|
||||
} while (0)
|
||||
|
||||
static bool join_path(char out[PATH_MAX], const char *a, const char *b) {
|
||||
int n = snprintf(out, PATH_MAX, "%s/%s", a, b);
|
||||
return n > 0 && (size_t)n < PATH_MAX;
|
||||
}
|
||||
static bool remove_tree(const char *path) {
|
||||
struct stat st;
|
||||
if (lstat(path, &st) != 0) {
|
||||
return errno == ENOENT;
|
||||
}
|
||||
if (!S_ISDIR(st.st_mode)) {
|
||||
return unlink(path) == 0;
|
||||
}
|
||||
DIR *d = opendir(path);
|
||||
if (!d) {
|
||||
return false;
|
||||
}
|
||||
bool ok = true;
|
||||
for (struct dirent *e = readdir(d); e; e = readdir(d)) {
|
||||
if (!strcmp(e->d_name, ".") || !strcmp(e->d_name, "..")) {
|
||||
continue;
|
||||
}
|
||||
char child[PATH_MAX];
|
||||
if (!join_path(child, path, e->d_name) || !remove_tree(child)) {
|
||||
ok = false;
|
||||
}
|
||||
}
|
||||
if (closedir(d) != 0 || rmdir(path) != 0) {
|
||||
ok = false;
|
||||
}
|
||||
return ok;
|
||||
}
|
||||
static size_t count_files(const char *path) {
|
||||
DIR *d = opendir(path);
|
||||
if (!d) {
|
||||
return 0;
|
||||
}
|
||||
size_t count = 0;
|
||||
for (struct dirent *e = readdir(d); e; e = readdir(d)) {
|
||||
if (!strcmp(e->d_name, ".") || !strcmp(e->d_name, "..")) {
|
||||
continue;
|
||||
}
|
||||
char child[PATH_MAX];
|
||||
struct stat s;
|
||||
if (join_path(child, path, e->d_name) && lstat(child, &s) == 0) {
|
||||
if (S_ISDIR(s.st_mode)) {
|
||||
count += count_files(child);
|
||||
} else if (S_ISREG(s.st_mode)) {
|
||||
++count;
|
||||
}
|
||||
}
|
||||
}
|
||||
closedir(d);
|
||||
return count;
|
||||
}
|
||||
|
||||
static bool write_pgm(const char *path, bool uniform) {
|
||||
FILE *f = fopen(path, "wb");
|
||||
if (!f) {
|
||||
return false;
|
||||
}
|
||||
if (fprintf(f, "P5\n128 128\n255\n") <= 0) {
|
||||
fclose(f);
|
||||
return false;
|
||||
}
|
||||
for (unsigned int y = 0; y < 128; ++y) {
|
||||
for (unsigned int x = 0; x < 128; ++x) {
|
||||
unsigned char v = uniform ? 128U : (unsigned char)((((x / 8U) ^ (y / 8U)) & 1U) ? 240U : 15U);
|
||||
if (fwrite(&v, 1, 1, f) != 1) {
|
||||
fclose(f);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
return fclose(f) == 0;
|
||||
}
|
||||
|
||||
static uint64_t read_u64_le(const unsigned char *bytes) {
|
||||
uint64_t value = 0;
|
||||
for (unsigned int index = 0; index < 8; ++index) {
|
||||
value |= (uint64_t)bytes[index] << (8U * index);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
static bool restore_file(const char *path, const unsigned char *bytes, size_t size) {
|
||||
int descriptor = open(path, O_WRONLY | O_TRUNC | O_CLOEXEC);
|
||||
if (descriptor < 0) {
|
||||
return false;
|
||||
}
|
||||
size_t written = 0;
|
||||
while (written < size) {
|
||||
ssize_t amount = write(descriptor, bytes + written, size - written);
|
||||
if (amount <= 0) {
|
||||
close(descriptor);
|
||||
return false;
|
||||
}
|
||||
written += (size_t)amount;
|
||||
}
|
||||
return close(descriptor) == 0;
|
||||
}
|
||||
|
||||
static bool expect_reader_result(const char *root, const Lardon3DProjectDbFeatureSet *feature_set,
|
||||
Lardon3DFeatureStoreResult expected) {
|
||||
Lardon3DFeatureReader *reader = NULL;
|
||||
Lardon3DFeatureFileMetadata metadata;
|
||||
Lardon3DFeatureStoreResult result =
|
||||
lardon3d_feature_reader_open(root, feature_set, &reader, &metadata);
|
||||
lardon3d_feature_reader_close(reader);
|
||||
return result == expected;
|
||||
}
|
||||
|
||||
static bool run_test(void) {
|
||||
char root[] = "/tmp/lardon3d-feature-store-XXXXXX";
|
||||
CHECK(mkdtemp(root));
|
||||
char dbpath[PATH_MAX], structured[PATH_MAX], uniform[PATH_MAX];
|
||||
CHECK(join_path(dbpath, root, "project.db") && join_path(structured, root, "structured.pgm") &&
|
||||
join_path(uniform, root, "uniform.pgm"));
|
||||
CHECK(write_pgm(structured, false) && write_pgm(uniform, true));
|
||||
Lardon3DProjectDb *db = NULL;
|
||||
char error[256];
|
||||
CHECK(lardon3d_project_db_open(dbpath, &db, error) == LARDON3D_PROJECT_DB_OK);
|
||||
Lardon3DAppState state;
|
||||
lardon3d_app_state_init(&state);
|
||||
state.project_loaded = true;
|
||||
state.project_db = db;
|
||||
CHECK(snprintf(state.project_path, sizeof(state.project_path), "%s", root) > 0);
|
||||
Lardon3DProjectDbScanSet a, b;
|
||||
CHECK(lardon3d_image_catalog_create_scanset(&state, "A", &a) &&
|
||||
lardon3d_image_catalog_create_scanset(&state, "B", &b));
|
||||
Lardon3DProjectDbImage ia, ib, iu;
|
||||
Lardon3DProjectDbImageAsset aa, ab, au;
|
||||
CHECK(lardon3d_image_catalog_import_file(&state, a.scanset_id, structured, 0, &ia, &aa) ==
|
||||
LARDON3D_IMAGE_CATALOG_IMPORTED);
|
||||
CHECK(lardon3d_image_catalog_import_file(&state, b.scanset_id, structured, 0, &ib, &ab) ==
|
||||
LARDON3D_IMAGE_CATALOG_IMPORTED &&
|
||||
aa.asset_id == ab.asset_id);
|
||||
CHECK(lardon3d_image_catalog_import_file(&state, a.scanset_id, uniform, 0, &iu, &au) ==
|
||||
LARDON3D_IMAGE_CATALOG_IMPORTED);
|
||||
Lardon3DFeatureExtractorParameters p = {
|
||||
.max_features = 512, .pyramid_levels = 4, .fast_threshold = 10};
|
||||
CHECK(lardon3d_feature_extractor_parameters_valid(&p));
|
||||
CHECK(!lardon3d_feature_extractor_parameters_valid(
|
||||
&(Lardon3DFeatureExtractorParameters){0, 4, 10}));
|
||||
CHECK(!lardon3d_feature_extractor_parameters_valid(
|
||||
&(Lardon3DFeatureExtractorParameters){8193, 4, 10}));
|
||||
CHECK(!lardon3d_feature_extractor_parameters_valid(
|
||||
&(Lardon3DFeatureExtractorParameters){512, 0, 10}));
|
||||
CHECK(!lardon3d_feature_extractor_parameters_valid(
|
||||
&(Lardon3DFeatureExtractorParameters){512, 17, 10}));
|
||||
CHECK(!lardon3d_feature_extractor_parameters_valid(
|
||||
&(Lardon3DFeatureExtractorParameters){512, 4, 0}));
|
||||
CHECK(!lardon3d_feature_extractor_parameters_valid(
|
||||
&(Lardon3DFeatureExtractorParameters){512, 4, 256}));
|
||||
unsigned char fp[32], fp2[32];
|
||||
lardon3d_feature_extractor_parameter_fingerprint(&p, fp);
|
||||
lardon3d_feature_extractor_parameter_fingerprint(&p, fp2);
|
||||
CHECK(memcmp(fp, fp2, 32) == 0);
|
||||
Lardon3DFeatureExtractorParameters p2 = p;
|
||||
p2.max_features = 256;
|
||||
lardon3d_feature_extractor_parameter_fingerprint(&p2, fp2);
|
||||
CHECK(memcmp(fp, fp2, 32) != 0);
|
||||
Lardon3DExtractedFeatures f;
|
||||
CHECK(lardon3d_feature_extract_orb(structured, &p, &f) == LARDON3D_FEATURE_EXTRACT_OK &&
|
||||
f.feature_count > 0 && f.feature_count <= 512);
|
||||
Lardon3DProjectDbFeatureSet sa, same, sb;
|
||||
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &p, &f, &sa) ==
|
||||
LARDON3D_FEATURE_STORE_OK);
|
||||
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &p, &f, &same) ==
|
||||
LARDON3D_FEATURE_STORE_ALREADY_PRESENT &&
|
||||
same.feature_set_id == sa.feature_set_id);
|
||||
CHECK(lardon3d_feature_store_publish(&state, ib.image_id, 0, &p, &f, &sb) ==
|
||||
LARDON3D_FEATURE_STORE_OK &&
|
||||
sb.feature_set_id != sa.feature_set_id && sb.feature_asset_id == sa.feature_asset_id);
|
||||
Lardon3DFeatureReader *reader = NULL;
|
||||
Lardon3DFeatureFileMetadata metadata;
|
||||
CHECK(lardon3d_feature_reader_open(root, &sa, &reader, &metadata) == LARDON3D_FEATURE_STORE_OK &&
|
||||
metadata.feature_count == f.feature_count);
|
||||
size_t chunk = metadata.feature_count < 8 ? metadata.feature_count : 8;
|
||||
Lardon3DFeatureKeypoint points[8];
|
||||
unsigned char descriptors[8 * 32];
|
||||
CHECK(lardon3d_feature_reader_keypoints(reader, 0, points, chunk) == LARDON3D_FEATURE_STORE_OK);
|
||||
CHECK(lardon3d_feature_reader_descriptors(reader, 0, descriptors, chunk, sizeof(descriptors)) ==
|
||||
LARDON3D_FEATURE_STORE_OK);
|
||||
for (size_t i = 0; i < chunk; ++i) {
|
||||
CHECK(isfinite(points[i].x) && memcmp(descriptors + i * 32, f.descriptors + i * 32, 32) == 0);
|
||||
}
|
||||
CHECK(lardon3d_feature_reader_keypoints(reader, metadata.feature_count, points, 1) ==
|
||||
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT);
|
||||
CHECK(lardon3d_feature_reader_descriptors(reader, UINT32_MAX, descriptors, 1,
|
||||
sizeof(descriptors)) ==
|
||||
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT);
|
||||
CHECK(lardon3d_feature_reader_keypoints(reader, 0, points, 257) ==
|
||||
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT);
|
||||
CHECK(lardon3d_feature_reader_descriptors(reader, 0, descriptors, 257, sizeof(descriptors)) ==
|
||||
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT);
|
||||
lardon3d_feature_reader_close(reader);
|
||||
|
||||
const char *invalid_paths[] = {
|
||||
"/assets/features/00/hash",
|
||||
"../assets/features/00/hash",
|
||||
"assets/features/AA/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
|
||||
"assets/features/aa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
|
||||
"assets/features/0/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
|
||||
"assets/features/aa/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
|
||||
};
|
||||
for (size_t index = 0; index < sizeof(invalid_paths) / sizeof(invalid_paths[0]); ++index) {
|
||||
Lardon3DProjectDbFeatureSet invalid_path_set = sa;
|
||||
CHECK(snprintf(invalid_path_set.asset.path, sizeof(invalid_path_set.asset.path), "%s",
|
||||
invalid_paths[index]) > 0);
|
||||
CHECK(lardon3d_feature_reader_open(root, &invalid_path_set, &reader, &metadata) ==
|
||||
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT);
|
||||
}
|
||||
|
||||
Lardon3DExtractedFeatures invalid_features = f;
|
||||
Lardon3DFeatureKeypoint invalid_point = f.keypoints[0];
|
||||
invalid_point.x = NAN;
|
||||
invalid_features.keypoints = &invalid_point;
|
||||
invalid_features.feature_count = 1;
|
||||
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &p2, &invalid_features, &same) ==
|
||||
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT);
|
||||
|
||||
Lardon3DFeatureExtractorParameters concurrent_parameters = {508, 4, 10};
|
||||
PublishThreadContext concurrent[2] = {
|
||||
{.state = &state,
|
||||
.image_id = ia.image_id,
|
||||
.parameters = &concurrent_parameters,
|
||||
.features = &f},
|
||||
{.state = &state,
|
||||
.image_id = ia.image_id,
|
||||
.parameters = &concurrent_parameters,
|
||||
.features = &f},
|
||||
};
|
||||
pthread_t publish_threads[2];
|
||||
CHECK(pthread_create(&publish_threads[0], NULL, publish_thread, &concurrent[0]) == 0 &&
|
||||
pthread_create(&publish_threads[1], NULL, publish_thread, &concurrent[1]) == 0);
|
||||
CHECK(pthread_join(publish_threads[0], NULL) == 0 && pthread_join(publish_threads[1], NULL) == 0);
|
||||
CHECK((concurrent[0].result == LARDON3D_FEATURE_STORE_OK ||
|
||||
concurrent[0].result == LARDON3D_FEATURE_STORE_ALREADY_PRESENT) &&
|
||||
(concurrent[1].result == LARDON3D_FEATURE_STORE_OK ||
|
||||
concurrent[1].result == LARDON3D_FEATURE_STORE_ALREADY_PRESENT) &&
|
||||
concurrent[0].feature_set.feature_set_id == concurrent[1].feature_set.feature_set_id);
|
||||
Lardon3DExtractedFeatures uf;
|
||||
CHECK(lardon3d_feature_extract_orb(uniform, &p, &uf) == LARDON3D_FEATURE_EXTRACT_OK &&
|
||||
uf.feature_count == 0);
|
||||
Lardon3DProjectDbFeatureSet us;
|
||||
CHECK(lardon3d_feature_store_publish(&state, iu.image_id, 0, &p, &uf, &us) ==
|
||||
LARDON3D_FEATURE_STORE_OK);
|
||||
CHECK(lardon3d_feature_reader_open(root, &us, &reader, &metadata) == LARDON3D_FEATURE_STORE_OK &&
|
||||
metadata.feature_count == 0);
|
||||
lardon3d_feature_reader_close(reader);
|
||||
Lardon3DFeatureExtractorParameters large_parameters = {8192, 4, 10};
|
||||
Lardon3DExtractedFeatures large = {
|
||||
.image_width = 128, .image_height = 128, .feature_count = 8192};
|
||||
large.keypoints = calloc(large.feature_count, sizeof(*large.keypoints));
|
||||
large.descriptors = calloc(large.feature_count, 32);
|
||||
CHECK(large.keypoints && large.descriptors);
|
||||
for (uint32_t i = 0; i < large.feature_count; ++i) {
|
||||
large.keypoints[i] =
|
||||
(Lardon3DFeatureKeypoint){(float)(i % 128), (float)((i / 128) % 64), 1, 0, 1, 0};
|
||||
memset(large.descriptors + (size_t)i * 32, (int)(i & 255), 32);
|
||||
}
|
||||
Lardon3DProjectDbFeatureSet large_set;
|
||||
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &large_parameters, &large,
|
||||
&large_set) == LARDON3D_FEATURE_STORE_OK);
|
||||
CHECK(lardon3d_feature_reader_open(root, &large_set, &reader, &metadata) ==
|
||||
LARDON3D_FEATURE_STORE_OK &&
|
||||
metadata.feature_count == 8192);
|
||||
Lardon3DFeatureKeypoint last;
|
||||
unsigned char last_descriptor[32];
|
||||
CHECK(lardon3d_feature_reader_keypoints(reader, 8191, &last, 1) == LARDON3D_FEATURE_STORE_OK &&
|
||||
lardon3d_feature_reader_descriptors(reader, 8191, last_descriptor, 1,
|
||||
sizeof(last_descriptor)) == LARDON3D_FEATURE_STORE_OK &&
|
||||
last_descriptor[0] == 255);
|
||||
lardon3d_feature_reader_close(reader);
|
||||
lardon3d_extracted_features_destroy(&large);
|
||||
char features_dir[PATH_MAX];
|
||||
CHECK(join_path(features_dir, root, "assets/features"));
|
||||
size_t before = count_files(features_dir);
|
||||
Lardon3DFeatureExtractorParameters failed_parameters = {511, 4, 10};
|
||||
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_FEATURE_REGISTER", "1", 1) == 0);
|
||||
Lardon3DProjectDbFeatureSet failed_set;
|
||||
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &failed_parameters, &f,
|
||||
&failed_set) == LARDON3D_FEATURE_STORE_DB_BUSY);
|
||||
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_FEATURE_REGISTER") == 0 &&
|
||||
count_files(features_dir) == before + 1);
|
||||
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &failed_parameters, &f,
|
||||
&failed_set) == LARDON3D_FEATURE_STORE_OK);
|
||||
Lardon3DFeatureExtractorParameters nondurable_parameters = {510, 4, 10};
|
||||
CHECK(setenv("LARDON3D_TEST_FEATURE_FAIL_DIRECTORY_SYNC", "1", 1) == 0);
|
||||
Lardon3DProjectDbFeatureSet nondurable;
|
||||
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &nondurable_parameters, &f,
|
||||
&nondurable) ==
|
||||
LARDON3D_FEATURE_STORE_PUBLISHED_NOT_DURABLE &&
|
||||
nondurable.asset.durability == LARDON3D_DB_FEATURE_ASSET_PUBLISHED_NOT_DURABLE);
|
||||
CHECK(unsetenv("LARDON3D_TEST_FEATURE_FAIL_DIRECTORY_SYNC") == 0);
|
||||
Lardon3DProjectDbFeatureSet promoted;
|
||||
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &nondurable_parameters, &f,
|
||||
&promoted) == LARDON3D_FEATURE_STORE_ALREADY_PRESENT &&
|
||||
promoted.feature_set_id == nondurable.feature_set_id &&
|
||||
promoted.asset.durability == LARDON3D_DB_FEATURE_ASSET_DURABLE);
|
||||
uint64_t deleted_set = nondurable.feature_set_id, deleted_asset = nondurable.feature_asset_id;
|
||||
CHECK(lardon3d_project_db_test_delete_feature_identity(db, deleted_set, deleted_asset) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
Lardon3DFeatureExtractorParameters replacement_parameters = {509, 4, 10};
|
||||
Lardon3DProjectDbFeatureSet replacement;
|
||||
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &replacement_parameters, &f,
|
||||
&replacement) == LARDON3D_FEATURE_STORE_OK &&
|
||||
replacement.feature_set_id > deleted_set && replacement.feature_asset_id > deleted_asset);
|
||||
Lardon3DProjectDbFeatureSet page[8];
|
||||
size_t count = 0;
|
||||
CHECK(lardon3d_project_db_list_feature_sets(db, 0, page, 1, &count) == LARDON3D_PROJECT_DB_OK &&
|
||||
count == 1);
|
||||
CHECK(lardon3d_project_db_list_feature_sets(db, page[0].feature_set_id, page, 8, &count) ==
|
||||
LARDON3D_PROJECT_DB_OK &&
|
||||
count == 6);
|
||||
for (uint32_t index = 0; index < 2000; ++index) {
|
||||
unsigned char metadata_fingerprint[32] = {0};
|
||||
metadata_fingerprint[0] = (unsigned char)index;
|
||||
metadata_fingerprint[1] = (unsigned char)(index >> 8);
|
||||
metadata_fingerprint[2] = 0xa5;
|
||||
Lardon3DProjectDbFeatureSet metadata_set;
|
||||
CHECK(lardon3d_project_db_register_feature_set(
|
||||
db, ia.image_id, "metadata.fixture", 1, metadata_fingerprint, aa.sha256,
|
||||
sa.feature_count, sa.descriptor_type, sa.descriptor_dimension, sa.asset.sha256,
|
||||
sa.asset.path, sa.asset.size_bytes, sa.asset.durability, 0, index,
|
||||
&metadata_set) == LARDON3D_PROJECT_DB_OK);
|
||||
}
|
||||
Lardon3DProjectDbFeatureSet *metadata_page = calloc(256, sizeof(*metadata_page));
|
||||
CHECK(metadata_page);
|
||||
const size_t page_sizes[] = {1, 8, 64, 256};
|
||||
for (size_t index = 0; index < sizeof(page_sizes) / sizeof(page_sizes[0]); ++index) {
|
||||
CHECK(lardon3d_project_db_list_feature_sets(db, 0, metadata_page, page_sizes[index], &count) ==
|
||||
LARDON3D_PROJECT_DB_OK &&
|
||||
count == page_sizes[index]);
|
||||
}
|
||||
CHECK(lardon3d_project_db_list_feature_sets(db, 0, metadata_page, 257, &count) ==
|
||||
LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||
free(metadata_page);
|
||||
char asset_path[PATH_MAX];
|
||||
CHECK(join_path(asset_path, root, sa.asset.path));
|
||||
int fd = open(asset_path, O_RDONLY | O_CLOEXEC);
|
||||
CHECK(fd >= 0);
|
||||
struct stat asset_info;
|
||||
CHECK(fstat(fd, &asset_info) == 0 && asset_info.st_size >= 160);
|
||||
size_t original_size = (size_t)asset_info.st_size;
|
||||
unsigned char *original = malloc(original_size);
|
||||
CHECK(original && read(fd, original, original_size) == (ssize_t)original_size && close(fd) == 0);
|
||||
|
||||
uint64_t descriptor_offset = read_u64_le(original + 48);
|
||||
const off_t truncated_sizes[] = {
|
||||
100,
|
||||
(off_t)descriptor_offset - 1,
|
||||
(off_t)descriptor_offset,
|
||||
(off_t)original_size - 1,
|
||||
};
|
||||
for (size_t index = 0; index < sizeof(truncated_sizes) / sizeof(truncated_sizes[0]); ++index) {
|
||||
CHECK(truncate(asset_path, truncated_sizes[index]) == 0 &&
|
||||
expect_reader_result(root, &sa, LARDON3D_FEATURE_STORE_CORRUPT) &&
|
||||
restore_file(asset_path, original, original_size));
|
||||
}
|
||||
|
||||
const size_t corrupt_offsets[] = {12, 16, 20, 24, 28, 40, 48, 56, 148, 144, 128};
|
||||
for (size_t index = 0; index < sizeof(corrupt_offsets) / sizeof(corrupt_offsets[0]); ++index) {
|
||||
unsigned char mutated = (unsigned char)(original[corrupt_offsets[index]] ^ 0x5aU);
|
||||
fd = open(asset_path, O_WRONLY | O_CLOEXEC);
|
||||
CHECK(fd >= 0 && pwrite(fd, &mutated, 1, (off_t)corrupt_offsets[index]) == 1 &&
|
||||
close(fd) == 0 && expect_reader_result(root, &sa, LARDON3D_FEATURE_STORE_CORRUPT) &&
|
||||
restore_file(asset_path, original, original_size));
|
||||
}
|
||||
|
||||
unsigned char future_version[4] = {2, 0, 0, 0};
|
||||
fd = open(asset_path, O_WRONLY | O_CLOEXEC);
|
||||
CHECK(fd >= 0 && pwrite(fd, future_version, sizeof(future_version), 8) == 4 && close(fd) == 0 &&
|
||||
expect_reader_result(root, &sa, LARDON3D_FEATURE_STORE_UNSUPPORTED_VERSION) &&
|
||||
restore_file(asset_path, original, original_size));
|
||||
|
||||
fd = open(asset_path, O_WRONLY | O_TRUNC | O_CLOEXEC);
|
||||
CHECK(fd >= 0 && write(fd, "bad", 3) == 3 && close(fd) == 0 &&
|
||||
expect_reader_result(root, &sa, LARDON3D_FEATURE_STORE_CORRUPT));
|
||||
free(original);
|
||||
lardon3d_extracted_features_destroy(&f);
|
||||
lardon3d_extracted_features_destroy(&uf);
|
||||
lardon3d_project_db_close(db);
|
||||
CHECK(remove_tree(root));
|
||||
return true;
|
||||
}
|
||||
|
||||
int main(void) { return run_test() ? 0 : 1; }
|
||||
188
tests/test_feature_task.c
Normal file
188
tests/test_feature_task.c
Normal file
|
|
@ -0,0 +1,188 @@
|
|||
#include <dirent.h>
|
||||
#include <errno.h>
|
||||
#include <sched.h>
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/stat.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <lardon3d/feature_store.h>
|
||||
#include <lardon3d/feature_task.h>
|
||||
#include <lardon3d/image_catalog.h>
|
||||
#include <lardon3d/project.h>
|
||||
#include <lardon3d/task_queue.h>
|
||||
|
||||
#define CHECK(x) \
|
||||
do { \
|
||||
if (!(x)) { \
|
||||
fprintf(stderr, "Échec ligne %d: %s\n", __LINE__, #x); \
|
||||
return false; \
|
||||
} \
|
||||
} while (0)
|
||||
static bool join_path(char out[PATH_MAX], const char *a, const char *b) {
|
||||
int n = snprintf(out, PATH_MAX, "%s/%s", a, b);
|
||||
return n > 0 && (size_t)n < PATH_MAX;
|
||||
}
|
||||
static bool remove_tree(const char *p) {
|
||||
struct stat s;
|
||||
if (lstat(p, &s) != 0) {
|
||||
return errno == ENOENT;
|
||||
}
|
||||
if (!S_ISDIR(s.st_mode)) {
|
||||
return unlink(p) == 0;
|
||||
}
|
||||
DIR *d = opendir(p);
|
||||
if (!d) {
|
||||
return false;
|
||||
}
|
||||
bool ok = true;
|
||||
for (struct dirent *e = readdir(d); e; e = readdir(d)) {
|
||||
if (!strcmp(e->d_name, ".") || !strcmp(e->d_name, "..")) {
|
||||
continue;
|
||||
}
|
||||
char c[PATH_MAX];
|
||||
if (!join_path(c, p, e->d_name) || !remove_tree(c)) {
|
||||
ok = false;
|
||||
}
|
||||
}
|
||||
if (closedir(d) || rmdir(p)) {
|
||||
ok = false;
|
||||
}
|
||||
return ok;
|
||||
}
|
||||
static bool write_pgm(const char *p) {
|
||||
FILE *f = fopen(p, "wb");
|
||||
if (!f) {
|
||||
return false;
|
||||
}
|
||||
fprintf(f, "P5\n192 192\n255\n");
|
||||
for (unsigned y = 0; y < 192; y++) {
|
||||
for (unsigned x = 0; x < 192; x++) {
|
||||
unsigned char v = (unsigned char)((((x / 6) ^ (y / 6)) & 1) ? 245 : 10);
|
||||
if (fwrite(&v, 1, 1, f) != 1) {
|
||||
fclose(f);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
return fclose(f) == 0;
|
||||
}
|
||||
static bool runtime(Lardon3DAppState *s) {
|
||||
s->hardware_profile = (Lardon3DHardwareProfile){.logical_cpu_count = 64,
|
||||
.page_size_bytes = 4096,
|
||||
.memory_total_bytes = UINT64_MAX,
|
||||
.cpu_architecture = "test"};
|
||||
Lardon3DResourcePolicy p = {
|
||||
.maximum_cpu_load_ratio = 1, .maximum_io_pressure_avg10 = 100, .io_slot_capacity = 2};
|
||||
s->resource_governor = lardon3d_resource_governor_create(&s->hardware_profile, &p);
|
||||
s->task_queue = s->resource_governor ? lardon3d_task_queue_create(s->resource_governor, 4) : NULL;
|
||||
return s->task_queue != NULL;
|
||||
}
|
||||
static bool wait_state(Lardon3DTaskQueue *q, uint64_t id, Lardon3DTaskState wanted,
|
||||
Lardon3DTaskSnapshot *out) {
|
||||
for (size_t i = 0; i < 2000000; i++) {
|
||||
if (lardon3d_task_queue_get(q, id, out) && out->state == wanted) {
|
||||
return true;
|
||||
}
|
||||
sched_yield();
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
static bool run_test(void) {
|
||||
char root[] = "/tmp/lardon3d-feature-task-XXXXXX";
|
||||
CHECK(mkdtemp(root) && setenv("LARDON3D_PROJECTS_ROOT", root, 1) == 0);
|
||||
char source[PATH_MAX];
|
||||
CHECK(join_path(source, root, "source.pgm") && write_pgm(source));
|
||||
Lardon3DAppState state;
|
||||
lardon3d_app_state_init(&state);
|
||||
CHECK(runtime(&state) && lardon3d_project_create(&state, "Features"));
|
||||
Lardon3DProjectDbScanSet scanset;
|
||||
CHECK(lardon3d_image_catalog_create_scanset(&state, "A", &scanset));
|
||||
Lardon3DProjectDbImage image;
|
||||
Lardon3DProjectDbImageAsset asset;
|
||||
CHECK(lardon3d_image_catalog_import_file(&state, scanset.scanset_id, source, 0, &image, &asset) ==
|
||||
LARDON3D_IMAGE_CATALOG_IMPORTED);
|
||||
Lardon3DFeatureExtractorParameters parameters = {512, 4, 10};
|
||||
CHECK(setenv("LARDON3D_TEST_FEATURE_PAUSE_BEFORE_PUBLISH", "1", 1) == 0 &&
|
||||
setenv("LARDON3D_TEST_FEATURE_SKIP_FINISHED_CHECKPOINT", "1", 1) == 0);
|
||||
uint64_t task_id = 0;
|
||||
CHECK(lardon3d_project_enqueue_feature_extract(&state, image.image_id, ¶meters, &task_id));
|
||||
Lardon3DTaskSnapshot snapshot;
|
||||
CHECK(wait_state(state.task_queue, task_id, TASK_PAUSED, &snapshot));
|
||||
lardon3d_task_queue_destroy(state.task_queue);
|
||||
state.task_queue = NULL;
|
||||
lardon3d_project_close(&state);
|
||||
lardon3d_resource_governor_destroy(state.resource_governor);
|
||||
state.resource_governor = NULL;
|
||||
CHECK(unsetenv("LARDON3D_TEST_FEATURE_PAUSE_BEFORE_PUBLISH") == 0 &&
|
||||
unsetenv("LARDON3D_TEST_FEATURE_SKIP_FINISHED_CHECKPOINT") == 0);
|
||||
lardon3d_app_state_init(&state);
|
||||
CHECK(runtime(&state) && lardon3d_project_open(&state, "Features"));
|
||||
Lardon3DProjectRecoverySummary summary;
|
||||
CHECK(lardon3d_project_last_recovery_summary(&state, &summary) && summary.resumed == 1);
|
||||
CHECK(wait_state(state.task_queue, task_id, TASK_COMPLETED, &snapshot) &&
|
||||
snapshot.progress == 100);
|
||||
unsigned char fp[32];
|
||||
lardon3d_feature_extractor_parameter_fingerprint(¶meters, fp);
|
||||
Lardon3DProjectDbFeatureSet set;
|
||||
CHECK(lardon3d_project_db_find_feature_set(state.project_db, image.image_id,
|
||||
LARDON3D_FEATURE_EXTRACTOR_KIND, 1, fp,
|
||||
&set) == LARDON3D_PROJECT_DB_OK);
|
||||
lardon3d_task_queue_destroy(state.task_queue);
|
||||
state.task_queue = NULL;
|
||||
lardon3d_project_close(&state);
|
||||
lardon3d_resource_governor_destroy(state.resource_governor);
|
||||
state.resource_governor = NULL;
|
||||
lardon3d_app_state_init(&state);
|
||||
CHECK(runtime(&state) && lardon3d_project_open(&state, "Features"));
|
||||
CHECK(lardon3d_project_last_recovery_summary(&state, &summary) && summary.resumed == 0);
|
||||
CHECK(lardon3d_project_db_load_feature_set(state.project_db, set.feature_set_id, &set) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(unlink(source) == 0);
|
||||
Lardon3DFeatureReader *reader = NULL;
|
||||
Lardon3DFeatureFileMetadata metadata;
|
||||
CHECK(lardon3d_feature_reader_open(state.project_path, &set, &reader, &metadata) ==
|
||||
LARDON3D_FEATURE_STORE_OK);
|
||||
lardon3d_feature_reader_close(reader);
|
||||
|
||||
CHECK(write_pgm(source));
|
||||
Lardon3DProjectDbImage missing_image;
|
||||
Lardon3DProjectDbImageAsset missing_asset;
|
||||
CHECK(lardon3d_image_catalog_import_file(&state, scanset.scanset_id, source, 0, &missing_image,
|
||||
&missing_asset) ==
|
||||
LARDON3D_IMAGE_CATALOG_ALREADY_PRESENT);
|
||||
char managed_path[PATH_MAX];
|
||||
CHECK(join_path(managed_path, state.project_path, missing_asset.path));
|
||||
CHECK(unlink(managed_path) == 0);
|
||||
uint64_t missing_task_id = 0;
|
||||
Lardon3DFeatureExtractorParameters missing_parameters = {511, 4, 10};
|
||||
CHECK(lardon3d_project_enqueue_feature_extract(&state, missing_image.image_id,
|
||||
&missing_parameters, &missing_task_id));
|
||||
CHECK(wait_state(state.task_queue, missing_task_id, TASK_FAILED, &snapshot));
|
||||
|
||||
char invalid_source[PATH_MAX];
|
||||
CHECK(join_path(invalid_source, root, "invalid.pgm"));
|
||||
FILE *invalid = fopen(invalid_source, "wb");
|
||||
CHECK(invalid && fwrite("not an image", 1, 12, invalid) == 12 && fclose(invalid) == 0);
|
||||
Lardon3DProjectDbImage invalid_image;
|
||||
Lardon3DProjectDbImageAsset invalid_asset;
|
||||
CHECK(lardon3d_image_catalog_import_file(&state, scanset.scanset_id, invalid_source, 0,
|
||||
&invalid_image,
|
||||
&invalid_asset) == LARDON3D_IMAGE_CATALOG_IMPORTED);
|
||||
uint64_t invalid_task_id = 0;
|
||||
Lardon3DFeatureExtractorParameters invalid_parameters = {510, 4, 10};
|
||||
CHECK(lardon3d_project_enqueue_feature_extract(&state, invalid_image.image_id,
|
||||
&invalid_parameters, &invalid_task_id));
|
||||
CHECK(wait_state(state.task_queue, invalid_task_id, TASK_FAILED, &snapshot));
|
||||
lardon3d_task_queue_destroy(state.task_queue);
|
||||
state.task_queue = NULL;
|
||||
lardon3d_project_close(&state);
|
||||
lardon3d_resource_governor_destroy(state.resource_governor);
|
||||
CHECK(remove_tree(root));
|
||||
return true;
|
||||
}
|
||||
int main(void) { return run_test() ? 0 : 1; }
|
||||
|
|
@ -11,19 +11,21 @@
|
|||
|
||||
#include <lardon3d/project_db.h>
|
||||
|
||||
#define CHECK(condition) do { if (!(condition)) { \
|
||||
(void)fprintf(stderr, "Échec ligne %d : %s\n", __LINE__, #condition); return false; \
|
||||
} } while (0)
|
||||
#define CHECK(condition) \
|
||||
do { \
|
||||
if (!(condition)) { \
|
||||
(void)fprintf(stderr, "Échec ligne %d : %s\n", __LINE__, #condition); \
|
||||
return false; \
|
||||
} \
|
||||
} while (0)
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_db_test_orphan_checkpoint(Lardon3DProjectDb *database);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_test_delete_catalog_identity(
|
||||
Lardon3DProjectDb *database, uint64_t scanset_id, uint64_t image_id,
|
||||
uint64_t asset_id);
|
||||
Lardon3DProjectDbResult
|
||||
lardon3d_project_db_test_delete_catalog_identity(Lardon3DProjectDb *database, uint64_t scanset_id,
|
||||
uint64_t image_id, uint64_t asset_id);
|
||||
|
||||
static void
|
||||
asset_path_for_hash(const unsigned char hash[LARDON3D_PROJECT_DB_SHA256_SIZE],
|
||||
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY])
|
||||
{
|
||||
static void asset_path_for_hash(const unsigned char hash[LARDON3D_PROJECT_DB_SHA256_SIZE],
|
||||
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY]) {
|
||||
static const char digits[] = "0123456789abcdef";
|
||||
char hex[65];
|
||||
for (size_t index = 0; index < sizeof(hex) / 2; ++index) {
|
||||
|
|
@ -31,13 +33,11 @@ asset_path_for_hash(const unsigned char hash[LARDON3D_PROJECT_DB_SHA256_SIZE],
|
|||
hex[index * 2 + 1] = digits[hash[index] & 15U];
|
||||
}
|
||||
hex[64] = '\0';
|
||||
(void)snprintf(path, LARDON3D_PROJECT_DB_PATH_CAPACITY,
|
||||
"assets/images/%c%c/%s", hex[0], hex[1], hex);
|
||||
(void)snprintf(path, LARDON3D_PROJECT_DB_PATH_CAPACITY, "assets/images/%c%c/%s", hex[0], hex[1],
|
||||
hex);
|
||||
}
|
||||
|
||||
static Lardon3DTaskDurableSnapshot
|
||||
task_snapshot(uint64_t id, Lardon3DTaskState saved)
|
||||
{
|
||||
static Lardon3DTaskDurableSnapshot task_snapshot(uint64_t id, Lardon3DTaskState saved) {
|
||||
Lardon3DTaskDurableSnapshot snapshot = {
|
||||
.id = id,
|
||||
.estimate = {.minimum_batch_size = 1, .maximum_batch_size = 1, .desired_cpu_threads = 1},
|
||||
|
|
@ -52,12 +52,17 @@ task_snapshot(uint64_t id, Lardon3DTaskState saved)
|
|||
return snapshot;
|
||||
}
|
||||
|
||||
typedef struct { Lardon3DProjectDb *database; bool success; } ThreadContext;
|
||||
typedef struct { Lardon3DProjectDb *database; uint64_t id; bool success; } IdThreadContext;
|
||||
typedef struct {
|
||||
Lardon3DProjectDb *database;
|
||||
bool success;
|
||||
} ThreadContext;
|
||||
typedef struct {
|
||||
Lardon3DProjectDb *database;
|
||||
uint64_t id;
|
||||
bool success;
|
||||
} IdThreadContext;
|
||||
|
||||
static void *
|
||||
read_thread(void *userdata)
|
||||
{
|
||||
static void *read_thread(void *userdata) {
|
||||
ThreadContext *context = userdata;
|
||||
context->success = true;
|
||||
for (size_t index = 0; index < 100; ++index) {
|
||||
|
|
@ -70,37 +75,49 @@ read_thread(void *userdata)
|
|||
return NULL;
|
||||
}
|
||||
|
||||
static void *
|
||||
allocate_id_thread(void *userdata)
|
||||
{
|
||||
static void *allocate_id_thread(void *userdata) {
|
||||
IdThreadContext *context = userdata;
|
||||
context->success = lardon3d_project_db_allocate_task_id(
|
||||
context->database, &context->id) == LARDON3D_PROJECT_DB_OK;
|
||||
context->success = lardon3d_project_db_allocate_task_id(context->database, &context->id) ==
|
||||
LARDON3D_PROJECT_DB_OK;
|
||||
return NULL;
|
||||
}
|
||||
|
||||
static bool
|
||||
create_future_database(const char *path)
|
||||
{
|
||||
static bool create_future_database(const char *path) {
|
||||
sqlite3 *connection = NULL;
|
||||
if (sqlite3_open(path, &connection) != SQLITE_OK) return false;
|
||||
bool ok = sqlite3_exec(connection, "CREATE TABLE metadata(key TEXT PRIMARY KEY,value INTEGER NOT NULL);"
|
||||
"INSERT INTO metadata VALUES('schema_version',5);", NULL, NULL, NULL) == SQLITE_OK;
|
||||
if (sqlite3_open(path, &connection) != SQLITE_OK) {
|
||||
return false;
|
||||
}
|
||||
bool ok = sqlite3_exec(connection,
|
||||
"CREATE TABLE metadata(key TEXT PRIMARY KEY,value INTEGER NOT NULL);"
|
||||
"INSERT INTO metadata VALUES('schema_version',6);",
|
||||
NULL, NULL, NULL) == SQLITE_OK;
|
||||
return sqlite3_close(connection) == SQLITE_OK && ok;
|
||||
}
|
||||
|
||||
static bool
|
||||
create_v1_database(const char *path)
|
||||
{
|
||||
static bool create_v1_database(const char *path) {
|
||||
static const char sql[] =
|
||||
"PRAGMA foreign_keys=ON;"
|
||||
"CREATE TABLE metadata(key TEXT PRIMARY KEY,value INTEGER NOT NULL);"
|
||||
"INSERT INTO metadata VALUES('schema_version',1);"
|
||||
"CREATE TABLE project(singleton INTEGER PRIMARY KEY CHECK(singleton=1),stable_id TEXT NOT NULL UNIQUE,name TEXT NOT NULL,created_at INTEGER NOT NULL,updated_at INTEGER NOT NULL);"
|
||||
"CREATE TABLE tasks(task_id INTEGER PRIMARY KEY CHECK(task_id>0),name TEXT NOT NULL,saved_state INTEGER NOT NULL CHECK(saved_state BETWEEN 0 AND 5),recovery_state INTEGER NOT NULL CHECK(recovery_state BETWEEN 0 AND 5),progress INTEGER NOT NULL CHECK(progress BETWEEN 0 AND 100),sequence_count INTEGER NOT NULL CHECK(sequence_count>=0),started_sec INTEGER NOT NULL,started_nsec INTEGER NOT NULL CHECK(started_nsec BETWEEN 0 AND 999999999),finished_sec INTEGER NOT NULL,finished_nsec INTEGER NOT NULL CHECK(finished_nsec BETWEEN 0 AND 999999999),updated_at INTEGER NOT NULL);"
|
||||
"CREATE TABLE project(singleton INTEGER PRIMARY KEY CHECK(singleton=1),stable_id TEXT NOT "
|
||||
"NULL UNIQUE,name TEXT NOT NULL,created_at INTEGER NOT NULL,updated_at INTEGER NOT NULL);"
|
||||
"CREATE TABLE tasks(task_id INTEGER PRIMARY KEY CHECK(task_id>0),name TEXT NOT "
|
||||
"NULL,saved_state INTEGER NOT NULL CHECK(saved_state BETWEEN 0 AND 5),recovery_state "
|
||||
"INTEGER NOT NULL CHECK(recovery_state BETWEEN 0 AND 5),progress INTEGER NOT NULL "
|
||||
"CHECK(progress BETWEEN 0 AND 100),sequence_count INTEGER NOT NULL "
|
||||
"CHECK(sequence_count>=0),started_sec INTEGER NOT NULL,started_nsec INTEGER NOT NULL "
|
||||
"CHECK(started_nsec BETWEEN 0 AND 999999999),finished_sec INTEGER NOT NULL,finished_nsec "
|
||||
"INTEGER NOT NULL CHECK(finished_nsec BETWEEN 0 AND 999999999),updated_at INTEGER NOT "
|
||||
"NULL);"
|
||||
"CREATE INDEX tasks_recovery_state_idx ON tasks(recovery_state,task_id);"
|
||||
"CREATE TABLE checkpoints(task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE CASCADE,path TEXT NOT NULL,format_version INTEGER NOT NULL CHECK(format_version>0),durability INTEGER NOT NULL CHECK(durability BETWEEN 0 AND 1),updated_at INTEGER NOT NULL);"
|
||||
"CREATE TABLE artifacts(artifact_id TEXT PRIMARY KEY,kind TEXT NOT NULL,path TEXT NOT NULL,state INTEGER NOT NULL CHECK(state BETWEEN 0 AND 1),size_bytes INTEGER NOT NULL CHECK(size_bytes>=0),producer_task_id INTEGER REFERENCES tasks(task_id),created_at INTEGER NOT NULL,updated_at INTEGER NOT NULL);"
|
||||
"CREATE TABLE checkpoints(task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE "
|
||||
"CASCADE,path TEXT NOT NULL,format_version INTEGER NOT NULL "
|
||||
"CHECK(format_version>0),durability INTEGER NOT NULL CHECK(durability BETWEEN 0 AND "
|
||||
"1),updated_at INTEGER NOT NULL);"
|
||||
"CREATE TABLE artifacts(artifact_id TEXT PRIMARY KEY,kind TEXT NOT NULL,path TEXT NOT "
|
||||
"NULL,state INTEGER NOT NULL CHECK(state BETWEEN 0 AND 1),size_bytes INTEGER NOT NULL "
|
||||
"CHECK(size_bytes>=0),producer_task_id INTEGER REFERENCES tasks(task_id),created_at "
|
||||
"INTEGER NOT NULL,updated_at INTEGER NOT NULL);"
|
||||
"CREATE INDEX artifacts_state_idx ON artifacts(state,artifact_id);"
|
||||
"CREATE INDEX artifacts_producer_idx ON artifacts(producer_task_id);"
|
||||
"INSERT INTO project VALUES(1,'legacy-project','Legacy',1,1);"
|
||||
|
|
@ -108,34 +125,43 @@ create_v1_database(const char *path)
|
|||
"INSERT INTO checkpoints VALUES(9,'legacy.chk',1,0,2);"
|
||||
"INSERT INTO artifacts VALUES('legacy-artifact','legacy','legacy.bin',0,0,9,2,2);";
|
||||
sqlite3 *connection = NULL;
|
||||
if (sqlite3_open(path, &connection) != SQLITE_OK) return false;
|
||||
if (sqlite3_open(path, &connection) != SQLITE_OK) {
|
||||
return false;
|
||||
}
|
||||
bool ok = sqlite3_exec(connection, sql, NULL, NULL, NULL) == SQLITE_OK;
|
||||
return sqlite3_close(connection) == SQLITE_OK && ok;
|
||||
}
|
||||
|
||||
static bool
|
||||
create_v2_database(const char *path)
|
||||
{
|
||||
if (!create_v1_database(path)) return false;
|
||||
static bool create_v2_database(const char *path) {
|
||||
if (!create_v1_database(path)) {
|
||||
return false;
|
||||
}
|
||||
sqlite3 *connection = NULL;
|
||||
if (sqlite3_open(path, &connection) != SQLITE_OK) return false;
|
||||
bool ok = sqlite3_exec(connection,
|
||||
if (sqlite3_open(path, &connection) != SQLITE_OK) {
|
||||
return false;
|
||||
}
|
||||
bool ok =
|
||||
sqlite3_exec(connection,
|
||||
"ALTER TABLE tasks ADD COLUMN task_kind TEXT;"
|
||||
"ALTER TABLE tasks ADD COLUMN task_kind_version INTEGER CHECK(task_kind_version IS NULL OR task_kind_version>0);"
|
||||
"ALTER TABLE tasks ADD COLUMN task_kind_version INTEGER "
|
||||
"CHECK(task_kind_version IS NULL OR task_kind_version>0);"
|
||||
"UPDATE tasks SET task_kind='test.work',task_kind_version=1 WHERE task_id=9;"
|
||||
"UPDATE metadata SET value=2 WHERE key='schema_version'",
|
||||
NULL, NULL, NULL) == SQLITE_OK;
|
||||
return sqlite3_close(connection) == SQLITE_OK && ok;
|
||||
}
|
||||
|
||||
static bool
|
||||
create_v3_database(const char *path)
|
||||
{
|
||||
if (!create_v2_database(path)) return false;
|
||||
static bool create_v3_database(const char *path) {
|
||||
if (!create_v2_database(path)) {
|
||||
return false;
|
||||
}
|
||||
sqlite3 *connection = NULL;
|
||||
if (sqlite3_open(path, &connection) != SQLITE_OK) return false;
|
||||
if (sqlite3_open(path, &connection) != SQLITE_OK) {
|
||||
return false;
|
||||
}
|
||||
bool ok = sqlite3_exec(connection,
|
||||
"CREATE TABLE image_import_tasks(task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE CASCADE,source_path TEXT NOT NULL);"
|
||||
"CREATE TABLE image_import_tasks(task_id INTEGER PRIMARY KEY REFERENCES "
|
||||
"tasks(task_id) ON DELETE CASCADE,source_path TEXT NOT NULL);"
|
||||
"INSERT INTO image_import_tasks VALUES(9,'/legacy/source');"
|
||||
"INSERT INTO metadata(key,value) VALUES('next_task_id',10);"
|
||||
"UPDATE metadata SET value=3 WHERE key='schema_version'",
|
||||
|
|
@ -143,82 +169,120 @@ create_v3_database(const char *path)
|
|||
return sqlite3_close(connection) == SQLITE_OK && ok;
|
||||
}
|
||||
|
||||
static bool
|
||||
query_integer(const char *path, const char *sql, sqlite3_int64 expected)
|
||||
{
|
||||
sqlite3 *connection = NULL;
|
||||
sqlite3_stmt *statement = NULL;
|
||||
if (sqlite3_open_v2(path, &connection, SQLITE_OPEN_READONLY, NULL) != SQLITE_OK
|
||||
|| sqlite3_prepare_v2(connection, sql, -1, &statement, NULL) != SQLITE_OK) {
|
||||
if (statement) (void)sqlite3_finalize(statement);
|
||||
if (connection) (void)sqlite3_close(connection);
|
||||
static bool create_v4_database(const char *path) {
|
||||
if (!create_v3_database(path)) {
|
||||
return false;
|
||||
}
|
||||
bool matches = sqlite3_step(statement) == SQLITE_ROW
|
||||
&& sqlite3_column_int64(statement, 0) == expected
|
||||
&& sqlite3_step(statement) == SQLITE_DONE;
|
||||
return sqlite3_finalize(statement) == SQLITE_OK
|
||||
&& sqlite3_close(connection) == SQLITE_OK && matches;
|
||||
sqlite3 *connection = NULL;
|
||||
if (sqlite3_open(path, &connection) != SQLITE_OK) {
|
||||
return false;
|
||||
}
|
||||
const char *sql =
|
||||
"CREATE TABLE scansets(scanset_id INTEGER PRIMARY KEY AUTOINCREMENT "
|
||||
"CHECK(scanset_id>0),name TEXT NOT NULL,created_at INTEGER NOT NULL,updated_at INTEGER NOT "
|
||||
"NULL);"
|
||||
"CREATE TABLE image_assets(asset_id INTEGER PRIMARY KEY AUTOINCREMENT "
|
||||
"CHECK(asset_id>0),sha256 BLOB NOT NULL UNIQUE,path TEXT NOT NULL UNIQUE,size_bytes "
|
||||
"INTEGER NOT NULL,state INTEGER NOT NULL,created_at INTEGER NOT NULL);"
|
||||
"CREATE TABLE images(image_id INTEGER PRIMARY KEY AUTOINCREMENT "
|
||||
"CHECK(image_id>0),scanset_id INTEGER NOT NULL REFERENCES scansets(scanset_id),asset_id "
|
||||
"INTEGER NOT NULL REFERENCES image_assets(asset_id),original_name TEXT NOT "
|
||||
"NULL,source_path TEXT NOT NULL,producer_task_id INTEGER REFERENCES "
|
||||
"tasks(task_id),imported_at INTEGER NOT NULL,UNIQUE(scanset_id,asset_id));"
|
||||
"CREATE INDEX images_scanset_idx ON images(scanset_id,image_id);"
|
||||
"CREATE INDEX images_producer_idx ON images(producer_task_id,image_id);"
|
||||
"ALTER TABLE image_import_tasks ADD COLUMN scanset_id INTEGER REFERENCES "
|
||||
"scansets(scanset_id);"
|
||||
"INSERT INTO scansets(name,created_at,updated_at) VALUES('Legacy',0,0);"
|
||||
"UPDATE image_import_tasks SET scanset_id=1;"
|
||||
"INSERT INTO metadata VALUES('legacy_image_catalog_pending',1);"
|
||||
"UPDATE metadata SET value=4 WHERE key='schema_version';";
|
||||
bool ok = sqlite3_exec(connection, sql, NULL, NULL, NULL) == SQLITE_OK;
|
||||
return sqlite3_close(connection) == SQLITE_OK && ok;
|
||||
}
|
||||
|
||||
static bool
|
||||
run_test(void)
|
||||
{
|
||||
static bool query_integer(const char *path, const char *sql, sqlite3_int64 expected) {
|
||||
sqlite3 *connection = NULL;
|
||||
sqlite3_stmt *statement = NULL;
|
||||
if (sqlite3_open_v2(path, &connection, SQLITE_OPEN_READONLY, NULL) != SQLITE_OK ||
|
||||
sqlite3_prepare_v2(connection, sql, -1, &statement, NULL) != SQLITE_OK) {
|
||||
if (statement) {
|
||||
(void)sqlite3_finalize(statement);
|
||||
}
|
||||
if (connection) {
|
||||
(void)sqlite3_close(connection);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
bool matches = sqlite3_step(statement) == SQLITE_ROW &&
|
||||
sqlite3_column_int64(statement, 0) == expected &&
|
||||
sqlite3_step(statement) == SQLITE_DONE;
|
||||
return sqlite3_finalize(statement) == SQLITE_OK && sqlite3_close(connection) == SQLITE_OK &&
|
||||
matches;
|
||||
}
|
||||
|
||||
static bool run_test(void) {
|
||||
char directory[] = "/tmp/lardon3d-project-db-XXXXXX";
|
||||
CHECK(mkdtemp(directory));
|
||||
char database_path[512], artifact_path[512], future_path[512], corrupt_path[512];
|
||||
char legacy_path[512], failed_migration_path[512], v2_path[512];
|
||||
char failed_v3_migration_path[512], v3_path[512], failed_v4_path[512];
|
||||
char v4_path[512], failed_v5_path[512];
|
||||
CHECK(snprintf(database_path, sizeof(database_path), "%s/project.db", directory) > 0);
|
||||
CHECK(snprintf(artifact_path, sizeof(artifact_path), "%s/artifact.bin", directory) > 0);
|
||||
CHECK(snprintf(future_path, sizeof(future_path), "%s/future.db", directory) > 0);
|
||||
CHECK(snprintf(corrupt_path, sizeof(corrupt_path), "%s/corrupt.db", directory) > 0);
|
||||
CHECK(snprintf(legacy_path, sizeof(legacy_path), "%s/legacy.db", directory) > 0);
|
||||
CHECK(snprintf(failed_migration_path, sizeof(failed_migration_path), "%s/failed-migration.db", directory) > 0);
|
||||
CHECK(snprintf(failed_migration_path, sizeof(failed_migration_path), "%s/failed-migration.db",
|
||||
directory) > 0);
|
||||
CHECK(snprintf(v2_path, sizeof(v2_path), "%s/v2.db", directory) > 0);
|
||||
CHECK(snprintf(failed_v3_migration_path, sizeof(failed_v3_migration_path),
|
||||
"%s/failed-v3-migration.db", directory) > 0);
|
||||
CHECK(snprintf(v3_path, sizeof(v3_path), "%s/v3.db", directory) > 0);
|
||||
CHECK(snprintf(failed_v4_path, sizeof(failed_v4_path),
|
||||
"%s/failed-v4-migration.db", directory) > 0);
|
||||
CHECK(snprintf(failed_v4_path, sizeof(failed_v4_path), "%s/failed-v4-migration.db", directory) >
|
||||
0);
|
||||
CHECK(snprintf(v4_path, sizeof(v4_path), "%s/v4.db", directory) > 0);
|
||||
CHECK(snprintf(failed_v5_path, sizeof(failed_v5_path), "%s/failed-v5-migration.db", directory) >
|
||||
0);
|
||||
|
||||
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY];
|
||||
Lardon3DProjectDb *database = NULL;
|
||||
CHECK(lardon3d_project_db_open(database_path, &database, error) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(database && lardon3d_project_db_schema_version(database) == 4);
|
||||
CHECK(database && lardon3d_project_db_schema_version(database) == 5);
|
||||
bool legacy_pending = true;
|
||||
CHECK(lardon3d_project_db_legacy_catalog_pending(database, &legacy_pending)
|
||||
== LARDON3D_PROJECT_DB_OK && !legacy_pending);
|
||||
CHECK(lardon3d_project_db_legacy_catalog_pending(database, &legacy_pending) ==
|
||||
LARDON3D_PROJECT_DB_OK &&
|
||||
!legacy_pending);
|
||||
|
||||
Lardon3DProjectDbScanSet deleted_scanset;
|
||||
CHECK(lardon3d_project_db_create_scanset(database, "Deleted identity",
|
||||
&deleted_scanset) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_create_scanset(database, "Deleted identity", &deleted_scanset) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
unsigned char first_hash[LARDON3D_PROJECT_DB_SHA256_SIZE] = {1};
|
||||
char first_asset_path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
||||
asset_path_for_hash(first_hash, first_asset_path);
|
||||
Lardon3DProjectDbImageRegisterStatus identity_status;
|
||||
Lardon3DProjectDbImage deleted_image;
|
||||
CHECK(lardon3d_project_db_register_image(database,
|
||||
deleted_scanset.scanset_id, first_hash, first_asset_path, 1,
|
||||
"deleted.jpg", "/source/deleted.jpg", 0, 1, &identity_status,
|
||||
CHECK(lardon3d_project_db_register_image(database, deleted_scanset.scanset_id, first_hash,
|
||||
first_asset_path, 1, "deleted.jpg",
|
||||
"/source/deleted.jpg", 0, 1, &identity_status,
|
||||
&deleted_image) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_test_delete_catalog_identity(database,
|
||||
deleted_scanset.scanset_id, deleted_image.image_id,
|
||||
deleted_image.asset_id) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_test_delete_catalog_identity(
|
||||
database, deleted_scanset.scanset_id, deleted_image.image_id, deleted_image.asset_id) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
Lardon3DProjectDbScanSet replacement_scanset;
|
||||
CHECK(lardon3d_project_db_create_scanset(database, "Replacement identity",
|
||||
&replacement_scanset) == LARDON3D_PROJECT_DB_OK
|
||||
&& replacement_scanset.scanset_id > deleted_scanset.scanset_id);
|
||||
&replacement_scanset) == LARDON3D_PROJECT_DB_OK &&
|
||||
replacement_scanset.scanset_id > deleted_scanset.scanset_id);
|
||||
unsigned char second_hash[LARDON3D_PROJECT_DB_SHA256_SIZE] = {2};
|
||||
char second_asset_path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
||||
asset_path_for_hash(second_hash, second_asset_path);
|
||||
Lardon3DProjectDbImage replacement_image;
|
||||
CHECK(lardon3d_project_db_register_image(database,
|
||||
replacement_scanset.scanset_id, second_hash, second_asset_path, 1,
|
||||
"replacement.jpg", "/source/replacement.jpg", 0, 2,
|
||||
&identity_status, &replacement_image) == LARDON3D_PROJECT_DB_OK
|
||||
&& replacement_image.image_id > deleted_image.image_id
|
||||
&& replacement_image.asset_id > deleted_image.asset_id);
|
||||
CHECK(lardon3d_project_db_register_image(database, replacement_scanset.scanset_id, second_hash,
|
||||
second_asset_path, 1, "replacement.jpg",
|
||||
"/source/replacement.jpg", 0, 2, &identity_status,
|
||||
&replacement_image) == LARDON3D_PROJECT_DB_OK &&
|
||||
replacement_image.image_id > deleted_image.image_id &&
|
||||
replacement_image.asset_id > deleted_image.asset_id);
|
||||
|
||||
Lardon3DProjectDbProject project = {.created_at = 100, .updated_at = 100};
|
||||
(void)snprintf(project.stable_id, sizeof(project.stable_id), "project-0001");
|
||||
|
|
@ -227,216 +291,281 @@ run_test(void)
|
|||
Lardon3DProjectDbProject loaded_project;
|
||||
CHECK(lardon3d_project_db_get_project(database, &loaded_project) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(strcmp(loaded_project.stable_id, project.stable_id) == 0);
|
||||
project.updated_at = 101; (void)snprintf(project.name, sizeof(project.name), "Projet renommé");
|
||||
project.updated_at = 101;
|
||||
(void)snprintf(project.name, sizeof(project.name), "Projet renommé");
|
||||
CHECK(lardon3d_project_db_set_project(database, &project) == LARDON3D_PROJECT_DB_OK);
|
||||
Lardon3DProjectDbProject conflict = project; (void)snprintf(conflict.stable_id, sizeof(conflict.stable_id), "other");
|
||||
Lardon3DProjectDbProject conflict = project;
|
||||
(void)snprintf(conflict.stable_id, sizeof(conflict.stable_id), "other");
|
||||
CHECK(lardon3d_project_db_set_project(database, &conflict) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||
char copied_error[LARDON3D_PROJECT_DB_ERROR_CAPACITY];
|
||||
CHECK(lardon3d_project_db_last_error(database, copied_error));
|
||||
CHECK(copied_error[0] != '\0');
|
||||
project = loaded_project; memset(project.name, 'x', sizeof(project.name));
|
||||
CHECK(lardon3d_project_db_set_project(database, &project) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||
project = loaded_project;
|
||||
memset(project.name, 'x', sizeof(project.name));
|
||||
CHECK(lardon3d_project_db_set_project(database, &project) ==
|
||||
LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||
|
||||
Lardon3DTaskDurableSnapshot running = task_snapshot(1, TASK_RUNNING);
|
||||
Lardon3DProjectDbCheckpoint checkpoint = {.format_version = 1, .durability = LARDON3D_DB_CHECKPOINT_DURABLE, .updated_at = 200};
|
||||
Lardon3DProjectDbCheckpoint checkpoint = {
|
||||
.format_version = 1, .durability = LARDON3D_DB_CHECKPOINT_DURABLE, .updated_at = 200};
|
||||
(void)snprintf(checkpoint.path, sizeof(checkpoint.path), "%s/checkpoints/task-1.chk", directory);
|
||||
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 200) == LARDON3D_PROJECT_DB_OK);
|
||||
running.progress = 30; running.sequence_count = 3;
|
||||
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 201) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 200) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
running.progress = 30;
|
||||
running.sequence_count = 3;
|
||||
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 201) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
Lardon3DProjectDbScanSet import_scanset;
|
||||
CHECK(lardon3d_project_db_create_scanset(database, "Legacy import",
|
||||
&import_scanset) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_create_scanset(database, "Legacy import", &import_scanset) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
Lardon3DProjectDbTask task;
|
||||
CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(task.recovery_state == TASK_PENDING && task.progress == 30 && task.sequence_count == 3);
|
||||
CHECK(task.has_task_kind && strcmp(task.task_kind, "test.work") == 0
|
||||
&& task.task_kind_version == 1);
|
||||
CHECK(task.has_task_kind && strcmp(task.task_kind, "test.work") == 0 &&
|
||||
task.task_kind_version == 1);
|
||||
CHECK(task.has_checkpoint && strcmp(task.checkpoint.path, checkpoint.path) == 0);
|
||||
checkpoint.durability = LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE;
|
||||
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 201) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 201) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(task.checkpoint.durability == LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE);
|
||||
CHECK(lardon3d_project_db_record_task(database, &running, "test.other", 1,
|
||||
&checkpoint, 202) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||
CHECK(lardon3d_project_db_record_task(database, &running, "Test.invalid", 1,
|
||||
&checkpoint, 202) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 0,
|
||||
&checkpoint, 202) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||
CHECK(lardon3d_project_db_record_image_import_task(database, &running,
|
||||
"import.images", 1, &checkpoint, "/tmp/source-a",
|
||||
import_scanset.scanset_id, 202)
|
||||
== LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||
CHECK(lardon3d_project_db_record_task(database, &running, "test.other", 1, &checkpoint, 202) ==
|
||||
LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||
CHECK(lardon3d_project_db_record_task(database, &running, "Test.invalid", 1, &checkpoint, 202) ==
|
||||
LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 0, &checkpoint, 202) ==
|
||||
LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||
CHECK(lardon3d_project_db_record_image_import_task(
|
||||
database, &running, "import.images", 1, &checkpoint, "/tmp/source-a",
|
||||
import_scanset.scanset_id, 202) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||
|
||||
Lardon3DTaskDurableSnapshot completed = task_snapshot(2, TASK_COMPLETED);
|
||||
CHECK(lardon3d_project_db_record_task(database, &completed, "test.work", 1, NULL, 202) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_record_task(database, &completed, "test.work", 1, NULL, 202) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
Lardon3DTaskDurableSnapshot no_checkpoint = task_snapshot(4, TASK_PENDING);
|
||||
CHECK(lardon3d_project_db_record_task(database, &no_checkpoint, NULL, 0, NULL, 202) == LARDON3D_PROJECT_DB_OK);
|
||||
Lardon3DProjectDbTask page[1]; size_t count = 0;
|
||||
CHECK(lardon3d_project_db_list_recoverable(database, 0, page, 1, &count) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_record_task(database, &no_checkpoint, NULL, 0, NULL, 202) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
Lardon3DProjectDbTask page[1];
|
||||
size_t count = 0;
|
||||
CHECK(lardon3d_project_db_list_recoverable(database, 0, page, 1, &count) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(count == 1 && page[0].task_id == 1);
|
||||
CHECK(lardon3d_project_db_list_recoverable(database, 1, page, 1, &count) == LARDON3D_PROJECT_DB_OK && count == 0);
|
||||
CHECK(lardon3d_project_db_list_recoverable(database, 0, page, LARDON3D_PROJECT_DB_RECOVERY_PAGE_MAX + 1, &count) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||
CHECK(lardon3d_project_db_list_recoverable(database, 1, page, 1, &count) ==
|
||||
LARDON3D_PROJECT_DB_OK &&
|
||||
count == 0);
|
||||
CHECK(lardon3d_project_db_list_recoverable(database, 0, page,
|
||||
LARDON3D_PROJECT_DB_RECOVERY_PAGE_MAX + 1,
|
||||
&count) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||
|
||||
Lardon3DTaskDurableSnapshot image_import = task_snapshot(5, TASK_PENDING);
|
||||
CHECK(lardon3d_project_db_record_image_import_task(database, &image_import,
|
||||
"import.images", 1, &checkpoint, "/tmp/source-a",
|
||||
import_scanset.scanset_id, 202)
|
||||
== LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_record_image_import_task(
|
||||
database, &image_import, "import.images", 1, &checkpoint, "/tmp/source-a",
|
||||
import_scanset.scanset_id, 202) == LARDON3D_PROJECT_DB_OK);
|
||||
Lardon3DProjectDbImageImport import_parameters;
|
||||
CHECK(lardon3d_project_db_load_image_import(database, 5,
|
||||
&import_parameters) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(import_parameters.task_id == 5
|
||||
&& strcmp(import_parameters.source_path, "/tmp/source-a") == 0
|
||||
&& import_parameters.scanset_id == import_scanset.scanset_id);
|
||||
CHECK(lardon3d_project_db_record_image_import_task(database, &image_import,
|
||||
"import.images", 1, &checkpoint, "/tmp/source-b",
|
||||
import_scanset.scanset_id, 203)
|
||||
== LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||
CHECK(lardon3d_project_db_load_image_import(database, 5, &import_parameters) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(import_parameters.task_id == 5 &&
|
||||
strcmp(import_parameters.source_path, "/tmp/source-a") == 0 &&
|
||||
import_parameters.scanset_id == import_scanset.scanset_id);
|
||||
CHECK(lardon3d_project_db_record_image_import_task(
|
||||
database, &image_import, "import.images", 1, &checkpoint, "/tmp/source-b",
|
||||
import_scanset.scanset_id, 203) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||
|
||||
Lardon3DTaskDurableSnapshot rollback_task = task_snapshot(3, TASK_PENDING);
|
||||
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_CHECKPOINT", "1", 1) == 0);
|
||||
CHECK(lardon3d_project_db_record_task(database, &rollback_task, "test.work", 1, &checkpoint, 203) == LARDON3D_PROJECT_DB_IO_ERROR);
|
||||
CHECK(lardon3d_project_db_record_task(database, &rollback_task, "test.work", 1, &checkpoint,
|
||||
203) == LARDON3D_PROJECT_DB_IO_ERROR);
|
||||
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_CHECKPOINT") == 0);
|
||||
CHECK(lardon3d_project_db_load_task(database, 3, &task) == LARDON3D_PROJECT_DB_NOT_FOUND);
|
||||
CHECK(lardon3d_project_db_test_orphan_checkpoint(database) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||
|
||||
Lardon3DProjectDbArtifact artifact = {.state = LARDON3D_DB_ARTIFACT_STAGED, .has_producer_task = true, .producer_task_id = 1, .created_at = 300, .updated_at = 300};
|
||||
Lardon3DProjectDbArtifact artifact = {.state = LARDON3D_DB_ARTIFACT_STAGED,
|
||||
.has_producer_task = true,
|
||||
.producer_task_id = 1,
|
||||
.created_at = 300,
|
||||
.updated_at = 300};
|
||||
(void)snprintf(artifact.artifact_id, sizeof(artifact.artifact_id), "artifact-1");
|
||||
(void)snprintf(artifact.kind, sizeof(artifact.kind), "generic-test");
|
||||
(void)snprintf(artifact.path, sizeof(artifact.path), "%s", artifact_path);
|
||||
CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_OK);
|
||||
Lardon3DProjectDbArtifact loaded_artifact;
|
||||
CHECK(lardon3d_project_db_load_artifact(database, artifact.artifact_id, &loaded_artifact) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_load_artifact(database, artifact.artifact_id, &loaded_artifact) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(loaded_artifact.state == LARDON3D_DB_ARTIFACT_STAGED);
|
||||
CHECK(lardon3d_project_db_mark_artifact_ready(database, artifact.artifact_id, 301) == LARDON3D_PROJECT_DB_IO_ERROR);
|
||||
int descriptor = open(artifact_path, O_WRONLY | O_CREAT | O_TRUNC, 0600); CHECK(descriptor >= 0);
|
||||
const char payload[] = "validated"; CHECK(write(descriptor, payload, sizeof(payload)) == (ssize_t)sizeof(payload)); CHECK(close(descriptor) == 0);
|
||||
CHECK(lardon3d_project_db_mark_artifact_ready(database, artifact.artifact_id, 301) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_load_artifact(database, artifact.artifact_id, &loaded_artifact) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(loaded_artifact.state == LARDON3D_DB_ARTIFACT_READY && loaded_artifact.size_bytes == sizeof(payload));
|
||||
CHECK(lardon3d_project_db_mark_artifact_ready(database, artifact.artifact_id, 301) ==
|
||||
LARDON3D_PROJECT_DB_IO_ERROR);
|
||||
int descriptor = open(artifact_path, O_WRONLY | O_CREAT | O_TRUNC, 0600);
|
||||
CHECK(descriptor >= 0);
|
||||
const char payload[] = "validated";
|
||||
CHECK(write(descriptor, payload, sizeof(payload)) == (ssize_t)sizeof(payload));
|
||||
CHECK(close(descriptor) == 0);
|
||||
CHECK(lardon3d_project_db_mark_artifact_ready(database, artifact.artifact_id, 301) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_load_artifact(database, artifact.artifact_id, &loaded_artifact) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(loaded_artifact.state == LARDON3D_DB_ARTIFACT_READY &&
|
||||
loaded_artifact.size_bytes == sizeof(payload));
|
||||
CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||
artifact.producer_task_id = 999;
|
||||
(void)snprintf(artifact.artifact_id, sizeof(artifact.artifact_id), "orphan-artifact");
|
||||
CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||
artifact = loaded_artifact; artifact.state = LARDON3D_DB_ARTIFACT_STAGED; artifact.size_bytes = 0;
|
||||
artifact = loaded_artifact;
|
||||
artifact.state = LARDON3D_DB_ARTIFACT_STAGED;
|
||||
artifact.size_bytes = 0;
|
||||
memset(artifact.path, 'x', sizeof(artifact.path));
|
||||
CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||
CHECK(lardon3d_project_db_create_artifact(database, &artifact) ==
|
||||
LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||
|
||||
ThreadContext contexts[2] = {{.database = database}, {.database = database}}; pthread_t threads[2];
|
||||
ThreadContext contexts[2] = {{.database = database}, {.database = database}};
|
||||
pthread_t threads[2];
|
||||
CHECK(pthread_create(&threads[0], NULL, read_thread, &contexts[0]) == 0);
|
||||
CHECK(pthread_create(&threads[1], NULL, read_thread, &contexts[1]) == 0);
|
||||
CHECK(pthread_join(threads[0], NULL) == 0 && pthread_join(threads[1], NULL) == 0);
|
||||
CHECK(contexts[0].success && contexts[1].success);
|
||||
IdThreadContext id_contexts[2] = {{.database = database}, {.database = database}};
|
||||
CHECK(pthread_create(&threads[0], NULL, allocate_id_thread,
|
||||
&id_contexts[0]) == 0);
|
||||
CHECK(pthread_create(&threads[1], NULL, allocate_id_thread,
|
||||
&id_contexts[1]) == 0);
|
||||
CHECK(pthread_join(threads[0], NULL) == 0
|
||||
&& pthread_join(threads[1], NULL) == 0);
|
||||
CHECK(id_contexts[0].success && id_contexts[1].success
|
||||
&& id_contexts[0].id != id_contexts[1].id
|
||||
&& id_contexts[0].id > 5 && id_contexts[1].id > 5);
|
||||
CHECK(pthread_create(&threads[0], NULL, allocate_id_thread, &id_contexts[0]) == 0);
|
||||
CHECK(pthread_create(&threads[1], NULL, allocate_id_thread, &id_contexts[1]) == 0);
|
||||
CHECK(pthread_join(threads[0], NULL) == 0 && pthread_join(threads[1], NULL) == 0);
|
||||
CHECK(id_contexts[0].success && id_contexts[1].success &&
|
||||
id_contexts[0].id != id_contexts[1].id && id_contexts[0].id > 5 && id_contexts[1].id > 5);
|
||||
|
||||
char too_long[LARDON3D_PROJECT_DB_PATH_CAPACITY + 1]; memset(too_long, 'x', sizeof(too_long)); too_long[sizeof(too_long) - 1] = '\0';
|
||||
CHECK(lardon3d_project_db_open(too_long, &database, error) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||
lardon3d_project_db_close(contexts[0].database); database = NULL;
|
||||
CHECK(query_integer(database_path, "SELECT value FROM metadata WHERE key='schema_version'", 4));
|
||||
char too_long[LARDON3D_PROJECT_DB_PATH_CAPACITY + 1];
|
||||
memset(too_long, 'x', sizeof(too_long));
|
||||
too_long[sizeof(too_long) - 1] = '\0';
|
||||
CHECK(lardon3d_project_db_open(too_long, &database, error) ==
|
||||
LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||
lardon3d_project_db_close(contexts[0].database);
|
||||
database = NULL;
|
||||
CHECK(query_integer(database_path, "SELECT value FROM metadata WHERE key='schema_version'", 5));
|
||||
CHECK(query_integer(database_path, "SELECT count(*) FROM tasks WHERE task_id=1", 1));
|
||||
CHECK(lardon3d_project_db_open(database_path, &database, error) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_load_artifact(database, "artifact-1", &loaded_artifact) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_load_artifact(database, "artifact-1", &loaded_artifact) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
lardon3d_project_db_close(database);
|
||||
|
||||
CHECK(create_future_database(future_path));
|
||||
CHECK(lardon3d_project_db_open(future_path, &database, error) == LARDON3D_PROJECT_DB_UNSUPPORTED_SCHEMA);
|
||||
descriptor = open(corrupt_path, O_WRONLY | O_CREAT | O_TRUNC, 0600); CHECK(descriptor >= 0);
|
||||
CHECK(lardon3d_project_db_open(future_path, &database, error) ==
|
||||
LARDON3D_PROJECT_DB_UNSUPPORTED_SCHEMA);
|
||||
descriptor = open(corrupt_path, O_WRONLY | O_CREAT | O_TRUNC, 0600);
|
||||
CHECK(descriptor >= 0);
|
||||
CHECK(write(descriptor, "not sqlite", 10) == 10 && close(descriptor) == 0);
|
||||
CHECK(lardon3d_project_db_open(corrupt_path, &database, error) == LARDON3D_PROJECT_DB_CORRUPT);
|
||||
|
||||
CHECK(create_v1_database(legacy_path));
|
||||
CHECK(lardon3d_project_db_open(legacy_path, &database, error) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_schema_version(database) == 4);
|
||||
CHECK(lardon3d_project_db_get_project(database, &loaded_project) == LARDON3D_PROJECT_DB_OK
|
||||
&& strcmp(loaded_project.stable_id, "legacy-project") == 0);
|
||||
CHECK(lardon3d_project_db_schema_version(database) == 5);
|
||||
CHECK(lardon3d_project_db_get_project(database, &loaded_project) == LARDON3D_PROJECT_DB_OK &&
|
||||
strcmp(loaded_project.stable_id, "legacy-project") == 0);
|
||||
CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(!task.has_task_kind && task.has_checkpoint
|
||||
&& strcmp(task.checkpoint.path, "legacy.chk") == 0);
|
||||
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact",
|
||||
&loaded_artifact) == LARDON3D_PROJECT_DB_OK);
|
||||
lardon3d_project_db_close(database); database = NULL;
|
||||
CHECK(query_integer(legacy_path, "SELECT value FROM metadata WHERE key='schema_version'", 4));
|
||||
CHECK(!task.has_task_kind && task.has_checkpoint &&
|
||||
strcmp(task.checkpoint.path, "legacy.chk") == 0);
|
||||
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact", &loaded_artifact) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
lardon3d_project_db_close(database);
|
||||
database = NULL;
|
||||
CHECK(query_integer(legacy_path, "SELECT value FROM metadata WHERE key='schema_version'", 5));
|
||||
|
||||
CHECK(create_v1_database(failed_migration_path));
|
||||
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V2", "1", 1) == 0);
|
||||
CHECK(lardon3d_project_db_open(failed_migration_path, &database, error)
|
||||
== LARDON3D_PROJECT_DB_IO_ERROR);
|
||||
CHECK(lardon3d_project_db_open(failed_migration_path, &database, error) ==
|
||||
LARDON3D_PROJECT_DB_IO_ERROR);
|
||||
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V2") == 0);
|
||||
CHECK(query_integer(failed_migration_path,
|
||||
"SELECT value FROM metadata WHERE key='schema_version'", 1));
|
||||
CHECK(lardon3d_project_db_open(failed_migration_path, &database, error)
|
||||
== LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK
|
||||
&& !task.has_task_kind);
|
||||
lardon3d_project_db_close(database); database = NULL;
|
||||
CHECK(lardon3d_project_db_open(failed_migration_path, &database, error) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK &&
|
||||
!task.has_task_kind);
|
||||
lardon3d_project_db_close(database);
|
||||
database = NULL;
|
||||
|
||||
CHECK(create_v2_database(v2_path));
|
||||
CHECK(lardon3d_project_db_open(v2_path, &database, error)
|
||||
== LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_load_task(database, 9, &task)
|
||||
== LARDON3D_PROJECT_DB_OK && task.has_task_kind
|
||||
&& strcmp(task.task_kind, "test.work") == 0);
|
||||
CHECK(task.has_checkpoint
|
||||
&& strcmp(task.checkpoint.path, "legacy.chk") == 0);
|
||||
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact",
|
||||
&loaded_artifact) == LARDON3D_PROJECT_DB_OK);
|
||||
lardon3d_project_db_close(database); database = NULL;
|
||||
CHECK(query_integer(v2_path,
|
||||
"SELECT value FROM metadata WHERE key='schema_version'", 4));
|
||||
CHECK(lardon3d_project_db_open(v2_path, &database, error) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK &&
|
||||
task.has_task_kind && strcmp(task.task_kind, "test.work") == 0);
|
||||
CHECK(task.has_checkpoint && strcmp(task.checkpoint.path, "legacy.chk") == 0);
|
||||
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact", &loaded_artifact) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
lardon3d_project_db_close(database);
|
||||
database = NULL;
|
||||
CHECK(query_integer(v2_path, "SELECT value FROM metadata WHERE key='schema_version'", 5));
|
||||
|
||||
CHECK(create_v2_database(failed_v3_migration_path));
|
||||
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V3", "1", 1) == 0);
|
||||
CHECK(lardon3d_project_db_open(failed_v3_migration_path, &database, error)
|
||||
== LARDON3D_PROJECT_DB_IO_ERROR);
|
||||
CHECK(lardon3d_project_db_open(failed_v3_migration_path, &database, error) ==
|
||||
LARDON3D_PROJECT_DB_IO_ERROR);
|
||||
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V3") == 0);
|
||||
CHECK(query_integer(failed_v3_migration_path,
|
||||
"SELECT value FROM metadata WHERE key='schema_version'", 2));
|
||||
CHECK(query_integer(failed_v3_migration_path,
|
||||
CHECK(query_integer(
|
||||
failed_v3_migration_path,
|
||||
"SELECT count(*) FROM sqlite_master WHERE type='table' AND name='image_import_tasks'", 0));
|
||||
|
||||
CHECK(create_v3_database(v3_path));
|
||||
CHECK(lardon3d_project_db_open(v3_path, &database, error)
|
||||
== LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_load_image_import(database, 9,
|
||||
&import_parameters) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(strcmp(import_parameters.source_path, "/legacy/source") == 0
|
||||
&& import_parameters.scanset_id > 0);
|
||||
CHECK(lardon3d_project_db_legacy_catalog_pending(database, &legacy_pending)
|
||||
== LARDON3D_PROJECT_DB_OK && legacy_pending);
|
||||
CHECK(lardon3d_project_db_load_task(database, 9, &task)
|
||||
== LARDON3D_PROJECT_DB_OK && task.has_task_kind);
|
||||
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact",
|
||||
&loaded_artifact) == LARDON3D_PROJECT_DB_OK);
|
||||
lardon3d_project_db_close(database); database = NULL;
|
||||
CHECK(query_integer(v3_path,
|
||||
"SELECT value FROM metadata WHERE key='schema_version'", 4));
|
||||
CHECK(lardon3d_project_db_open(v3_path, &database, error) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_load_image_import(database, 9, &import_parameters) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(strcmp(import_parameters.source_path, "/legacy/source") == 0 &&
|
||||
import_parameters.scanset_id > 0);
|
||||
CHECK(lardon3d_project_db_legacy_catalog_pending(database, &legacy_pending) ==
|
||||
LARDON3D_PROJECT_DB_OK &&
|
||||
legacy_pending);
|
||||
CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK &&
|
||||
task.has_task_kind);
|
||||
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact", &loaded_artifact) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
lardon3d_project_db_close(database);
|
||||
database = NULL;
|
||||
CHECK(query_integer(v3_path, "SELECT value FROM metadata WHERE key='schema_version'", 5));
|
||||
|
||||
CHECK(create_v3_database(failed_v4_path));
|
||||
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V4", "1", 1) == 0);
|
||||
CHECK(lardon3d_project_db_open(failed_v4_path, &database, error)
|
||||
== LARDON3D_PROJECT_DB_IO_ERROR);
|
||||
CHECK(lardon3d_project_db_open(failed_v4_path, &database, error) == LARDON3D_PROJECT_DB_IO_ERROR);
|
||||
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V4") == 0);
|
||||
CHECK(query_integer(failed_v4_path, "SELECT value FROM metadata WHERE key='schema_version'", 3));
|
||||
CHECK(query_integer(failed_v4_path,
|
||||
"SELECT value FROM metadata WHERE key='schema_version'", 3));
|
||||
CHECK(query_integer(failed_v4_path,
|
||||
"SELECT count(*) FROM sqlite_master WHERE type='table' AND name='scansets'", 0));
|
||||
"SELECT count(*) FROM sqlite_master WHERE type='table' AND name='scansets'",
|
||||
0));
|
||||
|
||||
CHECK(unlink(artifact_path) == 0); CHECK(unlink(database_path) == 0); CHECK(unlink(future_path) == 0); CHECK(unlink(corrupt_path) == 0);
|
||||
CHECK(unlink(legacy_path) == 0); CHECK(unlink(failed_migration_path) == 0);
|
||||
CHECK(unlink(v2_path) == 0); CHECK(unlink(failed_v3_migration_path) == 0);
|
||||
CHECK(unlink(v3_path) == 0); CHECK(unlink(failed_v4_path) == 0);
|
||||
CHECK(create_v4_database(v4_path));
|
||||
Lardon3DProjectDbResult v4_result = lardon3d_project_db_open(v4_path, &database, error);
|
||||
if (v4_result != LARDON3D_PROJECT_DB_OK) {
|
||||
fprintf(stderr, "Migration v4 (%d): %s\n", (int)v4_result, error);
|
||||
}
|
||||
CHECK(v4_result == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_schema_version(database) == 5);
|
||||
CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact", &loaded_artifact) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
lardon3d_project_db_close(database);
|
||||
database = NULL;
|
||||
|
||||
CHECK(create_v4_database(failed_v5_path));
|
||||
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V5", "1", 1) == 0);
|
||||
CHECK(lardon3d_project_db_open(failed_v5_path, &database, error) == LARDON3D_PROJECT_DB_IO_ERROR);
|
||||
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V5") == 0);
|
||||
CHECK(query_integer(failed_v5_path, "SELECT value FROM metadata WHERE key='schema_version'", 4));
|
||||
CHECK(query_integer(
|
||||
failed_v5_path,
|
||||
"SELECT count(*) FROM sqlite_master WHERE type='table' AND name='feature_sets'", 0));
|
||||
|
||||
CHECK(unlink(artifact_path) == 0);
|
||||
CHECK(unlink(database_path) == 0);
|
||||
CHECK(unlink(future_path) == 0);
|
||||
CHECK(unlink(corrupt_path) == 0);
|
||||
CHECK(unlink(legacy_path) == 0);
|
||||
CHECK(unlink(failed_migration_path) == 0);
|
||||
CHECK(unlink(v2_path) == 0);
|
||||
CHECK(unlink(failed_v3_migration_path) == 0);
|
||||
CHECK(unlink(v3_path) == 0);
|
||||
CHECK(unlink(failed_v4_path) == 0);
|
||||
CHECK(unlink(v4_path) == 0);
|
||||
CHECK(unlink(failed_v5_path) == 0);
|
||||
CHECK(rmdir(directory) == 0);
|
||||
return true;
|
||||
}
|
||||
|
||||
int main(void) { return run_test() ? EXIT_SUCCESS : EXIT_FAILURE; }
|
||||
int main(void) {
|
||||
return run_test() ? EXIT_SUCCESS : EXIT_FAILURE;
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue