feat: finalize feature store v1

This commit is contained in:
fy59 2026-08-08 13:22:57 +02:00
parent 0f99415317
commit b12af521e5
32 changed files with 5686 additions and 2973 deletions

View file

@ -23,10 +23,11 @@ Lardon3D ne vise pas simplement "dossier de photos → objet 3D", mais "ensemble
- **Project** : cycle de vie persistant, identité stable et Project Database ouverte
- **Import** : premier task kind de production, exécuté par la file générique en lots bornés et reprenables
- **ScanSet / Image Catalog v1** : acquisitions, images logiques, provenance et assets SHA-256 persistants et paginés
- **Feature Store v1** : extraction ORB réelle, assets binaires versionnés et lecture bornée
- **Image View** : vues triées et filtrées pour la TUI
- **Task** : moteur de tâches avec pause/reprise, annulation et séquences
- **Task Checkpoint v1** : snapshot durable, fichier atomique et reprise sûre
- **Project Database v4** : tâches, ScanSets, catalogue image, assets et migrations SQLite
- **Project Database v5** : tâches, ScanSets, images, Feature Sets/assets et migrations SQLite
- **Task Kind Registry** : identité métier durable et reconstruction runtime explicite
- **Recovery projet** : reprise automatique sélective et bornée des imports récupérables
- **Task Queue** : file FIFO avec sélection adaptative et backpressure
@ -102,6 +103,7 @@ Acquisitions
- [Pipeline de reconstruction](docs/architecture/reconstruction_pipeline.md)
- [Persistance](docs/architecture/persistence.md)
- [Base de données projet](docs/architecture/project_database.md)
- [Feature Store](docs/architecture/feature_store.md)
- [Viewer](docs/architecture/viewer.md)
- [Revue des fondations](docs/architecture/foundation_review.md)

View file

@ -0,0 +1,117 @@
# Feature Store v1
## Rôle et modèle
**IMPLEMENTED.** Le Feature Store est la mémoire visuelle locale persistante.
Un `FeatureSet` logique appartient à une `image_id` et identifie exactement
`orb` version 1 plus une configuration canonique. Il référence un
`FeatureAsset` physique immutable. Deux images logiques portant le même contenu
et la même configuration gardent deux `feature_set_id`, mais partagent le même
asset lorsque leurs octets sont identiques. Les futurs matches référenceront
`feature_set_id + feature_index`; ces deux valeurs sont immuables après
publication.
`feature_set_id` et `feature_asset_id` sont des identités SQLite
`AUTOINCREMENT`. Une identité validée n'est jamais réattribuée. SQLite ne
contient que les relations, empreintes, tailles et paramètres ; les tableaux de
points et descripteurs restent hors DB.
## Extracteur et configuration
L'extracteur production est ORB d'OpenCV 5, derrière une façade C. Aucune
exception C++ ni `cv::Mat` ne traverse cette frontière. La registry statique
expose `features.extract`, version 1. Une tâche traite exactement une image.
La configuration v1 contient trois `uint32` : `max_features` (1..8192),
`pyramid_levels` (1..16) et `fast_threshold` (1..255). Son fingerprint SHA-256
porte sur les 24 octets canoniques `L3DORBP1`, version et trois entiers
little-endian. Il ne dépend ni du padding, ni de la locale. Le contrat ORB v1
produit 32 octets binaires par point.
OpenCV n'est pas configuré via un état global par Lardon3D. Le scheduler actuel
n'a qu'un worker ; OpenCV peut néanmoins employer son backend parallèle
interne. Le contrôle fin de ce parallélisme devra précéder les futurs pools de
workers.
La limite de 100 000 000 pixels est vérifiée après `cv::imread` : l'API utilisée
ne fournit pas de sonde de dimensions multi-format fiable sans décodage. Le pic
mémoire du décodage peut donc précéder le rejet. Lardon3D ne revendique pas de
bornage pré-décodage et n'embarque pas un parseur JPEG/PNG parallèle.
## Feature File v1
Le fichier est little-endian et exige IEEE-754 binary32. Sa taille maximale est
16 Mio et son nombre maximal de points 8192.
| Offset | Taille | Champ |
|---:|---:|---|
| 0 | 8 | magic `L3DFEAT\0` |
| 8 | 4 | format version = 1 |
| 12 | 4 | header size = 160 |
| 16 | 4 | feature count |
| 20 | 4 | descriptor dimension = 32 |
| 24 | 4 | descriptor type = U8 |
| 28 | 4 | keypoint record size = 24 |
| 32,36 | 4+4 | largeur, hauteur décodées |
| 40,48,56 | 8+8+8 | offsets keypoints/descriptors, taille totale |
| 64 | 32 | SHA-256 de l'asset image source |
| 96 | 32 | fingerprint paramètres |
| 128 | 16 | `orb\0` puis zéros réservés |
| 144 | 4 | extractor version = 1 |
| 148 | 12 | réservés, zéro obligatoire |
Chaque keypoint contient six mots de 32 bits : `x`, `y`, `size`, orientation,
response en binary32, puis octave signé. `x/y` sont en pixels de l'image telle
que décodée par OpenCV, origine en haut à gauche. `size` est le diamètre du
voisinage en pixels. L'orientation est en degrés dans `[0,360)`. L'entrée `i`
du bloc keypoints correspond exactement aux 32 octets du descriptor `i`.
Le validateur contrôle magic, version, réservés, type, dimension, bornes,
offsets, taille exacte, multiplications, SHA-256 externe et cohérence DB. Une
version future est distinguée d'une corruption lorsque le fichier et son record
DB sont cohérents. Le reader utilise `pread`, accepte au plus 256 éléments par
appel et ne charge jamais le fichier entier.
Le chemin DB est validé sous sa forme canonique exacte dérivée du SHA-256 avant
toute ouverture. Une absence retourne `NOT_FOUND`; troncature, hash divergent,
header ou métadonnées DB divergents retournent `CORRUPT`. Une version future
retourne distinctement `UNSUPPORTED_VERSION`.
## Publication et persistance
Layout : `assets/features/<2 hex>/<sha256 complet lowercase>`. Le SHA-256 porte
sur le Feature File complet. Le protocole est : temporaire local, écriture,
`fsync`, hash, `link` atomique sans écrasement, validation complète lors d'une
adoption concurrente, `fsync` du répertoire, puis transaction SQLite. Un échec
du dernier `fsync` est enregistré `PUBLISHED_NOT_DURABLE`. Un échec SQLite après
publication laisse un fichier orphelin et aucune ligne logique partielle.
Un nouvel essai qui revalide l'asset et réussit le `fsync` promeut explicitement
sa durabilité vers `DURABLE` dans la transaction DB.
Le schéma v5 sépare `feature_assets`, `feature_sets` et
`feature_extract_tasks`. L'unicité logique porte sur image, kind, version et
fingerprint. Une tâche est persistée avant enqueue, passe par la queue et le
Governor (CPU, IO, 576 Mio conservateurs, lot 1), puis checkpointée initialement
et terminalement. La reprise recommence l'image entière : il n'existe pas de
fausse reprise intra-ORB. La publication est idempotente.
Pause et annulation sont coopératives avant/après l'appel ORB ; cet appel n'est
pas interruptible. L'image gérée est rehashée avant extraction. Une image
uniforme produit légitimement un Feature Set READY vide.
Lardon3D ne garantit pas des octets ORB identiques entre versions d'OpenCV,
plateformes ou backends. L'idempotence porte sur l'environnement courant et le
contrat `extractor_version`; une évolution qui change durablement la sémantique
ou les octets exige d'auditer et, si nécessaire, d'incrémenter cette version.
## Statut
**IMPLEMENTED** — ORB réel, format v1, assets content-addressed, DB v5,
publication atomique, reader borné, task kind production et reprise automatique.
**NOT_YET_WIRED** — commande de réconciliation/scrub des orphelins, contrôle fin
du backend parallèle OpenCV, orientation EXIF, lancement automatique de
l'extraction après import et planification multi-image/DAG.
**PLANNED** — Visual Index, paires candidates, matching, vérification
géométrique, tracks et SfM.

View file

@ -62,6 +62,12 @@ catalogue, le manifeste ou les images.
**Statut :** IMPLEMENTED
### Feature Store
Extraction ORB réelle par tâche persistante, Feature Sets logiques et assets
binaires content-addressed lisibles par plages bornées.
**Statut :** IMPLEMENTED
### Task
Moteur de tâches avec états, progression, pause/reprise coopérative,
annulation, checkpoints et estimations de ressources.

View file

@ -108,7 +108,7 @@ par le `time_t` local avant conversion. Le format reste donc lisible entre
plateformes uniquement pour les valeurs communes à leurs domaines `size_t` et
`time_t`.
## Project Database v4
## Project Database v5
SQLite contient l'état logique interrogable et les références aux fichiers ;
les checkpoints et artefacts volumineux restent externes. L'enregistrement du
@ -147,8 +147,8 @@ code.
**IMPLEMENTED** — modèle durable, codec v1, lecture validée, publication
atomique et restauration sûre d'une tâche isolée.
**IMPLEMENTED** — Project Database v4 pour identité, tâches typées, ScanSets,
images logiques, assets SHA-256, références checkpoint et artefacts génériques.
**IMPLEMENTED** — Project Database v5 pour identité, tâches typées, ScanSets,
images logiques, Feature Sets/assets SHA-256, checkpoints et artefacts génériques.
**IMPLEMENTED** — registry statique bornée et reconstruction explicite avec
ownership du userdata.
@ -205,5 +205,8 @@ retry piloté par l'utilisateur pour les sources indisponibles.
**NOT_YET_WIRED** — migration de la vue TUI en mémoire vers la pagination
SQLite, scrub des assets et réconciliation globale des orphelins.
**PLANNED** — Feature Store, Visual Index, migrations v5+ et reprise avec
**IMPLEMENTED** — Feature Store externe versionné, immutable, borné et relié
transactionnellement à ses métadonnées SQLite après publication.
**PLANNED** — Visual Index, migrations v6+ et reprise avec
dépendances.

View file

@ -168,9 +168,9 @@ référence vers le fichier checkpoint. Le fichier checkpoint validé reste la
source complète pour `lardon3d_task_restore()` ; la DB seule ne reconstruit
jamais une tâche. Un écart ou un fichier invalide interdit la reprise.
## Schéma v4 implémenté
## Schéma v5 implémenté
- `metadata(key PRIMARY KEY, value)` contient `schema_version=4` et
- `metadata(key PRIMARY KEY, value)` contient `schema_version=5` et
`next_task_id`, prochain ID durable allouable.
- `project(singleton=1, stable_id UNIQUE, name, created_at, updated_at)` décrit
l'unique identité logique de la DB.
@ -209,12 +209,12 @@ SHA-256 et le chemin asset sont déjà indexés par leurs contraintes `UNIQUE`.
## Ouverture et migrations
Une DB vide reçoit directement le schéma v4 dans une transaction
Une DB vide reçoit directement le schéma v5 dans une transaction
`BEGIN IMMEDIATE`. Une DB v1 reçoit transactionnellement les colonnes nullable
`task_kind` et `task_kind_version`, puis les migrations v2→v3. Les anciennes lignes restent
`NULL/NULL`, sans type inventé et sans perte des projets, tâches, checkpoints ou
artefacts. Une interruption ou erreur provoque un rollback complet. Les DB v1,
v2 et v3 sont migrées séquentiellement vers v4. Une version future est refusée et une DB contenant
v2, v3 et v4 sont migrées séquentiellement vers v5. Une version future est refusée et une DB contenant
des tables sans métadonnée de version est considérée corrompue. La fonction
interne de migration ne connaît que `0 → 4`, `1 → 2 → 3 → 4`,
`2 → 3 → 4` et `3 → 4`.
@ -293,7 +293,7 @@ UPDATE metadata SET value=4
WHERE key='schema_version' AND value=3;
```
Configuration v4 : `foreign_keys=ON`, `journal_mode=DELETE`,
Configuration v5 : `foreign_keys=ON`, `journal_mode=DELETE`,
`synchronous=FULL`, `busy_timeout=5000`. Le mode DELETE convient au propriétaire
unique actuel, évite les fichiers WAL/SHM durables et conserve la synchronisation
forte. Le timeout borne l'attente d'un verrou externe à cinq secondes.
@ -354,7 +354,7 @@ ouvert.
## Statut
**IMPLEMENTED** — SQLite système, schéma v4 et migrations v1→v2→v3→v4, identité
**IMPLEMENTED** — SQLite système, schéma v5 et migrations v1→v2→v3→v4→v5, identité
projet, transactions tâche+checkpoint, pagination de reprise et artefacts
génériques.
@ -382,7 +382,7 @@ cataloguées », pas « images migrées ».
**NOT_YET_WIRED** — autosave à toutes les transitions, retry UI des sources
indisponibles, migration de la TUI legacy et réconciliation des fichiers
orphelins, Feature Store et Visual Index.
orphelins et Visual Index. Le Feature Store v1 est implémenté.
**PLANNED** — migrations v5+, dépendances d'artefacts, graphe géométrique et
reconstruction incrémentale.

View file

@ -42,15 +42,16 @@ content-addressed. Les états Feature/Matching/Reconstruction restent planifiés
---
### C. Feature Store
### C. Feature Extraction et Feature Store
| Aspect | Description |
|--------|-------------|
| **Index / métadonnées persistants** | Les descripteurs de features (points clés, descripteurs, orientations) sont persistés sur disque dans un format binaire compact. Le rechargement évite la re-extraction. |
| **Données numériques massives** | Descripteurs floats, coordonnées de keypoints : volumes potentiellement importants. Doivent être stockés de manière séquentielle et indexée. |
| **Formats adaptés et bornés** | Format binaire avec en-tête (version, nombre de features, dimensions). Borné par le budget RAM du governor : si le store dépasse la capacité, seuls les N plus récents sont en mémoire. |
| **Extraction** | ORB OpenCV réel, une image par tâche `features.extract`. |
| **Métadonnées persistantes** | `FeatureSet` logique et `FeatureAsset` physique content-addressed dans ProjectDb v5. |
| **Données numériques massives** | Keypoints et descripteurs U8×32 restent hors SQLite dans Feature File v1. |
| **Lecture bornée** | Le reader lit au plus 256 features par plage et ne charge pas tout le fichier. |
**Statut :** PLANNED — aucune implémentation existante.
**Statut :** IMPLEMENTED v1 — extraction, publication atomique, reprise et reader borné.
---
@ -212,7 +213,11 @@ Image Catalog (B) ──► Feature Store (C)
---
## Statut : PLANNED (vision architecturale)
## Statut du pipeline
Import, Image Catalog, Feature Extraction et Feature Store sont
**IMPLEMENTED**. Visual Index, Candidate Pair, Matching, Tracks et SfM sont
**PLANNED**.
Ce document décrit la vision architecturale cible du pipeline de
reconstruction. Les modules listés ici ne sont pas tous implémentés.

View file

@ -73,6 +73,11 @@ Le Resource Governor est l'unique propriétaire des budgets (RAM, GPU, CPU, IO).
l'échantillon peut conserver taille/durée mais n'alimente jamais l'adaptation
mémoire.
- Pas de communication inter-classes de tâches
- `features.extract` réserve un lot de 1, un thread CPU et un slot I/O, avec
64 Mio fixes et 512 Mio par image. Cette estimation conservatrice couvre le
chemin actuel sans prétendre mesurer les allocations internes d'OpenCV.
`record_batch` couvre la validation source, le décodage, ORB, la publication
et la finalisation DB ; `peak_memory_bytes == 0` signifie « mesure inconnue ».
## Limites actuelles

View file

@ -26,7 +26,7 @@ fin de l'exécution. Le constructeur métier n'est jamais appelé sous mutex DB.
## Persistance et legacy
Le checkpoint générique reste en version 1. Project Database v4 conserve le
Le checkpoint générique reste en version 1. Project Database v5 conserve le
kind/version ; les lignes migrées depuis v1 restent `NULL/NULL` et sont classées
`LEGACY_UNTYPED`. Un kind inconnu ou une version non supportée reste inspectable
mais inexécutable. Aucun type n'est inventé et aucun code n'est sélectionné par
@ -46,5 +46,8 @@ pour restaurer hors mutex DB et transférer chaque tâche acceptée à la queue.
**NOT_YET_WIRED** — autosave complet et réconciliation orpheline.
**PLANNED** — kinds des tâches ScanSet, Image Catalog, Feature Store, Visual
**IMPLEMENTED** — `features.extract` version 1 reconstruit une extraction ORB
depuis `image_id` et ses paramètres bornés.
**PLANNED** — kinds des tâches Visual
Index et reconstruction lorsque ces traitements existeront réellement.

View file

@ -106,7 +106,7 @@ localement : son userdata est détruit, sans callback terminal ni écriture
durable d'une fausse annulation. Une annulation explicitement demandée conserve
le contrat de notification terminale.
La Project Database v4 peut enregistrer transactionnellement un résumé
La Project Database v5 peut enregistrer transactionnellement un résumé
`Lardon3DTaskDurableSnapshot` et la référence de son checkpoint. Elle ne stocke
ni estimation sérialisée complète, ni callback, ni réservation, et ne remplace
pas la validation du fichier checkpoint avant `task_restore()`.

View file

@ -33,6 +33,11 @@ au ScanSet, résultats du Visual Index et provenance. Une proximité temporelle
pourra servir de signal secondaire ; la proximité dans un dossier et le nom de
fichier ne constituent jamais l'identité principale.
Une observation de feature utilisera la référence durable
`feature_set_id + feature_index`. Une fois le Feature Set READY,
`feature_index` désigne immuablement la paire keypoint/descripteur de même
indice ; aucun chemin source n'entre dans cette identité.
## Concepts clés
### Étapes du matching

View file

@ -26,10 +26,10 @@ Matching & Tracks (appariement détaillé)
L'index visuel se situe entre l'extraction de features et la sélection de paires candidates. Il réduit drastiquement l'espace de recherche pour l'appariement.
Le futur index référencera exclusivement les `image_id` stables du catalogue
persistant. Un chemin d'asset ou un nom source n'est ni une identité d'image ni
une clé d'index durable. Le partage d'un asset entre deux ScanSets n'empêche pas
leurs deux images logiques d'être des observations distinctes.
Le futur index consommera exclusivement un `feature_set_id` et les descripteurs
obtenus par plages bornées du Feature Reader. Il ne dépendra ni des chemins
source, ni d'OpenCV, ni du runtime de la tâche productrice. Un chemin d'asset ou
un nom source n'est pas une clé d'index durable.
## Concepts clés

View file

@ -57,9 +57,8 @@ git diff --check
allocations :
```sh
meson setup build-asan --wipe \
-Db_sanitize=address,undefined \
-Db_static=false
CC=clang meson setup build-asan --wipe \
-Db_sanitize=address,undefined
meson compile -C build-asan -j8
meson test -C build-asan --print-errorlogs
```
@ -70,9 +69,9 @@ meson test -C build-asan --print-errorlogs
variables de condition, états partagés) :
```sh
meson setup build-tsan --wipe \
CC=clang meson setup build-tsan --wipe \
-Db_sanitize=thread \
-Db_static=false
-Db_lundef=false
meson compile -C build-tsan -j8
meson test -C build-tsan --print-errorlogs
```

View file

@ -160,7 +160,7 @@ Pour tout ticket touchant la concurrence, exécuter :
```sh
# Build TSan
meson setup build-tsan --wipe -Db_sanitize=thread -Db_static=false
CC=clang meson setup build-tsan --wipe -Db_sanitize=thread -Db_lundef=false
meson compile -C build-tsan -j8
meson test -C build-tsan --print-errorlogs
```

View file

@ -95,12 +95,12 @@ meson test -C build --print-errorlogs
git diff --check
# 4. Si mémoire/concurrence touchés
meson setup build-asan --wipe -Db_sanitize=address,undefined -Db_static=false
CC=clang meson setup build-asan --wipe -Db_sanitize=address,undefined
meson compile -C build-asan -j8
meson test -C build-asan --print-errorlogs
# 5. Si concurrence touchée
meson setup build-tsan --wipe -Db_sanitize=thread -Db_static=false
CC=clang meson setup build-tsan --wipe -Db_sanitize=thread -Db_lundef=false
meson compile -C build-tsan -j8
meson test -C build-tsan --print-errorlogs
```

View file

@ -29,7 +29,7 @@ Lardon3D suit une feuille de route ordonnée qui privilégie la stabilité et la
### Phase 3 : Persistance
- ✅ Fondation versionnée des checkpoints de tâches
- ✅ Project Database v4 (tâches, ScanSets, images, assets et migrations)
- ✅ Project Database v5 (tâches, ScanSets, images, Feature Sets/assets et migrations)
- ✅ Branchement Project Database au cycle de vie projet et inventaire de reprise
- ✅ Registry durable des types métier de tâches
- ✅ Premier type métier reconstructible (`import.images`)
@ -37,7 +37,7 @@ Lardon3D suit une feuille de route ordonnée qui privilégie la stabilité et la
- ✅ ScanSet v1 et Image Catalog persistant v1
### Phase 4 : Pipeline avancé
- 📋 Feature Store
- ✅ Feature Store v1 et extraction ORB persistante
- 📋 Visual Index
- 📋 Candidate Pair Generator
- 📋 Matching et vérification géométrique

View file

@ -0,0 +1,57 @@
#ifndef LARDON3D_FEATURE_EXTRACTOR_H
#define LARDON3D_FEATURE_EXTRACTOR_H
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#define LARDON3D_FEATURE_EXTRACTOR_KIND "orb"
enum {
LARDON3D_FEATURE_EXTRACTOR_VERSION = 1,
LARDON3D_FEATURE_DESCRIPTOR_DIMENSION = 32,
LARDON3D_FEATURE_MAX_FEATURES = 8192,
LARDON3D_FEATURE_MAX_IMAGE_PIXELS = 100000000,
};
typedef struct {
uint32_t max_features;
uint32_t pyramid_levels;
uint32_t fast_threshold;
} Lardon3DFeatureExtractorParameters;
typedef struct {
float x;
float y;
float size;
float angle_degrees;
float response;
int32_t octave;
} Lardon3DFeatureKeypoint;
typedef struct {
uint32_t image_width;
uint32_t image_height;
uint32_t feature_count;
Lardon3DFeatureKeypoint *keypoints;
unsigned char *descriptors;
} Lardon3DExtractedFeatures;
typedef enum {
LARDON3D_FEATURE_EXTRACT_OK = 0,
LARDON3D_FEATURE_EXTRACT_INVALID_ARGUMENT,
LARDON3D_FEATURE_EXTRACT_IMAGE_NOT_FOUND,
LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID,
LARDON3D_FEATURE_EXTRACT_OUT_OF_MEMORY,
LARDON3D_FEATURE_EXTRACT_ERROR
} Lardon3DFeatureExtractResult;
bool lardon3d_feature_extractor_parameters_valid(
const Lardon3DFeatureExtractorParameters *parameters);
void lardon3d_feature_extractor_parameter_fingerprint(
const Lardon3DFeatureExtractorParameters *parameters, unsigned char fingerprint[32]);
Lardon3DFeatureExtractResult
lardon3d_feature_extract_orb(const char *path, const Lardon3DFeatureExtractorParameters *parameters,
Lardon3DExtractedFeatures *features);
void lardon3d_extracted_features_destroy(Lardon3DExtractedFeatures *features);
#endif

View file

@ -0,0 +1,73 @@
#ifndef LARDON3D_FEATURE_STORE_H
#define LARDON3D_FEATURE_STORE_H
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#include <lardon3d/app_state.h>
#include <lardon3d/feature_extractor.h>
#include <lardon3d/project_db.h>
enum {
LARDON3D_FEATURE_FILE_VERSION = 1,
LARDON3D_FEATURE_FILE_HEADER_SIZE = 160,
LARDON3D_FEATURE_KEYPOINT_RECORD_SIZE = 24,
LARDON3D_FEATURE_READER_RANGE_MAX = 256,
LARDON3D_FEATURE_FILE_MAX_SIZE = 16 * 1024 * 1024,
};
typedef enum {
LARDON3D_FEATURE_DESCRIPTOR_U8 = 1,
LARDON3D_FEATURE_DESCRIPTOR_F32 = 2
} Lardon3DFeatureDescriptorType;
typedef enum {
LARDON3D_FEATURE_STORE_OK = 0,
LARDON3D_FEATURE_STORE_ALREADY_PRESENT,
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT,
LARDON3D_FEATURE_STORE_NOT_FOUND,
LARDON3D_FEATURE_STORE_INVALID,
LARDON3D_FEATURE_STORE_UNSUPPORTED_VERSION,
LARDON3D_FEATURE_STORE_CORRUPT,
LARDON3D_FEATURE_STORE_IO_ERROR,
LARDON3D_FEATURE_STORE_DB_BUSY,
LARDON3D_FEATURE_STORE_DB_ERROR,
LARDON3D_FEATURE_STORE_PUBLISHED_NOT_DURABLE
} Lardon3DFeatureStoreResult;
typedef struct Lardon3DFeatureReader Lardon3DFeatureReader;
typedef struct {
uint32_t format_version;
char extractor_kind[LARDON3D_TASK_KIND_CAPACITY];
uint32_t extractor_version;
uint32_t feature_count;
uint32_t descriptor_dimension;
Lardon3DFeatureDescriptorType descriptor_type;
uint32_t image_width;
uint32_t image_height;
unsigned char source_image_sha256[32];
unsigned char parameter_fingerprint[32];
} Lardon3DFeatureFileMetadata;
Lardon3DFeatureStoreResult lardon3d_feature_store_publish(
Lardon3DAppState *state, uint64_t image_id, uint64_t producer_task_id,
const Lardon3DFeatureExtractorParameters *parameters, const Lardon3DExtractedFeatures *features,
Lardon3DProjectDbFeatureSet *feature_set);
Lardon3DFeatureStoreResult
lardon3d_feature_reader_open(const char *project_path,
const Lardon3DProjectDbFeatureSet *feature_set,
Lardon3DFeatureReader **reader, Lardon3DFeatureFileMetadata *metadata);
void lardon3d_feature_reader_close(Lardon3DFeatureReader *reader);
Lardon3DFeatureStoreResult lardon3d_feature_reader_keypoints(Lardon3DFeatureReader *reader,
uint32_t start,
Lardon3DFeatureKeypoint *keypoints,
size_t capacity);
Lardon3DFeatureStoreResult lardon3d_feature_reader_descriptors(Lardon3DFeatureReader *reader,
uint32_t start,
unsigned char *descriptors,
size_t feature_capacity,
size_t descriptor_capacity);
#endif

View file

@ -0,0 +1,24 @@
#ifndef LARDON3D_FEATURE_TASK_H
#define LARDON3D_FEATURE_TASK_H
#include <stdbool.h>
#include <stdint.h>
#include <lardon3d/app_state.h>
#include <lardon3d/feature_extractor.h>
#include <lardon3d/task_kind_registry.h>
#define LARDON3D_FEATURE_EXTRACT_TASK_KIND "features.extract"
enum { LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION = 1 };
Lardon3DTask *
lardon3d_project_create_feature_extract_task(Lardon3DAppState *state, uint64_t image_id,
const Lardon3DFeatureExtractorParameters *parameters,
uint64_t *task_id);
bool lardon3d_project_enqueue_feature_extract(Lardon3DAppState *state, uint64_t image_id,
const Lardon3DFeatureExtractorParameters *parameters,
uint64_t *task_id);
bool lardon3d_feature_extract_reconstruct(const Lardon3DTaskDurableSnapshot *snapshot,
void *context, Lardon3DTaskKindBinding *binding);
#endif

View file

@ -12,61 +12,42 @@
enum { LARDON3D_IMAGE_IMPORT_TASK_KIND_VERSION = 1 };
typedef enum {
LARDON3D_IMPORT_TASK_IDLE = 0,
LARDON3D_IMPORT_TASK_RUNNING,
LARDON3D_IMPORT_TASK_SUCCEEDED,
LARDON3D_IMPORT_TASK_CANCELLED,
LARDON3D_IMPORT_TASK_FAILED
LARDON3D_IMPORT_TASK_IDLE = 0,
LARDON3D_IMPORT_TASK_RUNNING,
LARDON3D_IMPORT_TASK_SUCCEEDED,
LARDON3D_IMPORT_TASK_CANCELLED,
LARDON3D_IMPORT_TASK_FAILED
} Lardon3DImportTaskStatus;
typedef struct {
Lardon3DImportTaskStatus status;
size_t total;
size_t processed;
size_t copied;
size_t already_present;
size_t ignored;
char message[256];
Lardon3DImportTaskStatus status;
size_t total;
size_t processed;
size_t copied;
size_t already_present;
size_t ignored;
char message[256];
} Lardon3DImportTaskSnapshot;
typedef struct Lardon3DImportTask Lardon3DImportTask;
typedef struct {
const char *project_path;
Lardon3DProjectDb *project_db;
Lardon3DResourceGovernor *resource_governor;
} Lardon3DImageImportReconstructionContext;
typedef Lardon3DTaskReconstructionContext Lardon3DImageImportReconstructionContext;
Lardon3DTask *lardon3d_project_create_image_import_task(
Lardon3DAppState *state,
uint64_t scanset_id,
const char *source_directory,
uint64_t *task_id
);
bool lardon3d_project_enqueue_image_import(
Lardon3DAppState *state,
uint64_t scanset_id,
const char *source_directory,
uint64_t *task_id
);
bool lardon3d_image_import_reconstruct(
const Lardon3DTaskDurableSnapshot *snapshot,
void *context,
Lardon3DTaskKindBinding *binding
);
Lardon3DTask *lardon3d_project_create_image_import_task(Lardon3DAppState *state,
uint64_t scanset_id,
const char *source_directory,
uint64_t *task_id);
bool lardon3d_project_enqueue_image_import(Lardon3DAppState *state, uint64_t scanset_id,
const char *source_directory, uint64_t *task_id);
bool lardon3d_image_import_reconstruct(const Lardon3DTaskDurableSnapshot *snapshot, void *context,
Lardon3DTaskKindBinding *binding);
/* Compatibilité TUI : handle léger sur une tâche de la queue, sans thread. */
Lardon3DImportTask *lardon3d_import_task_create(void);
bool lardon3d_import_task_start(
Lardon3DImportTask *task,
Lardon3DAppState *state,
const char *source_directory
);
bool lardon3d_import_task_start(Lardon3DImportTask *task, Lardon3DAppState *state,
const char *source_directory);
void lardon3d_import_task_request_cancel(Lardon3DImportTask *task);
bool lardon3d_import_task_snapshot(
Lardon3DImportTask *task,
Lardon3DImportTaskSnapshot *snapshot
);
bool lardon3d_import_task_snapshot(Lardon3DImportTask *task, Lardon3DImportTaskSnapshot *snapshot);
bool lardon3d_import_task_is_finished(Lardon3DImportTask *task);
bool lardon3d_import_task_join(Lardon3DImportTask *task);
void lardon3d_import_task_destroy(Lardon3DImportTask *task);

View file

@ -8,84 +8,69 @@
#include <lardon3d/task_kind_registry.h>
typedef enum {
LARDON3D_PROJECT_TASK_CHECKPOINT_OK = 0,
LARDON3D_PROJECT_TASK_CHECKPOINT_PUBLISHED_NOT_DURABLE,
LARDON3D_PROJECT_TASK_CHECKPOINT_NO_PROJECT,
LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK,
LARDON3D_PROJECT_TASK_CHECKPOINT_IO_ERROR,
LARDON3D_PROJECT_TASK_CHECKPOINT_DB_BUSY,
LARDON3D_PROJECT_TASK_CHECKPOINT_DB_ERROR
LARDON3D_PROJECT_TASK_CHECKPOINT_OK = 0,
LARDON3D_PROJECT_TASK_CHECKPOINT_PUBLISHED_NOT_DURABLE,
LARDON3D_PROJECT_TASK_CHECKPOINT_NO_PROJECT,
LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK,
LARDON3D_PROJECT_TASK_CHECKPOINT_IO_ERROR,
LARDON3D_PROJECT_TASK_CHECKPOINT_DB_BUSY,
LARDON3D_PROJECT_TASK_CHECKPOINT_DB_ERROR
} Lardon3DProjectTaskCheckpointResult;
typedef enum {
LARDON3D_PROJECT_RECOVERABLE = 0,
LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE,
LARDON3D_PROJECT_RECOVERY_MISSING_CHECKPOINT,
LARDON3D_PROJECT_RECOVERY_INVALID_CHECKPOINT,
LARDON3D_PROJECT_RECOVERY_UNSUPPORTED_CHECKPOINT,
LARDON3D_PROJECT_RECOVERY_CHECKPOINT_IO_ERROR,
LARDON3D_PROJECT_RECOVERY_LEGACY_UNTYPED,
LARDON3D_PROJECT_RECOVERY_UNKNOWN_TASK_KIND,
LARDON3D_PROJECT_RECOVERY_UNSUPPORTED_TASK_KIND_VERSION
LARDON3D_PROJECT_RECOVERABLE = 0,
LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE,
LARDON3D_PROJECT_RECOVERY_MISSING_CHECKPOINT,
LARDON3D_PROJECT_RECOVERY_INVALID_CHECKPOINT,
LARDON3D_PROJECT_RECOVERY_UNSUPPORTED_CHECKPOINT,
LARDON3D_PROJECT_RECOVERY_CHECKPOINT_IO_ERROR,
LARDON3D_PROJECT_RECOVERY_LEGACY_UNTYPED,
LARDON3D_PROJECT_RECOVERY_UNKNOWN_TASK_KIND,
LARDON3D_PROJECT_RECOVERY_UNSUPPORTED_TASK_KIND_VERSION
} Lardon3DProjectRecoveryStatus;
typedef struct {
uint64_t task_id;
char name[LARDON3D_TASK_NAME_CAPACITY];
char task_kind[LARDON3D_TASK_KIND_CAPACITY];
uint32_t task_kind_version;
Lardon3DProjectRecoveryStatus status;
Lardon3DProjectDbCheckpointDurability durability;
Lardon3DTaskDurableSnapshot snapshot;
uint64_t task_id;
char name[LARDON3D_TASK_NAME_CAPACITY];
char task_kind[LARDON3D_TASK_KIND_CAPACITY];
uint32_t task_kind_version;
Lardon3DProjectRecoveryStatus status;
Lardon3DProjectDbCheckpointDurability durability;
Lardon3DTaskDurableSnapshot snapshot;
} Lardon3DProjectRecoveryEntry;
typedef struct {
size_t inspected;
size_t resumed;
size_t skipped;
size_t failed;
size_t published_not_durable;
bool queue_full;
size_t inspected;
size_t resumed;
size_t skipped;
size_t failed;
size_t published_not_durable;
bool queue_full;
} Lardon3DProjectRecoverySummary;
bool lardon3d_project_create(
Lardon3DAppState *state,
const char *name
);
bool lardon3d_project_create(Lardon3DAppState *state, const char *name);
bool lardon3d_project_open(
Lardon3DAppState *state,
const char *directory_name
);
bool lardon3d_project_open(Lardon3DAppState *state, const char *directory_name);
void lardon3d_project_close(Lardon3DAppState *state);
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_task(
Lardon3DAppState *state,
const Lardon3DTask *task
);
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_task(Lardon3DAppState *state,
const Lardon3DTask *task);
Lardon3DProjectTaskCheckpointResult
lardon3d_project_checkpoint_image_import_task(
Lardon3DAppState *state,
const Lardon3DTask *task,
const char *source_path,
uint64_t scanset_id
);
Lardon3DProjectDbResult lardon3d_project_list_recoverable(
Lardon3DAppState *state,
const Lardon3DTaskKindRegistry *registry,
uint64_t after_task_id,
Lardon3DProjectRecoveryEntry *entries,
size_t capacity,
size_t *count
);
Lardon3DProjectDbResult lardon3d_project_resume_recoverable_tasks(
Lardon3DAppState *state,
const Lardon3DTaskKindRegistry *registry,
Lardon3DProjectRecoverySummary *summary
);
bool lardon3d_project_last_recovery_summary(
const Lardon3DAppState *state,
Lardon3DProjectRecoverySummary *summary
);
lardon3d_project_checkpoint_image_import_task(Lardon3DAppState *state, const Lardon3DTask *task,
const char *source_path, uint64_t scanset_id);
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_task(
Lardon3DAppState *state, const Lardon3DTask *task,
const Lardon3DProjectDbFeatureExtractTask *parameters);
Lardon3DProjectDbResult lardon3d_project_list_recoverable(Lardon3DAppState *state,
const Lardon3DTaskKindRegistry *registry,
uint64_t after_task_id,
Lardon3DProjectRecoveryEntry *entries,
size_t capacity, size_t *count);
Lardon3DProjectDbResult
lardon3d_project_resume_recoverable_tasks(Lardon3DAppState *state,
const Lardon3DTaskKindRegistry *registry,
Lardon3DProjectRecoverySummary *summary);
bool lardon3d_project_last_recovery_summary(const Lardon3DAppState *state,
Lardon3DProjectRecoverySummary *summary);
#endif

View file

@ -9,233 +9,259 @@
#include <lardon3d/task.h>
enum {
LARDON3D_PROJECT_DB_SCHEMA_VERSION = 4,
LARDON3D_PROJECT_DB_ID_CAPACITY = 65,
LARDON3D_PROJECT_DB_KIND_CAPACITY = 65,
LARDON3D_PROJECT_DB_PATH_CAPACITY = 4096,
LARDON3D_PROJECT_DB_ERROR_CAPACITY = 256,
LARDON3D_PROJECT_DB_RECOVERY_PAGE_MAX = 256,
LARDON3D_PROJECT_DB_CATALOG_PAGE_MAX = 256,
LARDON3D_PROJECT_DB_SCANSET_NAME_CAPACITY = 256,
LARDON3D_PROJECT_DB_IMAGE_NAME_CAPACITY = 256,
LARDON3D_PROJECT_DB_SHA256_SIZE = 32,
LARDON3D_PROJECT_DB_SCHEMA_VERSION = 5,
LARDON3D_PROJECT_DB_ID_CAPACITY = 65,
LARDON3D_PROJECT_DB_KIND_CAPACITY = 65,
LARDON3D_PROJECT_DB_PATH_CAPACITY = 4096,
LARDON3D_PROJECT_DB_ERROR_CAPACITY = 256,
LARDON3D_PROJECT_DB_RECOVERY_PAGE_MAX = 256,
LARDON3D_PROJECT_DB_CATALOG_PAGE_MAX = 256,
LARDON3D_PROJECT_DB_SCANSET_NAME_CAPACITY = 256,
LARDON3D_PROJECT_DB_IMAGE_NAME_CAPACITY = 256,
LARDON3D_PROJECT_DB_SHA256_SIZE = 32,
};
typedef struct Lardon3DProjectDb Lardon3DProjectDb;
typedef enum {
LARDON3D_PROJECT_DB_OK = 0,
LARDON3D_PROJECT_DB_INVALID_ARGUMENT,
LARDON3D_PROJECT_DB_NOT_FOUND,
LARDON3D_PROJECT_DB_BUSY,
LARDON3D_PROJECT_DB_UNSUPPORTED_SCHEMA,
LARDON3D_PROJECT_DB_CORRUPT,
LARDON3D_PROJECT_DB_CONSTRAINT,
LARDON3D_PROJECT_DB_IO_ERROR
LARDON3D_PROJECT_DB_OK = 0,
LARDON3D_PROJECT_DB_INVALID_ARGUMENT,
LARDON3D_PROJECT_DB_NOT_FOUND,
LARDON3D_PROJECT_DB_BUSY,
LARDON3D_PROJECT_DB_UNSUPPORTED_SCHEMA,
LARDON3D_PROJECT_DB_CORRUPT,
LARDON3D_PROJECT_DB_CONSTRAINT,
LARDON3D_PROJECT_DB_IO_ERROR
} Lardon3DProjectDbResult;
typedef enum {
LARDON3D_DB_CHECKPOINT_DURABLE = 0,
LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE
LARDON3D_DB_CHECKPOINT_DURABLE = 0,
LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE
} Lardon3DProjectDbCheckpointDurability;
typedef enum {
LARDON3D_DB_ARTIFACT_STAGED = 0,
LARDON3D_DB_ARTIFACT_READY
LARDON3D_DB_ARTIFACT_STAGED = 0,
LARDON3D_DB_ARTIFACT_READY
} Lardon3DProjectDbArtifactState;
typedef struct {
char stable_id[LARDON3D_PROJECT_DB_ID_CAPACITY];
char name[LARDON3D_TASK_NAME_CAPACITY];
int64_t created_at;
int64_t updated_at;
char stable_id[LARDON3D_PROJECT_DB_ID_CAPACITY];
char name[LARDON3D_TASK_NAME_CAPACITY];
int64_t created_at;
int64_t updated_at;
} Lardon3DProjectDbProject;
typedef struct {
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
uint32_t format_version;
Lardon3DProjectDbCheckpointDurability durability;
int64_t updated_at;
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
uint32_t format_version;
Lardon3DProjectDbCheckpointDurability durability;
int64_t updated_at;
} Lardon3DProjectDbCheckpoint;
typedef struct {
uint64_t task_id;
char name[LARDON3D_TASK_NAME_CAPACITY];
bool has_task_kind;
char task_kind[LARDON3D_TASK_KIND_CAPACITY];
uint32_t task_kind_version;
Lardon3DTaskState saved_state;
Lardon3DTaskState recovery_state;
unsigned int progress;
unsigned int sequence_count;
struct timespec started_at;
struct timespec finished_at;
int64_t updated_at;
bool has_checkpoint;
Lardon3DProjectDbCheckpoint checkpoint;
uint64_t task_id;
char name[LARDON3D_TASK_NAME_CAPACITY];
bool has_task_kind;
char task_kind[LARDON3D_TASK_KIND_CAPACITY];
uint32_t task_kind_version;
Lardon3DTaskState saved_state;
Lardon3DTaskState recovery_state;
unsigned int progress;
unsigned int sequence_count;
struct timespec started_at;
struct timespec finished_at;
int64_t updated_at;
bool has_checkpoint;
Lardon3DProjectDbCheckpoint checkpoint;
} Lardon3DProjectDbTask;
typedef struct {
char artifact_id[LARDON3D_PROJECT_DB_ID_CAPACITY];
char kind[LARDON3D_PROJECT_DB_KIND_CAPACITY];
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
Lardon3DProjectDbArtifactState state;
uint64_t size_bytes;
bool has_producer_task;
uint64_t producer_task_id;
int64_t created_at;
int64_t updated_at;
char artifact_id[LARDON3D_PROJECT_DB_ID_CAPACITY];
char kind[LARDON3D_PROJECT_DB_KIND_CAPACITY];
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
Lardon3DProjectDbArtifactState state;
uint64_t size_bytes;
bool has_producer_task;
uint64_t producer_task_id;
int64_t created_at;
int64_t updated_at;
} Lardon3DProjectDbArtifact;
typedef struct {
uint64_t task_id;
char source_path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
uint64_t scanset_id;
uint64_t task_id;
char source_path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
uint64_t scanset_id;
} Lardon3DProjectDbImageImport;
typedef struct {
uint64_t scanset_id;
char name[LARDON3D_PROJECT_DB_SCANSET_NAME_CAPACITY];
int64_t created_at;
int64_t updated_at;
uint64_t scanset_id;
char name[LARDON3D_PROJECT_DB_SCANSET_NAME_CAPACITY];
int64_t created_at;
int64_t updated_at;
} Lardon3DProjectDbScanSet;
typedef enum {
LARDON3D_DB_IMAGE_ASSET_READY = 1
} Lardon3DProjectDbImageAssetState;
typedef enum { LARDON3D_DB_IMAGE_ASSET_READY = 1 } Lardon3DProjectDbImageAssetState;
typedef struct {
uint64_t asset_id;
unsigned char sha256[LARDON3D_PROJECT_DB_SHA256_SIZE];
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
uint64_t size_bytes;
Lardon3DProjectDbImageAssetState state;
int64_t created_at;
uint64_t asset_id;
unsigned char sha256[LARDON3D_PROJECT_DB_SHA256_SIZE];
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
uint64_t size_bytes;
Lardon3DProjectDbImageAssetState state;
int64_t created_at;
} Lardon3DProjectDbImageAsset;
typedef struct {
uint64_t image_id;
uint64_t scanset_id;
uint64_t asset_id;
char original_name[LARDON3D_PROJECT_DB_IMAGE_NAME_CAPACITY];
char source_path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
bool has_producer_task;
uint64_t producer_task_id;
int64_t imported_at;
uint64_t image_id;
uint64_t scanset_id;
uint64_t asset_id;
char original_name[LARDON3D_PROJECT_DB_IMAGE_NAME_CAPACITY];
char source_path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
bool has_producer_task;
uint64_t producer_task_id;
int64_t imported_at;
} Lardon3DProjectDbImage;
typedef enum {
LARDON3D_PROJECT_DB_IMAGE_REGISTERED = 0,
LARDON3D_PROJECT_DB_IMAGE_ALREADY_PRESENT
LARDON3D_PROJECT_DB_IMAGE_REGISTERED = 0,
LARDON3D_PROJECT_DB_IMAGE_ALREADY_PRESENT
} Lardon3DProjectDbImageRegisterStatus;
Lardon3DProjectDbResult lardon3d_project_db_open(
const char *path,
Lardon3DProjectDb **database,
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]
);
typedef enum {
LARDON3D_DB_FEATURE_ASSET_DURABLE = 0,
LARDON3D_DB_FEATURE_ASSET_PUBLISHED_NOT_DURABLE = 1
} Lardon3DProjectDbFeatureDurability;
typedef struct {
uint64_t feature_asset_id;
unsigned char sha256[32];
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
uint64_t size_bytes;
Lardon3DProjectDbFeatureDurability durability;
int64_t created_at;
} Lardon3DProjectDbFeatureAsset;
typedef struct {
uint64_t feature_set_id;
uint64_t image_id;
uint64_t feature_asset_id;
char extractor_kind[LARDON3D_PROJECT_DB_KIND_CAPACITY];
uint32_t extractor_version;
unsigned char parameter_fingerprint[32];
unsigned char source_image_sha256[32];
uint32_t feature_count;
uint32_t descriptor_type;
uint32_t descriptor_dimension;
bool has_producer_task;
uint64_t producer_task_id;
int64_t created_at;
Lardon3DProjectDbFeatureAsset asset;
} Lardon3DProjectDbFeatureSet;
typedef struct {
uint64_t task_id;
uint64_t image_id;
char extractor_kind[LARDON3D_PROJECT_DB_KIND_CAPACITY];
uint32_t extractor_version;
uint32_t max_features;
uint32_t pyramid_levels;
uint32_t fast_threshold;
unsigned char parameter_fingerprint[32];
} Lardon3DProjectDbFeatureExtractTask;
Lardon3DProjectDbResult lardon3d_project_db_open(const char *path, Lardon3DProjectDb **database,
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]);
void lardon3d_project_db_close(Lardon3DProjectDb *database);
bool lardon3d_project_db_last_error(
Lardon3DProjectDb *database,
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]
);
Lardon3DProjectDbResult lardon3d_project_db_legacy_catalog_pending(
Lardon3DProjectDb *database, bool *pending
);
bool lardon3d_project_db_last_error(Lardon3DProjectDb *database,
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]);
Lardon3DProjectDbResult lardon3d_project_db_legacy_catalog_pending(Lardon3DProjectDb *database,
bool *pending);
unsigned int lardon3d_project_db_schema_version(Lardon3DProjectDb *database);
Lardon3DProjectDbResult lardon3d_project_db_set_project(
Lardon3DProjectDb *database,
const Lardon3DProjectDbProject *project
);
Lardon3DProjectDbResult lardon3d_project_db_get_project(
Lardon3DProjectDb *database,
Lardon3DProjectDbProject *project
);
Lardon3DProjectDbResult lardon3d_project_db_set_project(Lardon3DProjectDb *database,
const Lardon3DProjectDbProject *project);
Lardon3DProjectDbResult lardon3d_project_db_get_project(Lardon3DProjectDb *database,
Lardon3DProjectDbProject *project);
Lardon3DProjectDbResult lardon3d_project_db_record_task(
Lardon3DProjectDb *database,
const Lardon3DTaskDurableSnapshot *snapshot,
const char *task_kind,
uint32_t task_kind_version,
const Lardon3DProjectDbCheckpoint *checkpoint,
int64_t updated_at
);
Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, const char *task_kind,
uint32_t task_kind_version, const Lardon3DProjectDbCheckpoint *checkpoint, int64_t updated_at);
Lardon3DProjectDbResult lardon3d_project_db_record_image_import_task(
Lardon3DProjectDb *database,
const Lardon3DTaskDurableSnapshot *snapshot,
const char *task_kind,
uint32_t task_kind_version,
const Lardon3DProjectDbCheckpoint *checkpoint,
const char *source_path,
uint64_t scanset_id,
int64_t updated_at
);
Lardon3DProjectDbResult lardon3d_project_db_create_scanset(
Lardon3DProjectDb *database, const char *name,
Lardon3DProjectDbScanSet *scanset
);
Lardon3DProjectDbResult lardon3d_project_db_load_scanset(
Lardon3DProjectDb *database, uint64_t scanset_id,
Lardon3DProjectDbScanSet *scanset
);
Lardon3DProjectDbResult lardon3d_project_db_list_scansets(
Lardon3DProjectDb *database, uint64_t after_scanset_id,
Lardon3DProjectDbScanSet *scansets, size_t capacity, size_t *count
);
Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, const char *task_kind,
uint32_t task_kind_version, const Lardon3DProjectDbCheckpoint *checkpoint,
const char *source_path, uint64_t scanset_id, int64_t updated_at);
Lardon3DProjectDbResult lardon3d_project_db_create_scanset(Lardon3DProjectDb *database,
const char *name,
Lardon3DProjectDbScanSet *scanset);
Lardon3DProjectDbResult lardon3d_project_db_load_scanset(Lardon3DProjectDb *database,
uint64_t scanset_id,
Lardon3DProjectDbScanSet *scanset);
Lardon3DProjectDbResult lardon3d_project_db_list_scansets(Lardon3DProjectDb *database,
uint64_t after_scanset_id,
Lardon3DProjectDbScanSet *scansets,
size_t capacity, size_t *count);
Lardon3DProjectDbResult lardon3d_project_db_register_image(
Lardon3DProjectDb *database, uint64_t scanset_id,
const unsigned char sha256[LARDON3D_PROJECT_DB_SHA256_SIZE],
const char *asset_path, uint64_t size_bytes, const char *original_name,
const char *source_path, uint64_t producer_task_id, int64_t imported_at,
Lardon3DProjectDbImageRegisterStatus *status,
Lardon3DProjectDbImage *image
);
Lardon3DProjectDbResult lardon3d_project_db_load_image(
Lardon3DProjectDb *database, uint64_t image_id,
Lardon3DProjectDbImage *image, Lardon3DProjectDbImageAsset *asset
);
Lardon3DProjectDbResult lardon3d_project_db_list_images(
Lardon3DProjectDb *database, uint64_t scanset_id, uint64_t after_image_id,
Lardon3DProjectDbImage *images, Lardon3DProjectDbImageAsset *assets,
size_t capacity, size_t *count
);
Lardon3DProjectDbResult lardon3d_project_db_count_images(
Lardon3DProjectDb *database, uint64_t scanset_id, uint64_t *count
);
Lardon3DProjectDbResult lardon3d_project_db_load_image_import(
Lardon3DProjectDb *database,
uint64_t task_id,
Lardon3DProjectDbImageImport *parameters
);
Lardon3DProjectDbResult lardon3d_project_db_allocate_task_id(
Lardon3DProjectDb *database,
uint64_t *task_id
);
Lardon3DProjectDbResult lardon3d_project_db_load_task(
Lardon3DProjectDb *database,
uint64_t task_id,
Lardon3DProjectDbTask *task
);
Lardon3DProjectDbResult lardon3d_project_db_list_recoverable(
Lardon3DProjectDb *database,
uint64_t after_task_id,
Lardon3DProjectDbTask *tasks,
size_t capacity,
size_t *count
);
Lardon3DProjectDbResult lardon3d_project_db_create_artifact(
Lardon3DProjectDb *database,
const Lardon3DProjectDbArtifact *artifact
);
Lardon3DProjectDbResult lardon3d_project_db_mark_artifact_ready(
Lardon3DProjectDb *database,
const char *artifact_id,
int64_t updated_at
);
Lardon3DProjectDbResult lardon3d_project_db_load_artifact(
Lardon3DProjectDb *database,
const char *artifact_id,
Lardon3DProjectDbArtifact *artifact
);
const unsigned char sha256[LARDON3D_PROJECT_DB_SHA256_SIZE], const char *asset_path,
uint64_t size_bytes, const char *original_name, const char *source_path,
uint64_t producer_task_id, int64_t imported_at, Lardon3DProjectDbImageRegisterStatus *status,
Lardon3DProjectDbImage *image);
Lardon3DProjectDbResult lardon3d_project_db_load_image(Lardon3DProjectDb *database,
uint64_t image_id,
Lardon3DProjectDbImage *image,
Lardon3DProjectDbImageAsset *asset);
Lardon3DProjectDbResult lardon3d_project_db_list_images(Lardon3DProjectDb *database,
uint64_t scanset_id,
uint64_t after_image_id,
Lardon3DProjectDbImage *images,
Lardon3DProjectDbImageAsset *assets,
size_t capacity, size_t *count);
Lardon3DProjectDbResult lardon3d_project_db_count_images(Lardon3DProjectDb *database,
uint64_t scanset_id, uint64_t *count);
Lardon3DProjectDbResult
lardon3d_project_db_load_image_import(Lardon3DProjectDb *database, uint64_t task_id,
Lardon3DProjectDbImageImport *parameters);
Lardon3DProjectDbResult lardon3d_project_db_allocate_task_id(Lardon3DProjectDb *database,
uint64_t *task_id);
Lardon3DProjectDbResult lardon3d_project_db_load_task(Lardon3DProjectDb *database, uint64_t task_id,
Lardon3DProjectDbTask *task);
Lardon3DProjectDbResult lardon3d_project_db_list_recoverable(Lardon3DProjectDb *database,
uint64_t after_task_id,
Lardon3DProjectDbTask *tasks,
size_t capacity, size_t *count);
Lardon3DProjectDbResult
lardon3d_project_db_create_artifact(Lardon3DProjectDb *database,
const Lardon3DProjectDbArtifact *artifact);
Lardon3DProjectDbResult lardon3d_project_db_mark_artifact_ready(Lardon3DProjectDb *database,
const char *artifact_id,
int64_t updated_at);
Lardon3DProjectDbResult lardon3d_project_db_load_artifact(Lardon3DProjectDb *database,
const char *artifact_id,
Lardon3DProjectDbArtifact *artifact);
Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_task(
Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, const char *task_kind,
uint32_t task_kind_version, const Lardon3DProjectDbCheckpoint *checkpoint,
const Lardon3DProjectDbFeatureExtractTask *parameters, int64_t updated_at);
Lardon3DProjectDbResult
lardon3d_project_db_load_feature_extract_task(Lardon3DProjectDb *database, uint64_t task_id,
Lardon3DProjectDbFeatureExtractTask *parameters);
Lardon3DProjectDbResult lardon3d_project_db_register_feature_set(
Lardon3DProjectDb *database, uint64_t image_id, const char *extractor_kind,
uint32_t extractor_version, const unsigned char parameter_fingerprint[32],
const unsigned char source_image_sha256[32], uint32_t feature_count, uint32_t descriptor_type,
uint32_t descriptor_dimension, const unsigned char asset_sha256[32], const char *asset_path,
uint64_t asset_size_bytes, Lardon3DProjectDbFeatureDurability durability,
uint64_t producer_task_id, int64_t created_at, Lardon3DProjectDbFeatureSet *feature_set);
Lardon3DProjectDbResult
lardon3d_project_db_find_feature_set(Lardon3DProjectDb *database, uint64_t image_id,
const char *extractor_kind, uint32_t extractor_version,
const unsigned char parameter_fingerprint[32],
Lardon3DProjectDbFeatureSet *feature_set);
Lardon3DProjectDbResult
lardon3d_project_db_load_feature_set(Lardon3DProjectDb *database, uint64_t feature_set_id,
Lardon3DProjectDbFeatureSet *feature_set);
Lardon3DProjectDbResult
lardon3d_project_db_list_feature_sets(Lardon3DProjectDb *database, uint64_t after_feature_set_id,
Lardon3DProjectDbFeatureSet *feature_sets, size_t capacity,
size_t *count);
#endif

View file

@ -6,63 +6,59 @@
#include <lardon3d/task.h>
typedef struct Lardon3DProjectDb Lardon3DProjectDb;
typedef struct Lardon3DResourceGovernor Lardon3DResourceGovernor;
enum {
LARDON3D_TASK_KIND_REGISTRY_MAX = 64,
LARDON3D_TASK_KIND_REGISTRY_MAX = 64,
};
typedef struct {
Lardon3DTaskCallback callback;
void *userdata;
Lardon3DTaskUserdataDestroy userdata_destroy;
Lardon3DTaskFinishedCallback finished_callback;
void *finished_userdata;
Lardon3DTaskCallback callback;
void *userdata;
Lardon3DTaskUserdataDestroy userdata_destroy;
Lardon3DTaskFinishedCallback finished_callback;
void *finished_userdata;
} Lardon3DTaskKindBinding;
typedef bool (*Lardon3DTaskKindReconstruct)(
const Lardon3DTaskDurableSnapshot *snapshot,
void *context,
Lardon3DTaskKindBinding *binding
);
typedef bool (*Lardon3DTaskKindReconstruct)(const Lardon3DTaskDurableSnapshot *snapshot,
void *context, Lardon3DTaskKindBinding *binding);
typedef struct {
const char *kind;
uint32_t kind_version;
Lardon3DTaskKindReconstruct reconstruct;
const char *kind;
uint32_t kind_version;
Lardon3DTaskKindReconstruct reconstruct;
} Lardon3DTaskKindDescriptor;
typedef struct {
const Lardon3DTaskKindDescriptor *descriptors;
size_t count;
const Lardon3DTaskKindDescriptor *descriptors;
size_t count;
} Lardon3DTaskKindRegistry;
typedef struct {
const char *project_path;
Lardon3DProjectDb *project_db;
Lardon3DResourceGovernor *resource_governor;
} Lardon3DTaskReconstructionContext;
typedef enum {
LARDON3D_TASK_KIND_OK = 0,
LARDON3D_TASK_KIND_INVALID_ARGUMENT,
LARDON3D_TASK_KIND_UNKNOWN,
LARDON3D_TASK_KIND_UNSUPPORTED_VERSION,
LARDON3D_TASK_KIND_RECONSTRUCTION_FAILED,
LARDON3D_TASK_KIND_RESTORE_FAILED
LARDON3D_TASK_KIND_OK = 0,
LARDON3D_TASK_KIND_INVALID_ARGUMENT,
LARDON3D_TASK_KIND_UNKNOWN,
LARDON3D_TASK_KIND_UNSUPPORTED_VERSION,
LARDON3D_TASK_KIND_RECONSTRUCTION_FAILED,
LARDON3D_TASK_KIND_RESTORE_FAILED
} Lardon3DTaskKindResult;
bool lardon3d_task_kind_registry_init(
Lardon3DTaskKindRegistry *registry,
const Lardon3DTaskKindDescriptor *descriptors,
size_t count
);
Lardon3DTaskKindResult lardon3d_task_kind_registry_lookup(
const Lardon3DTaskKindRegistry *registry,
const char *kind,
uint32_t kind_version,
const Lardon3DTaskKindDescriptor **descriptor
);
bool lardon3d_task_kind_registry_init(Lardon3DTaskKindRegistry *registry,
const Lardon3DTaskKindDescriptor *descriptors, size_t count);
Lardon3DTaskKindResult
lardon3d_task_kind_registry_lookup(const Lardon3DTaskKindRegistry *registry, const char *kind,
uint32_t kind_version,
const Lardon3DTaskKindDescriptor **descriptor);
Lardon3DTaskKindResult lardon3d_task_kind_registry_restore(
const Lardon3DTaskKindRegistry *registry,
const char *kind,
uint32_t kind_version,
const Lardon3DTaskDurableSnapshot *snapshot,
void *context,
Lardon3DTask **task
);
const Lardon3DTaskKindRegistry *registry, const char *kind, uint32_t kind_version,
const Lardon3DTaskDurableSnapshot *snapshot, void *context, Lardon3DTask **task);
const Lardon3DTaskKindRegistry *lardon3d_task_kind_registry_production(void);
#endif

View file

@ -1,6 +1,6 @@
project(
'lardon3d',
'c',
'c', 'cpp',
version: '0.1.0',
license: 'MIT',
default_options: [
@ -17,11 +17,13 @@ add_project_arguments(
'-Wshadow',
language: 'c',
)
add_project_arguments('-std=c++17', '-Wpedantic', '-Wconversion', '-Wshadow', language: 'cpp')
ncursesw = dependency('ncursesw', required: true)
threads = dependency('threads')
sqlite3 = dependency('sqlite3', required: true)
openssl = dependency('openssl', required: true)
opencv = dependency('opencv5', required: true, include_type: 'system', modules: ['opencv_core', 'opencv_imgcodecs', 'opencv_features2d'])
executable(
'lardon3d',
@ -35,6 +37,9 @@ executable(
'src/import_task.c',
'src/image_catalog.c',
'src/image_catalog_persistent.c',
'src/feature_extractor_opencv.cpp',
'src/feature_store.c',
'src/feature_task.c',
'src/image_view.c',
'src/project.c',
'src/project_db.c',
@ -48,7 +53,7 @@ executable(
'src/hardware_profile.c',
],
include_directories: include_directories('include'),
dependencies: [ncursesw, threads, sqlite3, openssl],
dependencies: [ncursesw, threads, sqlite3, openssl, opencv],
)
import_test = executable(
@ -82,6 +87,9 @@ import_task_test = executable(
'src/task_checkpoint.c',
'src/task_kind_registry.c',
'src/task_kinds.c',
'src/feature_task.c',
'src/feature_store.c',
'src/feature_extractor_opencv.cpp',
'src/task_queue.c',
'src/resource_governor.c',
'src/resource_snapshot.c',
@ -95,7 +103,7 @@ import_task_test = executable(
'-DLARDON3D_CHECKPOINT_TESTING',
],
include_directories: include_directories('include'),
dependencies: [threads, sqlite3, openssl],
dependencies: [threads, sqlite3, openssl, opencv],
)
test('import-task', import_task_test, timeout: 30)
@ -136,6 +144,44 @@ persistent_image_catalog_test = executable(
test('persistent-image-catalog', persistent_image_catalog_test, timeout: 60)
feature_store_test = executable(
'test-feature-store',
sources: [
'tests/test_feature_store.c',
'src/app_state.c',
'src/image_catalog_persistent.c',
'src/feature_extractor_opencv.cpp',
'src/feature_store.c',
'src/project_db.c',
'src/task.c',
'src/resource_governor.c',
'src/resource_snapshot.c',
],
c_args: ['-DLARDON3D_PROJECT_DB_TESTING', '-DLARDON3D_FEATURE_STORE_TESTING'],
include_directories: include_directories('include'),
dependencies: [threads, sqlite3, openssl, opencv],
)
test('feature-store', feature_store_test, timeout: 60)
feature_task_test = executable(
'test-feature-task',
sources: [
'tests/test_feature_task.c', 'src/app_state.c', 'src/project.c',
'src/project_db.c', 'src/task.c', 'src/task_checkpoint.c',
'src/task_kind_registry.c', 'src/task_kinds.c', 'src/task_queue.c',
'src/import.c', 'src/import_task.c', 'src/image_catalog.c',
'src/image_catalog_persistent.c', 'src/image_view.c',
'src/feature_task.c', 'src/feature_store.c',
'src/feature_extractor_opencv.cpp', 'src/resource_governor.c',
'src/resource_snapshot.c',
],
c_args: ['-DLARDON3D_PROJECT_DB_TESTING', '-DLARDON3D_FEATURE_TASK_TESTING'],
include_directories: include_directories('include'),
dependencies: [threads, sqlite3, openssl, opencv],
)
test('feature-task', feature_task_test, timeout: 60)
image_view_test = executable(
'test-image-view',
sources: [
@ -206,6 +252,9 @@ project_test = executable(
'src/task_checkpoint.c',
'src/task_kind_registry.c',
'src/task_kinds.c',
'src/feature_task.c',
'src/feature_store.c',
'src/feature_extractor_opencv.cpp',
'src/task_queue.c',
'src/import.c',
'src/import_task.c',
@ -220,7 +269,7 @@ project_test = executable(
'-DLARDON3D_CHECKPOINT_TESTING',
],
include_directories: include_directories('include'),
dependencies: [threads, sqlite3, openssl],
dependencies: [threads, sqlite3, openssl, opencv],
)
test('project', project_test, timeout: 30)

View file

@ -0,0 +1,142 @@
#include <cerrno>
#include <cmath>
#include <cstdlib>
#include <cstring>
#include <limits>
#include <new>
#include <sys/stat.h>
#include <vector>
#include <opencv2/features2d.hpp>
#include <opencv2/imgcodecs.hpp>
#include <openssl/sha.h>
extern "C" {
#include <lardon3d/feature_extractor.h>
}
extern "C" bool
lardon3d_feature_extractor_parameters_valid(const Lardon3DFeatureExtractorParameters *parameters) {
return parameters && parameters->max_features > 0 &&
parameters->max_features <= LARDON3D_FEATURE_MAX_FEATURES &&
parameters->pyramid_levels >= 1 && parameters->pyramid_levels <= 16 &&
parameters->fast_threshold >= 1 && parameters->fast_threshold <= 255;
}
static void put_u32(unsigned char *output, uint32_t value) {
output[0] = static_cast<unsigned char>(value);
output[1] = static_cast<unsigned char>(value >> 8);
output[2] = static_cast<unsigned char>(value >> 16);
output[3] = static_cast<unsigned char>(value >> 24);
}
extern "C" void lardon3d_feature_extractor_parameter_fingerprint(
const Lardon3DFeatureExtractorParameters *parameters, unsigned char fingerprint[32]) {
if (!fingerprint) {
return;
}
std::memset(fingerprint, 0, 32);
if (!lardon3d_feature_extractor_parameters_valid(parameters)) {
return;
}
unsigned char canonical[24] = {'L', '3', 'D', 'O', 'R', 'B', 'P', '1'};
put_u32(canonical + 8, LARDON3D_FEATURE_EXTRACTOR_VERSION);
put_u32(canonical + 12, parameters->max_features);
put_u32(canonical + 16, parameters->pyramid_levels);
put_u32(canonical + 20, parameters->fast_threshold);
(void)SHA256(canonical, sizeof(canonical), fingerprint);
}
extern "C" void lardon3d_extracted_features_destroy(Lardon3DExtractedFeatures *features) {
if (!features) {
return;
}
std::free(features->keypoints);
std::free(features->descriptors);
std::memset(features, 0, sizeof(*features));
}
extern "C" Lardon3DFeatureExtractResult
lardon3d_feature_extract_orb(const char *path, const Lardon3DFeatureExtractorParameters *parameters,
Lardon3DExtractedFeatures *features) {
if (features) {
std::memset(features, 0, sizeof(*features));
}
if (!path || !path[0] || !features || !lardon3d_feature_extractor_parameters_valid(parameters)) {
return LARDON3D_FEATURE_EXTRACT_INVALID_ARGUMENT;
}
struct stat information{};
if (lstat(path, &information) != 0) {
return errno == ENOENT ? LARDON3D_FEATURE_EXTRACT_IMAGE_NOT_FOUND
: LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID;
}
if (!S_ISREG(information.st_mode) || S_ISLNK(information.st_mode)) {
return LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID;
}
try {
cv::Mat image = cv::imread(path, cv::IMREAD_GRAYSCALE);
if (image.empty()) {
return LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID;
}
if (image.cols <= 0 || image.rows <= 0 ||
static_cast<uint64_t>(image.cols) * static_cast<uint64_t>(image.rows) >
LARDON3D_FEATURE_MAX_IMAGE_PIXELS) {
return LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID;
}
cv::Ptr<cv::ORB> extractor =
cv::ORB::create(static_cast<int>(parameters->max_features), 1.2F,
static_cast<int>(parameters->pyramid_levels), 31, 0, 2,
cv::ORB::HARRIS_SCORE, 31, static_cast<int>(parameters->fast_threshold));
std::vector<cv::KeyPoint> points;
cv::Mat descriptors;
extractor->detectAndCompute(image, cv::noArray(), points, descriptors);
if (points.size() > parameters->max_features ||
points.size() > std::numeric_limits<uint32_t>::max() ||
(!points.empty() && (descriptors.rows != static_cast<int>(points.size()) ||
descriptors.cols != LARDON3D_FEATURE_DESCRIPTOR_DIMENSION ||
descriptors.type() != CV_8UC1))) {
return LARDON3D_FEATURE_EXTRACT_ERROR;
}
size_t count = points.size();
Lardon3DFeatureKeypoint *keypoints =
count == 0 ? nullptr
: static_cast<Lardon3DFeatureKeypoint *>(
std::calloc(count, sizeof(Lardon3DFeatureKeypoint)));
unsigned char *bytes = count == 0 ? nullptr
: static_cast<unsigned char *>(std::malloc(
count * LARDON3D_FEATURE_DESCRIPTOR_DIMENSION));
if (count > 0 && (!keypoints || !bytes)) {
std::free(keypoints);
std::free(bytes);
return LARDON3D_FEATURE_EXTRACT_OUT_OF_MEMORY;
}
for (size_t index = 0; index < count; ++index) {
const cv::KeyPoint &point = points[index];
if (!std::isfinite(point.pt.x) || !std::isfinite(point.pt.y) || !std::isfinite(point.size) ||
!std::isfinite(point.angle) || !std::isfinite(point.response) || point.pt.x < 0.0F ||
point.pt.y < 0.0F || point.pt.x >= static_cast<float>(image.cols) ||
point.pt.y >= static_cast<float>(image.rows) || point.size <= 0.0F ||
point.angle < 0.0F || point.angle >= 360.0F) {
std::free(keypoints);
std::free(bytes);
return LARDON3D_FEATURE_EXTRACT_ERROR;
}
keypoints[index] = {point.pt.x, point.pt.y, point.size,
point.angle, point.response, point.octave};
std::memcpy(bytes + index * LARDON3D_FEATURE_DESCRIPTOR_DIMENSION,
descriptors.ptr(static_cast<int>(index)), LARDON3D_FEATURE_DESCRIPTOR_DIMENSION);
}
features->image_width = static_cast<uint32_t>(image.cols);
features->image_height = static_cast<uint32_t>(image.rows);
features->feature_count = static_cast<uint32_t>(count);
features->keypoints = keypoints;
features->descriptors = bytes;
return LARDON3D_FEATURE_EXTRACT_OK;
} catch (const cv::Exception &) {
return LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID;
} catch (const std::bad_alloc &) {
return LARDON3D_FEATURE_EXTRACT_OUT_OF_MEMORY;
} catch (...) {
return LARDON3D_FEATURE_EXTRACT_ERROR;
}
}

575
src/feature_store.c Normal file
View file

@ -0,0 +1,575 @@
#include <errno.h>
#include <fcntl.h>
#include <float.h>
#include <math.h>
#include <openssl/evp.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <unistd.h>
#include <lardon3d/feature_store.h>
static const unsigned char feature_magic[8] = {'L', '3', 'D', 'F', 'E', 'A', 'T', 0};
_Static_assert(sizeof(float) == 4 && FLT_RADIX == 2 && FLT_MANT_DIG == 24 && FLT_MAX_EXP == 128,
"Feature File v1 exige IEEE-754 binary32.");
struct Lardon3DFeatureReader {
int descriptor;
uint64_t keypoint_offset;
uint64_t descriptor_offset;
Lardon3DFeatureFileMetadata metadata;
};
static void put_u32(unsigned char *p, uint32_t v) {
p[0] = (unsigned char)v;
p[1] = (unsigned char)(v >> 8);
p[2] = (unsigned char)(v >> 16);
p[3] = (unsigned char)(v >> 24);
}
static void put_u64(unsigned char *p, uint64_t v) {
for (unsigned int i = 0; i < 8; ++i) {
p[i] = (unsigned char)(v >> (8U * i));
}
}
static uint32_t get_u32(const unsigned char *p) {
return (uint32_t)p[0] | ((uint32_t)p[1] << 8) | ((uint32_t)p[2] << 16) | ((uint32_t)p[3] << 24);
}
static uint64_t get_u64(const unsigned char *p) {
uint64_t v = 0;
for (unsigned int i = 0; i < 8; ++i) {
v |= (uint64_t)p[i] << (8U * i);
}
return v;
}
static void put_float(unsigned char *p, float value) {
uint32_t bits = 0;
memcpy(&bits, &value, sizeof(bits));
put_u32(p, bits);
}
static float get_float(const unsigned char *p) {
uint32_t bits = get_u32(p);
float value = 0;
memcpy(&value, &bits, sizeof(value));
return value;
}
static bool write_all(int fd, const void *data, size_t size) {
const unsigned char *bytes = data;
size_t used = 0;
while (used < size) {
ssize_t n = write(fd, bytes + used, size - used);
if (n < 0 && errno == EINTR) {
continue;
}
if (n <= 0) {
return false;
}
used += (size_t)n;
}
return true;
}
static bool read_exact(int fd, void *data, size_t size, off_t offset) {
unsigned char *bytes = data;
size_t used = 0;
while (used < size) {
ssize_t n = pread(fd, bytes + used, size - used, offset + (off_t)used);
if (n < 0 && errno == EINTR) {
continue;
}
if (n <= 0) {
return false;
}
used += (size_t)n;
}
return true;
}
static bool sha256_fd(int fd, unsigned char output[32]) {
EVP_MD_CTX *context = EVP_MD_CTX_new();
if (!context) {
return false;
}
bool ok = EVP_DigestInit_ex(context, EVP_sha256(), NULL) == 1;
unsigned char buffer[65536];
off_t offset = 0;
while (ok) {
ssize_t n = pread(fd, buffer, sizeof(buffer), offset);
if (n < 0 && errno == EINTR) {
continue;
}
if (n < 0) {
ok = false;
break;
}
if (n == 0) {
break;
}
ok = EVP_DigestUpdate(context, buffer, (size_t)n) == 1;
offset += (off_t)n;
}
unsigned int length = 0;
ok = ok && EVP_DigestFinal_ex(context, output, &length) == 1 && length == 32;
EVP_MD_CTX_free(context);
return ok;
}
static void hex_sha(const unsigned char hash[32], char text[65]) {
static const char digits[] = "0123456789abcdef";
for (size_t i = 0; i < 32; ++i) {
text[2 * i] = digits[hash[i] >> 4];
text[2 * i + 1] = digits[hash[i] & 15];
}
text[64] = '\0';
}
static bool canonical_feature_path(const unsigned char hash[32], const char *path) {
if (!hash || !path) {
return false;
}
char hex[65];
hex_sha(hash, hex);
char expected[LARDON3D_PROJECT_DB_PATH_CAPACITY];
int written = snprintf(expected, sizeof(expected), "assets/features/%.2s/%s", hex, hex);
return written > 0 && (size_t)written < sizeof(expected) && strcmp(path, expected) == 0;
}
static bool join(char output[PATH_MAX], const char *a, const char *b) {
int n = snprintf(output, PATH_MAX, "%s/%s", a, b);
return n > 0 && (size_t)n < PATH_MAX;
}
static bool ensure_directory(const char *path) {
if (mkdir(path, 0755) == 0) {
return true;
}
if (errno != EEXIST) {
return false;
}
struct stat info;
return lstat(path, &info) == 0 && S_ISDIR(info.st_mode) && !S_ISLNK(info.st_mode);
}
static bool sync_directory(const char *path) {
int fd = open(path, O_RDONLY | O_DIRECTORY | O_CLOEXEC);
if (fd < 0) {
return false;
}
bool ok = fsync(fd) == 0;
if (close(fd) != 0) {
ok = false;
}
return ok;
}
static bool sync_feature_asset_directory(const char *project_path,
const Lardon3DProjectDbFeatureSet *feature_set) {
char relative[LARDON3D_PROJECT_DB_PATH_CAPACITY];
int written = snprintf(relative, sizeof(relative), "%s", feature_set->asset.path);
if (written <= 0 || (size_t)written >= sizeof(relative)) {
return false;
}
char *separator = strrchr(relative, '/');
if (!separator) {
return false;
}
*separator = '\0';
char directory[PATH_MAX];
return join(directory, project_path, relative) && sync_directory(directory);
}
static bool valid_features(const Lardon3DExtractedFeatures *features) {
if (!features || features->feature_count > LARDON3D_FEATURE_MAX_FEATURES ||
features->image_width == 0 || features->image_height == 0 ||
(features->feature_count > 0 && (!features->keypoints || !features->descriptors))) {
return false;
}
for (uint32_t i = 0; i < features->feature_count; ++i) {
const Lardon3DFeatureKeypoint *p = &features->keypoints[i];
if (!isfinite(p->x) || !isfinite(p->y) || !isfinite(p->size) || !isfinite(p->angle_degrees) ||
!isfinite(p->response) || p->x < 0 || p->y < 0 || p->x >= (float)features->image_width ||
p->y >= (float)features->image_height || p->size <= 0 || p->angle_degrees < 0 ||
p->angle_degrees >= 360 || p->octave < 0 || p->octave >= 16) {
return false;
}
}
return true;
}
static bool write_feature_file(int fd, const Lardon3DProjectDbImageAsset *image_asset,
const unsigned char fingerprint[32],
const Lardon3DExtractedFeatures *features, uint64_t *file_size) {
uint64_t kp_bytes = (uint64_t)features->feature_count * LARDON3D_FEATURE_KEYPOINT_RECORD_SIZE;
uint64_t descriptor_bytes =
(uint64_t)features->feature_count * LARDON3D_FEATURE_DESCRIPTOR_DIMENSION;
uint64_t descriptor_offset = LARDON3D_FEATURE_FILE_HEADER_SIZE + kp_bytes;
uint64_t total = descriptor_offset + descriptor_bytes;
if (total > LARDON3D_FEATURE_FILE_MAX_SIZE) {
return false;
}
unsigned char header[LARDON3D_FEATURE_FILE_HEADER_SIZE] = {0};
memcpy(header, feature_magic, 8);
put_u32(header + 8, LARDON3D_FEATURE_FILE_VERSION);
put_u32(header + 12, LARDON3D_FEATURE_FILE_HEADER_SIZE);
put_u32(header + 16, features->feature_count);
put_u32(header + 20, LARDON3D_FEATURE_DESCRIPTOR_DIMENSION);
put_u32(header + 24, LARDON3D_FEATURE_DESCRIPTOR_U8);
put_u32(header + 28, LARDON3D_FEATURE_KEYPOINT_RECORD_SIZE);
put_u32(header + 32, features->image_width);
put_u32(header + 36, features->image_height);
put_u64(header + 40, LARDON3D_FEATURE_FILE_HEADER_SIZE);
put_u64(header + 48, descriptor_offset);
put_u64(header + 56, total);
memcpy(header + 64, image_asset->sha256, 32);
memcpy(header + 96, fingerprint, 32);
memcpy(header + 128, LARDON3D_FEATURE_EXTRACTOR_KIND, 4);
put_u32(header + 144, LARDON3D_FEATURE_EXTRACTOR_VERSION);
if (!write_all(fd, header, sizeof(header))) {
return false;
}
unsigned char record[LARDON3D_FEATURE_KEYPOINT_RECORD_SIZE];
for (uint32_t i = 0; i < features->feature_count; ++i) {
const Lardon3DFeatureKeypoint *p = &features->keypoints[i];
put_float(record, p->x);
put_float(record + 4, p->y);
put_float(record + 8, p->size);
put_float(record + 12, p->angle_degrees);
put_float(record + 16, p->response);
put_u32(record + 20, (uint32_t)p->octave);
if (!write_all(fd, record, sizeof(record))) {
return false;
}
}
if (descriptor_bytes > 0 && !write_all(fd, features->descriptors, (size_t)descriptor_bytes)) {
return false;
}
*file_size = total;
return true;
}
static Lardon3DFeatureStoreResult
reuse_existing_feature_set(Lardon3DAppState *state, uint64_t image_id, uint64_t producer_task_id,
const unsigned char fingerprint[32],
Lardon3DProjectDbFeatureSet *feature_set) {
if (feature_set->asset.durability == LARDON3D_DB_FEATURE_ASSET_DURABLE) {
return LARDON3D_FEATURE_STORE_ALREADY_PRESENT;
}
Lardon3DFeatureReader *reader = NULL;
Lardon3DFeatureFileMetadata metadata;
Lardon3DFeatureStoreResult validation =
lardon3d_feature_reader_open(state->project_path, feature_set, &reader, &metadata);
lardon3d_feature_reader_close(reader);
if (validation != LARDON3D_FEATURE_STORE_OK ||
!sync_feature_asset_directory(state->project_path, feature_set)) {
return LARDON3D_FEATURE_STORE_PUBLISHED_NOT_DURABLE;
}
int64_t now = (int64_t)time(NULL);
if (now < 0) {
return LARDON3D_FEATURE_STORE_IO_ERROR;
}
Lardon3DProjectDbResult promoted = lardon3d_project_db_register_feature_set(
state->project_db, image_id, LARDON3D_FEATURE_EXTRACTOR_KIND,
LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, feature_set->source_image_sha256,
feature_set->feature_count, feature_set->descriptor_type, feature_set->descriptor_dimension,
feature_set->asset.sha256, feature_set->asset.path, feature_set->asset.size_bytes,
LARDON3D_DB_FEATURE_ASSET_DURABLE, producer_task_id, now, feature_set);
return promoted == LARDON3D_PROJECT_DB_OK ? LARDON3D_FEATURE_STORE_ALREADY_PRESENT
: LARDON3D_FEATURE_STORE_DB_ERROR;
}
static Lardon3DFeatureStoreResult register_published_feature_set(
Lardon3DAppState *state, uint64_t image_id, uint64_t producer_task_id,
const Lardon3DProjectDbImageAsset *image_asset, const unsigned char fingerprint[32],
const Lardon3DExtractedFeatures *features, const unsigned char file_hash[32],
const char *relative_path, uint64_t file_size, bool durable,
Lardon3DProjectDbFeatureSet *feature_set) {
int64_t now = (int64_t)time(NULL);
if (now < 0) {
return LARDON3D_FEATURE_STORE_IO_ERROR;
}
Lardon3DProjectDbResult result = lardon3d_project_db_register_feature_set(
state->project_db, image_id, LARDON3D_FEATURE_EXTRACTOR_KIND,
LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, image_asset->sha256, features->feature_count,
LARDON3D_FEATURE_DESCRIPTOR_U8, LARDON3D_FEATURE_DESCRIPTOR_DIMENSION, file_hash,
relative_path, file_size,
durable ? LARDON3D_DB_FEATURE_ASSET_DURABLE : LARDON3D_DB_FEATURE_ASSET_PUBLISHED_NOT_DURABLE,
producer_task_id, now, feature_set);
if (result != LARDON3D_PROJECT_DB_OK) {
return result == LARDON3D_PROJECT_DB_BUSY ? LARDON3D_FEATURE_STORE_DB_BUSY
: LARDON3D_FEATURE_STORE_DB_ERROR;
}
return durable ? LARDON3D_FEATURE_STORE_OK : LARDON3D_FEATURE_STORE_PUBLISHED_NOT_DURABLE;
}
Lardon3DFeatureStoreResult lardon3d_feature_store_publish(
Lardon3DAppState *state, uint64_t image_id, uint64_t producer_task_id,
const Lardon3DFeatureExtractorParameters *parameters, const Lardon3DExtractedFeatures *features,
Lardon3DProjectDbFeatureSet *feature_set) {
if (feature_set) {
memset(feature_set, 0, sizeof(*feature_set));
}
if (!state || !state->project_loaded || !state->project_db || !feature_set || image_id == 0 ||
!lardon3d_feature_extractor_parameters_valid(parameters) || !valid_features(features)) {
return LARDON3D_FEATURE_STORE_INVALID_ARGUMENT;
}
unsigned char fingerprint[32];
lardon3d_feature_extractor_parameter_fingerprint(parameters, fingerprint);
Lardon3DProjectDbResult found = lardon3d_project_db_find_feature_set(
state->project_db, image_id, LARDON3D_FEATURE_EXTRACTOR_KIND,
LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, feature_set);
if (found == LARDON3D_PROJECT_DB_OK) {
return reuse_existing_feature_set(state, image_id, producer_task_id, fingerprint, feature_set);
}
if (found != LARDON3D_PROJECT_DB_NOT_FOUND) {
return found == LARDON3D_PROJECT_DB_BUSY ? LARDON3D_FEATURE_STORE_DB_BUSY
: LARDON3D_FEATURE_STORE_DB_ERROR;
}
Lardon3DProjectDbImage image;
Lardon3DProjectDbImageAsset image_asset;
if (lardon3d_project_db_load_image(state->project_db, image_id, &image, &image_asset) !=
LARDON3D_PROJECT_DB_OK) {
return LARDON3D_FEATURE_STORE_NOT_FOUND;
}
char assets[PATH_MAX], features_dir[PATH_MAX];
if (!join(assets, state->project_path, "assets") || !ensure_directory(assets) ||
!join(features_dir, assets, "features") || !ensure_directory(features_dir)) {
return LARDON3D_FEATURE_STORE_IO_ERROR;
}
char temporary[PATH_MAX];
int n = snprintf(temporary, sizeof(temporary), "%s/.feature-XXXXXX", features_dir);
if (n <= 0 || (size_t)n >= sizeof(temporary)) {
return LARDON3D_FEATURE_STORE_IO_ERROR;
}
int fd = mkstemp(temporary);
if (fd < 0) {
return LARDON3D_FEATURE_STORE_IO_ERROR;
}
uint64_t file_size = 0;
bool written =
write_feature_file(fd, &image_asset, fingerprint, features, &file_size) && fsync(fd) == 0;
unsigned char file_hash[32];
if (written) {
written = sha256_fd(fd, file_hash);
}
if (close(fd) != 0) {
written = false;
}
if (!written) {
unlink(temporary);
return LARDON3D_FEATURE_STORE_IO_ERROR;
}
char hex[65];
hex_sha(file_hash, hex);
char prefix[PATH_MAX];
n = snprintf(prefix, sizeof(prefix), "%s/%.2s", features_dir, hex);
if (n <= 0 || (size_t)n >= sizeof(prefix) || !ensure_directory(prefix)) {
unlink(temporary);
return LARDON3D_FEATURE_STORE_IO_ERROR;
}
char relative[LARDON3D_PROJECT_DB_PATH_CAPACITY];
n = snprintf(relative, sizeof(relative), "assets/features/%.2s/%s", hex, hex);
char final[PATH_MAX];
if (n <= 0 || (size_t)n >= sizeof(relative) || !join(final, state->project_path, relative)) {
unlink(temporary);
return LARDON3D_FEATURE_STORE_IO_ERROR;
}
if (link(temporary, final) != 0) {
if (errno != EEXIST) {
unlink(temporary);
return LARDON3D_FEATURE_STORE_IO_ERROR;
}
int existing = open(final, O_RDONLY | O_NOFOLLOW | O_CLOEXEC);
struct stat info;
unsigned char existing_hash[32];
bool same = existing >= 0 && fstat(existing, &info) == 0 && S_ISREG(info.st_mode) &&
info.st_size >= 0 && (uint64_t)info.st_size == file_size &&
sha256_fd(existing, existing_hash) && memcmp(existing_hash, file_hash, 32) == 0;
if (existing >= 0) {
(void)close(existing);
}
if (!same) {
unlink(temporary);
return LARDON3D_FEATURE_STORE_CORRUPT;
}
}
unlink(temporary);
bool durable = sync_directory(prefix);
#ifdef LARDON3D_FEATURE_STORE_TESTING
const char *fail_sync = getenv("LARDON3D_TEST_FEATURE_FAIL_DIRECTORY_SYNC");
if (fail_sync && strcmp(fail_sync, "1") == 0) {
durable = false;
}
#endif
return register_published_feature_set(state, image_id, producer_task_id, &image_asset,
fingerprint, features, file_hash, relative, file_size,
durable, feature_set);
}
static Lardon3DFeatureStoreResult validate_header(const unsigned char h[160], uint64_t actual,
Lardon3DFeatureFileMetadata *m, uint64_t *ko,
uint64_t *doff) {
if (memcmp(h, feature_magic, 8) != 0) {
return LARDON3D_FEATURE_STORE_INVALID;
}
uint32_t version = get_u32(h + 8);
if (version != 1) {
return version > 1 ? LARDON3D_FEATURE_STORE_UNSUPPORTED_VERSION
: LARDON3D_FEATURE_STORE_INVALID;
}
uint32_t count = get_u32(h + 16), dim = get_u32(h + 20), type = get_u32(h + 24),
record = get_u32(h + 28), width = get_u32(h + 32), height = get_u32(h + 36);
uint64_t key = get_u64(h + 40), desc = get_u64(h + 48), declared = get_u64(h + 56);
uint64_t kb = (uint64_t)count * record, db = (uint64_t)count * dim;
if (get_u32(h + 12) != 160 || count > LARDON3D_FEATURE_MAX_FEATURES || dim != 32 ||
type != LARDON3D_FEATURE_DESCRIPTOR_U8 || record != 24 || width == 0 || height == 0 ||
key != 160 || desc != key + kb || declared != desc + db || declared != actual ||
actual > LARDON3D_FEATURE_FILE_MAX_SIZE || memcmp(h + 128, "orb\0", 4) != 0 ||
get_u32(h + 144) != 1) {
return LARDON3D_FEATURE_STORE_CORRUPT;
}
for (size_t i = 132; i < 144; ++i) {
if (h[i] != 0) {
return LARDON3D_FEATURE_STORE_CORRUPT;
}
}
for (size_t i = 148; i < 160; ++i) {
if (h[i] != 0) {
return LARDON3D_FEATURE_STORE_CORRUPT;
}
}
*m = (Lardon3DFeatureFileMetadata){.format_version = version,
.extractor_version = get_u32(h + 144),
.feature_count = count,
.descriptor_dimension = dim,
.descriptor_type = (Lardon3DFeatureDescriptorType)type,
.image_width = width,
.image_height = height};
memcpy(m->source_image_sha256, h + 64, 32);
memcpy(m->parameter_fingerprint, h + 96, 32);
memcpy(m->extractor_kind, h + 128, 4);
*ko = key;
*doff = desc;
return LARDON3D_FEATURE_STORE_OK;
}
Lardon3DFeatureStoreResult lardon3d_feature_reader_open(const char *project_path,
const Lardon3DProjectDbFeatureSet *set,
Lardon3DFeatureReader **out,
Lardon3DFeatureFileMetadata *metadata) {
if (out) {
*out = NULL;
}
if (!project_path || !set || !out || !metadata ||
!canonical_feature_path(set->asset.sha256, set->asset.path)) {
return LARDON3D_FEATURE_STORE_INVALID_ARGUMENT;
}
char path[PATH_MAX];
if (!join(path, project_path, set->asset.path)) {
return LARDON3D_FEATURE_STORE_INVALID_ARGUMENT;
}
int fd = open(path, O_RDONLY | O_NOFOLLOW | O_CLOEXEC);
if (fd < 0) {
return errno == ENOENT ? LARDON3D_FEATURE_STORE_NOT_FOUND : LARDON3D_FEATURE_STORE_IO_ERROR;
}
struct stat st;
if (fstat(fd, &st) != 0 || !S_ISREG(st.st_mode) || st.st_size < 160 ||
(uint64_t)st.st_size > LARDON3D_FEATURE_FILE_MAX_SIZE ||
(uint64_t)st.st_size != set->asset.size_bytes) {
close(fd);
return LARDON3D_FEATURE_STORE_CORRUPT;
}
unsigned char header[160];
if (!read_exact(fd, header, sizeof(header), 0)) {
close(fd);
return LARDON3D_FEATURE_STORE_CORRUPT;
}
Lardon3DFeatureReader *reader = calloc(1, sizeof(*reader));
if (!reader) {
close(fd);
return LARDON3D_FEATURE_STORE_IO_ERROR;
}
Lardon3DFeatureStoreResult result =
validate_header(header, (uint64_t)st.st_size, &reader->metadata, &reader->keypoint_offset,
&reader->descriptor_offset);
if (result != LARDON3D_FEATURE_STORE_OK) {
close(fd);
free(reader);
return result;
}
unsigned char hash[32];
if (!sha256_fd(fd, hash) || memcmp(hash, set->asset.sha256, 32) != 0) {
close(fd);
free(reader);
return LARDON3D_FEATURE_STORE_CORRUPT;
}
if (reader->metadata.feature_count != set->feature_count ||
strcmp(reader->metadata.extractor_kind, set->extractor_kind) != 0 ||
reader->metadata.extractor_version != set->extractor_version ||
reader->metadata.descriptor_dimension != set->descriptor_dimension ||
reader->metadata.descriptor_type != (Lardon3DFeatureDescriptorType)set->descriptor_type ||
memcmp(reader->metadata.source_image_sha256, set->source_image_sha256, 32) != 0 ||
memcmp(reader->metadata.parameter_fingerprint, set->parameter_fingerprint, 32) != 0) {
close(fd);
free(reader);
return LARDON3D_FEATURE_STORE_CORRUPT;
}
reader->descriptor = fd;
*metadata = reader->metadata;
*out = reader;
return LARDON3D_FEATURE_STORE_OK;
}
void lardon3d_feature_reader_close(Lardon3DFeatureReader *r) {
if (!r) {
return;
}
(void)close(r->descriptor);
free(r);
}
Lardon3DFeatureStoreResult lardon3d_feature_reader_keypoints(Lardon3DFeatureReader *r,
uint32_t start,
Lardon3DFeatureKeypoint *out,
size_t capacity) {
if (!r || (!out && capacity > 0) || capacity > LARDON3D_FEATURE_READER_RANGE_MAX ||
start > r->metadata.feature_count || capacity > (size_t)(r->metadata.feature_count - start)) {
return LARDON3D_FEATURE_STORE_INVALID_ARGUMENT;
}
unsigned char record[24];
for (size_t i = 0; i < capacity; ++i) {
uint64_t index = (uint64_t)start + i;
if (!read_exact(r->descriptor, record, 24, (off_t)(r->keypoint_offset + index * 24))) {
return LARDON3D_FEATURE_STORE_IO_ERROR;
}
out[i] = (Lardon3DFeatureKeypoint){get_float(record), get_float(record + 4),
get_float(record + 8), get_float(record + 12),
get_float(record + 16), (int32_t)get_u32(record + 20)};
if (!isfinite(out[i].x) || !isfinite(out[i].y) || !isfinite(out[i].size) ||
!isfinite(out[i].angle_degrees) || !isfinite(out[i].response)) {
return LARDON3D_FEATURE_STORE_CORRUPT;
}
}
return LARDON3D_FEATURE_STORE_OK;
}
Lardon3DFeatureStoreResult lardon3d_feature_reader_descriptors(Lardon3DFeatureReader *r,
uint32_t start, unsigned char *out,
size_t capacity, size_t bytes) {
if (!r || (!out && capacity > 0) || capacity > LARDON3D_FEATURE_READER_RANGE_MAX ||
start > r->metadata.feature_count || capacity > (size_t)(r->metadata.feature_count - start) ||
capacity > SIZE_MAX / r->metadata.descriptor_dimension ||
bytes < capacity * r->metadata.descriptor_dimension) {
return LARDON3D_FEATURE_STORE_INVALID_ARGUMENT;
}
size_t amount = capacity * r->metadata.descriptor_dimension;
if (amount && !read_exact(r->descriptor, out, amount,
(off_t)(r->descriptor_offset +
(uint64_t)start * r->metadata.descriptor_dimension))) {
return LARDON3D_FEATURE_STORE_IO_ERROR;
}
return LARDON3D_FEATURE_STORE_OK;
}

325
src/feature_task.c Normal file
View file

@ -0,0 +1,325 @@
#include <errno.h>
#include <fcntl.h>
#include <openssl/evp.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
#include <lardon3d/feature_store.h>
#include <lardon3d/feature_task.h>
#include <lardon3d/project.h>
#include <lardon3d/task_queue.h>
typedef struct {
char project_path[PATH_MAX];
Lardon3DProjectDb *database;
Lardon3DResourceGovernor *governor;
Lardon3DProjectDbFeatureExtractTask parameters;
} Lardon3DFeatureTaskContext;
static void destroy_context(void *userdata) { free(userdata); }
static void runtime_state(const Lardon3DFeatureTaskContext *context, Lardon3DAppState *state) {
lardon3d_app_state_init(state);
state->project_loaded = true;
state->project_db = context->database;
state->resource_governor = context->governor;
(void)snprintf(state->project_path, sizeof(state->project_path), "%s", context->project_path);
}
static bool file_hash(const char *path, unsigned char output[32]) {
int fd = open(path, O_RDONLY | O_NOFOLLOW | O_CLOEXEC);
if (fd < 0) {
return false;
}
struct stat st;
if (fstat(fd, &st) != 0 || !S_ISREG(st.st_mode)) {
close(fd);
return false;
}
EVP_MD_CTX *ctx = EVP_MD_CTX_new();
bool ok = ctx && EVP_DigestInit_ex(ctx, EVP_sha256(), NULL) == 1;
unsigned char buffer[65536];
while (ok) {
ssize_t n = read(fd, buffer, sizeof(buffer));
if (n < 0 && errno == EINTR) {
continue;
}
if (n < 0) {
ok = false;
break;
}
if (n == 0) {
break;
}
ok = EVP_DigestUpdate(ctx, buffer, (size_t)n) == 1;
}
unsigned int size = 0;
ok = ok && EVP_DigestFinal_ex(ctx, output, &size) == 1 && size == 32;
if (ctx) {
EVP_MD_CTX_free(ctx);
}
if (close(fd) != 0) {
ok = false;
}
return ok;
}
static uint64_t elapsed_ns(struct timespec a, struct timespec b) {
uint64_t s = b.tv_sec >= a.tv_sec ? (uint64_t)(b.tv_sec - a.tv_sec) : 0;
long n = b.tv_nsec - a.tv_nsec;
if (n < 0 && s) {
--s;
n += 1000000000L;
}
return s <= UINT64_MAX / 1000000000ULL ? s * 1000000000ULL + (uint64_t)n : UINT64_MAX;
}
static void finished(const Lardon3DTask *task, void *userdata) {
#ifdef LARDON3D_FEATURE_TASK_TESTING
const char *skip = getenv("LARDON3D_TEST_FEATURE_SKIP_FINISHED_CHECKPOINT");
if (skip && strcmp(skip, "1") == 0) {
return;
}
#endif
Lardon3DFeatureTaskContext *context = userdata;
Lardon3DAppState state;
runtime_state(context, &state);
(void)lardon3d_project_checkpoint_feature_extract_task(&state, task, &context->parameters);
}
static Lardon3DProjectDbResult find_valid_feature_set(const Lardon3DFeatureTaskContext *context,
const unsigned char fingerprint[32]) {
Lardon3DProjectDbFeatureSet existing;
Lardon3DProjectDbResult found = lardon3d_project_db_find_feature_set(
context->database, context->parameters.image_id, LARDON3D_FEATURE_EXTRACTOR_KIND,
LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, &existing);
if (found != LARDON3D_PROJECT_DB_OK) {
return found;
}
Lardon3DFeatureReader *reader = NULL;
Lardon3DFeatureFileMetadata metadata;
Lardon3DFeatureStoreResult valid =
lardon3d_feature_reader_open(context->project_path, &existing, &reader, &metadata);
lardon3d_feature_reader_close(reader);
return valid == LARDON3D_FEATURE_STORE_OK ? LARDON3D_PROJECT_DB_OK : LARDON3D_PROJECT_DB_CORRUPT;
}
static bool load_validated_source(const Lardon3DFeatureTaskContext *context, char path[PATH_MAX]) {
Lardon3DProjectDbImage image;
Lardon3DProjectDbImageAsset asset;
if (lardon3d_project_db_load_image(context->database, context->parameters.image_id, &image,
&asset) != LARDON3D_PROJECT_DB_OK) {
return false;
}
int written = snprintf(path, PATH_MAX, "%s/%s", context->project_path, asset.path);
if (written <= 0 || (size_t)written >= PATH_MAX) {
return false;
}
unsigned char actual[32];
return file_hash(path, actual) && memcmp(actual, asset.sha256, 32) == 0;
}
static bool run(Lardon3DTask *task, void *userdata) {
Lardon3DFeatureTaskContext *context = userdata;
if (!lardon3d_task_checkpoint(task)) {
return false;
}
struct timespec begin;
(void)clock_gettime(CLOCK_MONOTONIC, &begin);
unsigned char fingerprint[32];
lardon3d_feature_extractor_parameter_fingerprint(
&(Lardon3DFeatureExtractorParameters){context->parameters.max_features,
context->parameters.pyramid_levels,
context->parameters.fast_threshold},
fingerprint);
Lardon3DProjectDbResult found = find_valid_feature_set(context, fingerprint);
if (found == LARDON3D_PROJECT_DB_OK) {
return lardon3d_task_set_progress(task, 100, "Features déjà présentes.");
}
if (found == LARDON3D_PROJECT_DB_CORRUPT) {
return lardon3d_task_fail(task, "Feature Set READY absent ou corrompu.");
}
if (found != LARDON3D_PROJECT_DB_NOT_FOUND) {
return lardon3d_task_fail(task, "Recherche Feature Store impossible.");
}
char path[PATH_MAX];
if (!load_validated_source(context, path)) {
return lardon3d_task_fail(task, "Asset image absent ou corrompu.");
}
Lardon3DFeatureExtractorParameters parameters = {context->parameters.max_features,
context->parameters.pyramid_levels,
context->parameters.fast_threshold};
Lardon3DExtractedFeatures features;
Lardon3DFeatureExtractResult extracted =
lardon3d_feature_extract_orb(path, &parameters, &features);
if (extracted != LARDON3D_FEATURE_EXTRACT_OK) {
return lardon3d_task_fail(task, "Décodage ou extraction ORB impossible.");
}
if (!lardon3d_task_checkpoint(task)) {
lardon3d_extracted_features_destroy(&features);
return false;
}
#ifdef LARDON3D_FEATURE_TASK_TESTING
const char *pause = getenv("LARDON3D_TEST_FEATURE_PAUSE_BEFORE_PUBLISH");
if (pause && strcmp(pause, "1") == 0) {
(void)lardon3d_task_pause(task);
if (!lardon3d_task_checkpoint(task)) {
lardon3d_extracted_features_destroy(&features);
return false;
}
}
#endif
Lardon3DAppState state;
runtime_state(context, &state);
Lardon3DProjectDbFeatureSet set;
Lardon3DFeatureStoreResult published = lardon3d_feature_store_publish(
&state, context->parameters.image_id, lardon3d_task_id(task), &parameters, &features, &set);
lardon3d_extracted_features_destroy(&features);
if (published != LARDON3D_FEATURE_STORE_OK &&
published != LARDON3D_FEATURE_STORE_ALREADY_PRESENT &&
published != LARDON3D_FEATURE_STORE_PUBLISHED_NOT_DURABLE) {
return lardon3d_task_fail(task, "Publication Feature Store impossible.");
}
struct timespec end;
(void)clock_gettime(CLOCK_MONOTONIC, &end);
(void)lardon3d_resource_governor_record_batch(context->governor, LARDON3D_RESOURCE_TASK_CPU, 1,
elapsed_ns(begin, end), 0);
return lardon3d_task_set_progress(task, 100, "Feature Set publié.");
}
static Lardon3DFeatureTaskContext *
make_context(const Lardon3DTaskReconstructionContext *runtime,
const Lardon3DProjectDbFeatureExtractTask *parameters) {
if (!runtime || !runtime->project_path || !runtime->project_db || !runtime->resource_governor ||
!parameters) {
return NULL;
}
Lardon3DFeatureTaskContext *context = calloc(1, sizeof(*context));
if (!context) {
return NULL;
}
int n =
snprintf(context->project_path, sizeof(context->project_path), "%s", runtime->project_path);
if (n <= 0 || (size_t)n >= sizeof(context->project_path)) {
free(context);
return NULL;
}
context->database = runtime->project_db;
context->governor = runtime->resource_governor;
context->parameters = *parameters;
return context;
}
bool lardon3d_feature_extract_reconstruct(const Lardon3DTaskDurableSnapshot *snapshot,
void *userdata, Lardon3DTaskKindBinding *binding) {
Lardon3DTaskReconstructionContext *runtime = userdata;
if (!snapshot || !runtime || !binding) {
return false;
}
Lardon3DProjectDbFeatureExtractTask parameters;
if (lardon3d_project_db_load_feature_extract_task(runtime->project_db, snapshot->id,
&parameters) != LARDON3D_PROJECT_DB_OK ||
strcmp(parameters.extractor_kind, LARDON3D_FEATURE_EXTRACTOR_KIND) != 0 ||
parameters.extractor_version != 1) {
return false;
}
unsigned char expected[32];
lardon3d_feature_extractor_parameter_fingerprint(
&(Lardon3DFeatureExtractorParameters){parameters.max_features, parameters.pyramid_levels,
parameters.fast_threshold},
expected);
if (memcmp(expected, parameters.parameter_fingerprint, 32) != 0) {
return false;
}
Lardon3DFeatureTaskContext *context = make_context(runtime, &parameters);
if (!context) {
return false;
}
*binding = (Lardon3DTaskKindBinding){.callback = run,
.userdata = context,
.userdata_destroy = destroy_context,
.finished_callback = finished,
.finished_userdata = context};
return true;
}
Lardon3DTask *
lardon3d_project_create_feature_extract_task(Lardon3DAppState *state, uint64_t image_id,
const Lardon3DFeatureExtractorParameters *parameters,
uint64_t *task_id) {
if (task_id) {
*task_id = 0;
}
if (!state || !state->project_loaded || !state->project_db || !state->resource_governor ||
!parameters || !task_id || !lardon3d_feature_extractor_parameters_valid(parameters)) {
return NULL;
}
Lardon3DProjectDbImage image;
Lardon3DProjectDbImageAsset asset;
if (lardon3d_project_db_load_image(state->project_db, image_id, &image, &asset) !=
LARDON3D_PROJECT_DB_OK) {
return NULL;
}
uint64_t id;
if (lardon3d_project_db_allocate_task_id(state->project_db, &id) != LARDON3D_PROJECT_DB_OK) {
return NULL;
}
Lardon3DProjectDbFeatureExtractTask durable = {.task_id = id,
.image_id = image_id,
.extractor_version = 1,
.max_features = parameters->max_features,
.pyramid_levels = parameters->pyramid_levels,
.fast_threshold = parameters->fast_threshold};
snprintf(durable.extractor_kind, sizeof(durable.extractor_kind), "%s",
LARDON3D_FEATURE_EXTRACTOR_KIND);
lardon3d_feature_extractor_parameter_fingerprint(parameters, durable.parameter_fingerprint);
Lardon3DTaskReconstructionContext runtime = {state->project_path, state->project_db,
state->resource_governor};
Lardon3DFeatureTaskContext *context = make_context(&runtime, &durable);
if (!context) {
return NULL;
}
Lardon3DResourceEstimate estimate = {.memory_fixed_bytes = 64ULL * 1024 * 1024,
.memory_bytes_per_item = 512ULL * 1024 * 1024,
.minimum_batch_size = 1,
.maximum_batch_size = 1,
.desired_cpu_threads = 1,
.desired_io_slots = 1,
.task_class = LARDON3D_RESOURCE_TASK_CPU};
Lardon3DTask *task = lardon3d_task_create_typed("Extraction de features", &estimate,
LARDON3D_FEATURE_EXTRACT_TASK_KIND, 1, run,
context, destroy_context);
if (!task || !lardon3d_task_assign_id(task, id) ||
!lardon3d_task_set_finished_callback(task, finished, context) ||
lardon3d_project_checkpoint_feature_extract_task(state, task, &durable) !=
LARDON3D_PROJECT_TASK_CHECKPOINT_OK) {
lardon3d_task_destroy(task);
return NULL;
}
*task_id = id;
return task;
}
bool lardon3d_project_enqueue_feature_extract(Lardon3DAppState *state, uint64_t image_id,
const Lardon3DFeatureExtractorParameters *parameters,
uint64_t *task_id) {
if (!state || !state->task_queue) {
return false;
}
Lardon3DTask *task =
lardon3d_project_create_feature_extract_task(state, image_id, parameters, task_id);
if (!task) {
return false;
}
if (!lardon3d_task_queue_add(state->task_queue, task, NULL)) {
lardon3d_task_destroy(task);
return false;
}
return true;
}

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

View file

@ -1,17 +1,22 @@
#include <lardon3d/feature_task.h>
#include <lardon3d/import_task.h>
#include <lardon3d/task_kind_registry.h>
const Lardon3DTaskKindRegistry *
lardon3d_task_kind_registry_production(void)
{
static const Lardon3DTaskKindDescriptor descriptors[] = {{
.kind = LARDON3D_IMAGE_IMPORT_TASK_KIND,
.kind_version = LARDON3D_IMAGE_IMPORT_TASK_KIND_VERSION,
.reconstruct = lardon3d_image_import_reconstruct,
}};
static const Lardon3DTaskKindRegistry registry = {
.descriptors = descriptors,
.count = sizeof(descriptors) / sizeof(descriptors[0]),
};
return &registry;
const Lardon3DTaskKindRegistry *lardon3d_task_kind_registry_production(void) {
static const Lardon3DTaskKindDescriptor descriptors[] = {
{
.kind = LARDON3D_IMAGE_IMPORT_TASK_KIND,
.kind_version = LARDON3D_IMAGE_IMPORT_TASK_KIND_VERSION,
.reconstruct = lardon3d_image_import_reconstruct,
},
{
.kind = LARDON3D_FEATURE_EXTRACT_TASK_KIND,
.kind_version = LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION,
.reconstruct = lardon3d_feature_extract_reconstruct,
}};
static const Lardon3DTaskKindRegistry registry = {
.descriptors = descriptors,
.count = sizeof(descriptors) / sizeof(descriptors[0]),
};
return &registry;
}

432
tests/test_feature_store.c Normal file
View file

@ -0,0 +1,432 @@
#include <dirent.h>
#include <errno.h>
#include <fcntl.h>
#include <math.h>
#include <pthread.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <unistd.h>
#include <lardon3d/feature_store.h>
#include <lardon3d/image_catalog.h>
Lardon3DProjectDbResult lardon3d_project_db_test_delete_feature_identity(Lardon3DProjectDb *,
uint64_t, uint64_t);
typedef struct {
Lardon3DAppState *state;
uint64_t image_id;
const Lardon3DFeatureExtractorParameters *parameters;
const Lardon3DExtractedFeatures *features;
Lardon3DFeatureStoreResult result;
Lardon3DProjectDbFeatureSet feature_set;
} PublishThreadContext;
static void *publish_thread(void *userdata) {
PublishThreadContext *context = userdata;
context->result =
lardon3d_feature_store_publish(context->state, context->image_id, 0, context->parameters,
context->features, &context->feature_set);
return NULL;
}
#define CHECK(x) \
do { \
if (!(x)) { \
fprintf(stderr, "Échec ligne %d: %s\n", __LINE__, #x); \
return false; \
} \
} while (0)
static bool join_path(char out[PATH_MAX], const char *a, const char *b) {
int n = snprintf(out, PATH_MAX, "%s/%s", a, b);
return n > 0 && (size_t)n < PATH_MAX;
}
static bool remove_tree(const char *path) {
struct stat st;
if (lstat(path, &st) != 0) {
return errno == ENOENT;
}
if (!S_ISDIR(st.st_mode)) {
return unlink(path) == 0;
}
DIR *d = opendir(path);
if (!d) {
return false;
}
bool ok = true;
for (struct dirent *e = readdir(d); e; e = readdir(d)) {
if (!strcmp(e->d_name, ".") || !strcmp(e->d_name, "..")) {
continue;
}
char child[PATH_MAX];
if (!join_path(child, path, e->d_name) || !remove_tree(child)) {
ok = false;
}
}
if (closedir(d) != 0 || rmdir(path) != 0) {
ok = false;
}
return ok;
}
static size_t count_files(const char *path) {
DIR *d = opendir(path);
if (!d) {
return 0;
}
size_t count = 0;
for (struct dirent *e = readdir(d); e; e = readdir(d)) {
if (!strcmp(e->d_name, ".") || !strcmp(e->d_name, "..")) {
continue;
}
char child[PATH_MAX];
struct stat s;
if (join_path(child, path, e->d_name) && lstat(child, &s) == 0) {
if (S_ISDIR(s.st_mode)) {
count += count_files(child);
} else if (S_ISREG(s.st_mode)) {
++count;
}
}
}
closedir(d);
return count;
}
static bool write_pgm(const char *path, bool uniform) {
FILE *f = fopen(path, "wb");
if (!f) {
return false;
}
if (fprintf(f, "P5\n128 128\n255\n") <= 0) {
fclose(f);
return false;
}
for (unsigned int y = 0; y < 128; ++y) {
for (unsigned int x = 0; x < 128; ++x) {
unsigned char v = uniform ? 128U : (unsigned char)((((x / 8U) ^ (y / 8U)) & 1U) ? 240U : 15U);
if (fwrite(&v, 1, 1, f) != 1) {
fclose(f);
return false;
}
}
}
return fclose(f) == 0;
}
static uint64_t read_u64_le(const unsigned char *bytes) {
uint64_t value = 0;
for (unsigned int index = 0; index < 8; ++index) {
value |= (uint64_t)bytes[index] << (8U * index);
}
return value;
}
static bool restore_file(const char *path, const unsigned char *bytes, size_t size) {
int descriptor = open(path, O_WRONLY | O_TRUNC | O_CLOEXEC);
if (descriptor < 0) {
return false;
}
size_t written = 0;
while (written < size) {
ssize_t amount = write(descriptor, bytes + written, size - written);
if (amount <= 0) {
close(descriptor);
return false;
}
written += (size_t)amount;
}
return close(descriptor) == 0;
}
static bool expect_reader_result(const char *root, const Lardon3DProjectDbFeatureSet *feature_set,
Lardon3DFeatureStoreResult expected) {
Lardon3DFeatureReader *reader = NULL;
Lardon3DFeatureFileMetadata metadata;
Lardon3DFeatureStoreResult result =
lardon3d_feature_reader_open(root, feature_set, &reader, &metadata);
lardon3d_feature_reader_close(reader);
return result == expected;
}
static bool run_test(void) {
char root[] = "/tmp/lardon3d-feature-store-XXXXXX";
CHECK(mkdtemp(root));
char dbpath[PATH_MAX], structured[PATH_MAX], uniform[PATH_MAX];
CHECK(join_path(dbpath, root, "project.db") && join_path(structured, root, "structured.pgm") &&
join_path(uniform, root, "uniform.pgm"));
CHECK(write_pgm(structured, false) && write_pgm(uniform, true));
Lardon3DProjectDb *db = NULL;
char error[256];
CHECK(lardon3d_project_db_open(dbpath, &db, error) == LARDON3D_PROJECT_DB_OK);
Lardon3DAppState state;
lardon3d_app_state_init(&state);
state.project_loaded = true;
state.project_db = db;
CHECK(snprintf(state.project_path, sizeof(state.project_path), "%s", root) > 0);
Lardon3DProjectDbScanSet a, b;
CHECK(lardon3d_image_catalog_create_scanset(&state, "A", &a) &&
lardon3d_image_catalog_create_scanset(&state, "B", &b));
Lardon3DProjectDbImage ia, ib, iu;
Lardon3DProjectDbImageAsset aa, ab, au;
CHECK(lardon3d_image_catalog_import_file(&state, a.scanset_id, structured, 0, &ia, &aa) ==
LARDON3D_IMAGE_CATALOG_IMPORTED);
CHECK(lardon3d_image_catalog_import_file(&state, b.scanset_id, structured, 0, &ib, &ab) ==
LARDON3D_IMAGE_CATALOG_IMPORTED &&
aa.asset_id == ab.asset_id);
CHECK(lardon3d_image_catalog_import_file(&state, a.scanset_id, uniform, 0, &iu, &au) ==
LARDON3D_IMAGE_CATALOG_IMPORTED);
Lardon3DFeatureExtractorParameters p = {
.max_features = 512, .pyramid_levels = 4, .fast_threshold = 10};
CHECK(lardon3d_feature_extractor_parameters_valid(&p));
CHECK(!lardon3d_feature_extractor_parameters_valid(
&(Lardon3DFeatureExtractorParameters){0, 4, 10}));
CHECK(!lardon3d_feature_extractor_parameters_valid(
&(Lardon3DFeatureExtractorParameters){8193, 4, 10}));
CHECK(!lardon3d_feature_extractor_parameters_valid(
&(Lardon3DFeatureExtractorParameters){512, 0, 10}));
CHECK(!lardon3d_feature_extractor_parameters_valid(
&(Lardon3DFeatureExtractorParameters){512, 17, 10}));
CHECK(!lardon3d_feature_extractor_parameters_valid(
&(Lardon3DFeatureExtractorParameters){512, 4, 0}));
CHECK(!lardon3d_feature_extractor_parameters_valid(
&(Lardon3DFeatureExtractorParameters){512, 4, 256}));
unsigned char fp[32], fp2[32];
lardon3d_feature_extractor_parameter_fingerprint(&p, fp);
lardon3d_feature_extractor_parameter_fingerprint(&p, fp2);
CHECK(memcmp(fp, fp2, 32) == 0);
Lardon3DFeatureExtractorParameters p2 = p;
p2.max_features = 256;
lardon3d_feature_extractor_parameter_fingerprint(&p2, fp2);
CHECK(memcmp(fp, fp2, 32) != 0);
Lardon3DExtractedFeatures f;
CHECK(lardon3d_feature_extract_orb(structured, &p, &f) == LARDON3D_FEATURE_EXTRACT_OK &&
f.feature_count > 0 && f.feature_count <= 512);
Lardon3DProjectDbFeatureSet sa, same, sb;
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &p, &f, &sa) ==
LARDON3D_FEATURE_STORE_OK);
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &p, &f, &same) ==
LARDON3D_FEATURE_STORE_ALREADY_PRESENT &&
same.feature_set_id == sa.feature_set_id);
CHECK(lardon3d_feature_store_publish(&state, ib.image_id, 0, &p, &f, &sb) ==
LARDON3D_FEATURE_STORE_OK &&
sb.feature_set_id != sa.feature_set_id && sb.feature_asset_id == sa.feature_asset_id);
Lardon3DFeatureReader *reader = NULL;
Lardon3DFeatureFileMetadata metadata;
CHECK(lardon3d_feature_reader_open(root, &sa, &reader, &metadata) == LARDON3D_FEATURE_STORE_OK &&
metadata.feature_count == f.feature_count);
size_t chunk = metadata.feature_count < 8 ? metadata.feature_count : 8;
Lardon3DFeatureKeypoint points[8];
unsigned char descriptors[8 * 32];
CHECK(lardon3d_feature_reader_keypoints(reader, 0, points, chunk) == LARDON3D_FEATURE_STORE_OK);
CHECK(lardon3d_feature_reader_descriptors(reader, 0, descriptors, chunk, sizeof(descriptors)) ==
LARDON3D_FEATURE_STORE_OK);
for (size_t i = 0; i < chunk; ++i) {
CHECK(isfinite(points[i].x) && memcmp(descriptors + i * 32, f.descriptors + i * 32, 32) == 0);
}
CHECK(lardon3d_feature_reader_keypoints(reader, metadata.feature_count, points, 1) ==
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT);
CHECK(lardon3d_feature_reader_descriptors(reader, UINT32_MAX, descriptors, 1,
sizeof(descriptors)) ==
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT);
CHECK(lardon3d_feature_reader_keypoints(reader, 0, points, 257) ==
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT);
CHECK(lardon3d_feature_reader_descriptors(reader, 0, descriptors, 257, sizeof(descriptors)) ==
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT);
lardon3d_feature_reader_close(reader);
const char *invalid_paths[] = {
"/assets/features/00/hash",
"../assets/features/00/hash",
"assets/features/AA/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"assets/features/aa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"assets/features/0/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"assets/features/aa/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
};
for (size_t index = 0; index < sizeof(invalid_paths) / sizeof(invalid_paths[0]); ++index) {
Lardon3DProjectDbFeatureSet invalid_path_set = sa;
CHECK(snprintf(invalid_path_set.asset.path, sizeof(invalid_path_set.asset.path), "%s",
invalid_paths[index]) > 0);
CHECK(lardon3d_feature_reader_open(root, &invalid_path_set, &reader, &metadata) ==
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT);
}
Lardon3DExtractedFeatures invalid_features = f;
Lardon3DFeatureKeypoint invalid_point = f.keypoints[0];
invalid_point.x = NAN;
invalid_features.keypoints = &invalid_point;
invalid_features.feature_count = 1;
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &p2, &invalid_features, &same) ==
LARDON3D_FEATURE_STORE_INVALID_ARGUMENT);
Lardon3DFeatureExtractorParameters concurrent_parameters = {508, 4, 10};
PublishThreadContext concurrent[2] = {
{.state = &state,
.image_id = ia.image_id,
.parameters = &concurrent_parameters,
.features = &f},
{.state = &state,
.image_id = ia.image_id,
.parameters = &concurrent_parameters,
.features = &f},
};
pthread_t publish_threads[2];
CHECK(pthread_create(&publish_threads[0], NULL, publish_thread, &concurrent[0]) == 0 &&
pthread_create(&publish_threads[1], NULL, publish_thread, &concurrent[1]) == 0);
CHECK(pthread_join(publish_threads[0], NULL) == 0 && pthread_join(publish_threads[1], NULL) == 0);
CHECK((concurrent[0].result == LARDON3D_FEATURE_STORE_OK ||
concurrent[0].result == LARDON3D_FEATURE_STORE_ALREADY_PRESENT) &&
(concurrent[1].result == LARDON3D_FEATURE_STORE_OK ||
concurrent[1].result == LARDON3D_FEATURE_STORE_ALREADY_PRESENT) &&
concurrent[0].feature_set.feature_set_id == concurrent[1].feature_set.feature_set_id);
Lardon3DExtractedFeatures uf;
CHECK(lardon3d_feature_extract_orb(uniform, &p, &uf) == LARDON3D_FEATURE_EXTRACT_OK &&
uf.feature_count == 0);
Lardon3DProjectDbFeatureSet us;
CHECK(lardon3d_feature_store_publish(&state, iu.image_id, 0, &p, &uf, &us) ==
LARDON3D_FEATURE_STORE_OK);
CHECK(lardon3d_feature_reader_open(root, &us, &reader, &metadata) == LARDON3D_FEATURE_STORE_OK &&
metadata.feature_count == 0);
lardon3d_feature_reader_close(reader);
Lardon3DFeatureExtractorParameters large_parameters = {8192, 4, 10};
Lardon3DExtractedFeatures large = {
.image_width = 128, .image_height = 128, .feature_count = 8192};
large.keypoints = calloc(large.feature_count, sizeof(*large.keypoints));
large.descriptors = calloc(large.feature_count, 32);
CHECK(large.keypoints && large.descriptors);
for (uint32_t i = 0; i < large.feature_count; ++i) {
large.keypoints[i] =
(Lardon3DFeatureKeypoint){(float)(i % 128), (float)((i / 128) % 64), 1, 0, 1, 0};
memset(large.descriptors + (size_t)i * 32, (int)(i & 255), 32);
}
Lardon3DProjectDbFeatureSet large_set;
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &large_parameters, &large,
&large_set) == LARDON3D_FEATURE_STORE_OK);
CHECK(lardon3d_feature_reader_open(root, &large_set, &reader, &metadata) ==
LARDON3D_FEATURE_STORE_OK &&
metadata.feature_count == 8192);
Lardon3DFeatureKeypoint last;
unsigned char last_descriptor[32];
CHECK(lardon3d_feature_reader_keypoints(reader, 8191, &last, 1) == LARDON3D_FEATURE_STORE_OK &&
lardon3d_feature_reader_descriptors(reader, 8191, last_descriptor, 1,
sizeof(last_descriptor)) == LARDON3D_FEATURE_STORE_OK &&
last_descriptor[0] == 255);
lardon3d_feature_reader_close(reader);
lardon3d_extracted_features_destroy(&large);
char features_dir[PATH_MAX];
CHECK(join_path(features_dir, root, "assets/features"));
size_t before = count_files(features_dir);
Lardon3DFeatureExtractorParameters failed_parameters = {511, 4, 10};
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_FEATURE_REGISTER", "1", 1) == 0);
Lardon3DProjectDbFeatureSet failed_set;
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &failed_parameters, &f,
&failed_set) == LARDON3D_FEATURE_STORE_DB_BUSY);
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_FEATURE_REGISTER") == 0 &&
count_files(features_dir) == before + 1);
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &failed_parameters, &f,
&failed_set) == LARDON3D_FEATURE_STORE_OK);
Lardon3DFeatureExtractorParameters nondurable_parameters = {510, 4, 10};
CHECK(setenv("LARDON3D_TEST_FEATURE_FAIL_DIRECTORY_SYNC", "1", 1) == 0);
Lardon3DProjectDbFeatureSet nondurable;
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &nondurable_parameters, &f,
&nondurable) ==
LARDON3D_FEATURE_STORE_PUBLISHED_NOT_DURABLE &&
nondurable.asset.durability == LARDON3D_DB_FEATURE_ASSET_PUBLISHED_NOT_DURABLE);
CHECK(unsetenv("LARDON3D_TEST_FEATURE_FAIL_DIRECTORY_SYNC") == 0);
Lardon3DProjectDbFeatureSet promoted;
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &nondurable_parameters, &f,
&promoted) == LARDON3D_FEATURE_STORE_ALREADY_PRESENT &&
promoted.feature_set_id == nondurable.feature_set_id &&
promoted.asset.durability == LARDON3D_DB_FEATURE_ASSET_DURABLE);
uint64_t deleted_set = nondurable.feature_set_id, deleted_asset = nondurable.feature_asset_id;
CHECK(lardon3d_project_db_test_delete_feature_identity(db, deleted_set, deleted_asset) ==
LARDON3D_PROJECT_DB_OK);
Lardon3DFeatureExtractorParameters replacement_parameters = {509, 4, 10};
Lardon3DProjectDbFeatureSet replacement;
CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &replacement_parameters, &f,
&replacement) == LARDON3D_FEATURE_STORE_OK &&
replacement.feature_set_id > deleted_set && replacement.feature_asset_id > deleted_asset);
Lardon3DProjectDbFeatureSet page[8];
size_t count = 0;
CHECK(lardon3d_project_db_list_feature_sets(db, 0, page, 1, &count) == LARDON3D_PROJECT_DB_OK &&
count == 1);
CHECK(lardon3d_project_db_list_feature_sets(db, page[0].feature_set_id, page, 8, &count) ==
LARDON3D_PROJECT_DB_OK &&
count == 6);
for (uint32_t index = 0; index < 2000; ++index) {
unsigned char metadata_fingerprint[32] = {0};
metadata_fingerprint[0] = (unsigned char)index;
metadata_fingerprint[1] = (unsigned char)(index >> 8);
metadata_fingerprint[2] = 0xa5;
Lardon3DProjectDbFeatureSet metadata_set;
CHECK(lardon3d_project_db_register_feature_set(
db, ia.image_id, "metadata.fixture", 1, metadata_fingerprint, aa.sha256,
sa.feature_count, sa.descriptor_type, sa.descriptor_dimension, sa.asset.sha256,
sa.asset.path, sa.asset.size_bytes, sa.asset.durability, 0, index,
&metadata_set) == LARDON3D_PROJECT_DB_OK);
}
Lardon3DProjectDbFeatureSet *metadata_page = calloc(256, sizeof(*metadata_page));
CHECK(metadata_page);
const size_t page_sizes[] = {1, 8, 64, 256};
for (size_t index = 0; index < sizeof(page_sizes) / sizeof(page_sizes[0]); ++index) {
CHECK(lardon3d_project_db_list_feature_sets(db, 0, metadata_page, page_sizes[index], &count) ==
LARDON3D_PROJECT_DB_OK &&
count == page_sizes[index]);
}
CHECK(lardon3d_project_db_list_feature_sets(db, 0, metadata_page, 257, &count) ==
LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
free(metadata_page);
char asset_path[PATH_MAX];
CHECK(join_path(asset_path, root, sa.asset.path));
int fd = open(asset_path, O_RDONLY | O_CLOEXEC);
CHECK(fd >= 0);
struct stat asset_info;
CHECK(fstat(fd, &asset_info) == 0 && asset_info.st_size >= 160);
size_t original_size = (size_t)asset_info.st_size;
unsigned char *original = malloc(original_size);
CHECK(original && read(fd, original, original_size) == (ssize_t)original_size && close(fd) == 0);
uint64_t descriptor_offset = read_u64_le(original + 48);
const off_t truncated_sizes[] = {
100,
(off_t)descriptor_offset - 1,
(off_t)descriptor_offset,
(off_t)original_size - 1,
};
for (size_t index = 0; index < sizeof(truncated_sizes) / sizeof(truncated_sizes[0]); ++index) {
CHECK(truncate(asset_path, truncated_sizes[index]) == 0 &&
expect_reader_result(root, &sa, LARDON3D_FEATURE_STORE_CORRUPT) &&
restore_file(asset_path, original, original_size));
}
const size_t corrupt_offsets[] = {12, 16, 20, 24, 28, 40, 48, 56, 148, 144, 128};
for (size_t index = 0; index < sizeof(corrupt_offsets) / sizeof(corrupt_offsets[0]); ++index) {
unsigned char mutated = (unsigned char)(original[corrupt_offsets[index]] ^ 0x5aU);
fd = open(asset_path, O_WRONLY | O_CLOEXEC);
CHECK(fd >= 0 && pwrite(fd, &mutated, 1, (off_t)corrupt_offsets[index]) == 1 &&
close(fd) == 0 && expect_reader_result(root, &sa, LARDON3D_FEATURE_STORE_CORRUPT) &&
restore_file(asset_path, original, original_size));
}
unsigned char future_version[4] = {2, 0, 0, 0};
fd = open(asset_path, O_WRONLY | O_CLOEXEC);
CHECK(fd >= 0 && pwrite(fd, future_version, sizeof(future_version), 8) == 4 && close(fd) == 0 &&
expect_reader_result(root, &sa, LARDON3D_FEATURE_STORE_UNSUPPORTED_VERSION) &&
restore_file(asset_path, original, original_size));
fd = open(asset_path, O_WRONLY | O_TRUNC | O_CLOEXEC);
CHECK(fd >= 0 && write(fd, "bad", 3) == 3 && close(fd) == 0 &&
expect_reader_result(root, &sa, LARDON3D_FEATURE_STORE_CORRUPT));
free(original);
lardon3d_extracted_features_destroy(&f);
lardon3d_extracted_features_destroy(&uf);
lardon3d_project_db_close(db);
CHECK(remove_tree(root));
return true;
}
int main(void) { return run_test() ? 0 : 1; }

188
tests/test_feature_task.c Normal file
View file

@ -0,0 +1,188 @@
#include <dirent.h>
#include <errno.h>
#include <sched.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <unistd.h>
#include <lardon3d/feature_store.h>
#include <lardon3d/feature_task.h>
#include <lardon3d/image_catalog.h>
#include <lardon3d/project.h>
#include <lardon3d/task_queue.h>
#define CHECK(x) \
do { \
if (!(x)) { \
fprintf(stderr, "Échec ligne %d: %s\n", __LINE__, #x); \
return false; \
} \
} while (0)
static bool join_path(char out[PATH_MAX], const char *a, const char *b) {
int n = snprintf(out, PATH_MAX, "%s/%s", a, b);
return n > 0 && (size_t)n < PATH_MAX;
}
static bool remove_tree(const char *p) {
struct stat s;
if (lstat(p, &s) != 0) {
return errno == ENOENT;
}
if (!S_ISDIR(s.st_mode)) {
return unlink(p) == 0;
}
DIR *d = opendir(p);
if (!d) {
return false;
}
bool ok = true;
for (struct dirent *e = readdir(d); e; e = readdir(d)) {
if (!strcmp(e->d_name, ".") || !strcmp(e->d_name, "..")) {
continue;
}
char c[PATH_MAX];
if (!join_path(c, p, e->d_name) || !remove_tree(c)) {
ok = false;
}
}
if (closedir(d) || rmdir(p)) {
ok = false;
}
return ok;
}
static bool write_pgm(const char *p) {
FILE *f = fopen(p, "wb");
if (!f) {
return false;
}
fprintf(f, "P5\n192 192\n255\n");
for (unsigned y = 0; y < 192; y++) {
for (unsigned x = 0; x < 192; x++) {
unsigned char v = (unsigned char)((((x / 6) ^ (y / 6)) & 1) ? 245 : 10);
if (fwrite(&v, 1, 1, f) != 1) {
fclose(f);
return false;
}
}
}
return fclose(f) == 0;
}
static bool runtime(Lardon3DAppState *s) {
s->hardware_profile = (Lardon3DHardwareProfile){.logical_cpu_count = 64,
.page_size_bytes = 4096,
.memory_total_bytes = UINT64_MAX,
.cpu_architecture = "test"};
Lardon3DResourcePolicy p = {
.maximum_cpu_load_ratio = 1, .maximum_io_pressure_avg10 = 100, .io_slot_capacity = 2};
s->resource_governor = lardon3d_resource_governor_create(&s->hardware_profile, &p);
s->task_queue = s->resource_governor ? lardon3d_task_queue_create(s->resource_governor, 4) : NULL;
return s->task_queue != NULL;
}
static bool wait_state(Lardon3DTaskQueue *q, uint64_t id, Lardon3DTaskState wanted,
Lardon3DTaskSnapshot *out) {
for (size_t i = 0; i < 2000000; i++) {
if (lardon3d_task_queue_get(q, id, out) && out->state == wanted) {
return true;
}
sched_yield();
}
return false;
}
static bool run_test(void) {
char root[] = "/tmp/lardon3d-feature-task-XXXXXX";
CHECK(mkdtemp(root) && setenv("LARDON3D_PROJECTS_ROOT", root, 1) == 0);
char source[PATH_MAX];
CHECK(join_path(source, root, "source.pgm") && write_pgm(source));
Lardon3DAppState state;
lardon3d_app_state_init(&state);
CHECK(runtime(&state) && lardon3d_project_create(&state, "Features"));
Lardon3DProjectDbScanSet scanset;
CHECK(lardon3d_image_catalog_create_scanset(&state, "A", &scanset));
Lardon3DProjectDbImage image;
Lardon3DProjectDbImageAsset asset;
CHECK(lardon3d_image_catalog_import_file(&state, scanset.scanset_id, source, 0, &image, &asset) ==
LARDON3D_IMAGE_CATALOG_IMPORTED);
Lardon3DFeatureExtractorParameters parameters = {512, 4, 10};
CHECK(setenv("LARDON3D_TEST_FEATURE_PAUSE_BEFORE_PUBLISH", "1", 1) == 0 &&
setenv("LARDON3D_TEST_FEATURE_SKIP_FINISHED_CHECKPOINT", "1", 1) == 0);
uint64_t task_id = 0;
CHECK(lardon3d_project_enqueue_feature_extract(&state, image.image_id, &parameters, &task_id));
Lardon3DTaskSnapshot snapshot;
CHECK(wait_state(state.task_queue, task_id, TASK_PAUSED, &snapshot));
lardon3d_task_queue_destroy(state.task_queue);
state.task_queue = NULL;
lardon3d_project_close(&state);
lardon3d_resource_governor_destroy(state.resource_governor);
state.resource_governor = NULL;
CHECK(unsetenv("LARDON3D_TEST_FEATURE_PAUSE_BEFORE_PUBLISH") == 0 &&
unsetenv("LARDON3D_TEST_FEATURE_SKIP_FINISHED_CHECKPOINT") == 0);
lardon3d_app_state_init(&state);
CHECK(runtime(&state) && lardon3d_project_open(&state, "Features"));
Lardon3DProjectRecoverySummary summary;
CHECK(lardon3d_project_last_recovery_summary(&state, &summary) && summary.resumed == 1);
CHECK(wait_state(state.task_queue, task_id, TASK_COMPLETED, &snapshot) &&
snapshot.progress == 100);
unsigned char fp[32];
lardon3d_feature_extractor_parameter_fingerprint(&parameters, fp);
Lardon3DProjectDbFeatureSet set;
CHECK(lardon3d_project_db_find_feature_set(state.project_db, image.image_id,
LARDON3D_FEATURE_EXTRACTOR_KIND, 1, fp,
&set) == LARDON3D_PROJECT_DB_OK);
lardon3d_task_queue_destroy(state.task_queue);
state.task_queue = NULL;
lardon3d_project_close(&state);
lardon3d_resource_governor_destroy(state.resource_governor);
state.resource_governor = NULL;
lardon3d_app_state_init(&state);
CHECK(runtime(&state) && lardon3d_project_open(&state, "Features"));
CHECK(lardon3d_project_last_recovery_summary(&state, &summary) && summary.resumed == 0);
CHECK(lardon3d_project_db_load_feature_set(state.project_db, set.feature_set_id, &set) ==
LARDON3D_PROJECT_DB_OK);
CHECK(unlink(source) == 0);
Lardon3DFeatureReader *reader = NULL;
Lardon3DFeatureFileMetadata metadata;
CHECK(lardon3d_feature_reader_open(state.project_path, &set, &reader, &metadata) ==
LARDON3D_FEATURE_STORE_OK);
lardon3d_feature_reader_close(reader);
CHECK(write_pgm(source));
Lardon3DProjectDbImage missing_image;
Lardon3DProjectDbImageAsset missing_asset;
CHECK(lardon3d_image_catalog_import_file(&state, scanset.scanset_id, source, 0, &missing_image,
&missing_asset) ==
LARDON3D_IMAGE_CATALOG_ALREADY_PRESENT);
char managed_path[PATH_MAX];
CHECK(join_path(managed_path, state.project_path, missing_asset.path));
CHECK(unlink(managed_path) == 0);
uint64_t missing_task_id = 0;
Lardon3DFeatureExtractorParameters missing_parameters = {511, 4, 10};
CHECK(lardon3d_project_enqueue_feature_extract(&state, missing_image.image_id,
&missing_parameters, &missing_task_id));
CHECK(wait_state(state.task_queue, missing_task_id, TASK_FAILED, &snapshot));
char invalid_source[PATH_MAX];
CHECK(join_path(invalid_source, root, "invalid.pgm"));
FILE *invalid = fopen(invalid_source, "wb");
CHECK(invalid && fwrite("not an image", 1, 12, invalid) == 12 && fclose(invalid) == 0);
Lardon3DProjectDbImage invalid_image;
Lardon3DProjectDbImageAsset invalid_asset;
CHECK(lardon3d_image_catalog_import_file(&state, scanset.scanset_id, invalid_source, 0,
&invalid_image,
&invalid_asset) == LARDON3D_IMAGE_CATALOG_IMPORTED);
uint64_t invalid_task_id = 0;
Lardon3DFeatureExtractorParameters invalid_parameters = {510, 4, 10};
CHECK(lardon3d_project_enqueue_feature_extract(&state, invalid_image.image_id,
&invalid_parameters, &invalid_task_id));
CHECK(wait_state(state.task_queue, invalid_task_id, TASK_FAILED, &snapshot));
lardon3d_task_queue_destroy(state.task_queue);
state.task_queue = NULL;
lardon3d_project_close(&state);
lardon3d_resource_governor_destroy(state.resource_governor);
CHECK(remove_tree(root));
return true;
}
int main(void) { return run_test() ? 0 : 1; }

View file

@ -11,432 +11,561 @@
#include <lardon3d/project_db.h>
#define CHECK(condition) do { if (!(condition)) { \
(void)fprintf(stderr, "Échec ligne %d : %s\n", __LINE__, #condition); return false; \
} } while (0)
#define CHECK(condition) \
do { \
if (!(condition)) { \
(void)fprintf(stderr, "Échec ligne %d : %s\n", __LINE__, #condition); \
return false; \
} \
} while (0)
Lardon3DProjectDbResult lardon3d_project_db_test_orphan_checkpoint(Lardon3DProjectDb *database);
Lardon3DProjectDbResult lardon3d_project_db_test_delete_catalog_identity(
Lardon3DProjectDb *database, uint64_t scanset_id, uint64_t image_id,
uint64_t asset_id);
Lardon3DProjectDbResult
lardon3d_project_db_test_delete_catalog_identity(Lardon3DProjectDb *database, uint64_t scanset_id,
uint64_t image_id, uint64_t asset_id);
static void
asset_path_for_hash(const unsigned char hash[LARDON3D_PROJECT_DB_SHA256_SIZE],
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY])
{
static const char digits[] = "0123456789abcdef";
char hex[65];
for (size_t index = 0; index < sizeof(hex) / 2; ++index) {
hex[index * 2] = digits[hash[index] >> 4];
hex[index * 2 + 1] = digits[hash[index] & 15U];
}
hex[64] = '\0';
(void)snprintf(path, LARDON3D_PROJECT_DB_PATH_CAPACITY,
"assets/images/%c%c/%s", hex[0], hex[1], hex);
static void asset_path_for_hash(const unsigned char hash[LARDON3D_PROJECT_DB_SHA256_SIZE],
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY]) {
static const char digits[] = "0123456789abcdef";
char hex[65];
for (size_t index = 0; index < sizeof(hex) / 2; ++index) {
hex[index * 2] = digits[hash[index] >> 4];
hex[index * 2 + 1] = digits[hash[index] & 15U];
}
hex[64] = '\0';
(void)snprintf(path, LARDON3D_PROJECT_DB_PATH_CAPACITY, "assets/images/%c%c/%s", hex[0], hex[1],
hex);
}
static Lardon3DTaskDurableSnapshot
task_snapshot(uint64_t id, Lardon3DTaskState saved)
{
Lardon3DTaskDurableSnapshot snapshot = {
.id = id,
.estimate = {.minimum_batch_size = 1, .maximum_batch_size = 1, .desired_cpu_threads = 1},
.progress = saved == TASK_COMPLETED ? 100 : 25,
.saved_state = saved,
.recovery_state = saved == TASK_RUNNING || saved == TASK_PAUSED ? TASK_PENDING : saved,
.started_at = {.tv_sec = 10, .tv_nsec = 20},
.finished_at = {.tv_sec = 30, .tv_nsec = 40},
.sequence_count = 2,
};
(void)snprintf(snapshot.name, sizeof(snapshot.name), "Tâche %llu", (unsigned long long)id);
return snapshot;
static Lardon3DTaskDurableSnapshot task_snapshot(uint64_t id, Lardon3DTaskState saved) {
Lardon3DTaskDurableSnapshot snapshot = {
.id = id,
.estimate = {.minimum_batch_size = 1, .maximum_batch_size = 1, .desired_cpu_threads = 1},
.progress = saved == TASK_COMPLETED ? 100 : 25,
.saved_state = saved,
.recovery_state = saved == TASK_RUNNING || saved == TASK_PAUSED ? TASK_PENDING : saved,
.started_at = {.tv_sec = 10, .tv_nsec = 20},
.finished_at = {.tv_sec = 30, .tv_nsec = 40},
.sequence_count = 2,
};
(void)snprintf(snapshot.name, sizeof(snapshot.name), "Tâche %llu", (unsigned long long)id);
return snapshot;
}
typedef struct { Lardon3DProjectDb *database; bool success; } ThreadContext;
typedef struct { Lardon3DProjectDb *database; uint64_t id; bool success; } IdThreadContext;
typedef struct {
Lardon3DProjectDb *database;
bool success;
} ThreadContext;
typedef struct {
Lardon3DProjectDb *database;
uint64_t id;
bool success;
} IdThreadContext;
static void *
read_thread(void *userdata)
{
ThreadContext *context = userdata;
context->success = true;
for (size_t index = 0; index < 100; ++index) {
Lardon3DProjectDbTask task;
if (lardon3d_project_db_load_task(context->database, 1, &task) != LARDON3D_PROJECT_DB_OK) {
context->success = false;
break;
}
}
return NULL;
}
static void *
allocate_id_thread(void *userdata)
{
IdThreadContext *context = userdata;
context->success = lardon3d_project_db_allocate_task_id(
context->database, &context->id) == LARDON3D_PROJECT_DB_OK;
return NULL;
}
static bool
create_future_database(const char *path)
{
sqlite3 *connection = NULL;
if (sqlite3_open(path, &connection) != SQLITE_OK) return false;
bool ok = sqlite3_exec(connection, "CREATE TABLE metadata(key TEXT PRIMARY KEY,value INTEGER NOT NULL);"
"INSERT INTO metadata VALUES('schema_version',5);", NULL, NULL, NULL) == SQLITE_OK;
return sqlite3_close(connection) == SQLITE_OK && ok;
}
static bool
create_v1_database(const char *path)
{
static const char sql[] =
"PRAGMA foreign_keys=ON;"
"CREATE TABLE metadata(key TEXT PRIMARY KEY,value INTEGER NOT NULL);"
"INSERT INTO metadata VALUES('schema_version',1);"
"CREATE TABLE project(singleton INTEGER PRIMARY KEY CHECK(singleton=1),stable_id TEXT NOT NULL UNIQUE,name TEXT NOT NULL,created_at INTEGER NOT NULL,updated_at INTEGER NOT NULL);"
"CREATE TABLE tasks(task_id INTEGER PRIMARY KEY CHECK(task_id>0),name TEXT NOT NULL,saved_state INTEGER NOT NULL CHECK(saved_state BETWEEN 0 AND 5),recovery_state INTEGER NOT NULL CHECK(recovery_state BETWEEN 0 AND 5),progress INTEGER NOT NULL CHECK(progress BETWEEN 0 AND 100),sequence_count INTEGER NOT NULL CHECK(sequence_count>=0),started_sec INTEGER NOT NULL,started_nsec INTEGER NOT NULL CHECK(started_nsec BETWEEN 0 AND 999999999),finished_sec INTEGER NOT NULL,finished_nsec INTEGER NOT NULL CHECK(finished_nsec BETWEEN 0 AND 999999999),updated_at INTEGER NOT NULL);"
"CREATE INDEX tasks_recovery_state_idx ON tasks(recovery_state,task_id);"
"CREATE TABLE checkpoints(task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE CASCADE,path TEXT NOT NULL,format_version INTEGER NOT NULL CHECK(format_version>0),durability INTEGER NOT NULL CHECK(durability BETWEEN 0 AND 1),updated_at INTEGER NOT NULL);"
"CREATE TABLE artifacts(artifact_id TEXT PRIMARY KEY,kind TEXT NOT NULL,path TEXT NOT NULL,state INTEGER NOT NULL CHECK(state BETWEEN 0 AND 1),size_bytes INTEGER NOT NULL CHECK(size_bytes>=0),producer_task_id INTEGER REFERENCES tasks(task_id),created_at INTEGER NOT NULL,updated_at INTEGER NOT NULL);"
"CREATE INDEX artifacts_state_idx ON artifacts(state,artifact_id);"
"CREATE INDEX artifacts_producer_idx ON artifacts(producer_task_id);"
"INSERT INTO project VALUES(1,'legacy-project','Legacy',1,1);"
"INSERT INTO tasks VALUES(9,'Legacy task',1,0,12,3,1,0,0,0,2);"
"INSERT INTO checkpoints VALUES(9,'legacy.chk',1,0,2);"
"INSERT INTO artifacts VALUES('legacy-artifact','legacy','legacy.bin',0,0,9,2,2);";
sqlite3 *connection = NULL;
if (sqlite3_open(path, &connection) != SQLITE_OK) return false;
bool ok = sqlite3_exec(connection, sql, NULL, NULL, NULL) == SQLITE_OK;
return sqlite3_close(connection) == SQLITE_OK && ok;
}
static bool
create_v2_database(const char *path)
{
if (!create_v1_database(path)) return false;
sqlite3 *connection = NULL;
if (sqlite3_open(path, &connection) != SQLITE_OK) return false;
bool ok = sqlite3_exec(connection,
"ALTER TABLE tasks ADD COLUMN task_kind TEXT;"
"ALTER TABLE tasks ADD COLUMN task_kind_version INTEGER CHECK(task_kind_version IS NULL OR task_kind_version>0);"
"UPDATE tasks SET task_kind='test.work',task_kind_version=1 WHERE task_id=9;"
"UPDATE metadata SET value=2 WHERE key='schema_version'",
NULL, NULL, NULL) == SQLITE_OK;
return sqlite3_close(connection) == SQLITE_OK && ok;
}
static bool
create_v3_database(const char *path)
{
if (!create_v2_database(path)) return false;
sqlite3 *connection = NULL;
if (sqlite3_open(path, &connection) != SQLITE_OK) return false;
bool ok = sqlite3_exec(connection,
"CREATE TABLE image_import_tasks(task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE CASCADE,source_path TEXT NOT NULL);"
"INSERT INTO image_import_tasks VALUES(9,'/legacy/source');"
"INSERT INTO metadata(key,value) VALUES('next_task_id',10);"
"UPDATE metadata SET value=3 WHERE key='schema_version'",
NULL, NULL, NULL) == SQLITE_OK;
return sqlite3_close(connection) == SQLITE_OK && ok;
}
static bool
query_integer(const char *path, const char *sql, sqlite3_int64 expected)
{
sqlite3 *connection = NULL;
sqlite3_stmt *statement = NULL;
if (sqlite3_open_v2(path, &connection, SQLITE_OPEN_READONLY, NULL) != SQLITE_OK
|| sqlite3_prepare_v2(connection, sql, -1, &statement, NULL) != SQLITE_OK) {
if (statement) (void)sqlite3_finalize(statement);
if (connection) (void)sqlite3_close(connection);
return false;
}
bool matches = sqlite3_step(statement) == SQLITE_ROW
&& sqlite3_column_int64(statement, 0) == expected
&& sqlite3_step(statement) == SQLITE_DONE;
return sqlite3_finalize(statement) == SQLITE_OK
&& sqlite3_close(connection) == SQLITE_OK && matches;
}
static bool
run_test(void)
{
char directory[] = "/tmp/lardon3d-project-db-XXXXXX";
CHECK(mkdtemp(directory));
char database_path[512], artifact_path[512], future_path[512], corrupt_path[512];
char legacy_path[512], failed_migration_path[512], v2_path[512];
char failed_v3_migration_path[512], v3_path[512], failed_v4_path[512];
CHECK(snprintf(database_path, sizeof(database_path), "%s/project.db", directory) > 0);
CHECK(snprintf(artifact_path, sizeof(artifact_path), "%s/artifact.bin", directory) > 0);
CHECK(snprintf(future_path, sizeof(future_path), "%s/future.db", directory) > 0);
CHECK(snprintf(corrupt_path, sizeof(corrupt_path), "%s/corrupt.db", directory) > 0);
CHECK(snprintf(legacy_path, sizeof(legacy_path), "%s/legacy.db", directory) > 0);
CHECK(snprintf(failed_migration_path, sizeof(failed_migration_path), "%s/failed-migration.db", directory) > 0);
CHECK(snprintf(v2_path, sizeof(v2_path), "%s/v2.db", directory) > 0);
CHECK(snprintf(failed_v3_migration_path, sizeof(failed_v3_migration_path),
"%s/failed-v3-migration.db", directory) > 0);
CHECK(snprintf(v3_path, sizeof(v3_path), "%s/v3.db", directory) > 0);
CHECK(snprintf(failed_v4_path, sizeof(failed_v4_path),
"%s/failed-v4-migration.db", directory) > 0);
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY];
Lardon3DProjectDb *database = NULL;
CHECK(lardon3d_project_db_open(database_path, &database, error) == LARDON3D_PROJECT_DB_OK);
CHECK(database && lardon3d_project_db_schema_version(database) == 4);
bool legacy_pending = true;
CHECK(lardon3d_project_db_legacy_catalog_pending(database, &legacy_pending)
== LARDON3D_PROJECT_DB_OK && !legacy_pending);
Lardon3DProjectDbScanSet deleted_scanset;
CHECK(lardon3d_project_db_create_scanset(database, "Deleted identity",
&deleted_scanset) == LARDON3D_PROJECT_DB_OK);
unsigned char first_hash[LARDON3D_PROJECT_DB_SHA256_SIZE] = {1};
char first_asset_path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
asset_path_for_hash(first_hash, first_asset_path);
Lardon3DProjectDbImageRegisterStatus identity_status;
Lardon3DProjectDbImage deleted_image;
CHECK(lardon3d_project_db_register_image(database,
deleted_scanset.scanset_id, first_hash, first_asset_path, 1,
"deleted.jpg", "/source/deleted.jpg", 0, 1, &identity_status,
&deleted_image) == LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_test_delete_catalog_identity(database,
deleted_scanset.scanset_id, deleted_image.image_id,
deleted_image.asset_id) == LARDON3D_PROJECT_DB_OK);
Lardon3DProjectDbScanSet replacement_scanset;
CHECK(lardon3d_project_db_create_scanset(database, "Replacement identity",
&replacement_scanset) == LARDON3D_PROJECT_DB_OK
&& replacement_scanset.scanset_id > deleted_scanset.scanset_id);
unsigned char second_hash[LARDON3D_PROJECT_DB_SHA256_SIZE] = {2};
char second_asset_path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
asset_path_for_hash(second_hash, second_asset_path);
Lardon3DProjectDbImage replacement_image;
CHECK(lardon3d_project_db_register_image(database,
replacement_scanset.scanset_id, second_hash, second_asset_path, 1,
"replacement.jpg", "/source/replacement.jpg", 0, 2,
&identity_status, &replacement_image) == LARDON3D_PROJECT_DB_OK
&& replacement_image.image_id > deleted_image.image_id
&& replacement_image.asset_id > deleted_image.asset_id);
Lardon3DProjectDbProject project = {.created_at = 100, .updated_at = 100};
(void)snprintf(project.stable_id, sizeof(project.stable_id), "project-0001");
(void)snprintf(project.name, sizeof(project.name), "Projet test");
CHECK(lardon3d_project_db_set_project(database, &project) == LARDON3D_PROJECT_DB_OK);
Lardon3DProjectDbProject loaded_project;
CHECK(lardon3d_project_db_get_project(database, &loaded_project) == LARDON3D_PROJECT_DB_OK);
CHECK(strcmp(loaded_project.stable_id, project.stable_id) == 0);
project.updated_at = 101; (void)snprintf(project.name, sizeof(project.name), "Projet renommé");
CHECK(lardon3d_project_db_set_project(database, &project) == LARDON3D_PROJECT_DB_OK);
Lardon3DProjectDbProject conflict = project; (void)snprintf(conflict.stable_id, sizeof(conflict.stable_id), "other");
CHECK(lardon3d_project_db_set_project(database, &conflict) == LARDON3D_PROJECT_DB_CONSTRAINT);
char copied_error[LARDON3D_PROJECT_DB_ERROR_CAPACITY];
CHECK(lardon3d_project_db_last_error(database, copied_error));
CHECK(copied_error[0] != '\0');
project = loaded_project; memset(project.name, 'x', sizeof(project.name));
CHECK(lardon3d_project_db_set_project(database, &project) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
Lardon3DTaskDurableSnapshot running = task_snapshot(1, TASK_RUNNING);
Lardon3DProjectDbCheckpoint checkpoint = {.format_version = 1, .durability = LARDON3D_DB_CHECKPOINT_DURABLE, .updated_at = 200};
(void)snprintf(checkpoint.path, sizeof(checkpoint.path), "%s/checkpoints/task-1.chk", directory);
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 200) == LARDON3D_PROJECT_DB_OK);
running.progress = 30; running.sequence_count = 3;
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 201) == LARDON3D_PROJECT_DB_OK);
Lardon3DProjectDbScanSet import_scanset;
CHECK(lardon3d_project_db_create_scanset(database, "Legacy import",
&import_scanset) == LARDON3D_PROJECT_DB_OK);
static void *read_thread(void *userdata) {
ThreadContext *context = userdata;
context->success = true;
for (size_t index = 0; index < 100; ++index) {
Lardon3DProjectDbTask task;
CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK);
CHECK(task.recovery_state == TASK_PENDING && task.progress == 30 && task.sequence_count == 3);
CHECK(task.has_task_kind && strcmp(task.task_kind, "test.work") == 0
&& task.task_kind_version == 1);
CHECK(task.has_checkpoint && strcmp(task.checkpoint.path, checkpoint.path) == 0);
checkpoint.durability = LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE;
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 201) == LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK);
CHECK(task.checkpoint.durability == LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE);
CHECK(lardon3d_project_db_record_task(database, &running, "test.other", 1,
&checkpoint, 202) == LARDON3D_PROJECT_DB_CONSTRAINT);
CHECK(lardon3d_project_db_record_task(database, &running, "Test.invalid", 1,
&checkpoint, 202) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 0,
&checkpoint, 202) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
CHECK(lardon3d_project_db_record_image_import_task(database, &running,
"import.images", 1, &checkpoint, "/tmp/source-a",
import_scanset.scanset_id, 202)
== LARDON3D_PROJECT_DB_CONSTRAINT);
Lardon3DTaskDurableSnapshot completed = task_snapshot(2, TASK_COMPLETED);
CHECK(lardon3d_project_db_record_task(database, &completed, "test.work", 1, NULL, 202) == LARDON3D_PROJECT_DB_OK);
Lardon3DTaskDurableSnapshot no_checkpoint = task_snapshot(4, TASK_PENDING);
CHECK(lardon3d_project_db_record_task(database, &no_checkpoint, NULL, 0, NULL, 202) == LARDON3D_PROJECT_DB_OK);
Lardon3DProjectDbTask page[1]; size_t count = 0;
CHECK(lardon3d_project_db_list_recoverable(database, 0, page, 1, &count) == LARDON3D_PROJECT_DB_OK);
CHECK(count == 1 && page[0].task_id == 1);
CHECK(lardon3d_project_db_list_recoverable(database, 1, page, 1, &count) == LARDON3D_PROJECT_DB_OK && count == 0);
CHECK(lardon3d_project_db_list_recoverable(database, 0, page, LARDON3D_PROJECT_DB_RECOVERY_PAGE_MAX + 1, &count) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
Lardon3DTaskDurableSnapshot image_import = task_snapshot(5, TASK_PENDING);
CHECK(lardon3d_project_db_record_image_import_task(database, &image_import,
"import.images", 1, &checkpoint, "/tmp/source-a",
import_scanset.scanset_id, 202)
== LARDON3D_PROJECT_DB_OK);
Lardon3DProjectDbImageImport import_parameters;
CHECK(lardon3d_project_db_load_image_import(database, 5,
&import_parameters) == LARDON3D_PROJECT_DB_OK);
CHECK(import_parameters.task_id == 5
&& strcmp(import_parameters.source_path, "/tmp/source-a") == 0
&& import_parameters.scanset_id == import_scanset.scanset_id);
CHECK(lardon3d_project_db_record_image_import_task(database, &image_import,
"import.images", 1, &checkpoint, "/tmp/source-b",
import_scanset.scanset_id, 203)
== LARDON3D_PROJECT_DB_CONSTRAINT);
Lardon3DTaskDurableSnapshot rollback_task = task_snapshot(3, TASK_PENDING);
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_CHECKPOINT", "1", 1) == 0);
CHECK(lardon3d_project_db_record_task(database, &rollback_task, "test.work", 1, &checkpoint, 203) == LARDON3D_PROJECT_DB_IO_ERROR);
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_CHECKPOINT") == 0);
CHECK(lardon3d_project_db_load_task(database, 3, &task) == LARDON3D_PROJECT_DB_NOT_FOUND);
CHECK(lardon3d_project_db_test_orphan_checkpoint(database) == LARDON3D_PROJECT_DB_CONSTRAINT);
Lardon3DProjectDbArtifact artifact = {.state = LARDON3D_DB_ARTIFACT_STAGED, .has_producer_task = true, .producer_task_id = 1, .created_at = 300, .updated_at = 300};
(void)snprintf(artifact.artifact_id, sizeof(artifact.artifact_id), "artifact-1");
(void)snprintf(artifact.kind, sizeof(artifact.kind), "generic-test");
(void)snprintf(artifact.path, sizeof(artifact.path), "%s", artifact_path);
CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_OK);
Lardon3DProjectDbArtifact loaded_artifact;
CHECK(lardon3d_project_db_load_artifact(database, artifact.artifact_id, &loaded_artifact) == LARDON3D_PROJECT_DB_OK);
CHECK(loaded_artifact.state == LARDON3D_DB_ARTIFACT_STAGED);
CHECK(lardon3d_project_db_mark_artifact_ready(database, artifact.artifact_id, 301) == LARDON3D_PROJECT_DB_IO_ERROR);
int descriptor = open(artifact_path, O_WRONLY | O_CREAT | O_TRUNC, 0600); CHECK(descriptor >= 0);
const char payload[] = "validated"; CHECK(write(descriptor, payload, sizeof(payload)) == (ssize_t)sizeof(payload)); CHECK(close(descriptor) == 0);
CHECK(lardon3d_project_db_mark_artifact_ready(database, artifact.artifact_id, 301) == LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_load_artifact(database, artifact.artifact_id, &loaded_artifact) == LARDON3D_PROJECT_DB_OK);
CHECK(loaded_artifact.state == LARDON3D_DB_ARTIFACT_READY && loaded_artifact.size_bytes == sizeof(payload));
CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_CONSTRAINT);
artifact.producer_task_id = 999;
(void)snprintf(artifact.artifact_id, sizeof(artifact.artifact_id), "orphan-artifact");
CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_CONSTRAINT);
artifact = loaded_artifact; artifact.state = LARDON3D_DB_ARTIFACT_STAGED; artifact.size_bytes = 0;
memset(artifact.path, 'x', sizeof(artifact.path));
CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
ThreadContext contexts[2] = {{.database = database}, {.database = database}}; pthread_t threads[2];
CHECK(pthread_create(&threads[0], NULL, read_thread, &contexts[0]) == 0);
CHECK(pthread_create(&threads[1], NULL, read_thread, &contexts[1]) == 0);
CHECK(pthread_join(threads[0], NULL) == 0 && pthread_join(threads[1], NULL) == 0);
CHECK(contexts[0].success && contexts[1].success);
IdThreadContext id_contexts[2] = {{.database = database}, {.database = database}};
CHECK(pthread_create(&threads[0], NULL, allocate_id_thread,
&id_contexts[0]) == 0);
CHECK(pthread_create(&threads[1], NULL, allocate_id_thread,
&id_contexts[1]) == 0);
CHECK(pthread_join(threads[0], NULL) == 0
&& pthread_join(threads[1], NULL) == 0);
CHECK(id_contexts[0].success && id_contexts[1].success
&& id_contexts[0].id != id_contexts[1].id
&& id_contexts[0].id > 5 && id_contexts[1].id > 5);
char too_long[LARDON3D_PROJECT_DB_PATH_CAPACITY + 1]; memset(too_long, 'x', sizeof(too_long)); too_long[sizeof(too_long) - 1] = '\0';
CHECK(lardon3d_project_db_open(too_long, &database, error) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
lardon3d_project_db_close(contexts[0].database); database = NULL;
CHECK(query_integer(database_path, "SELECT value FROM metadata WHERE key='schema_version'", 4));
CHECK(query_integer(database_path, "SELECT count(*) FROM tasks WHERE task_id=1", 1));
CHECK(lardon3d_project_db_open(database_path, &database, error) == LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_load_artifact(database, "artifact-1", &loaded_artifact) == LARDON3D_PROJECT_DB_OK);
lardon3d_project_db_close(database);
CHECK(create_future_database(future_path));
CHECK(lardon3d_project_db_open(future_path, &database, error) == LARDON3D_PROJECT_DB_UNSUPPORTED_SCHEMA);
descriptor = open(corrupt_path, O_WRONLY | O_CREAT | O_TRUNC, 0600); CHECK(descriptor >= 0);
CHECK(write(descriptor, "not sqlite", 10) == 10 && close(descriptor) == 0);
CHECK(lardon3d_project_db_open(corrupt_path, &database, error) == LARDON3D_PROJECT_DB_CORRUPT);
CHECK(create_v1_database(legacy_path));
CHECK(lardon3d_project_db_open(legacy_path, &database, error) == LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_schema_version(database) == 4);
CHECK(lardon3d_project_db_get_project(database, &loaded_project) == LARDON3D_PROJECT_DB_OK
&& strcmp(loaded_project.stable_id, "legacy-project") == 0);
CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK);
CHECK(!task.has_task_kind && task.has_checkpoint
&& strcmp(task.checkpoint.path, "legacy.chk") == 0);
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact",
&loaded_artifact) == LARDON3D_PROJECT_DB_OK);
lardon3d_project_db_close(database); database = NULL;
CHECK(query_integer(legacy_path, "SELECT value FROM metadata WHERE key='schema_version'", 4));
CHECK(create_v1_database(failed_migration_path));
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V2", "1", 1) == 0);
CHECK(lardon3d_project_db_open(failed_migration_path, &database, error)
== LARDON3D_PROJECT_DB_IO_ERROR);
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V2") == 0);
CHECK(query_integer(failed_migration_path,
"SELECT value FROM metadata WHERE key='schema_version'", 1));
CHECK(lardon3d_project_db_open(failed_migration_path, &database, error)
== LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK
&& !task.has_task_kind);
lardon3d_project_db_close(database); database = NULL;
CHECK(create_v2_database(v2_path));
CHECK(lardon3d_project_db_open(v2_path, &database, error)
== LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_load_task(database, 9, &task)
== LARDON3D_PROJECT_DB_OK && task.has_task_kind
&& strcmp(task.task_kind, "test.work") == 0);
CHECK(task.has_checkpoint
&& strcmp(task.checkpoint.path, "legacy.chk") == 0);
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact",
&loaded_artifact) == LARDON3D_PROJECT_DB_OK);
lardon3d_project_db_close(database); database = NULL;
CHECK(query_integer(v2_path,
"SELECT value FROM metadata WHERE key='schema_version'", 4));
CHECK(create_v2_database(failed_v3_migration_path));
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V3", "1", 1) == 0);
CHECK(lardon3d_project_db_open(failed_v3_migration_path, &database, error)
== LARDON3D_PROJECT_DB_IO_ERROR);
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V3") == 0);
CHECK(query_integer(failed_v3_migration_path,
"SELECT value FROM metadata WHERE key='schema_version'", 2));
CHECK(query_integer(failed_v3_migration_path,
"SELECT count(*) FROM sqlite_master WHERE type='table' AND name='image_import_tasks'", 0));
CHECK(create_v3_database(v3_path));
CHECK(lardon3d_project_db_open(v3_path, &database, error)
== LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_load_image_import(database, 9,
&import_parameters) == LARDON3D_PROJECT_DB_OK);
CHECK(strcmp(import_parameters.source_path, "/legacy/source") == 0
&& import_parameters.scanset_id > 0);
CHECK(lardon3d_project_db_legacy_catalog_pending(database, &legacy_pending)
== LARDON3D_PROJECT_DB_OK && legacy_pending);
CHECK(lardon3d_project_db_load_task(database, 9, &task)
== LARDON3D_PROJECT_DB_OK && task.has_task_kind);
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact",
&loaded_artifact) == LARDON3D_PROJECT_DB_OK);
lardon3d_project_db_close(database); database = NULL;
CHECK(query_integer(v3_path,
"SELECT value FROM metadata WHERE key='schema_version'", 4));
CHECK(create_v3_database(failed_v4_path));
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V4", "1", 1) == 0);
CHECK(lardon3d_project_db_open(failed_v4_path, &database, error)
== LARDON3D_PROJECT_DB_IO_ERROR);
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V4") == 0);
CHECK(query_integer(failed_v4_path,
"SELECT value FROM metadata WHERE key='schema_version'", 3));
CHECK(query_integer(failed_v4_path,
"SELECT count(*) FROM sqlite_master WHERE type='table' AND name='scansets'", 0));
CHECK(unlink(artifact_path) == 0); CHECK(unlink(database_path) == 0); CHECK(unlink(future_path) == 0); CHECK(unlink(corrupt_path) == 0);
CHECK(unlink(legacy_path) == 0); CHECK(unlink(failed_migration_path) == 0);
CHECK(unlink(v2_path) == 0); CHECK(unlink(failed_v3_migration_path) == 0);
CHECK(unlink(v3_path) == 0); CHECK(unlink(failed_v4_path) == 0);
CHECK(rmdir(directory) == 0);
return true;
if (lardon3d_project_db_load_task(context->database, 1, &task) != LARDON3D_PROJECT_DB_OK) {
context->success = false;
break;
}
}
return NULL;
}
int main(void) { return run_test() ? EXIT_SUCCESS : EXIT_FAILURE; }
static void *allocate_id_thread(void *userdata) {
IdThreadContext *context = userdata;
context->success = lardon3d_project_db_allocate_task_id(context->database, &context->id) ==
LARDON3D_PROJECT_DB_OK;
return NULL;
}
static bool create_future_database(const char *path) {
sqlite3 *connection = NULL;
if (sqlite3_open(path, &connection) != SQLITE_OK) {
return false;
}
bool ok = sqlite3_exec(connection,
"CREATE TABLE metadata(key TEXT PRIMARY KEY,value INTEGER NOT NULL);"
"INSERT INTO metadata VALUES('schema_version',6);",
NULL, NULL, NULL) == SQLITE_OK;
return sqlite3_close(connection) == SQLITE_OK && ok;
}
static bool create_v1_database(const char *path) {
static const char sql[] =
"PRAGMA foreign_keys=ON;"
"CREATE TABLE metadata(key TEXT PRIMARY KEY,value INTEGER NOT NULL);"
"INSERT INTO metadata VALUES('schema_version',1);"
"CREATE TABLE project(singleton INTEGER PRIMARY KEY CHECK(singleton=1),stable_id TEXT NOT "
"NULL UNIQUE,name TEXT NOT NULL,created_at INTEGER NOT NULL,updated_at INTEGER NOT NULL);"
"CREATE TABLE tasks(task_id INTEGER PRIMARY KEY CHECK(task_id>0),name TEXT NOT "
"NULL,saved_state INTEGER NOT NULL CHECK(saved_state BETWEEN 0 AND 5),recovery_state "
"INTEGER NOT NULL CHECK(recovery_state BETWEEN 0 AND 5),progress INTEGER NOT NULL "
"CHECK(progress BETWEEN 0 AND 100),sequence_count INTEGER NOT NULL "
"CHECK(sequence_count>=0),started_sec INTEGER NOT NULL,started_nsec INTEGER NOT NULL "
"CHECK(started_nsec BETWEEN 0 AND 999999999),finished_sec INTEGER NOT NULL,finished_nsec "
"INTEGER NOT NULL CHECK(finished_nsec BETWEEN 0 AND 999999999),updated_at INTEGER NOT "
"NULL);"
"CREATE INDEX tasks_recovery_state_idx ON tasks(recovery_state,task_id);"
"CREATE TABLE checkpoints(task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE "
"CASCADE,path TEXT NOT NULL,format_version INTEGER NOT NULL "
"CHECK(format_version>0),durability INTEGER NOT NULL CHECK(durability BETWEEN 0 AND "
"1),updated_at INTEGER NOT NULL);"
"CREATE TABLE artifacts(artifact_id TEXT PRIMARY KEY,kind TEXT NOT NULL,path TEXT NOT "
"NULL,state INTEGER NOT NULL CHECK(state BETWEEN 0 AND 1),size_bytes INTEGER NOT NULL "
"CHECK(size_bytes>=0),producer_task_id INTEGER REFERENCES tasks(task_id),created_at "
"INTEGER NOT NULL,updated_at INTEGER NOT NULL);"
"CREATE INDEX artifacts_state_idx ON artifacts(state,artifact_id);"
"CREATE INDEX artifacts_producer_idx ON artifacts(producer_task_id);"
"INSERT INTO project VALUES(1,'legacy-project','Legacy',1,1);"
"INSERT INTO tasks VALUES(9,'Legacy task',1,0,12,3,1,0,0,0,2);"
"INSERT INTO checkpoints VALUES(9,'legacy.chk',1,0,2);"
"INSERT INTO artifacts VALUES('legacy-artifact','legacy','legacy.bin',0,0,9,2,2);";
sqlite3 *connection = NULL;
if (sqlite3_open(path, &connection) != SQLITE_OK) {
return false;
}
bool ok = sqlite3_exec(connection, sql, NULL, NULL, NULL) == SQLITE_OK;
return sqlite3_close(connection) == SQLITE_OK && ok;
}
static bool create_v2_database(const char *path) {
if (!create_v1_database(path)) {
return false;
}
sqlite3 *connection = NULL;
if (sqlite3_open(path, &connection) != SQLITE_OK) {
return false;
}
bool ok =
sqlite3_exec(connection,
"ALTER TABLE tasks ADD COLUMN task_kind TEXT;"
"ALTER TABLE tasks ADD COLUMN task_kind_version INTEGER "
"CHECK(task_kind_version IS NULL OR task_kind_version>0);"
"UPDATE tasks SET task_kind='test.work',task_kind_version=1 WHERE task_id=9;"
"UPDATE metadata SET value=2 WHERE key='schema_version'",
NULL, NULL, NULL) == SQLITE_OK;
return sqlite3_close(connection) == SQLITE_OK && ok;
}
static bool create_v3_database(const char *path) {
if (!create_v2_database(path)) {
return false;
}
sqlite3 *connection = NULL;
if (sqlite3_open(path, &connection) != SQLITE_OK) {
return false;
}
bool ok = sqlite3_exec(connection,
"CREATE TABLE image_import_tasks(task_id INTEGER PRIMARY KEY REFERENCES "
"tasks(task_id) ON DELETE CASCADE,source_path TEXT NOT NULL);"
"INSERT INTO image_import_tasks VALUES(9,'/legacy/source');"
"INSERT INTO metadata(key,value) VALUES('next_task_id',10);"
"UPDATE metadata SET value=3 WHERE key='schema_version'",
NULL, NULL, NULL) == SQLITE_OK;
return sqlite3_close(connection) == SQLITE_OK && ok;
}
static bool create_v4_database(const char *path) {
if (!create_v3_database(path)) {
return false;
}
sqlite3 *connection = NULL;
if (sqlite3_open(path, &connection) != SQLITE_OK) {
return false;
}
const char *sql =
"CREATE TABLE scansets(scanset_id INTEGER PRIMARY KEY AUTOINCREMENT "
"CHECK(scanset_id>0),name TEXT NOT NULL,created_at INTEGER NOT NULL,updated_at INTEGER NOT "
"NULL);"
"CREATE TABLE image_assets(asset_id INTEGER PRIMARY KEY AUTOINCREMENT "
"CHECK(asset_id>0),sha256 BLOB NOT NULL UNIQUE,path TEXT NOT NULL UNIQUE,size_bytes "
"INTEGER NOT NULL,state INTEGER NOT NULL,created_at INTEGER NOT NULL);"
"CREATE TABLE images(image_id INTEGER PRIMARY KEY AUTOINCREMENT "
"CHECK(image_id>0),scanset_id INTEGER NOT NULL REFERENCES scansets(scanset_id),asset_id "
"INTEGER NOT NULL REFERENCES image_assets(asset_id),original_name TEXT NOT "
"NULL,source_path TEXT NOT NULL,producer_task_id INTEGER REFERENCES "
"tasks(task_id),imported_at INTEGER NOT NULL,UNIQUE(scanset_id,asset_id));"
"CREATE INDEX images_scanset_idx ON images(scanset_id,image_id);"
"CREATE INDEX images_producer_idx ON images(producer_task_id,image_id);"
"ALTER TABLE image_import_tasks ADD COLUMN scanset_id INTEGER REFERENCES "
"scansets(scanset_id);"
"INSERT INTO scansets(name,created_at,updated_at) VALUES('Legacy',0,0);"
"UPDATE image_import_tasks SET scanset_id=1;"
"INSERT INTO metadata VALUES('legacy_image_catalog_pending',1);"
"UPDATE metadata SET value=4 WHERE key='schema_version';";
bool ok = sqlite3_exec(connection, sql, NULL, NULL, NULL) == SQLITE_OK;
return sqlite3_close(connection) == SQLITE_OK && ok;
}
static bool query_integer(const char *path, const char *sql, sqlite3_int64 expected) {
sqlite3 *connection = NULL;
sqlite3_stmt *statement = NULL;
if (sqlite3_open_v2(path, &connection, SQLITE_OPEN_READONLY, NULL) != SQLITE_OK ||
sqlite3_prepare_v2(connection, sql, -1, &statement, NULL) != SQLITE_OK) {
if (statement) {
(void)sqlite3_finalize(statement);
}
if (connection) {
(void)sqlite3_close(connection);
}
return false;
}
bool matches = sqlite3_step(statement) == SQLITE_ROW &&
sqlite3_column_int64(statement, 0) == expected &&
sqlite3_step(statement) == SQLITE_DONE;
return sqlite3_finalize(statement) == SQLITE_OK && sqlite3_close(connection) == SQLITE_OK &&
matches;
}
static bool run_test(void) {
char directory[] = "/tmp/lardon3d-project-db-XXXXXX";
CHECK(mkdtemp(directory));
char database_path[512], artifact_path[512], future_path[512], corrupt_path[512];
char legacy_path[512], failed_migration_path[512], v2_path[512];
char failed_v3_migration_path[512], v3_path[512], failed_v4_path[512];
char v4_path[512], failed_v5_path[512];
CHECK(snprintf(database_path, sizeof(database_path), "%s/project.db", directory) > 0);
CHECK(snprintf(artifact_path, sizeof(artifact_path), "%s/artifact.bin", directory) > 0);
CHECK(snprintf(future_path, sizeof(future_path), "%s/future.db", directory) > 0);
CHECK(snprintf(corrupt_path, sizeof(corrupt_path), "%s/corrupt.db", directory) > 0);
CHECK(snprintf(legacy_path, sizeof(legacy_path), "%s/legacy.db", directory) > 0);
CHECK(snprintf(failed_migration_path, sizeof(failed_migration_path), "%s/failed-migration.db",
directory) > 0);
CHECK(snprintf(v2_path, sizeof(v2_path), "%s/v2.db", directory) > 0);
CHECK(snprintf(failed_v3_migration_path, sizeof(failed_v3_migration_path),
"%s/failed-v3-migration.db", directory) > 0);
CHECK(snprintf(v3_path, sizeof(v3_path), "%s/v3.db", directory) > 0);
CHECK(snprintf(failed_v4_path, sizeof(failed_v4_path), "%s/failed-v4-migration.db", directory) >
0);
CHECK(snprintf(v4_path, sizeof(v4_path), "%s/v4.db", directory) > 0);
CHECK(snprintf(failed_v5_path, sizeof(failed_v5_path), "%s/failed-v5-migration.db", directory) >
0);
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY];
Lardon3DProjectDb *database = NULL;
CHECK(lardon3d_project_db_open(database_path, &database, error) == LARDON3D_PROJECT_DB_OK);
CHECK(database && lardon3d_project_db_schema_version(database) == 5);
bool legacy_pending = true;
CHECK(lardon3d_project_db_legacy_catalog_pending(database, &legacy_pending) ==
LARDON3D_PROJECT_DB_OK &&
!legacy_pending);
Lardon3DProjectDbScanSet deleted_scanset;
CHECK(lardon3d_project_db_create_scanset(database, "Deleted identity", &deleted_scanset) ==
LARDON3D_PROJECT_DB_OK);
unsigned char first_hash[LARDON3D_PROJECT_DB_SHA256_SIZE] = {1};
char first_asset_path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
asset_path_for_hash(first_hash, first_asset_path);
Lardon3DProjectDbImageRegisterStatus identity_status;
Lardon3DProjectDbImage deleted_image;
CHECK(lardon3d_project_db_register_image(database, deleted_scanset.scanset_id, first_hash,
first_asset_path, 1, "deleted.jpg",
"/source/deleted.jpg", 0, 1, &identity_status,
&deleted_image) == LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_test_delete_catalog_identity(
database, deleted_scanset.scanset_id, deleted_image.image_id, deleted_image.asset_id) ==
LARDON3D_PROJECT_DB_OK);
Lardon3DProjectDbScanSet replacement_scanset;
CHECK(lardon3d_project_db_create_scanset(database, "Replacement identity",
&replacement_scanset) == LARDON3D_PROJECT_DB_OK &&
replacement_scanset.scanset_id > deleted_scanset.scanset_id);
unsigned char second_hash[LARDON3D_PROJECT_DB_SHA256_SIZE] = {2};
char second_asset_path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
asset_path_for_hash(second_hash, second_asset_path);
Lardon3DProjectDbImage replacement_image;
CHECK(lardon3d_project_db_register_image(database, replacement_scanset.scanset_id, second_hash,
second_asset_path, 1, "replacement.jpg",
"/source/replacement.jpg", 0, 2, &identity_status,
&replacement_image) == LARDON3D_PROJECT_DB_OK &&
replacement_image.image_id > deleted_image.image_id &&
replacement_image.asset_id > deleted_image.asset_id);
Lardon3DProjectDbProject project = {.created_at = 100, .updated_at = 100};
(void)snprintf(project.stable_id, sizeof(project.stable_id), "project-0001");
(void)snprintf(project.name, sizeof(project.name), "Projet test");
CHECK(lardon3d_project_db_set_project(database, &project) == LARDON3D_PROJECT_DB_OK);
Lardon3DProjectDbProject loaded_project;
CHECK(lardon3d_project_db_get_project(database, &loaded_project) == LARDON3D_PROJECT_DB_OK);
CHECK(strcmp(loaded_project.stable_id, project.stable_id) == 0);
project.updated_at = 101;
(void)snprintf(project.name, sizeof(project.name), "Projet renommé");
CHECK(lardon3d_project_db_set_project(database, &project) == LARDON3D_PROJECT_DB_OK);
Lardon3DProjectDbProject conflict = project;
(void)snprintf(conflict.stable_id, sizeof(conflict.stable_id), "other");
CHECK(lardon3d_project_db_set_project(database, &conflict) == LARDON3D_PROJECT_DB_CONSTRAINT);
char copied_error[LARDON3D_PROJECT_DB_ERROR_CAPACITY];
CHECK(lardon3d_project_db_last_error(database, copied_error));
CHECK(copied_error[0] != '\0');
project = loaded_project;
memset(project.name, 'x', sizeof(project.name));
CHECK(lardon3d_project_db_set_project(database, &project) ==
LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
Lardon3DTaskDurableSnapshot running = task_snapshot(1, TASK_RUNNING);
Lardon3DProjectDbCheckpoint checkpoint = {
.format_version = 1, .durability = LARDON3D_DB_CHECKPOINT_DURABLE, .updated_at = 200};
(void)snprintf(checkpoint.path, sizeof(checkpoint.path), "%s/checkpoints/task-1.chk", directory);
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 200) ==
LARDON3D_PROJECT_DB_OK);
running.progress = 30;
running.sequence_count = 3;
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 201) ==
LARDON3D_PROJECT_DB_OK);
Lardon3DProjectDbScanSet import_scanset;
CHECK(lardon3d_project_db_create_scanset(database, "Legacy import", &import_scanset) ==
LARDON3D_PROJECT_DB_OK);
Lardon3DProjectDbTask task;
CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK);
CHECK(task.recovery_state == TASK_PENDING && task.progress == 30 && task.sequence_count == 3);
CHECK(task.has_task_kind && strcmp(task.task_kind, "test.work") == 0 &&
task.task_kind_version == 1);
CHECK(task.has_checkpoint && strcmp(task.checkpoint.path, checkpoint.path) == 0);
checkpoint.durability = LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE;
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 201) ==
LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK);
CHECK(task.checkpoint.durability == LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE);
CHECK(lardon3d_project_db_record_task(database, &running, "test.other", 1, &checkpoint, 202) ==
LARDON3D_PROJECT_DB_CONSTRAINT);
CHECK(lardon3d_project_db_record_task(database, &running, "Test.invalid", 1, &checkpoint, 202) ==
LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 0, &checkpoint, 202) ==
LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
CHECK(lardon3d_project_db_record_image_import_task(
database, &running, "import.images", 1, &checkpoint, "/tmp/source-a",
import_scanset.scanset_id, 202) == LARDON3D_PROJECT_DB_CONSTRAINT);
Lardon3DTaskDurableSnapshot completed = task_snapshot(2, TASK_COMPLETED);
CHECK(lardon3d_project_db_record_task(database, &completed, "test.work", 1, NULL, 202) ==
LARDON3D_PROJECT_DB_OK);
Lardon3DTaskDurableSnapshot no_checkpoint = task_snapshot(4, TASK_PENDING);
CHECK(lardon3d_project_db_record_task(database, &no_checkpoint, NULL, 0, NULL, 202) ==
LARDON3D_PROJECT_DB_OK);
Lardon3DProjectDbTask page[1];
size_t count = 0;
CHECK(lardon3d_project_db_list_recoverable(database, 0, page, 1, &count) ==
LARDON3D_PROJECT_DB_OK);
CHECK(count == 1 && page[0].task_id == 1);
CHECK(lardon3d_project_db_list_recoverable(database, 1, page, 1, &count) ==
LARDON3D_PROJECT_DB_OK &&
count == 0);
CHECK(lardon3d_project_db_list_recoverable(database, 0, page,
LARDON3D_PROJECT_DB_RECOVERY_PAGE_MAX + 1,
&count) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
Lardon3DTaskDurableSnapshot image_import = task_snapshot(5, TASK_PENDING);
CHECK(lardon3d_project_db_record_image_import_task(
database, &image_import, "import.images", 1, &checkpoint, "/tmp/source-a",
import_scanset.scanset_id, 202) == LARDON3D_PROJECT_DB_OK);
Lardon3DProjectDbImageImport import_parameters;
CHECK(lardon3d_project_db_load_image_import(database, 5, &import_parameters) ==
LARDON3D_PROJECT_DB_OK);
CHECK(import_parameters.task_id == 5 &&
strcmp(import_parameters.source_path, "/tmp/source-a") == 0 &&
import_parameters.scanset_id == import_scanset.scanset_id);
CHECK(lardon3d_project_db_record_image_import_task(
database, &image_import, "import.images", 1, &checkpoint, "/tmp/source-b",
import_scanset.scanset_id, 203) == LARDON3D_PROJECT_DB_CONSTRAINT);
Lardon3DTaskDurableSnapshot rollback_task = task_snapshot(3, TASK_PENDING);
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_CHECKPOINT", "1", 1) == 0);
CHECK(lardon3d_project_db_record_task(database, &rollback_task, "test.work", 1, &checkpoint,
203) == LARDON3D_PROJECT_DB_IO_ERROR);
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_CHECKPOINT") == 0);
CHECK(lardon3d_project_db_load_task(database, 3, &task) == LARDON3D_PROJECT_DB_NOT_FOUND);
CHECK(lardon3d_project_db_test_orphan_checkpoint(database) == LARDON3D_PROJECT_DB_CONSTRAINT);
Lardon3DProjectDbArtifact artifact = {.state = LARDON3D_DB_ARTIFACT_STAGED,
.has_producer_task = true,
.producer_task_id = 1,
.created_at = 300,
.updated_at = 300};
(void)snprintf(artifact.artifact_id, sizeof(artifact.artifact_id), "artifact-1");
(void)snprintf(artifact.kind, sizeof(artifact.kind), "generic-test");
(void)snprintf(artifact.path, sizeof(artifact.path), "%s", artifact_path);
CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_OK);
Lardon3DProjectDbArtifact loaded_artifact;
CHECK(lardon3d_project_db_load_artifact(database, artifact.artifact_id, &loaded_artifact) ==
LARDON3D_PROJECT_DB_OK);
CHECK(loaded_artifact.state == LARDON3D_DB_ARTIFACT_STAGED);
CHECK(lardon3d_project_db_mark_artifact_ready(database, artifact.artifact_id, 301) ==
LARDON3D_PROJECT_DB_IO_ERROR);
int descriptor = open(artifact_path, O_WRONLY | O_CREAT | O_TRUNC, 0600);
CHECK(descriptor >= 0);
const char payload[] = "validated";
CHECK(write(descriptor, payload, sizeof(payload)) == (ssize_t)sizeof(payload));
CHECK(close(descriptor) == 0);
CHECK(lardon3d_project_db_mark_artifact_ready(database, artifact.artifact_id, 301) ==
LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_load_artifact(database, artifact.artifact_id, &loaded_artifact) ==
LARDON3D_PROJECT_DB_OK);
CHECK(loaded_artifact.state == LARDON3D_DB_ARTIFACT_READY &&
loaded_artifact.size_bytes == sizeof(payload));
CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_CONSTRAINT);
artifact.producer_task_id = 999;
(void)snprintf(artifact.artifact_id, sizeof(artifact.artifact_id), "orphan-artifact");
CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_CONSTRAINT);
artifact = loaded_artifact;
artifact.state = LARDON3D_DB_ARTIFACT_STAGED;
artifact.size_bytes = 0;
memset(artifact.path, 'x', sizeof(artifact.path));
CHECK(lardon3d_project_db_create_artifact(database, &artifact) ==
LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
ThreadContext contexts[2] = {{.database = database}, {.database = database}};
pthread_t threads[2];
CHECK(pthread_create(&threads[0], NULL, read_thread, &contexts[0]) == 0);
CHECK(pthread_create(&threads[1], NULL, read_thread, &contexts[1]) == 0);
CHECK(pthread_join(threads[0], NULL) == 0 && pthread_join(threads[1], NULL) == 0);
CHECK(contexts[0].success && contexts[1].success);
IdThreadContext id_contexts[2] = {{.database = database}, {.database = database}};
CHECK(pthread_create(&threads[0], NULL, allocate_id_thread, &id_contexts[0]) == 0);
CHECK(pthread_create(&threads[1], NULL, allocate_id_thread, &id_contexts[1]) == 0);
CHECK(pthread_join(threads[0], NULL) == 0 && pthread_join(threads[1], NULL) == 0);
CHECK(id_contexts[0].success && id_contexts[1].success &&
id_contexts[0].id != id_contexts[1].id && id_contexts[0].id > 5 && id_contexts[1].id > 5);
char too_long[LARDON3D_PROJECT_DB_PATH_CAPACITY + 1];
memset(too_long, 'x', sizeof(too_long));
too_long[sizeof(too_long) - 1] = '\0';
CHECK(lardon3d_project_db_open(too_long, &database, error) ==
LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
lardon3d_project_db_close(contexts[0].database);
database = NULL;
CHECK(query_integer(database_path, "SELECT value FROM metadata WHERE key='schema_version'", 5));
CHECK(query_integer(database_path, "SELECT count(*) FROM tasks WHERE task_id=1", 1));
CHECK(lardon3d_project_db_open(database_path, &database, error) == LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_load_artifact(database, "artifact-1", &loaded_artifact) ==
LARDON3D_PROJECT_DB_OK);
lardon3d_project_db_close(database);
CHECK(create_future_database(future_path));
CHECK(lardon3d_project_db_open(future_path, &database, error) ==
LARDON3D_PROJECT_DB_UNSUPPORTED_SCHEMA);
descriptor = open(corrupt_path, O_WRONLY | O_CREAT | O_TRUNC, 0600);
CHECK(descriptor >= 0);
CHECK(write(descriptor, "not sqlite", 10) == 10 && close(descriptor) == 0);
CHECK(lardon3d_project_db_open(corrupt_path, &database, error) == LARDON3D_PROJECT_DB_CORRUPT);
CHECK(create_v1_database(legacy_path));
CHECK(lardon3d_project_db_open(legacy_path, &database, error) == LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_schema_version(database) == 5);
CHECK(lardon3d_project_db_get_project(database, &loaded_project) == LARDON3D_PROJECT_DB_OK &&
strcmp(loaded_project.stable_id, "legacy-project") == 0);
CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK);
CHECK(!task.has_task_kind && task.has_checkpoint &&
strcmp(task.checkpoint.path, "legacy.chk") == 0);
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact", &loaded_artifact) ==
LARDON3D_PROJECT_DB_OK);
lardon3d_project_db_close(database);
database = NULL;
CHECK(query_integer(legacy_path, "SELECT value FROM metadata WHERE key='schema_version'", 5));
CHECK(create_v1_database(failed_migration_path));
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V2", "1", 1) == 0);
CHECK(lardon3d_project_db_open(failed_migration_path, &database, error) ==
LARDON3D_PROJECT_DB_IO_ERROR);
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V2") == 0);
CHECK(query_integer(failed_migration_path,
"SELECT value FROM metadata WHERE key='schema_version'", 1));
CHECK(lardon3d_project_db_open(failed_migration_path, &database, error) ==
LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK &&
!task.has_task_kind);
lardon3d_project_db_close(database);
database = NULL;
CHECK(create_v2_database(v2_path));
CHECK(lardon3d_project_db_open(v2_path, &database, error) == LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK &&
task.has_task_kind && strcmp(task.task_kind, "test.work") == 0);
CHECK(task.has_checkpoint && strcmp(task.checkpoint.path, "legacy.chk") == 0);
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact", &loaded_artifact) ==
LARDON3D_PROJECT_DB_OK);
lardon3d_project_db_close(database);
database = NULL;
CHECK(query_integer(v2_path, "SELECT value FROM metadata WHERE key='schema_version'", 5));
CHECK(create_v2_database(failed_v3_migration_path));
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V3", "1", 1) == 0);
CHECK(lardon3d_project_db_open(failed_v3_migration_path, &database, error) ==
LARDON3D_PROJECT_DB_IO_ERROR);
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V3") == 0);
CHECK(query_integer(failed_v3_migration_path,
"SELECT value FROM metadata WHERE key='schema_version'", 2));
CHECK(query_integer(
failed_v3_migration_path,
"SELECT count(*) FROM sqlite_master WHERE type='table' AND name='image_import_tasks'", 0));
CHECK(create_v3_database(v3_path));
CHECK(lardon3d_project_db_open(v3_path, &database, error) == LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_load_image_import(database, 9, &import_parameters) ==
LARDON3D_PROJECT_DB_OK);
CHECK(strcmp(import_parameters.source_path, "/legacy/source") == 0 &&
import_parameters.scanset_id > 0);
CHECK(lardon3d_project_db_legacy_catalog_pending(database, &legacy_pending) ==
LARDON3D_PROJECT_DB_OK &&
legacy_pending);
CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK &&
task.has_task_kind);
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact", &loaded_artifact) ==
LARDON3D_PROJECT_DB_OK);
lardon3d_project_db_close(database);
database = NULL;
CHECK(query_integer(v3_path, "SELECT value FROM metadata WHERE key='schema_version'", 5));
CHECK(create_v3_database(failed_v4_path));
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V4", "1", 1) == 0);
CHECK(lardon3d_project_db_open(failed_v4_path, &database, error) == LARDON3D_PROJECT_DB_IO_ERROR);
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V4") == 0);
CHECK(query_integer(failed_v4_path, "SELECT value FROM metadata WHERE key='schema_version'", 3));
CHECK(query_integer(failed_v4_path,
"SELECT count(*) FROM sqlite_master WHERE type='table' AND name='scansets'",
0));
CHECK(create_v4_database(v4_path));
Lardon3DProjectDbResult v4_result = lardon3d_project_db_open(v4_path, &database, error);
if (v4_result != LARDON3D_PROJECT_DB_OK) {
fprintf(stderr, "Migration v4 (%d): %s\n", (int)v4_result, error);
}
CHECK(v4_result == LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_schema_version(database) == 5);
CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact", &loaded_artifact) ==
LARDON3D_PROJECT_DB_OK);
lardon3d_project_db_close(database);
database = NULL;
CHECK(create_v4_database(failed_v5_path));
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V5", "1", 1) == 0);
CHECK(lardon3d_project_db_open(failed_v5_path, &database, error) == LARDON3D_PROJECT_DB_IO_ERROR);
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V5") == 0);
CHECK(query_integer(failed_v5_path, "SELECT value FROM metadata WHERE key='schema_version'", 4));
CHECK(query_integer(
failed_v5_path,
"SELECT count(*) FROM sqlite_master WHERE type='table' AND name='feature_sets'", 0));
CHECK(unlink(artifact_path) == 0);
CHECK(unlink(database_path) == 0);
CHECK(unlink(future_path) == 0);
CHECK(unlink(corrupt_path) == 0);
CHECK(unlink(legacy_path) == 0);
CHECK(unlink(failed_migration_path) == 0);
CHECK(unlink(v2_path) == 0);
CHECK(unlink(failed_v3_migration_path) == 0);
CHECK(unlink(v3_path) == 0);
CHECK(unlink(failed_v4_path) == 0);
CHECK(unlink(v4_path) == 0);
CHECK(unlink(failed_v5_path) == 0);
CHECK(rmdir(directory) == 0);
return true;
}
int main(void) {
return run_test() ? EXIT_SUCCESS : EXIT_FAILURE;
}