diff --git a/README.md b/README.md index 9276dd4..e8e1e91 100644 --- a/README.md +++ b/README.md @@ -23,10 +23,11 @@ Lardon3D ne vise pas simplement "dossier de photos → objet 3D", mais "ensemble - **Project** : cycle de vie persistant, identité stable et Project Database ouverte - **Import** : premier task kind de production, exécuté par la file générique en lots bornés et reprenables - **ScanSet / Image Catalog v1** : acquisitions, images logiques, provenance et assets SHA-256 persistants et paginés +- **Feature Store v1** : extraction ORB réelle, assets binaires versionnés et lecture bornée - **Image View** : vues triées et filtrées pour la TUI - **Task** : moteur de tâches avec pause/reprise, annulation et séquences - **Task Checkpoint v1** : snapshot durable, fichier atomique et reprise sûre -- **Project Database v4** : tâches, ScanSets, catalogue image, assets et migrations SQLite +- **Project Database v5** : tâches, ScanSets, images, Feature Sets/assets et migrations SQLite - **Task Kind Registry** : identité métier durable et reconstruction runtime explicite - **Recovery projet** : reprise automatique sélective et bornée des imports récupérables - **Task Queue** : file FIFO avec sélection adaptative et backpressure @@ -102,6 +103,7 @@ Acquisitions - [Pipeline de reconstruction](docs/architecture/reconstruction_pipeline.md) - [Persistance](docs/architecture/persistence.md) - [Base de données projet](docs/architecture/project_database.md) +- [Feature Store](docs/architecture/feature_store.md) - [Viewer](docs/architecture/viewer.md) - [Revue des fondations](docs/architecture/foundation_review.md) diff --git a/docs/architecture/feature_store.md b/docs/architecture/feature_store.md new file mode 100644 index 0000000..14bd4dd --- /dev/null +++ b/docs/architecture/feature_store.md @@ -0,0 +1,117 @@ +# Feature Store v1 + +## Rôle et modèle + +**IMPLEMENTED.** Le Feature Store est la mémoire visuelle locale persistante. +Un `FeatureSet` logique appartient à une `image_id` et identifie exactement +`orb` version 1 plus une configuration canonique. Il référence un +`FeatureAsset` physique immutable. Deux images logiques portant le même contenu +et la même configuration gardent deux `feature_set_id`, mais partagent le même +asset lorsque leurs octets sont identiques. Les futurs matches référenceront +`feature_set_id + feature_index`; ces deux valeurs sont immuables après +publication. + +`feature_set_id` et `feature_asset_id` sont des identités SQLite +`AUTOINCREMENT`. Une identité validée n'est jamais réattribuée. SQLite ne +contient que les relations, empreintes, tailles et paramètres ; les tableaux de +points et descripteurs restent hors DB. + +## Extracteur et configuration + +L'extracteur production est ORB d'OpenCV 5, derrière une façade C. Aucune +exception C++ ni `cv::Mat` ne traverse cette frontière. La registry statique +expose `features.extract`, version 1. Une tâche traite exactement une image. + +La configuration v1 contient trois `uint32` : `max_features` (1..8192), +`pyramid_levels` (1..16) et `fast_threshold` (1..255). Son fingerprint SHA-256 +porte sur les 24 octets canoniques `L3DORBP1`, version et trois entiers +little-endian. Il ne dépend ni du padding, ni de la locale. Le contrat ORB v1 +produit 32 octets binaires par point. + +OpenCV n'est pas configuré via un état global par Lardon3D. Le scheduler actuel +n'a qu'un worker ; OpenCV peut néanmoins employer son backend parallèle +interne. Le contrôle fin de ce parallélisme devra précéder les futurs pools de +workers. + +La limite de 100 000 000 pixels est vérifiée après `cv::imread` : l'API utilisée +ne fournit pas de sonde de dimensions multi-format fiable sans décodage. Le pic +mémoire du décodage peut donc précéder le rejet. Lardon3D ne revendique pas de +bornage pré-décodage et n'embarque pas un parseur JPEG/PNG parallèle. + +## Feature File v1 + +Le fichier est little-endian et exige IEEE-754 binary32. Sa taille maximale est +16 Mio et son nombre maximal de points 8192. + +| Offset | Taille | Champ | +|---:|---:|---| +| 0 | 8 | magic `L3DFEAT\0` | +| 8 | 4 | format version = 1 | +| 12 | 4 | header size = 160 | +| 16 | 4 | feature count | +| 20 | 4 | descriptor dimension = 32 | +| 24 | 4 | descriptor type = U8 | +| 28 | 4 | keypoint record size = 24 | +| 32,36 | 4+4 | largeur, hauteur décodées | +| 40,48,56 | 8+8+8 | offsets keypoints/descriptors, taille totale | +| 64 | 32 | SHA-256 de l'asset image source | +| 96 | 32 | fingerprint paramètres | +| 128 | 16 | `orb\0` puis zéros réservés | +| 144 | 4 | extractor version = 1 | +| 148 | 12 | réservés, zéro obligatoire | + +Chaque keypoint contient six mots de 32 bits : `x`, `y`, `size`, orientation, +response en binary32, puis octave signé. `x/y` sont en pixels de l'image telle +que décodée par OpenCV, origine en haut à gauche. `size` est le diamètre du +voisinage en pixels. L'orientation est en degrés dans `[0,360)`. L'entrée `i` +du bloc keypoints correspond exactement aux 32 octets du descriptor `i`. + +Le validateur contrôle magic, version, réservés, type, dimension, bornes, +offsets, taille exacte, multiplications, SHA-256 externe et cohérence DB. Une +version future est distinguée d'une corruption lorsque le fichier et son record +DB sont cohérents. Le reader utilise `pread`, accepte au plus 256 éléments par +appel et ne charge jamais le fichier entier. + +Le chemin DB est validé sous sa forme canonique exacte dérivée du SHA-256 avant +toute ouverture. Une absence retourne `NOT_FOUND`; troncature, hash divergent, +header ou métadonnées DB divergents retournent `CORRUPT`. Une version future +retourne distinctement `UNSUPPORTED_VERSION`. + +## Publication et persistance + +Layout : `assets/features/<2 hex>/`. Le SHA-256 porte +sur le Feature File complet. Le protocole est : temporaire local, écriture, +`fsync`, hash, `link` atomique sans écrasement, validation complète lors d'une +adoption concurrente, `fsync` du répertoire, puis transaction SQLite. Un échec +du dernier `fsync` est enregistré `PUBLISHED_NOT_DURABLE`. Un échec SQLite après +publication laisse un fichier orphelin et aucune ligne logique partielle. +Un nouvel essai qui revalide l'asset et réussit le `fsync` promeut explicitement +sa durabilité vers `DURABLE` dans la transaction DB. + +Le schéma v5 sépare `feature_assets`, `feature_sets` et +`feature_extract_tasks`. L'unicité logique porte sur image, kind, version et +fingerprint. Une tâche est persistée avant enqueue, passe par la queue et le +Governor (CPU, IO, 576 Mio conservateurs, lot 1), puis checkpointée initialement +et terminalement. La reprise recommence l'image entière : il n'existe pas de +fausse reprise intra-ORB. La publication est idempotente. + +Pause et annulation sont coopératives avant/après l'appel ORB ; cet appel n'est +pas interruptible. L'image gérée est rehashée avant extraction. Une image +uniforme produit légitimement un Feature Set READY vide. + +Lardon3D ne garantit pas des octets ORB identiques entre versions d'OpenCV, +plateformes ou backends. L'idempotence porte sur l'environnement courant et le +contrat `extractor_version`; une évolution qui change durablement la sémantique +ou les octets exige d'auditer et, si nécessaire, d'incrémenter cette version. + +## Statut + +**IMPLEMENTED** — ORB réel, format v1, assets content-addressed, DB v5, +publication atomique, reader borné, task kind production et reprise automatique. + +**NOT_YET_WIRED** — commande de réconciliation/scrub des orphelins, contrôle fin +du backend parallèle OpenCV, orientation EXIF, lancement automatique de +l'extraction après import et planification multi-image/DAG. + +**PLANNED** — Visual Index, paires candidates, matching, vérification +géométrique, tracks et SfM. diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 8d341ab..3b568ed 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -62,6 +62,12 @@ catalogue, le manifeste ou les images. **Statut :** IMPLEMENTED +### Feature Store +Extraction ORB réelle par tâche persistante, Feature Sets logiques et assets +binaires content-addressed lisibles par plages bornées. + +**Statut :** IMPLEMENTED + ### Task Moteur de tâches avec états, progression, pause/reprise coopérative, annulation, checkpoints et estimations de ressources. diff --git a/docs/architecture/persistence.md b/docs/architecture/persistence.md index a314f55..d59cb88 100644 --- a/docs/architecture/persistence.md +++ b/docs/architecture/persistence.md @@ -108,7 +108,7 @@ par le `time_t` local avant conversion. Le format reste donc lisible entre plateformes uniquement pour les valeurs communes à leurs domaines `size_t` et `time_t`. -## Project Database v4 +## Project Database v5 SQLite contient l'état logique interrogable et les références aux fichiers ; les checkpoints et artefacts volumineux restent externes. L'enregistrement du @@ -147,8 +147,8 @@ code. **IMPLEMENTED** — modèle durable, codec v1, lecture validée, publication atomique et restauration sûre d'une tâche isolée. -**IMPLEMENTED** — Project Database v4 pour identité, tâches typées, ScanSets, -images logiques, assets SHA-256, références checkpoint et artefacts génériques. +**IMPLEMENTED** — Project Database v5 pour identité, tâches typées, ScanSets, +images logiques, Feature Sets/assets SHA-256, checkpoints et artefacts génériques. **IMPLEMENTED** — registry statique bornée et reconstruction explicite avec ownership du userdata. @@ -205,5 +205,8 @@ retry piloté par l'utilisateur pour les sources indisponibles. **NOT_YET_WIRED** — migration de la vue TUI en mémoire vers la pagination SQLite, scrub des assets et réconciliation globale des orphelins. -**PLANNED** — Feature Store, Visual Index, migrations v5+ et reprise avec +**IMPLEMENTED** — Feature Store externe versionné, immutable, borné et relié +transactionnellement à ses métadonnées SQLite après publication. + +**PLANNED** — Visual Index, migrations v6+ et reprise avec dépendances. diff --git a/docs/architecture/project_database.md b/docs/architecture/project_database.md index 6accae0..e1d5ea8 100644 --- a/docs/architecture/project_database.md +++ b/docs/architecture/project_database.md @@ -168,9 +168,9 @@ référence vers le fichier checkpoint. Le fichier checkpoint validé reste la source complète pour `lardon3d_task_restore()` ; la DB seule ne reconstruit jamais une tâche. Un écart ou un fichier invalide interdit la reprise. -## Schéma v4 implémenté +## Schéma v5 implémenté -- `metadata(key PRIMARY KEY, value)` contient `schema_version=4` et +- `metadata(key PRIMARY KEY, value)` contient `schema_version=5` et `next_task_id`, prochain ID durable allouable. - `project(singleton=1, stable_id UNIQUE, name, created_at, updated_at)` décrit l'unique identité logique de la DB. @@ -209,12 +209,12 @@ SHA-256 et le chemin asset sont déjà indexés par leurs contraintes `UNIQUE`. ## Ouverture et migrations -Une DB vide reçoit directement le schéma v4 dans une transaction +Une DB vide reçoit directement le schéma v5 dans une transaction `BEGIN IMMEDIATE`. Une DB v1 reçoit transactionnellement les colonnes nullable `task_kind` et `task_kind_version`, puis les migrations v2→v3. Les anciennes lignes restent `NULL/NULL`, sans type inventé et sans perte des projets, tâches, checkpoints ou artefacts. Une interruption ou erreur provoque un rollback complet. Les DB v1, -v2 et v3 sont migrées séquentiellement vers v4. Une version future est refusée et une DB contenant +v2, v3 et v4 sont migrées séquentiellement vers v5. Une version future est refusée et une DB contenant des tables sans métadonnée de version est considérée corrompue. La fonction interne de migration ne connaît que `0 → 4`, `1 → 2 → 3 → 4`, `2 → 3 → 4` et `3 → 4`. @@ -293,7 +293,7 @@ UPDATE metadata SET value=4 WHERE key='schema_version' AND value=3; ``` -Configuration v4 : `foreign_keys=ON`, `journal_mode=DELETE`, +Configuration v5 : `foreign_keys=ON`, `journal_mode=DELETE`, `synchronous=FULL`, `busy_timeout=5000`. Le mode DELETE convient au propriétaire unique actuel, évite les fichiers WAL/SHM durables et conserve la synchronisation forte. Le timeout borne l'attente d'un verrou externe à cinq secondes. @@ -354,7 +354,7 @@ ouvert. ## Statut -**IMPLEMENTED** — SQLite système, schéma v4 et migrations v1→v2→v3→v4, identité +**IMPLEMENTED** — SQLite système, schéma v5 et migrations v1→v2→v3→v4→v5, identité projet, transactions tâche+checkpoint, pagination de reprise et artefacts génériques. @@ -382,7 +382,7 @@ cataloguées », pas « images migrées ». **NOT_YET_WIRED** — autosave à toutes les transitions, retry UI des sources indisponibles, migration de la TUI legacy et réconciliation des fichiers -orphelins, Feature Store et Visual Index. +orphelins et Visual Index. Le Feature Store v1 est implémenté. **PLANNED** — migrations v5+, dépendances d'artefacts, graphe géométrique et reconstruction incrémentale. diff --git a/docs/architecture/reconstruction_pipeline.md b/docs/architecture/reconstruction_pipeline.md index e5f21d8..fdb0257 100644 --- a/docs/architecture/reconstruction_pipeline.md +++ b/docs/architecture/reconstruction_pipeline.md @@ -42,15 +42,16 @@ content-addressed. Les états Feature/Matching/Reconstruction restent planifiés --- -### C. Feature Store +### C. Feature Extraction et Feature Store | Aspect | Description | |--------|-------------| -| **Index / métadonnées persistants** | Les descripteurs de features (points clés, descripteurs, orientations) sont persistés sur disque dans un format binaire compact. Le rechargement évite la re-extraction. | -| **Données numériques massives** | Descripteurs floats, coordonnées de keypoints : volumes potentiellement importants. Doivent être stockés de manière séquentielle et indexée. | -| **Formats adaptés et bornés** | Format binaire avec en-tête (version, nombre de features, dimensions). Borné par le budget RAM du governor : si le store dépasse la capacité, seuls les N plus récents sont en mémoire. | +| **Extraction** | ORB OpenCV réel, une image par tâche `features.extract`. | +| **Métadonnées persistantes** | `FeatureSet` logique et `FeatureAsset` physique content-addressed dans ProjectDb v5. | +| **Données numériques massives** | Keypoints et descripteurs U8×32 restent hors SQLite dans Feature File v1. | +| **Lecture bornée** | Le reader lit au plus 256 features par plage et ne charge pas tout le fichier. | -**Statut :** PLANNED — aucune implémentation existante. +**Statut :** IMPLEMENTED v1 — extraction, publication atomique, reprise et reader borné. --- @@ -212,7 +213,11 @@ Image Catalog (B) ──► Feature Store (C) --- -## Statut : PLANNED (vision architecturale) +## Statut du pipeline + +Import, Image Catalog, Feature Extraction et Feature Store sont +**IMPLEMENTED**. Visual Index, Candidate Pair, Matching, Tracks et SfM sont +**PLANNED**. Ce document décrit la vision architecturale cible du pipeline de reconstruction. Les modules listés ici ne sont pas tous implémentés. diff --git a/docs/architecture/resource_governor.md b/docs/architecture/resource_governor.md index 4d265d9..55dcfa3 100644 --- a/docs/architecture/resource_governor.md +++ b/docs/architecture/resource_governor.md @@ -73,6 +73,11 @@ Le Resource Governor est l'unique propriétaire des budgets (RAM, GPU, CPU, IO). l'échantillon peut conserver taille/durée mais n'alimente jamais l'adaptation mémoire. - Pas de communication inter-classes de tâches +- `features.extract` réserve un lot de 1, un thread CPU et un slot I/O, avec + 64 Mio fixes et 512 Mio par image. Cette estimation conservatrice couvre le + chemin actuel sans prétendre mesurer les allocations internes d'OpenCV. + `record_batch` couvre la validation source, le décodage, ORB, la publication + et la finalisation DB ; `peak_memory_bytes == 0` signifie « mesure inconnue ». ## Limites actuelles diff --git a/docs/architecture/task_kind_registry.md b/docs/architecture/task_kind_registry.md index 5a80c9a..47f1a2d 100644 --- a/docs/architecture/task_kind_registry.md +++ b/docs/architecture/task_kind_registry.md @@ -26,7 +26,7 @@ fin de l'exécution. Le constructeur métier n'est jamais appelé sous mutex DB. ## Persistance et legacy -Le checkpoint générique reste en version 1. Project Database v4 conserve le +Le checkpoint générique reste en version 1. Project Database v5 conserve le kind/version ; les lignes migrées depuis v1 restent `NULL/NULL` et sont classées `LEGACY_UNTYPED`. Un kind inconnu ou une version non supportée reste inspectable mais inexécutable. Aucun type n'est inventé et aucun code n'est sélectionné par @@ -46,5 +46,8 @@ pour restaurer hors mutex DB et transférer chaque tâche acceptée à la queue. **NOT_YET_WIRED** — autosave complet et réconciliation orpheline. -**PLANNED** — kinds des tâches ScanSet, Image Catalog, Feature Store, Visual +**IMPLEMENTED** — `features.extract` version 1 reconstruit une extraction ORB +depuis `image_id` et ses paramètres bornés. + +**PLANNED** — kinds des tâches Visual Index et reconstruction lorsque ces traitements existeront réellement. diff --git a/docs/architecture/task_system.md b/docs/architecture/task_system.md index 3d5a9a2..2bd24ff 100644 --- a/docs/architecture/task_system.md +++ b/docs/architecture/task_system.md @@ -106,7 +106,7 @@ localement : son userdata est détruit, sans callback terminal ni écriture durable d'une fausse annulation. Une annulation explicitement demandée conserve le contrat de notification terminale. -La Project Database v4 peut enregistrer transactionnellement un résumé +La Project Database v5 peut enregistrer transactionnellement un résumé `Lardon3DTaskDurableSnapshot` et la référence de son checkpoint. Elle ne stocke ni estimation sérialisée complète, ni callback, ni réservation, et ne remplace pas la validation du fichier checkpoint avant `task_restore()`. diff --git a/docs/concepts/matching_and_tracks.md b/docs/concepts/matching_and_tracks.md index abd83b1..12c92e0 100644 --- a/docs/concepts/matching_and_tracks.md +++ b/docs/concepts/matching_and_tracks.md @@ -33,6 +33,11 @@ au ScanSet, résultats du Visual Index et provenance. Une proximité temporelle pourra servir de signal secondaire ; la proximité dans un dossier et le nom de fichier ne constituent jamais l'identité principale. +Une observation de feature utilisera la référence durable +`feature_set_id + feature_index`. Une fois le Feature Set READY, +`feature_index` désigne immuablement la paire keypoint/descripteur de même +indice ; aucun chemin source n'entre dans cette identité. + ## Concepts clés ### Étapes du matching diff --git a/docs/concepts/visual_index.md b/docs/concepts/visual_index.md index 65218f7..9b68ae4 100644 --- a/docs/concepts/visual_index.md +++ b/docs/concepts/visual_index.md @@ -26,10 +26,10 @@ Matching & Tracks (appariement détaillé) L'index visuel se situe entre l'extraction de features et la sélection de paires candidates. Il réduit drastiquement l'espace de recherche pour l'appariement. -Le futur index référencera exclusivement les `image_id` stables du catalogue -persistant. Un chemin d'asset ou un nom source n'est ni une identité d'image ni -une clé d'index durable. Le partage d'un asset entre deux ScanSets n'empêche pas -leurs deux images logiques d'être des observations distinctes. +Le futur index consommera exclusivement un `feature_set_id` et les descripteurs +obtenus par plages bornées du Feature Reader. Il ne dépendra ni des chemins +source, ni d'OpenCV, ni du runtime de la tâche productrice. Un chemin d'asset ou +un nom source n'est pas une clé d'index durable. ## Concepts clés diff --git a/docs/development/build.md b/docs/development/build.md index 925c2a5..66627a2 100644 --- a/docs/development/build.md +++ b/docs/development/build.md @@ -57,9 +57,8 @@ git diff --check allocations : ```sh -meson setup build-asan --wipe \ - -Db_sanitize=address,undefined \ - -Db_static=false +CC=clang meson setup build-asan --wipe \ + -Db_sanitize=address,undefined meson compile -C build-asan -j8 meson test -C build-asan --print-errorlogs ``` @@ -70,9 +69,9 @@ meson test -C build-asan --print-errorlogs variables de condition, états partagés) : ```sh -meson setup build-tsan --wipe \ +CC=clang meson setup build-tsan --wipe \ -Db_sanitize=thread \ - -Db_static=false + -Db_lundef=false meson compile -C build-tsan -j8 meson test -C build-tsan --print-errorlogs ``` diff --git a/docs/development/concurrency.md b/docs/development/concurrency.md index 6361519..94c09f1 100644 --- a/docs/development/concurrency.md +++ b/docs/development/concurrency.md @@ -160,7 +160,7 @@ Pour tout ticket touchant la concurrence, exécuter : ```sh # Build TSan -meson setup build-tsan --wipe -Db_sanitize=thread -Db_static=false +CC=clang meson setup build-tsan --wipe -Db_sanitize=thread -Db_lundef=false meson compile -C build-tsan -j8 meson test -C build-tsan --print-errorlogs ``` diff --git a/docs/development/testing.md b/docs/development/testing.md index 6beac38..1d85c9b 100644 --- a/docs/development/testing.md +++ b/docs/development/testing.md @@ -95,12 +95,12 @@ meson test -C build --print-errorlogs git diff --check # 4. Si mémoire/concurrence touchés -meson setup build-asan --wipe -Db_sanitize=address,undefined -Db_static=false +CC=clang meson setup build-asan --wipe -Db_sanitize=address,undefined meson compile -C build-asan -j8 meson test -C build-asan --print-errorlogs # 5. Si concurrence touchée -meson setup build-tsan --wipe -Db_sanitize=thread -Db_static=false +CC=clang meson setup build-tsan --wipe -Db_sanitize=thread -Db_lundef=false meson compile -C build-tsan -j8 meson test -C build-tsan --print-errorlogs ``` diff --git a/docs/roadmap/roadmap.md b/docs/roadmap/roadmap.md index a832ae0..7cb1995 100644 --- a/docs/roadmap/roadmap.md +++ b/docs/roadmap/roadmap.md @@ -29,7 +29,7 @@ Lardon3D suit une feuille de route ordonnée qui privilégie la stabilité et la ### Phase 3 : Persistance - ✅ Fondation versionnée des checkpoints de tâches -- ✅ Project Database v4 (tâches, ScanSets, images, assets et migrations) +- ✅ Project Database v5 (tâches, ScanSets, images, Feature Sets/assets et migrations) - ✅ Branchement Project Database au cycle de vie projet et inventaire de reprise - ✅ Registry durable des types métier de tâches - ✅ Premier type métier reconstructible (`import.images`) @@ -37,7 +37,7 @@ Lardon3D suit une feuille de route ordonnée qui privilégie la stabilité et la - ✅ ScanSet v1 et Image Catalog persistant v1 ### Phase 4 : Pipeline avancé -- 📋 Feature Store +- ✅ Feature Store v1 et extraction ORB persistante - 📋 Visual Index - 📋 Candidate Pair Generator - 📋 Matching et vérification géométrique diff --git a/include/lardon3d/feature_extractor.h b/include/lardon3d/feature_extractor.h new file mode 100644 index 0000000..a44da3f --- /dev/null +++ b/include/lardon3d/feature_extractor.h @@ -0,0 +1,57 @@ +#ifndef LARDON3D_FEATURE_EXTRACTOR_H +#define LARDON3D_FEATURE_EXTRACTOR_H + +#include +#include +#include + +#define LARDON3D_FEATURE_EXTRACTOR_KIND "orb" +enum { + LARDON3D_FEATURE_EXTRACTOR_VERSION = 1, + LARDON3D_FEATURE_DESCRIPTOR_DIMENSION = 32, + LARDON3D_FEATURE_MAX_FEATURES = 8192, + LARDON3D_FEATURE_MAX_IMAGE_PIXELS = 100000000, +}; + +typedef struct { + uint32_t max_features; + uint32_t pyramid_levels; + uint32_t fast_threshold; +} Lardon3DFeatureExtractorParameters; + +typedef struct { + float x; + float y; + float size; + float angle_degrees; + float response; + int32_t octave; +} Lardon3DFeatureKeypoint; + +typedef struct { + uint32_t image_width; + uint32_t image_height; + uint32_t feature_count; + Lardon3DFeatureKeypoint *keypoints; + unsigned char *descriptors; +} Lardon3DExtractedFeatures; + +typedef enum { + LARDON3D_FEATURE_EXTRACT_OK = 0, + LARDON3D_FEATURE_EXTRACT_INVALID_ARGUMENT, + LARDON3D_FEATURE_EXTRACT_IMAGE_NOT_FOUND, + LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID, + LARDON3D_FEATURE_EXTRACT_OUT_OF_MEMORY, + LARDON3D_FEATURE_EXTRACT_ERROR +} Lardon3DFeatureExtractResult; + +bool lardon3d_feature_extractor_parameters_valid( + const Lardon3DFeatureExtractorParameters *parameters); +void lardon3d_feature_extractor_parameter_fingerprint( + const Lardon3DFeatureExtractorParameters *parameters, unsigned char fingerprint[32]); +Lardon3DFeatureExtractResult +lardon3d_feature_extract_orb(const char *path, const Lardon3DFeatureExtractorParameters *parameters, + Lardon3DExtractedFeatures *features); +void lardon3d_extracted_features_destroy(Lardon3DExtractedFeatures *features); + +#endif diff --git a/include/lardon3d/feature_store.h b/include/lardon3d/feature_store.h new file mode 100644 index 0000000..2b4e488 --- /dev/null +++ b/include/lardon3d/feature_store.h @@ -0,0 +1,73 @@ +#ifndef LARDON3D_FEATURE_STORE_H +#define LARDON3D_FEATURE_STORE_H + +#include +#include +#include + +#include +#include +#include + +enum { + LARDON3D_FEATURE_FILE_VERSION = 1, + LARDON3D_FEATURE_FILE_HEADER_SIZE = 160, + LARDON3D_FEATURE_KEYPOINT_RECORD_SIZE = 24, + LARDON3D_FEATURE_READER_RANGE_MAX = 256, + LARDON3D_FEATURE_FILE_MAX_SIZE = 16 * 1024 * 1024, +}; + +typedef enum { + LARDON3D_FEATURE_DESCRIPTOR_U8 = 1, + LARDON3D_FEATURE_DESCRIPTOR_F32 = 2 +} Lardon3DFeatureDescriptorType; + +typedef enum { + LARDON3D_FEATURE_STORE_OK = 0, + LARDON3D_FEATURE_STORE_ALREADY_PRESENT, + LARDON3D_FEATURE_STORE_INVALID_ARGUMENT, + LARDON3D_FEATURE_STORE_NOT_FOUND, + LARDON3D_FEATURE_STORE_INVALID, + LARDON3D_FEATURE_STORE_UNSUPPORTED_VERSION, + LARDON3D_FEATURE_STORE_CORRUPT, + LARDON3D_FEATURE_STORE_IO_ERROR, + LARDON3D_FEATURE_STORE_DB_BUSY, + LARDON3D_FEATURE_STORE_DB_ERROR, + LARDON3D_FEATURE_STORE_PUBLISHED_NOT_DURABLE +} Lardon3DFeatureStoreResult; + +typedef struct Lardon3DFeatureReader Lardon3DFeatureReader; + +typedef struct { + uint32_t format_version; + char extractor_kind[LARDON3D_TASK_KIND_CAPACITY]; + uint32_t extractor_version; + uint32_t feature_count; + uint32_t descriptor_dimension; + Lardon3DFeatureDescriptorType descriptor_type; + uint32_t image_width; + uint32_t image_height; + unsigned char source_image_sha256[32]; + unsigned char parameter_fingerprint[32]; +} Lardon3DFeatureFileMetadata; + +Lardon3DFeatureStoreResult lardon3d_feature_store_publish( + Lardon3DAppState *state, uint64_t image_id, uint64_t producer_task_id, + const Lardon3DFeatureExtractorParameters *parameters, const Lardon3DExtractedFeatures *features, + Lardon3DProjectDbFeatureSet *feature_set); +Lardon3DFeatureStoreResult +lardon3d_feature_reader_open(const char *project_path, + const Lardon3DProjectDbFeatureSet *feature_set, + Lardon3DFeatureReader **reader, Lardon3DFeatureFileMetadata *metadata); +void lardon3d_feature_reader_close(Lardon3DFeatureReader *reader); +Lardon3DFeatureStoreResult lardon3d_feature_reader_keypoints(Lardon3DFeatureReader *reader, + uint32_t start, + Lardon3DFeatureKeypoint *keypoints, + size_t capacity); +Lardon3DFeatureStoreResult lardon3d_feature_reader_descriptors(Lardon3DFeatureReader *reader, + uint32_t start, + unsigned char *descriptors, + size_t feature_capacity, + size_t descriptor_capacity); + +#endif diff --git a/include/lardon3d/feature_task.h b/include/lardon3d/feature_task.h new file mode 100644 index 0000000..625cdc1 --- /dev/null +++ b/include/lardon3d/feature_task.h @@ -0,0 +1,24 @@ +#ifndef LARDON3D_FEATURE_TASK_H +#define LARDON3D_FEATURE_TASK_H + +#include +#include + +#include +#include +#include + +#define LARDON3D_FEATURE_EXTRACT_TASK_KIND "features.extract" +enum { LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION = 1 }; + +Lardon3DTask * +lardon3d_project_create_feature_extract_task(Lardon3DAppState *state, uint64_t image_id, + const Lardon3DFeatureExtractorParameters *parameters, + uint64_t *task_id); +bool lardon3d_project_enqueue_feature_extract(Lardon3DAppState *state, uint64_t image_id, + const Lardon3DFeatureExtractorParameters *parameters, + uint64_t *task_id); +bool lardon3d_feature_extract_reconstruct(const Lardon3DTaskDurableSnapshot *snapshot, + void *context, Lardon3DTaskKindBinding *binding); + +#endif diff --git a/include/lardon3d/import_task.h b/include/lardon3d/import_task.h index 4b35b4f..ad1d61a 100644 --- a/include/lardon3d/import_task.h +++ b/include/lardon3d/import_task.h @@ -12,61 +12,42 @@ enum { LARDON3D_IMAGE_IMPORT_TASK_KIND_VERSION = 1 }; typedef enum { - LARDON3D_IMPORT_TASK_IDLE = 0, - LARDON3D_IMPORT_TASK_RUNNING, - LARDON3D_IMPORT_TASK_SUCCEEDED, - LARDON3D_IMPORT_TASK_CANCELLED, - LARDON3D_IMPORT_TASK_FAILED + LARDON3D_IMPORT_TASK_IDLE = 0, + LARDON3D_IMPORT_TASK_RUNNING, + LARDON3D_IMPORT_TASK_SUCCEEDED, + LARDON3D_IMPORT_TASK_CANCELLED, + LARDON3D_IMPORT_TASK_FAILED } Lardon3DImportTaskStatus; typedef struct { - Lardon3DImportTaskStatus status; - size_t total; - size_t processed; - size_t copied; - size_t already_present; - size_t ignored; - char message[256]; + Lardon3DImportTaskStatus status; + size_t total; + size_t processed; + size_t copied; + size_t already_present; + size_t ignored; + char message[256]; } Lardon3DImportTaskSnapshot; typedef struct Lardon3DImportTask Lardon3DImportTask; -typedef struct { - const char *project_path; - Lardon3DProjectDb *project_db; - Lardon3DResourceGovernor *resource_governor; -} Lardon3DImageImportReconstructionContext; +typedef Lardon3DTaskReconstructionContext Lardon3DImageImportReconstructionContext; -Lardon3DTask *lardon3d_project_create_image_import_task( - Lardon3DAppState *state, - uint64_t scanset_id, - const char *source_directory, - uint64_t *task_id -); -bool lardon3d_project_enqueue_image_import( - Lardon3DAppState *state, - uint64_t scanset_id, - const char *source_directory, - uint64_t *task_id -); -bool lardon3d_image_import_reconstruct( - const Lardon3DTaskDurableSnapshot *snapshot, - void *context, - Lardon3DTaskKindBinding *binding -); +Lardon3DTask *lardon3d_project_create_image_import_task(Lardon3DAppState *state, + uint64_t scanset_id, + const char *source_directory, + uint64_t *task_id); +bool lardon3d_project_enqueue_image_import(Lardon3DAppState *state, uint64_t scanset_id, + const char *source_directory, uint64_t *task_id); +bool lardon3d_image_import_reconstruct(const Lardon3DTaskDurableSnapshot *snapshot, void *context, + Lardon3DTaskKindBinding *binding); /* Compatibilité TUI : handle léger sur une tâche de la queue, sans thread. */ Lardon3DImportTask *lardon3d_import_task_create(void); -bool lardon3d_import_task_start( - Lardon3DImportTask *task, - Lardon3DAppState *state, - const char *source_directory -); +bool lardon3d_import_task_start(Lardon3DImportTask *task, Lardon3DAppState *state, + const char *source_directory); void lardon3d_import_task_request_cancel(Lardon3DImportTask *task); -bool lardon3d_import_task_snapshot( - Lardon3DImportTask *task, - Lardon3DImportTaskSnapshot *snapshot -); +bool lardon3d_import_task_snapshot(Lardon3DImportTask *task, Lardon3DImportTaskSnapshot *snapshot); bool lardon3d_import_task_is_finished(Lardon3DImportTask *task); bool lardon3d_import_task_join(Lardon3DImportTask *task); void lardon3d_import_task_destroy(Lardon3DImportTask *task); diff --git a/include/lardon3d/project.h b/include/lardon3d/project.h index d0f6a4e..0848f81 100644 --- a/include/lardon3d/project.h +++ b/include/lardon3d/project.h @@ -8,84 +8,69 @@ #include typedef enum { - LARDON3D_PROJECT_TASK_CHECKPOINT_OK = 0, - LARDON3D_PROJECT_TASK_CHECKPOINT_PUBLISHED_NOT_DURABLE, - LARDON3D_PROJECT_TASK_CHECKPOINT_NO_PROJECT, - LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK, - LARDON3D_PROJECT_TASK_CHECKPOINT_IO_ERROR, - LARDON3D_PROJECT_TASK_CHECKPOINT_DB_BUSY, - LARDON3D_PROJECT_TASK_CHECKPOINT_DB_ERROR + LARDON3D_PROJECT_TASK_CHECKPOINT_OK = 0, + LARDON3D_PROJECT_TASK_CHECKPOINT_PUBLISHED_NOT_DURABLE, + LARDON3D_PROJECT_TASK_CHECKPOINT_NO_PROJECT, + LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK, + LARDON3D_PROJECT_TASK_CHECKPOINT_IO_ERROR, + LARDON3D_PROJECT_TASK_CHECKPOINT_DB_BUSY, + LARDON3D_PROJECT_TASK_CHECKPOINT_DB_ERROR } Lardon3DProjectTaskCheckpointResult; typedef enum { - LARDON3D_PROJECT_RECOVERABLE = 0, - LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE, - LARDON3D_PROJECT_RECOVERY_MISSING_CHECKPOINT, - LARDON3D_PROJECT_RECOVERY_INVALID_CHECKPOINT, - LARDON3D_PROJECT_RECOVERY_UNSUPPORTED_CHECKPOINT, - LARDON3D_PROJECT_RECOVERY_CHECKPOINT_IO_ERROR, - LARDON3D_PROJECT_RECOVERY_LEGACY_UNTYPED, - LARDON3D_PROJECT_RECOVERY_UNKNOWN_TASK_KIND, - LARDON3D_PROJECT_RECOVERY_UNSUPPORTED_TASK_KIND_VERSION + LARDON3D_PROJECT_RECOVERABLE = 0, + LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE, + LARDON3D_PROJECT_RECOVERY_MISSING_CHECKPOINT, + LARDON3D_PROJECT_RECOVERY_INVALID_CHECKPOINT, + LARDON3D_PROJECT_RECOVERY_UNSUPPORTED_CHECKPOINT, + LARDON3D_PROJECT_RECOVERY_CHECKPOINT_IO_ERROR, + LARDON3D_PROJECT_RECOVERY_LEGACY_UNTYPED, + LARDON3D_PROJECT_RECOVERY_UNKNOWN_TASK_KIND, + LARDON3D_PROJECT_RECOVERY_UNSUPPORTED_TASK_KIND_VERSION } Lardon3DProjectRecoveryStatus; typedef struct { - uint64_t task_id; - char name[LARDON3D_TASK_NAME_CAPACITY]; - char task_kind[LARDON3D_TASK_KIND_CAPACITY]; - uint32_t task_kind_version; - Lardon3DProjectRecoveryStatus status; - Lardon3DProjectDbCheckpointDurability durability; - Lardon3DTaskDurableSnapshot snapshot; + uint64_t task_id; + char name[LARDON3D_TASK_NAME_CAPACITY]; + char task_kind[LARDON3D_TASK_KIND_CAPACITY]; + uint32_t task_kind_version; + Lardon3DProjectRecoveryStatus status; + Lardon3DProjectDbCheckpointDurability durability; + Lardon3DTaskDurableSnapshot snapshot; } Lardon3DProjectRecoveryEntry; typedef struct { - size_t inspected; - size_t resumed; - size_t skipped; - size_t failed; - size_t published_not_durable; - bool queue_full; + size_t inspected; + size_t resumed; + size_t skipped; + size_t failed; + size_t published_not_durable; + bool queue_full; } Lardon3DProjectRecoverySummary; -bool lardon3d_project_create( - Lardon3DAppState *state, - const char *name -); +bool lardon3d_project_create(Lardon3DAppState *state, const char *name); -bool lardon3d_project_open( - Lardon3DAppState *state, - const char *directory_name -); +bool lardon3d_project_open(Lardon3DAppState *state, const char *directory_name); void lardon3d_project_close(Lardon3DAppState *state); -Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_task( - Lardon3DAppState *state, - const Lardon3DTask *task -); +Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_task(Lardon3DAppState *state, + const Lardon3DTask *task); Lardon3DProjectTaskCheckpointResult -lardon3d_project_checkpoint_image_import_task( - Lardon3DAppState *state, - const Lardon3DTask *task, - const char *source_path, - uint64_t scanset_id -); -Lardon3DProjectDbResult lardon3d_project_list_recoverable( - Lardon3DAppState *state, - const Lardon3DTaskKindRegistry *registry, - uint64_t after_task_id, - Lardon3DProjectRecoveryEntry *entries, - size_t capacity, - size_t *count -); -Lardon3DProjectDbResult lardon3d_project_resume_recoverable_tasks( - Lardon3DAppState *state, - const Lardon3DTaskKindRegistry *registry, - Lardon3DProjectRecoverySummary *summary -); -bool lardon3d_project_last_recovery_summary( - const Lardon3DAppState *state, - Lardon3DProjectRecoverySummary *summary -); +lardon3d_project_checkpoint_image_import_task(Lardon3DAppState *state, const Lardon3DTask *task, + const char *source_path, uint64_t scanset_id); +Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_task( + Lardon3DAppState *state, const Lardon3DTask *task, + const Lardon3DProjectDbFeatureExtractTask *parameters); +Lardon3DProjectDbResult lardon3d_project_list_recoverable(Lardon3DAppState *state, + const Lardon3DTaskKindRegistry *registry, + uint64_t after_task_id, + Lardon3DProjectRecoveryEntry *entries, + size_t capacity, size_t *count); +Lardon3DProjectDbResult +lardon3d_project_resume_recoverable_tasks(Lardon3DAppState *state, + const Lardon3DTaskKindRegistry *registry, + Lardon3DProjectRecoverySummary *summary); +bool lardon3d_project_last_recovery_summary(const Lardon3DAppState *state, + Lardon3DProjectRecoverySummary *summary); #endif diff --git a/include/lardon3d/project_db.h b/include/lardon3d/project_db.h index af3582e..0ea185a 100644 --- a/include/lardon3d/project_db.h +++ b/include/lardon3d/project_db.h @@ -9,233 +9,259 @@ #include enum { - LARDON3D_PROJECT_DB_SCHEMA_VERSION = 4, - LARDON3D_PROJECT_DB_ID_CAPACITY = 65, - LARDON3D_PROJECT_DB_KIND_CAPACITY = 65, - LARDON3D_PROJECT_DB_PATH_CAPACITY = 4096, - LARDON3D_PROJECT_DB_ERROR_CAPACITY = 256, - LARDON3D_PROJECT_DB_RECOVERY_PAGE_MAX = 256, - LARDON3D_PROJECT_DB_CATALOG_PAGE_MAX = 256, - LARDON3D_PROJECT_DB_SCANSET_NAME_CAPACITY = 256, - LARDON3D_PROJECT_DB_IMAGE_NAME_CAPACITY = 256, - LARDON3D_PROJECT_DB_SHA256_SIZE = 32, + LARDON3D_PROJECT_DB_SCHEMA_VERSION = 5, + LARDON3D_PROJECT_DB_ID_CAPACITY = 65, + LARDON3D_PROJECT_DB_KIND_CAPACITY = 65, + LARDON3D_PROJECT_DB_PATH_CAPACITY = 4096, + LARDON3D_PROJECT_DB_ERROR_CAPACITY = 256, + LARDON3D_PROJECT_DB_RECOVERY_PAGE_MAX = 256, + LARDON3D_PROJECT_DB_CATALOG_PAGE_MAX = 256, + LARDON3D_PROJECT_DB_SCANSET_NAME_CAPACITY = 256, + LARDON3D_PROJECT_DB_IMAGE_NAME_CAPACITY = 256, + LARDON3D_PROJECT_DB_SHA256_SIZE = 32, }; typedef struct Lardon3DProjectDb Lardon3DProjectDb; typedef enum { - LARDON3D_PROJECT_DB_OK = 0, - LARDON3D_PROJECT_DB_INVALID_ARGUMENT, - LARDON3D_PROJECT_DB_NOT_FOUND, - LARDON3D_PROJECT_DB_BUSY, - LARDON3D_PROJECT_DB_UNSUPPORTED_SCHEMA, - LARDON3D_PROJECT_DB_CORRUPT, - LARDON3D_PROJECT_DB_CONSTRAINT, - LARDON3D_PROJECT_DB_IO_ERROR + LARDON3D_PROJECT_DB_OK = 0, + LARDON3D_PROJECT_DB_INVALID_ARGUMENT, + LARDON3D_PROJECT_DB_NOT_FOUND, + LARDON3D_PROJECT_DB_BUSY, + LARDON3D_PROJECT_DB_UNSUPPORTED_SCHEMA, + LARDON3D_PROJECT_DB_CORRUPT, + LARDON3D_PROJECT_DB_CONSTRAINT, + LARDON3D_PROJECT_DB_IO_ERROR } Lardon3DProjectDbResult; typedef enum { - LARDON3D_DB_CHECKPOINT_DURABLE = 0, - LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE + LARDON3D_DB_CHECKPOINT_DURABLE = 0, + LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE } Lardon3DProjectDbCheckpointDurability; typedef enum { - LARDON3D_DB_ARTIFACT_STAGED = 0, - LARDON3D_DB_ARTIFACT_READY + LARDON3D_DB_ARTIFACT_STAGED = 0, + LARDON3D_DB_ARTIFACT_READY } Lardon3DProjectDbArtifactState; typedef struct { - char stable_id[LARDON3D_PROJECT_DB_ID_CAPACITY]; - char name[LARDON3D_TASK_NAME_CAPACITY]; - int64_t created_at; - int64_t updated_at; + char stable_id[LARDON3D_PROJECT_DB_ID_CAPACITY]; + char name[LARDON3D_TASK_NAME_CAPACITY]; + int64_t created_at; + int64_t updated_at; } Lardon3DProjectDbProject; typedef struct { - char path[LARDON3D_PROJECT_DB_PATH_CAPACITY]; - uint32_t format_version; - Lardon3DProjectDbCheckpointDurability durability; - int64_t updated_at; + char path[LARDON3D_PROJECT_DB_PATH_CAPACITY]; + uint32_t format_version; + Lardon3DProjectDbCheckpointDurability durability; + int64_t updated_at; } Lardon3DProjectDbCheckpoint; typedef struct { - uint64_t task_id; - char name[LARDON3D_TASK_NAME_CAPACITY]; - bool has_task_kind; - char task_kind[LARDON3D_TASK_KIND_CAPACITY]; - uint32_t task_kind_version; - Lardon3DTaskState saved_state; - Lardon3DTaskState recovery_state; - unsigned int progress; - unsigned int sequence_count; - struct timespec started_at; - struct timespec finished_at; - int64_t updated_at; - bool has_checkpoint; - Lardon3DProjectDbCheckpoint checkpoint; + uint64_t task_id; + char name[LARDON3D_TASK_NAME_CAPACITY]; + bool has_task_kind; + char task_kind[LARDON3D_TASK_KIND_CAPACITY]; + uint32_t task_kind_version; + Lardon3DTaskState saved_state; + Lardon3DTaskState recovery_state; + unsigned int progress; + unsigned int sequence_count; + struct timespec started_at; + struct timespec finished_at; + int64_t updated_at; + bool has_checkpoint; + Lardon3DProjectDbCheckpoint checkpoint; } Lardon3DProjectDbTask; typedef struct { - char artifact_id[LARDON3D_PROJECT_DB_ID_CAPACITY]; - char kind[LARDON3D_PROJECT_DB_KIND_CAPACITY]; - char path[LARDON3D_PROJECT_DB_PATH_CAPACITY]; - Lardon3DProjectDbArtifactState state; - uint64_t size_bytes; - bool has_producer_task; - uint64_t producer_task_id; - int64_t created_at; - int64_t updated_at; + char artifact_id[LARDON3D_PROJECT_DB_ID_CAPACITY]; + char kind[LARDON3D_PROJECT_DB_KIND_CAPACITY]; + char path[LARDON3D_PROJECT_DB_PATH_CAPACITY]; + Lardon3DProjectDbArtifactState state; + uint64_t size_bytes; + bool has_producer_task; + uint64_t producer_task_id; + int64_t created_at; + int64_t updated_at; } Lardon3DProjectDbArtifact; typedef struct { - uint64_t task_id; - char source_path[LARDON3D_PROJECT_DB_PATH_CAPACITY]; - uint64_t scanset_id; + uint64_t task_id; + char source_path[LARDON3D_PROJECT_DB_PATH_CAPACITY]; + uint64_t scanset_id; } Lardon3DProjectDbImageImport; typedef struct { - uint64_t scanset_id; - char name[LARDON3D_PROJECT_DB_SCANSET_NAME_CAPACITY]; - int64_t created_at; - int64_t updated_at; + uint64_t scanset_id; + char name[LARDON3D_PROJECT_DB_SCANSET_NAME_CAPACITY]; + int64_t created_at; + int64_t updated_at; } Lardon3DProjectDbScanSet; -typedef enum { - LARDON3D_DB_IMAGE_ASSET_READY = 1 -} Lardon3DProjectDbImageAssetState; +typedef enum { LARDON3D_DB_IMAGE_ASSET_READY = 1 } Lardon3DProjectDbImageAssetState; typedef struct { - uint64_t asset_id; - unsigned char sha256[LARDON3D_PROJECT_DB_SHA256_SIZE]; - char path[LARDON3D_PROJECT_DB_PATH_CAPACITY]; - uint64_t size_bytes; - Lardon3DProjectDbImageAssetState state; - int64_t created_at; + uint64_t asset_id; + unsigned char sha256[LARDON3D_PROJECT_DB_SHA256_SIZE]; + char path[LARDON3D_PROJECT_DB_PATH_CAPACITY]; + uint64_t size_bytes; + Lardon3DProjectDbImageAssetState state; + int64_t created_at; } Lardon3DProjectDbImageAsset; typedef struct { - uint64_t image_id; - uint64_t scanset_id; - uint64_t asset_id; - char original_name[LARDON3D_PROJECT_DB_IMAGE_NAME_CAPACITY]; - char source_path[LARDON3D_PROJECT_DB_PATH_CAPACITY]; - bool has_producer_task; - uint64_t producer_task_id; - int64_t imported_at; + uint64_t image_id; + uint64_t scanset_id; + uint64_t asset_id; + char original_name[LARDON3D_PROJECT_DB_IMAGE_NAME_CAPACITY]; + char source_path[LARDON3D_PROJECT_DB_PATH_CAPACITY]; + bool has_producer_task; + uint64_t producer_task_id; + int64_t imported_at; } Lardon3DProjectDbImage; typedef enum { - LARDON3D_PROJECT_DB_IMAGE_REGISTERED = 0, - LARDON3D_PROJECT_DB_IMAGE_ALREADY_PRESENT + LARDON3D_PROJECT_DB_IMAGE_REGISTERED = 0, + LARDON3D_PROJECT_DB_IMAGE_ALREADY_PRESENT } Lardon3DProjectDbImageRegisterStatus; -Lardon3DProjectDbResult lardon3d_project_db_open( - const char *path, - Lardon3DProjectDb **database, - char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY] -); +typedef enum { + LARDON3D_DB_FEATURE_ASSET_DURABLE = 0, + LARDON3D_DB_FEATURE_ASSET_PUBLISHED_NOT_DURABLE = 1 +} Lardon3DProjectDbFeatureDurability; + +typedef struct { + uint64_t feature_asset_id; + unsigned char sha256[32]; + char path[LARDON3D_PROJECT_DB_PATH_CAPACITY]; + uint64_t size_bytes; + Lardon3DProjectDbFeatureDurability durability; + int64_t created_at; +} Lardon3DProjectDbFeatureAsset; + +typedef struct { + uint64_t feature_set_id; + uint64_t image_id; + uint64_t feature_asset_id; + char extractor_kind[LARDON3D_PROJECT_DB_KIND_CAPACITY]; + uint32_t extractor_version; + unsigned char parameter_fingerprint[32]; + unsigned char source_image_sha256[32]; + uint32_t feature_count; + uint32_t descriptor_type; + uint32_t descriptor_dimension; + bool has_producer_task; + uint64_t producer_task_id; + int64_t created_at; + Lardon3DProjectDbFeatureAsset asset; +} Lardon3DProjectDbFeatureSet; + +typedef struct { + uint64_t task_id; + uint64_t image_id; + char extractor_kind[LARDON3D_PROJECT_DB_KIND_CAPACITY]; + uint32_t extractor_version; + uint32_t max_features; + uint32_t pyramid_levels; + uint32_t fast_threshold; + unsigned char parameter_fingerprint[32]; +} Lardon3DProjectDbFeatureExtractTask; + +Lardon3DProjectDbResult lardon3d_project_db_open(const char *path, Lardon3DProjectDb **database, + char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]); void lardon3d_project_db_close(Lardon3DProjectDb *database); -bool lardon3d_project_db_last_error( - Lardon3DProjectDb *database, - char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY] -); -Lardon3DProjectDbResult lardon3d_project_db_legacy_catalog_pending( - Lardon3DProjectDb *database, bool *pending -); +bool lardon3d_project_db_last_error(Lardon3DProjectDb *database, + char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]); +Lardon3DProjectDbResult lardon3d_project_db_legacy_catalog_pending(Lardon3DProjectDb *database, + bool *pending); unsigned int lardon3d_project_db_schema_version(Lardon3DProjectDb *database); -Lardon3DProjectDbResult lardon3d_project_db_set_project( - Lardon3DProjectDb *database, - const Lardon3DProjectDbProject *project -); -Lardon3DProjectDbResult lardon3d_project_db_get_project( - Lardon3DProjectDb *database, - Lardon3DProjectDbProject *project -); +Lardon3DProjectDbResult lardon3d_project_db_set_project(Lardon3DProjectDb *database, + const Lardon3DProjectDbProject *project); +Lardon3DProjectDbResult lardon3d_project_db_get_project(Lardon3DProjectDb *database, + Lardon3DProjectDbProject *project); Lardon3DProjectDbResult lardon3d_project_db_record_task( - Lardon3DProjectDb *database, - const Lardon3DTaskDurableSnapshot *snapshot, - const char *task_kind, - uint32_t task_kind_version, - const Lardon3DProjectDbCheckpoint *checkpoint, - int64_t updated_at -); + Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, const char *task_kind, + uint32_t task_kind_version, const Lardon3DProjectDbCheckpoint *checkpoint, int64_t updated_at); Lardon3DProjectDbResult lardon3d_project_db_record_image_import_task( - Lardon3DProjectDb *database, - const Lardon3DTaskDurableSnapshot *snapshot, - const char *task_kind, - uint32_t task_kind_version, - const Lardon3DProjectDbCheckpoint *checkpoint, - const char *source_path, - uint64_t scanset_id, - int64_t updated_at -); -Lardon3DProjectDbResult lardon3d_project_db_create_scanset( - Lardon3DProjectDb *database, const char *name, - Lardon3DProjectDbScanSet *scanset -); -Lardon3DProjectDbResult lardon3d_project_db_load_scanset( - Lardon3DProjectDb *database, uint64_t scanset_id, - Lardon3DProjectDbScanSet *scanset -); -Lardon3DProjectDbResult lardon3d_project_db_list_scansets( - Lardon3DProjectDb *database, uint64_t after_scanset_id, - Lardon3DProjectDbScanSet *scansets, size_t capacity, size_t *count -); + Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, const char *task_kind, + uint32_t task_kind_version, const Lardon3DProjectDbCheckpoint *checkpoint, + const char *source_path, uint64_t scanset_id, int64_t updated_at); +Lardon3DProjectDbResult lardon3d_project_db_create_scanset(Lardon3DProjectDb *database, + const char *name, + Lardon3DProjectDbScanSet *scanset); +Lardon3DProjectDbResult lardon3d_project_db_load_scanset(Lardon3DProjectDb *database, + uint64_t scanset_id, + Lardon3DProjectDbScanSet *scanset); +Lardon3DProjectDbResult lardon3d_project_db_list_scansets(Lardon3DProjectDb *database, + uint64_t after_scanset_id, + Lardon3DProjectDbScanSet *scansets, + size_t capacity, size_t *count); Lardon3DProjectDbResult lardon3d_project_db_register_image( Lardon3DProjectDb *database, uint64_t scanset_id, - const unsigned char sha256[LARDON3D_PROJECT_DB_SHA256_SIZE], - const char *asset_path, uint64_t size_bytes, const char *original_name, - const char *source_path, uint64_t producer_task_id, int64_t imported_at, - Lardon3DProjectDbImageRegisterStatus *status, - Lardon3DProjectDbImage *image -); -Lardon3DProjectDbResult lardon3d_project_db_load_image( - Lardon3DProjectDb *database, uint64_t image_id, - Lardon3DProjectDbImage *image, Lardon3DProjectDbImageAsset *asset -); -Lardon3DProjectDbResult lardon3d_project_db_list_images( - Lardon3DProjectDb *database, uint64_t scanset_id, uint64_t after_image_id, - Lardon3DProjectDbImage *images, Lardon3DProjectDbImageAsset *assets, - size_t capacity, size_t *count -); -Lardon3DProjectDbResult lardon3d_project_db_count_images( - Lardon3DProjectDb *database, uint64_t scanset_id, uint64_t *count -); -Lardon3DProjectDbResult lardon3d_project_db_load_image_import( - Lardon3DProjectDb *database, - uint64_t task_id, - Lardon3DProjectDbImageImport *parameters -); -Lardon3DProjectDbResult lardon3d_project_db_allocate_task_id( - Lardon3DProjectDb *database, - uint64_t *task_id -); -Lardon3DProjectDbResult lardon3d_project_db_load_task( - Lardon3DProjectDb *database, - uint64_t task_id, - Lardon3DProjectDbTask *task -); -Lardon3DProjectDbResult lardon3d_project_db_list_recoverable( - Lardon3DProjectDb *database, - uint64_t after_task_id, - Lardon3DProjectDbTask *tasks, - size_t capacity, - size_t *count -); -Lardon3DProjectDbResult lardon3d_project_db_create_artifact( - Lardon3DProjectDb *database, - const Lardon3DProjectDbArtifact *artifact -); -Lardon3DProjectDbResult lardon3d_project_db_mark_artifact_ready( - Lardon3DProjectDb *database, - const char *artifact_id, - int64_t updated_at -); -Lardon3DProjectDbResult lardon3d_project_db_load_artifact( - Lardon3DProjectDb *database, - const char *artifact_id, - Lardon3DProjectDbArtifact *artifact -); + const unsigned char sha256[LARDON3D_PROJECT_DB_SHA256_SIZE], const char *asset_path, + uint64_t size_bytes, const char *original_name, const char *source_path, + uint64_t producer_task_id, int64_t imported_at, Lardon3DProjectDbImageRegisterStatus *status, + Lardon3DProjectDbImage *image); +Lardon3DProjectDbResult lardon3d_project_db_load_image(Lardon3DProjectDb *database, + uint64_t image_id, + Lardon3DProjectDbImage *image, + Lardon3DProjectDbImageAsset *asset); +Lardon3DProjectDbResult lardon3d_project_db_list_images(Lardon3DProjectDb *database, + uint64_t scanset_id, + uint64_t after_image_id, + Lardon3DProjectDbImage *images, + Lardon3DProjectDbImageAsset *assets, + size_t capacity, size_t *count); +Lardon3DProjectDbResult lardon3d_project_db_count_images(Lardon3DProjectDb *database, + uint64_t scanset_id, uint64_t *count); +Lardon3DProjectDbResult +lardon3d_project_db_load_image_import(Lardon3DProjectDb *database, uint64_t task_id, + Lardon3DProjectDbImageImport *parameters); +Lardon3DProjectDbResult lardon3d_project_db_allocate_task_id(Lardon3DProjectDb *database, + uint64_t *task_id); +Lardon3DProjectDbResult lardon3d_project_db_load_task(Lardon3DProjectDb *database, uint64_t task_id, + Lardon3DProjectDbTask *task); +Lardon3DProjectDbResult lardon3d_project_db_list_recoverable(Lardon3DProjectDb *database, + uint64_t after_task_id, + Lardon3DProjectDbTask *tasks, + size_t capacity, size_t *count); +Lardon3DProjectDbResult +lardon3d_project_db_create_artifact(Lardon3DProjectDb *database, + const Lardon3DProjectDbArtifact *artifact); +Lardon3DProjectDbResult lardon3d_project_db_mark_artifact_ready(Lardon3DProjectDb *database, + const char *artifact_id, + int64_t updated_at); +Lardon3DProjectDbResult lardon3d_project_db_load_artifact(Lardon3DProjectDb *database, + const char *artifact_id, + Lardon3DProjectDbArtifact *artifact); +Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_task( + Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, const char *task_kind, + uint32_t task_kind_version, const Lardon3DProjectDbCheckpoint *checkpoint, + const Lardon3DProjectDbFeatureExtractTask *parameters, int64_t updated_at); +Lardon3DProjectDbResult +lardon3d_project_db_load_feature_extract_task(Lardon3DProjectDb *database, uint64_t task_id, + Lardon3DProjectDbFeatureExtractTask *parameters); +Lardon3DProjectDbResult lardon3d_project_db_register_feature_set( + Lardon3DProjectDb *database, uint64_t image_id, const char *extractor_kind, + uint32_t extractor_version, const unsigned char parameter_fingerprint[32], + const unsigned char source_image_sha256[32], uint32_t feature_count, uint32_t descriptor_type, + uint32_t descriptor_dimension, const unsigned char asset_sha256[32], const char *asset_path, + uint64_t asset_size_bytes, Lardon3DProjectDbFeatureDurability durability, + uint64_t producer_task_id, int64_t created_at, Lardon3DProjectDbFeatureSet *feature_set); +Lardon3DProjectDbResult +lardon3d_project_db_find_feature_set(Lardon3DProjectDb *database, uint64_t image_id, + const char *extractor_kind, uint32_t extractor_version, + const unsigned char parameter_fingerprint[32], + Lardon3DProjectDbFeatureSet *feature_set); +Lardon3DProjectDbResult +lardon3d_project_db_load_feature_set(Lardon3DProjectDb *database, uint64_t feature_set_id, + Lardon3DProjectDbFeatureSet *feature_set); +Lardon3DProjectDbResult +lardon3d_project_db_list_feature_sets(Lardon3DProjectDb *database, uint64_t after_feature_set_id, + Lardon3DProjectDbFeatureSet *feature_sets, size_t capacity, + size_t *count); #endif diff --git a/include/lardon3d/task_kind_registry.h b/include/lardon3d/task_kind_registry.h index 10ffba8..69daa63 100644 --- a/include/lardon3d/task_kind_registry.h +++ b/include/lardon3d/task_kind_registry.h @@ -6,63 +6,59 @@ #include +typedef struct Lardon3DProjectDb Lardon3DProjectDb; +typedef struct Lardon3DResourceGovernor Lardon3DResourceGovernor; + enum { - LARDON3D_TASK_KIND_REGISTRY_MAX = 64, + LARDON3D_TASK_KIND_REGISTRY_MAX = 64, }; typedef struct { - Lardon3DTaskCallback callback; - void *userdata; - Lardon3DTaskUserdataDestroy userdata_destroy; - Lardon3DTaskFinishedCallback finished_callback; - void *finished_userdata; + Lardon3DTaskCallback callback; + void *userdata; + Lardon3DTaskUserdataDestroy userdata_destroy; + Lardon3DTaskFinishedCallback finished_callback; + void *finished_userdata; } Lardon3DTaskKindBinding; -typedef bool (*Lardon3DTaskKindReconstruct)( - const Lardon3DTaskDurableSnapshot *snapshot, - void *context, - Lardon3DTaskKindBinding *binding -); +typedef bool (*Lardon3DTaskKindReconstruct)(const Lardon3DTaskDurableSnapshot *snapshot, + void *context, Lardon3DTaskKindBinding *binding); typedef struct { - const char *kind; - uint32_t kind_version; - Lardon3DTaskKindReconstruct reconstruct; + const char *kind; + uint32_t kind_version; + Lardon3DTaskKindReconstruct reconstruct; } Lardon3DTaskKindDescriptor; typedef struct { - const Lardon3DTaskKindDescriptor *descriptors; - size_t count; + const Lardon3DTaskKindDescriptor *descriptors; + size_t count; } Lardon3DTaskKindRegistry; +typedef struct { + const char *project_path; + Lardon3DProjectDb *project_db; + Lardon3DResourceGovernor *resource_governor; +} Lardon3DTaskReconstructionContext; + typedef enum { - LARDON3D_TASK_KIND_OK = 0, - LARDON3D_TASK_KIND_INVALID_ARGUMENT, - LARDON3D_TASK_KIND_UNKNOWN, - LARDON3D_TASK_KIND_UNSUPPORTED_VERSION, - LARDON3D_TASK_KIND_RECONSTRUCTION_FAILED, - LARDON3D_TASK_KIND_RESTORE_FAILED + LARDON3D_TASK_KIND_OK = 0, + LARDON3D_TASK_KIND_INVALID_ARGUMENT, + LARDON3D_TASK_KIND_UNKNOWN, + LARDON3D_TASK_KIND_UNSUPPORTED_VERSION, + LARDON3D_TASK_KIND_RECONSTRUCTION_FAILED, + LARDON3D_TASK_KIND_RESTORE_FAILED } Lardon3DTaskKindResult; -bool lardon3d_task_kind_registry_init( - Lardon3DTaskKindRegistry *registry, - const Lardon3DTaskKindDescriptor *descriptors, - size_t count -); -Lardon3DTaskKindResult lardon3d_task_kind_registry_lookup( - const Lardon3DTaskKindRegistry *registry, - const char *kind, - uint32_t kind_version, - const Lardon3DTaskKindDescriptor **descriptor -); +bool lardon3d_task_kind_registry_init(Lardon3DTaskKindRegistry *registry, + const Lardon3DTaskKindDescriptor *descriptors, size_t count); +Lardon3DTaskKindResult +lardon3d_task_kind_registry_lookup(const Lardon3DTaskKindRegistry *registry, const char *kind, + uint32_t kind_version, + const Lardon3DTaskKindDescriptor **descriptor); Lardon3DTaskKindResult lardon3d_task_kind_registry_restore( - const Lardon3DTaskKindRegistry *registry, - const char *kind, - uint32_t kind_version, - const Lardon3DTaskDurableSnapshot *snapshot, - void *context, - Lardon3DTask **task -); + const Lardon3DTaskKindRegistry *registry, const char *kind, uint32_t kind_version, + const Lardon3DTaskDurableSnapshot *snapshot, void *context, Lardon3DTask **task); const Lardon3DTaskKindRegistry *lardon3d_task_kind_registry_production(void); #endif diff --git a/meson.build b/meson.build index e0fec0e..eea062f 100644 --- a/meson.build +++ b/meson.build @@ -1,6 +1,6 @@ project( 'lardon3d', - 'c', + 'c', 'cpp', version: '0.1.0', license: 'MIT', default_options: [ @@ -17,11 +17,13 @@ add_project_arguments( '-Wshadow', language: 'c', ) +add_project_arguments('-std=c++17', '-Wpedantic', '-Wconversion', '-Wshadow', language: 'cpp') ncursesw = dependency('ncursesw', required: true) threads = dependency('threads') sqlite3 = dependency('sqlite3', required: true) openssl = dependency('openssl', required: true) +opencv = dependency('opencv5', required: true, include_type: 'system', modules: ['opencv_core', 'opencv_imgcodecs', 'opencv_features2d']) executable( 'lardon3d', @@ -35,6 +37,9 @@ executable( 'src/import_task.c', 'src/image_catalog.c', 'src/image_catalog_persistent.c', + 'src/feature_extractor_opencv.cpp', + 'src/feature_store.c', + 'src/feature_task.c', 'src/image_view.c', 'src/project.c', 'src/project_db.c', @@ -48,7 +53,7 @@ executable( 'src/hardware_profile.c', ], include_directories: include_directories('include'), - dependencies: [ncursesw, threads, sqlite3, openssl], + dependencies: [ncursesw, threads, sqlite3, openssl, opencv], ) import_test = executable( @@ -82,6 +87,9 @@ import_task_test = executable( 'src/task_checkpoint.c', 'src/task_kind_registry.c', 'src/task_kinds.c', + 'src/feature_task.c', + 'src/feature_store.c', + 'src/feature_extractor_opencv.cpp', 'src/task_queue.c', 'src/resource_governor.c', 'src/resource_snapshot.c', @@ -95,7 +103,7 @@ import_task_test = executable( '-DLARDON3D_CHECKPOINT_TESTING', ], include_directories: include_directories('include'), - dependencies: [threads, sqlite3, openssl], + dependencies: [threads, sqlite3, openssl, opencv], ) test('import-task', import_task_test, timeout: 30) @@ -136,6 +144,44 @@ persistent_image_catalog_test = executable( test('persistent-image-catalog', persistent_image_catalog_test, timeout: 60) +feature_store_test = executable( + 'test-feature-store', + sources: [ + 'tests/test_feature_store.c', + 'src/app_state.c', + 'src/image_catalog_persistent.c', + 'src/feature_extractor_opencv.cpp', + 'src/feature_store.c', + 'src/project_db.c', + 'src/task.c', + 'src/resource_governor.c', + 'src/resource_snapshot.c', + ], + c_args: ['-DLARDON3D_PROJECT_DB_TESTING', '-DLARDON3D_FEATURE_STORE_TESTING'], + include_directories: include_directories('include'), + dependencies: [threads, sqlite3, openssl, opencv], +) + +test('feature-store', feature_store_test, timeout: 60) + +feature_task_test = executable( + 'test-feature-task', + sources: [ + 'tests/test_feature_task.c', 'src/app_state.c', 'src/project.c', + 'src/project_db.c', 'src/task.c', 'src/task_checkpoint.c', + 'src/task_kind_registry.c', 'src/task_kinds.c', 'src/task_queue.c', + 'src/import.c', 'src/import_task.c', 'src/image_catalog.c', + 'src/image_catalog_persistent.c', 'src/image_view.c', + 'src/feature_task.c', 'src/feature_store.c', + 'src/feature_extractor_opencv.cpp', 'src/resource_governor.c', + 'src/resource_snapshot.c', + ], + c_args: ['-DLARDON3D_PROJECT_DB_TESTING', '-DLARDON3D_FEATURE_TASK_TESTING'], + include_directories: include_directories('include'), + dependencies: [threads, sqlite3, openssl, opencv], +) +test('feature-task', feature_task_test, timeout: 60) + image_view_test = executable( 'test-image-view', sources: [ @@ -206,6 +252,9 @@ project_test = executable( 'src/task_checkpoint.c', 'src/task_kind_registry.c', 'src/task_kinds.c', + 'src/feature_task.c', + 'src/feature_store.c', + 'src/feature_extractor_opencv.cpp', 'src/task_queue.c', 'src/import.c', 'src/import_task.c', @@ -220,7 +269,7 @@ project_test = executable( '-DLARDON3D_CHECKPOINT_TESTING', ], include_directories: include_directories('include'), - dependencies: [threads, sqlite3, openssl], + dependencies: [threads, sqlite3, openssl, opencv], ) test('project', project_test, timeout: 30) diff --git a/src/feature_extractor_opencv.cpp b/src/feature_extractor_opencv.cpp new file mode 100644 index 0000000..abf0e84 --- /dev/null +++ b/src/feature_extractor_opencv.cpp @@ -0,0 +1,142 @@ +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +extern "C" { +#include +} + +extern "C" bool +lardon3d_feature_extractor_parameters_valid(const Lardon3DFeatureExtractorParameters *parameters) { + return parameters && parameters->max_features > 0 && + parameters->max_features <= LARDON3D_FEATURE_MAX_FEATURES && + parameters->pyramid_levels >= 1 && parameters->pyramid_levels <= 16 && + parameters->fast_threshold >= 1 && parameters->fast_threshold <= 255; +} + +static void put_u32(unsigned char *output, uint32_t value) { + output[0] = static_cast(value); + output[1] = static_cast(value >> 8); + output[2] = static_cast(value >> 16); + output[3] = static_cast(value >> 24); +} + +extern "C" void lardon3d_feature_extractor_parameter_fingerprint( + const Lardon3DFeatureExtractorParameters *parameters, unsigned char fingerprint[32]) { + if (!fingerprint) { + return; + } + std::memset(fingerprint, 0, 32); + if (!lardon3d_feature_extractor_parameters_valid(parameters)) { + return; + } + unsigned char canonical[24] = {'L', '3', 'D', 'O', 'R', 'B', 'P', '1'}; + put_u32(canonical + 8, LARDON3D_FEATURE_EXTRACTOR_VERSION); + put_u32(canonical + 12, parameters->max_features); + put_u32(canonical + 16, parameters->pyramid_levels); + put_u32(canonical + 20, parameters->fast_threshold); + (void)SHA256(canonical, sizeof(canonical), fingerprint); +} + +extern "C" void lardon3d_extracted_features_destroy(Lardon3DExtractedFeatures *features) { + if (!features) { + return; + } + std::free(features->keypoints); + std::free(features->descriptors); + std::memset(features, 0, sizeof(*features)); +} + +extern "C" Lardon3DFeatureExtractResult +lardon3d_feature_extract_orb(const char *path, const Lardon3DFeatureExtractorParameters *parameters, + Lardon3DExtractedFeatures *features) { + if (features) { + std::memset(features, 0, sizeof(*features)); + } + if (!path || !path[0] || !features || !lardon3d_feature_extractor_parameters_valid(parameters)) { + return LARDON3D_FEATURE_EXTRACT_INVALID_ARGUMENT; + } + struct stat information{}; + if (lstat(path, &information) != 0) { + return errno == ENOENT ? LARDON3D_FEATURE_EXTRACT_IMAGE_NOT_FOUND + : LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID; + } + if (!S_ISREG(information.st_mode) || S_ISLNK(information.st_mode)) { + return LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID; + } + try { + cv::Mat image = cv::imread(path, cv::IMREAD_GRAYSCALE); + if (image.empty()) { + return LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID; + } + if (image.cols <= 0 || image.rows <= 0 || + static_cast(image.cols) * static_cast(image.rows) > + LARDON3D_FEATURE_MAX_IMAGE_PIXELS) { + return LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID; + } + cv::Ptr extractor = + cv::ORB::create(static_cast(parameters->max_features), 1.2F, + static_cast(parameters->pyramid_levels), 31, 0, 2, + cv::ORB::HARRIS_SCORE, 31, static_cast(parameters->fast_threshold)); + std::vector points; + cv::Mat descriptors; + extractor->detectAndCompute(image, cv::noArray(), points, descriptors); + if (points.size() > parameters->max_features || + points.size() > std::numeric_limits::max() || + (!points.empty() && (descriptors.rows != static_cast(points.size()) || + descriptors.cols != LARDON3D_FEATURE_DESCRIPTOR_DIMENSION || + descriptors.type() != CV_8UC1))) { + return LARDON3D_FEATURE_EXTRACT_ERROR; + } + size_t count = points.size(); + Lardon3DFeatureKeypoint *keypoints = + count == 0 ? nullptr + : static_cast( + std::calloc(count, sizeof(Lardon3DFeatureKeypoint))); + unsigned char *bytes = count == 0 ? nullptr + : static_cast(std::malloc( + count * LARDON3D_FEATURE_DESCRIPTOR_DIMENSION)); + if (count > 0 && (!keypoints || !bytes)) { + std::free(keypoints); + std::free(bytes); + return LARDON3D_FEATURE_EXTRACT_OUT_OF_MEMORY; + } + for (size_t index = 0; index < count; ++index) { + const cv::KeyPoint &point = points[index]; + if (!std::isfinite(point.pt.x) || !std::isfinite(point.pt.y) || !std::isfinite(point.size) || + !std::isfinite(point.angle) || !std::isfinite(point.response) || point.pt.x < 0.0F || + point.pt.y < 0.0F || point.pt.x >= static_cast(image.cols) || + point.pt.y >= static_cast(image.rows) || point.size <= 0.0F || + point.angle < 0.0F || point.angle >= 360.0F) { + std::free(keypoints); + std::free(bytes); + return LARDON3D_FEATURE_EXTRACT_ERROR; + } + keypoints[index] = {point.pt.x, point.pt.y, point.size, + point.angle, point.response, point.octave}; + std::memcpy(bytes + index * LARDON3D_FEATURE_DESCRIPTOR_DIMENSION, + descriptors.ptr(static_cast(index)), LARDON3D_FEATURE_DESCRIPTOR_DIMENSION); + } + features->image_width = static_cast(image.cols); + features->image_height = static_cast(image.rows); + features->feature_count = static_cast(count); + features->keypoints = keypoints; + features->descriptors = bytes; + return LARDON3D_FEATURE_EXTRACT_OK; + } catch (const cv::Exception &) { + return LARDON3D_FEATURE_EXTRACT_IMAGE_INVALID; + } catch (const std::bad_alloc &) { + return LARDON3D_FEATURE_EXTRACT_OUT_OF_MEMORY; + } catch (...) { + return LARDON3D_FEATURE_EXTRACT_ERROR; + } +} diff --git a/src/feature_store.c b/src/feature_store.c new file mode 100644 index 0000000..fc6dedb --- /dev/null +++ b/src/feature_store.c @@ -0,0 +1,575 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +static const unsigned char feature_magic[8] = {'L', '3', 'D', 'F', 'E', 'A', 'T', 0}; +_Static_assert(sizeof(float) == 4 && FLT_RADIX == 2 && FLT_MANT_DIG == 24 && FLT_MAX_EXP == 128, + "Feature File v1 exige IEEE-754 binary32."); + +struct Lardon3DFeatureReader { + int descriptor; + uint64_t keypoint_offset; + uint64_t descriptor_offset; + Lardon3DFeatureFileMetadata metadata; +}; + +static void put_u32(unsigned char *p, uint32_t v) { + p[0] = (unsigned char)v; + p[1] = (unsigned char)(v >> 8); + p[2] = (unsigned char)(v >> 16); + p[3] = (unsigned char)(v >> 24); +} +static void put_u64(unsigned char *p, uint64_t v) { + for (unsigned int i = 0; i < 8; ++i) { + p[i] = (unsigned char)(v >> (8U * i)); + } +} +static uint32_t get_u32(const unsigned char *p) { + return (uint32_t)p[0] | ((uint32_t)p[1] << 8) | ((uint32_t)p[2] << 16) | ((uint32_t)p[3] << 24); +} +static uint64_t get_u64(const unsigned char *p) { + uint64_t v = 0; + for (unsigned int i = 0; i < 8; ++i) { + v |= (uint64_t)p[i] << (8U * i); + } + return v; +} +static void put_float(unsigned char *p, float value) { + uint32_t bits = 0; + memcpy(&bits, &value, sizeof(bits)); + put_u32(p, bits); +} +static float get_float(const unsigned char *p) { + uint32_t bits = get_u32(p); + float value = 0; + memcpy(&value, &bits, sizeof(value)); + return value; +} + +static bool write_all(int fd, const void *data, size_t size) { + const unsigned char *bytes = data; + size_t used = 0; + while (used < size) { + ssize_t n = write(fd, bytes + used, size - used); + if (n < 0 && errno == EINTR) { + continue; + } + if (n <= 0) { + return false; + } + used += (size_t)n; + } + return true; +} + +static bool read_exact(int fd, void *data, size_t size, off_t offset) { + unsigned char *bytes = data; + size_t used = 0; + while (used < size) { + ssize_t n = pread(fd, bytes + used, size - used, offset + (off_t)used); + if (n < 0 && errno == EINTR) { + continue; + } + if (n <= 0) { + return false; + } + used += (size_t)n; + } + return true; +} + +static bool sha256_fd(int fd, unsigned char output[32]) { + EVP_MD_CTX *context = EVP_MD_CTX_new(); + if (!context) { + return false; + } + bool ok = EVP_DigestInit_ex(context, EVP_sha256(), NULL) == 1; + unsigned char buffer[65536]; + off_t offset = 0; + while (ok) { + ssize_t n = pread(fd, buffer, sizeof(buffer), offset); + if (n < 0 && errno == EINTR) { + continue; + } + if (n < 0) { + ok = false; + break; + } + if (n == 0) { + break; + } + ok = EVP_DigestUpdate(context, buffer, (size_t)n) == 1; + offset += (off_t)n; + } + unsigned int length = 0; + ok = ok && EVP_DigestFinal_ex(context, output, &length) == 1 && length == 32; + EVP_MD_CTX_free(context); + return ok; +} + +static void hex_sha(const unsigned char hash[32], char text[65]) { + static const char digits[] = "0123456789abcdef"; + for (size_t i = 0; i < 32; ++i) { + text[2 * i] = digits[hash[i] >> 4]; + text[2 * i + 1] = digits[hash[i] & 15]; + } + text[64] = '\0'; +} + +static bool canonical_feature_path(const unsigned char hash[32], const char *path) { + if (!hash || !path) { + return false; + } + char hex[65]; + hex_sha(hash, hex); + char expected[LARDON3D_PROJECT_DB_PATH_CAPACITY]; + int written = snprintf(expected, sizeof(expected), "assets/features/%.2s/%s", hex, hex); + return written > 0 && (size_t)written < sizeof(expected) && strcmp(path, expected) == 0; +} + +static bool join(char output[PATH_MAX], const char *a, const char *b) { + int n = snprintf(output, PATH_MAX, "%s/%s", a, b); + return n > 0 && (size_t)n < PATH_MAX; +} + +static bool ensure_directory(const char *path) { + if (mkdir(path, 0755) == 0) { + return true; + } + if (errno != EEXIST) { + return false; + } + struct stat info; + return lstat(path, &info) == 0 && S_ISDIR(info.st_mode) && !S_ISLNK(info.st_mode); +} + +static bool sync_directory(const char *path) { + int fd = open(path, O_RDONLY | O_DIRECTORY | O_CLOEXEC); + if (fd < 0) { + return false; + } + bool ok = fsync(fd) == 0; + if (close(fd) != 0) { + ok = false; + } + return ok; +} + +static bool sync_feature_asset_directory(const char *project_path, + const Lardon3DProjectDbFeatureSet *feature_set) { + char relative[LARDON3D_PROJECT_DB_PATH_CAPACITY]; + int written = snprintf(relative, sizeof(relative), "%s", feature_set->asset.path); + if (written <= 0 || (size_t)written >= sizeof(relative)) { + return false; + } + char *separator = strrchr(relative, '/'); + if (!separator) { + return false; + } + *separator = '\0'; + char directory[PATH_MAX]; + return join(directory, project_path, relative) && sync_directory(directory); +} + +static bool valid_features(const Lardon3DExtractedFeatures *features) { + if (!features || features->feature_count > LARDON3D_FEATURE_MAX_FEATURES || + features->image_width == 0 || features->image_height == 0 || + (features->feature_count > 0 && (!features->keypoints || !features->descriptors))) { + return false; + } + for (uint32_t i = 0; i < features->feature_count; ++i) { + const Lardon3DFeatureKeypoint *p = &features->keypoints[i]; + if (!isfinite(p->x) || !isfinite(p->y) || !isfinite(p->size) || !isfinite(p->angle_degrees) || + !isfinite(p->response) || p->x < 0 || p->y < 0 || p->x >= (float)features->image_width || + p->y >= (float)features->image_height || p->size <= 0 || p->angle_degrees < 0 || + p->angle_degrees >= 360 || p->octave < 0 || p->octave >= 16) { + return false; + } + } + return true; +} + +static bool write_feature_file(int fd, const Lardon3DProjectDbImageAsset *image_asset, + const unsigned char fingerprint[32], + const Lardon3DExtractedFeatures *features, uint64_t *file_size) { + uint64_t kp_bytes = (uint64_t)features->feature_count * LARDON3D_FEATURE_KEYPOINT_RECORD_SIZE; + uint64_t descriptor_bytes = + (uint64_t)features->feature_count * LARDON3D_FEATURE_DESCRIPTOR_DIMENSION; + uint64_t descriptor_offset = LARDON3D_FEATURE_FILE_HEADER_SIZE + kp_bytes; + uint64_t total = descriptor_offset + descriptor_bytes; + if (total > LARDON3D_FEATURE_FILE_MAX_SIZE) { + return false; + } + unsigned char header[LARDON3D_FEATURE_FILE_HEADER_SIZE] = {0}; + memcpy(header, feature_magic, 8); + put_u32(header + 8, LARDON3D_FEATURE_FILE_VERSION); + put_u32(header + 12, LARDON3D_FEATURE_FILE_HEADER_SIZE); + put_u32(header + 16, features->feature_count); + put_u32(header + 20, LARDON3D_FEATURE_DESCRIPTOR_DIMENSION); + put_u32(header + 24, LARDON3D_FEATURE_DESCRIPTOR_U8); + put_u32(header + 28, LARDON3D_FEATURE_KEYPOINT_RECORD_SIZE); + put_u32(header + 32, features->image_width); + put_u32(header + 36, features->image_height); + put_u64(header + 40, LARDON3D_FEATURE_FILE_HEADER_SIZE); + put_u64(header + 48, descriptor_offset); + put_u64(header + 56, total); + memcpy(header + 64, image_asset->sha256, 32); + memcpy(header + 96, fingerprint, 32); + memcpy(header + 128, LARDON3D_FEATURE_EXTRACTOR_KIND, 4); + put_u32(header + 144, LARDON3D_FEATURE_EXTRACTOR_VERSION); + if (!write_all(fd, header, sizeof(header))) { + return false; + } + unsigned char record[LARDON3D_FEATURE_KEYPOINT_RECORD_SIZE]; + for (uint32_t i = 0; i < features->feature_count; ++i) { + const Lardon3DFeatureKeypoint *p = &features->keypoints[i]; + put_float(record, p->x); + put_float(record + 4, p->y); + put_float(record + 8, p->size); + put_float(record + 12, p->angle_degrees); + put_float(record + 16, p->response); + put_u32(record + 20, (uint32_t)p->octave); + if (!write_all(fd, record, sizeof(record))) { + return false; + } + } + if (descriptor_bytes > 0 && !write_all(fd, features->descriptors, (size_t)descriptor_bytes)) { + return false; + } + *file_size = total; + return true; +} + +static Lardon3DFeatureStoreResult +reuse_existing_feature_set(Lardon3DAppState *state, uint64_t image_id, uint64_t producer_task_id, + const unsigned char fingerprint[32], + Lardon3DProjectDbFeatureSet *feature_set) { + if (feature_set->asset.durability == LARDON3D_DB_FEATURE_ASSET_DURABLE) { + return LARDON3D_FEATURE_STORE_ALREADY_PRESENT; + } + Lardon3DFeatureReader *reader = NULL; + Lardon3DFeatureFileMetadata metadata; + Lardon3DFeatureStoreResult validation = + lardon3d_feature_reader_open(state->project_path, feature_set, &reader, &metadata); + lardon3d_feature_reader_close(reader); + if (validation != LARDON3D_FEATURE_STORE_OK || + !sync_feature_asset_directory(state->project_path, feature_set)) { + return LARDON3D_FEATURE_STORE_PUBLISHED_NOT_DURABLE; + } + int64_t now = (int64_t)time(NULL); + if (now < 0) { + return LARDON3D_FEATURE_STORE_IO_ERROR; + } + Lardon3DProjectDbResult promoted = lardon3d_project_db_register_feature_set( + state->project_db, image_id, LARDON3D_FEATURE_EXTRACTOR_KIND, + LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, feature_set->source_image_sha256, + feature_set->feature_count, feature_set->descriptor_type, feature_set->descriptor_dimension, + feature_set->asset.sha256, feature_set->asset.path, feature_set->asset.size_bytes, + LARDON3D_DB_FEATURE_ASSET_DURABLE, producer_task_id, now, feature_set); + return promoted == LARDON3D_PROJECT_DB_OK ? LARDON3D_FEATURE_STORE_ALREADY_PRESENT + : LARDON3D_FEATURE_STORE_DB_ERROR; +} + +static Lardon3DFeatureStoreResult register_published_feature_set( + Lardon3DAppState *state, uint64_t image_id, uint64_t producer_task_id, + const Lardon3DProjectDbImageAsset *image_asset, const unsigned char fingerprint[32], + const Lardon3DExtractedFeatures *features, const unsigned char file_hash[32], + const char *relative_path, uint64_t file_size, bool durable, + Lardon3DProjectDbFeatureSet *feature_set) { + int64_t now = (int64_t)time(NULL); + if (now < 0) { + return LARDON3D_FEATURE_STORE_IO_ERROR; + } + Lardon3DProjectDbResult result = lardon3d_project_db_register_feature_set( + state->project_db, image_id, LARDON3D_FEATURE_EXTRACTOR_KIND, + LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, image_asset->sha256, features->feature_count, + LARDON3D_FEATURE_DESCRIPTOR_U8, LARDON3D_FEATURE_DESCRIPTOR_DIMENSION, file_hash, + relative_path, file_size, + durable ? LARDON3D_DB_FEATURE_ASSET_DURABLE : LARDON3D_DB_FEATURE_ASSET_PUBLISHED_NOT_DURABLE, + producer_task_id, now, feature_set); + if (result != LARDON3D_PROJECT_DB_OK) { + return result == LARDON3D_PROJECT_DB_BUSY ? LARDON3D_FEATURE_STORE_DB_BUSY + : LARDON3D_FEATURE_STORE_DB_ERROR; + } + return durable ? LARDON3D_FEATURE_STORE_OK : LARDON3D_FEATURE_STORE_PUBLISHED_NOT_DURABLE; +} + +Lardon3DFeatureStoreResult lardon3d_feature_store_publish( + Lardon3DAppState *state, uint64_t image_id, uint64_t producer_task_id, + const Lardon3DFeatureExtractorParameters *parameters, const Lardon3DExtractedFeatures *features, + Lardon3DProjectDbFeatureSet *feature_set) { + if (feature_set) { + memset(feature_set, 0, sizeof(*feature_set)); + } + if (!state || !state->project_loaded || !state->project_db || !feature_set || image_id == 0 || + !lardon3d_feature_extractor_parameters_valid(parameters) || !valid_features(features)) { + return LARDON3D_FEATURE_STORE_INVALID_ARGUMENT; + } + unsigned char fingerprint[32]; + lardon3d_feature_extractor_parameter_fingerprint(parameters, fingerprint); + Lardon3DProjectDbResult found = lardon3d_project_db_find_feature_set( + state->project_db, image_id, LARDON3D_FEATURE_EXTRACTOR_KIND, + LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, feature_set); + if (found == LARDON3D_PROJECT_DB_OK) { + return reuse_existing_feature_set(state, image_id, producer_task_id, fingerprint, feature_set); + } + if (found != LARDON3D_PROJECT_DB_NOT_FOUND) { + return found == LARDON3D_PROJECT_DB_BUSY ? LARDON3D_FEATURE_STORE_DB_BUSY + : LARDON3D_FEATURE_STORE_DB_ERROR; + } + Lardon3DProjectDbImage image; + Lardon3DProjectDbImageAsset image_asset; + if (lardon3d_project_db_load_image(state->project_db, image_id, &image, &image_asset) != + LARDON3D_PROJECT_DB_OK) { + return LARDON3D_FEATURE_STORE_NOT_FOUND; + } + char assets[PATH_MAX], features_dir[PATH_MAX]; + if (!join(assets, state->project_path, "assets") || !ensure_directory(assets) || + !join(features_dir, assets, "features") || !ensure_directory(features_dir)) { + return LARDON3D_FEATURE_STORE_IO_ERROR; + } + char temporary[PATH_MAX]; + int n = snprintf(temporary, sizeof(temporary), "%s/.feature-XXXXXX", features_dir); + if (n <= 0 || (size_t)n >= sizeof(temporary)) { + return LARDON3D_FEATURE_STORE_IO_ERROR; + } + int fd = mkstemp(temporary); + if (fd < 0) { + return LARDON3D_FEATURE_STORE_IO_ERROR; + } + uint64_t file_size = 0; + bool written = + write_feature_file(fd, &image_asset, fingerprint, features, &file_size) && fsync(fd) == 0; + unsigned char file_hash[32]; + if (written) { + written = sha256_fd(fd, file_hash); + } + if (close(fd) != 0) { + written = false; + } + if (!written) { + unlink(temporary); + return LARDON3D_FEATURE_STORE_IO_ERROR; + } + char hex[65]; + hex_sha(file_hash, hex); + char prefix[PATH_MAX]; + n = snprintf(prefix, sizeof(prefix), "%s/%.2s", features_dir, hex); + if (n <= 0 || (size_t)n >= sizeof(prefix) || !ensure_directory(prefix)) { + unlink(temporary); + return LARDON3D_FEATURE_STORE_IO_ERROR; + } + char relative[LARDON3D_PROJECT_DB_PATH_CAPACITY]; + n = snprintf(relative, sizeof(relative), "assets/features/%.2s/%s", hex, hex); + char final[PATH_MAX]; + if (n <= 0 || (size_t)n >= sizeof(relative) || !join(final, state->project_path, relative)) { + unlink(temporary); + return LARDON3D_FEATURE_STORE_IO_ERROR; + } + if (link(temporary, final) != 0) { + if (errno != EEXIST) { + unlink(temporary); + return LARDON3D_FEATURE_STORE_IO_ERROR; + } + int existing = open(final, O_RDONLY | O_NOFOLLOW | O_CLOEXEC); + struct stat info; + unsigned char existing_hash[32]; + bool same = existing >= 0 && fstat(existing, &info) == 0 && S_ISREG(info.st_mode) && + info.st_size >= 0 && (uint64_t)info.st_size == file_size && + sha256_fd(existing, existing_hash) && memcmp(existing_hash, file_hash, 32) == 0; + if (existing >= 0) { + (void)close(existing); + } + if (!same) { + unlink(temporary); + return LARDON3D_FEATURE_STORE_CORRUPT; + } + } + unlink(temporary); + bool durable = sync_directory(prefix); +#ifdef LARDON3D_FEATURE_STORE_TESTING + const char *fail_sync = getenv("LARDON3D_TEST_FEATURE_FAIL_DIRECTORY_SYNC"); + if (fail_sync && strcmp(fail_sync, "1") == 0) { + durable = false; + } +#endif + return register_published_feature_set(state, image_id, producer_task_id, &image_asset, + fingerprint, features, file_hash, relative, file_size, + durable, feature_set); +} + +static Lardon3DFeatureStoreResult validate_header(const unsigned char h[160], uint64_t actual, + Lardon3DFeatureFileMetadata *m, uint64_t *ko, + uint64_t *doff) { + if (memcmp(h, feature_magic, 8) != 0) { + return LARDON3D_FEATURE_STORE_INVALID; + } + uint32_t version = get_u32(h + 8); + if (version != 1) { + return version > 1 ? LARDON3D_FEATURE_STORE_UNSUPPORTED_VERSION + : LARDON3D_FEATURE_STORE_INVALID; + } + uint32_t count = get_u32(h + 16), dim = get_u32(h + 20), type = get_u32(h + 24), + record = get_u32(h + 28), width = get_u32(h + 32), height = get_u32(h + 36); + uint64_t key = get_u64(h + 40), desc = get_u64(h + 48), declared = get_u64(h + 56); + uint64_t kb = (uint64_t)count * record, db = (uint64_t)count * dim; + if (get_u32(h + 12) != 160 || count > LARDON3D_FEATURE_MAX_FEATURES || dim != 32 || + type != LARDON3D_FEATURE_DESCRIPTOR_U8 || record != 24 || width == 0 || height == 0 || + key != 160 || desc != key + kb || declared != desc + db || declared != actual || + actual > LARDON3D_FEATURE_FILE_MAX_SIZE || memcmp(h + 128, "orb\0", 4) != 0 || + get_u32(h + 144) != 1) { + return LARDON3D_FEATURE_STORE_CORRUPT; + } + for (size_t i = 132; i < 144; ++i) { + if (h[i] != 0) { + return LARDON3D_FEATURE_STORE_CORRUPT; + } + } + for (size_t i = 148; i < 160; ++i) { + if (h[i] != 0) { + return LARDON3D_FEATURE_STORE_CORRUPT; + } + } + *m = (Lardon3DFeatureFileMetadata){.format_version = version, + .extractor_version = get_u32(h + 144), + .feature_count = count, + .descriptor_dimension = dim, + .descriptor_type = (Lardon3DFeatureDescriptorType)type, + .image_width = width, + .image_height = height}; + memcpy(m->source_image_sha256, h + 64, 32); + memcpy(m->parameter_fingerprint, h + 96, 32); + memcpy(m->extractor_kind, h + 128, 4); + *ko = key; + *doff = desc; + return LARDON3D_FEATURE_STORE_OK; +} + +Lardon3DFeatureStoreResult lardon3d_feature_reader_open(const char *project_path, + const Lardon3DProjectDbFeatureSet *set, + Lardon3DFeatureReader **out, + Lardon3DFeatureFileMetadata *metadata) { + if (out) { + *out = NULL; + } + if (!project_path || !set || !out || !metadata || + !canonical_feature_path(set->asset.sha256, set->asset.path)) { + return LARDON3D_FEATURE_STORE_INVALID_ARGUMENT; + } + char path[PATH_MAX]; + if (!join(path, project_path, set->asset.path)) { + return LARDON3D_FEATURE_STORE_INVALID_ARGUMENT; + } + int fd = open(path, O_RDONLY | O_NOFOLLOW | O_CLOEXEC); + if (fd < 0) { + return errno == ENOENT ? LARDON3D_FEATURE_STORE_NOT_FOUND : LARDON3D_FEATURE_STORE_IO_ERROR; + } + struct stat st; + if (fstat(fd, &st) != 0 || !S_ISREG(st.st_mode) || st.st_size < 160 || + (uint64_t)st.st_size > LARDON3D_FEATURE_FILE_MAX_SIZE || + (uint64_t)st.st_size != set->asset.size_bytes) { + close(fd); + return LARDON3D_FEATURE_STORE_CORRUPT; + } + unsigned char header[160]; + if (!read_exact(fd, header, sizeof(header), 0)) { + close(fd); + return LARDON3D_FEATURE_STORE_CORRUPT; + } + Lardon3DFeatureReader *reader = calloc(1, sizeof(*reader)); + if (!reader) { + close(fd); + return LARDON3D_FEATURE_STORE_IO_ERROR; + } + Lardon3DFeatureStoreResult result = + validate_header(header, (uint64_t)st.st_size, &reader->metadata, &reader->keypoint_offset, + &reader->descriptor_offset); + if (result != LARDON3D_FEATURE_STORE_OK) { + close(fd); + free(reader); + return result; + } + unsigned char hash[32]; + if (!sha256_fd(fd, hash) || memcmp(hash, set->asset.sha256, 32) != 0) { + close(fd); + free(reader); + return LARDON3D_FEATURE_STORE_CORRUPT; + } + if (reader->metadata.feature_count != set->feature_count || + strcmp(reader->metadata.extractor_kind, set->extractor_kind) != 0 || + reader->metadata.extractor_version != set->extractor_version || + reader->metadata.descriptor_dimension != set->descriptor_dimension || + reader->metadata.descriptor_type != (Lardon3DFeatureDescriptorType)set->descriptor_type || + memcmp(reader->metadata.source_image_sha256, set->source_image_sha256, 32) != 0 || + memcmp(reader->metadata.parameter_fingerprint, set->parameter_fingerprint, 32) != 0) { + close(fd); + free(reader); + return LARDON3D_FEATURE_STORE_CORRUPT; + } + reader->descriptor = fd; + *metadata = reader->metadata; + *out = reader; + return LARDON3D_FEATURE_STORE_OK; +} + +void lardon3d_feature_reader_close(Lardon3DFeatureReader *r) { + if (!r) { + return; + } + (void)close(r->descriptor); + free(r); +} + +Lardon3DFeatureStoreResult lardon3d_feature_reader_keypoints(Lardon3DFeatureReader *r, + uint32_t start, + Lardon3DFeatureKeypoint *out, + size_t capacity) { + if (!r || (!out && capacity > 0) || capacity > LARDON3D_FEATURE_READER_RANGE_MAX || + start > r->metadata.feature_count || capacity > (size_t)(r->metadata.feature_count - start)) { + return LARDON3D_FEATURE_STORE_INVALID_ARGUMENT; + } + unsigned char record[24]; + for (size_t i = 0; i < capacity; ++i) { + uint64_t index = (uint64_t)start + i; + if (!read_exact(r->descriptor, record, 24, (off_t)(r->keypoint_offset + index * 24))) { + return LARDON3D_FEATURE_STORE_IO_ERROR; + } + out[i] = (Lardon3DFeatureKeypoint){get_float(record), get_float(record + 4), + get_float(record + 8), get_float(record + 12), + get_float(record + 16), (int32_t)get_u32(record + 20)}; + if (!isfinite(out[i].x) || !isfinite(out[i].y) || !isfinite(out[i].size) || + !isfinite(out[i].angle_degrees) || !isfinite(out[i].response)) { + return LARDON3D_FEATURE_STORE_CORRUPT; + } + } + return LARDON3D_FEATURE_STORE_OK; +} + +Lardon3DFeatureStoreResult lardon3d_feature_reader_descriptors(Lardon3DFeatureReader *r, + uint32_t start, unsigned char *out, + size_t capacity, size_t bytes) { + if (!r || (!out && capacity > 0) || capacity > LARDON3D_FEATURE_READER_RANGE_MAX || + start > r->metadata.feature_count || capacity > (size_t)(r->metadata.feature_count - start) || + capacity > SIZE_MAX / r->metadata.descriptor_dimension || + bytes < capacity * r->metadata.descriptor_dimension) { + return LARDON3D_FEATURE_STORE_INVALID_ARGUMENT; + } + size_t amount = capacity * r->metadata.descriptor_dimension; + if (amount && !read_exact(r->descriptor, out, amount, + (off_t)(r->descriptor_offset + + (uint64_t)start * r->metadata.descriptor_dimension))) { + return LARDON3D_FEATURE_STORE_IO_ERROR; + } + return LARDON3D_FEATURE_STORE_OK; +} diff --git a/src/feature_task.c b/src/feature_task.c new file mode 100644 index 0000000..47e9987 --- /dev/null +++ b/src/feature_task.c @@ -0,0 +1,325 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +typedef struct { + char project_path[PATH_MAX]; + Lardon3DProjectDb *database; + Lardon3DResourceGovernor *governor; + Lardon3DProjectDbFeatureExtractTask parameters; +} Lardon3DFeatureTaskContext; + +static void destroy_context(void *userdata) { free(userdata); } + +static void runtime_state(const Lardon3DFeatureTaskContext *context, Lardon3DAppState *state) { + lardon3d_app_state_init(state); + state->project_loaded = true; + state->project_db = context->database; + state->resource_governor = context->governor; + (void)snprintf(state->project_path, sizeof(state->project_path), "%s", context->project_path); +} + +static bool file_hash(const char *path, unsigned char output[32]) { + int fd = open(path, O_RDONLY | O_NOFOLLOW | O_CLOEXEC); + if (fd < 0) { + return false; + } + struct stat st; + if (fstat(fd, &st) != 0 || !S_ISREG(st.st_mode)) { + close(fd); + return false; + } + EVP_MD_CTX *ctx = EVP_MD_CTX_new(); + bool ok = ctx && EVP_DigestInit_ex(ctx, EVP_sha256(), NULL) == 1; + unsigned char buffer[65536]; + while (ok) { + ssize_t n = read(fd, buffer, sizeof(buffer)); + if (n < 0 && errno == EINTR) { + continue; + } + if (n < 0) { + ok = false; + break; + } + if (n == 0) { + break; + } + ok = EVP_DigestUpdate(ctx, buffer, (size_t)n) == 1; + } + unsigned int size = 0; + ok = ok && EVP_DigestFinal_ex(ctx, output, &size) == 1 && size == 32; + if (ctx) { + EVP_MD_CTX_free(ctx); + } + if (close(fd) != 0) { + ok = false; + } + return ok; +} + +static uint64_t elapsed_ns(struct timespec a, struct timespec b) { + uint64_t s = b.tv_sec >= a.tv_sec ? (uint64_t)(b.tv_sec - a.tv_sec) : 0; + long n = b.tv_nsec - a.tv_nsec; + if (n < 0 && s) { + --s; + n += 1000000000L; + } + return s <= UINT64_MAX / 1000000000ULL ? s * 1000000000ULL + (uint64_t)n : UINT64_MAX; +} + +static void finished(const Lardon3DTask *task, void *userdata) { +#ifdef LARDON3D_FEATURE_TASK_TESTING + const char *skip = getenv("LARDON3D_TEST_FEATURE_SKIP_FINISHED_CHECKPOINT"); + if (skip && strcmp(skip, "1") == 0) { + return; + } +#endif + Lardon3DFeatureTaskContext *context = userdata; + Lardon3DAppState state; + runtime_state(context, &state); + (void)lardon3d_project_checkpoint_feature_extract_task(&state, task, &context->parameters); +} + +static Lardon3DProjectDbResult find_valid_feature_set(const Lardon3DFeatureTaskContext *context, + const unsigned char fingerprint[32]) { + Lardon3DProjectDbFeatureSet existing; + Lardon3DProjectDbResult found = lardon3d_project_db_find_feature_set( + context->database, context->parameters.image_id, LARDON3D_FEATURE_EXTRACTOR_KIND, + LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, &existing); + if (found != LARDON3D_PROJECT_DB_OK) { + return found; + } + Lardon3DFeatureReader *reader = NULL; + Lardon3DFeatureFileMetadata metadata; + Lardon3DFeatureStoreResult valid = + lardon3d_feature_reader_open(context->project_path, &existing, &reader, &metadata); + lardon3d_feature_reader_close(reader); + return valid == LARDON3D_FEATURE_STORE_OK ? LARDON3D_PROJECT_DB_OK : LARDON3D_PROJECT_DB_CORRUPT; +} + +static bool load_validated_source(const Lardon3DFeatureTaskContext *context, char path[PATH_MAX]) { + Lardon3DProjectDbImage image; + Lardon3DProjectDbImageAsset asset; + if (lardon3d_project_db_load_image(context->database, context->parameters.image_id, &image, + &asset) != LARDON3D_PROJECT_DB_OK) { + return false; + } + int written = snprintf(path, PATH_MAX, "%s/%s", context->project_path, asset.path); + if (written <= 0 || (size_t)written >= PATH_MAX) { + return false; + } + unsigned char actual[32]; + return file_hash(path, actual) && memcmp(actual, asset.sha256, 32) == 0; +} + +static bool run(Lardon3DTask *task, void *userdata) { + Lardon3DFeatureTaskContext *context = userdata; + if (!lardon3d_task_checkpoint(task)) { + return false; + } + struct timespec begin; + (void)clock_gettime(CLOCK_MONOTONIC, &begin); + unsigned char fingerprint[32]; + lardon3d_feature_extractor_parameter_fingerprint( + &(Lardon3DFeatureExtractorParameters){context->parameters.max_features, + context->parameters.pyramid_levels, + context->parameters.fast_threshold}, + fingerprint); + Lardon3DProjectDbResult found = find_valid_feature_set(context, fingerprint); + if (found == LARDON3D_PROJECT_DB_OK) { + return lardon3d_task_set_progress(task, 100, "Features déjà présentes."); + } + if (found == LARDON3D_PROJECT_DB_CORRUPT) { + return lardon3d_task_fail(task, "Feature Set READY absent ou corrompu."); + } + if (found != LARDON3D_PROJECT_DB_NOT_FOUND) { + return lardon3d_task_fail(task, "Recherche Feature Store impossible."); + } + char path[PATH_MAX]; + if (!load_validated_source(context, path)) { + return lardon3d_task_fail(task, "Asset image absent ou corrompu."); + } + Lardon3DFeatureExtractorParameters parameters = {context->parameters.max_features, + context->parameters.pyramid_levels, + context->parameters.fast_threshold}; + Lardon3DExtractedFeatures features; + Lardon3DFeatureExtractResult extracted = + lardon3d_feature_extract_orb(path, ¶meters, &features); + if (extracted != LARDON3D_FEATURE_EXTRACT_OK) { + return lardon3d_task_fail(task, "Décodage ou extraction ORB impossible."); + } + if (!lardon3d_task_checkpoint(task)) { + lardon3d_extracted_features_destroy(&features); + return false; + } +#ifdef LARDON3D_FEATURE_TASK_TESTING + const char *pause = getenv("LARDON3D_TEST_FEATURE_PAUSE_BEFORE_PUBLISH"); + if (pause && strcmp(pause, "1") == 0) { + (void)lardon3d_task_pause(task); + if (!lardon3d_task_checkpoint(task)) { + lardon3d_extracted_features_destroy(&features); + return false; + } + } +#endif + Lardon3DAppState state; + runtime_state(context, &state); + Lardon3DProjectDbFeatureSet set; + Lardon3DFeatureStoreResult published = lardon3d_feature_store_publish( + &state, context->parameters.image_id, lardon3d_task_id(task), ¶meters, &features, &set); + lardon3d_extracted_features_destroy(&features); + if (published != LARDON3D_FEATURE_STORE_OK && + published != LARDON3D_FEATURE_STORE_ALREADY_PRESENT && + published != LARDON3D_FEATURE_STORE_PUBLISHED_NOT_DURABLE) { + return lardon3d_task_fail(task, "Publication Feature Store impossible."); + } + struct timespec end; + (void)clock_gettime(CLOCK_MONOTONIC, &end); + (void)lardon3d_resource_governor_record_batch(context->governor, LARDON3D_RESOURCE_TASK_CPU, 1, + elapsed_ns(begin, end), 0); + return lardon3d_task_set_progress(task, 100, "Feature Set publié."); +} + +static Lardon3DFeatureTaskContext * +make_context(const Lardon3DTaskReconstructionContext *runtime, + const Lardon3DProjectDbFeatureExtractTask *parameters) { + if (!runtime || !runtime->project_path || !runtime->project_db || !runtime->resource_governor || + !parameters) { + return NULL; + } + Lardon3DFeatureTaskContext *context = calloc(1, sizeof(*context)); + if (!context) { + return NULL; + } + int n = + snprintf(context->project_path, sizeof(context->project_path), "%s", runtime->project_path); + if (n <= 0 || (size_t)n >= sizeof(context->project_path)) { + free(context); + return NULL; + } + context->database = runtime->project_db; + context->governor = runtime->resource_governor; + context->parameters = *parameters; + return context; +} + +bool lardon3d_feature_extract_reconstruct(const Lardon3DTaskDurableSnapshot *snapshot, + void *userdata, Lardon3DTaskKindBinding *binding) { + Lardon3DTaskReconstructionContext *runtime = userdata; + if (!snapshot || !runtime || !binding) { + return false; + } + Lardon3DProjectDbFeatureExtractTask parameters; + if (lardon3d_project_db_load_feature_extract_task(runtime->project_db, snapshot->id, + ¶meters) != LARDON3D_PROJECT_DB_OK || + strcmp(parameters.extractor_kind, LARDON3D_FEATURE_EXTRACTOR_KIND) != 0 || + parameters.extractor_version != 1) { + return false; + } + unsigned char expected[32]; + lardon3d_feature_extractor_parameter_fingerprint( + &(Lardon3DFeatureExtractorParameters){parameters.max_features, parameters.pyramid_levels, + parameters.fast_threshold}, + expected); + if (memcmp(expected, parameters.parameter_fingerprint, 32) != 0) { + return false; + } + Lardon3DFeatureTaskContext *context = make_context(runtime, ¶meters); + if (!context) { + return false; + } + *binding = (Lardon3DTaskKindBinding){.callback = run, + .userdata = context, + .userdata_destroy = destroy_context, + .finished_callback = finished, + .finished_userdata = context}; + return true; +} + +Lardon3DTask * +lardon3d_project_create_feature_extract_task(Lardon3DAppState *state, uint64_t image_id, + const Lardon3DFeatureExtractorParameters *parameters, + uint64_t *task_id) { + if (task_id) { + *task_id = 0; + } + if (!state || !state->project_loaded || !state->project_db || !state->resource_governor || + !parameters || !task_id || !lardon3d_feature_extractor_parameters_valid(parameters)) { + return NULL; + } + Lardon3DProjectDbImage image; + Lardon3DProjectDbImageAsset asset; + if (lardon3d_project_db_load_image(state->project_db, image_id, &image, &asset) != + LARDON3D_PROJECT_DB_OK) { + return NULL; + } + uint64_t id; + if (lardon3d_project_db_allocate_task_id(state->project_db, &id) != LARDON3D_PROJECT_DB_OK) { + return NULL; + } + Lardon3DProjectDbFeatureExtractTask durable = {.task_id = id, + .image_id = image_id, + .extractor_version = 1, + .max_features = parameters->max_features, + .pyramid_levels = parameters->pyramid_levels, + .fast_threshold = parameters->fast_threshold}; + snprintf(durable.extractor_kind, sizeof(durable.extractor_kind), "%s", + LARDON3D_FEATURE_EXTRACTOR_KIND); + lardon3d_feature_extractor_parameter_fingerprint(parameters, durable.parameter_fingerprint); + Lardon3DTaskReconstructionContext runtime = {state->project_path, state->project_db, + state->resource_governor}; + Lardon3DFeatureTaskContext *context = make_context(&runtime, &durable); + if (!context) { + return NULL; + } + Lardon3DResourceEstimate estimate = {.memory_fixed_bytes = 64ULL * 1024 * 1024, + .memory_bytes_per_item = 512ULL * 1024 * 1024, + .minimum_batch_size = 1, + .maximum_batch_size = 1, + .desired_cpu_threads = 1, + .desired_io_slots = 1, + .task_class = LARDON3D_RESOURCE_TASK_CPU}; + Lardon3DTask *task = lardon3d_task_create_typed("Extraction de features", &estimate, + LARDON3D_FEATURE_EXTRACT_TASK_KIND, 1, run, + context, destroy_context); + if (!task || !lardon3d_task_assign_id(task, id) || + !lardon3d_task_set_finished_callback(task, finished, context) || + lardon3d_project_checkpoint_feature_extract_task(state, task, &durable) != + LARDON3D_PROJECT_TASK_CHECKPOINT_OK) { + lardon3d_task_destroy(task); + return NULL; + } + *task_id = id; + return task; +} + +bool lardon3d_project_enqueue_feature_extract(Lardon3DAppState *state, uint64_t image_id, + const Lardon3DFeatureExtractorParameters *parameters, + uint64_t *task_id) { + if (!state || !state->task_queue) { + return false; + } + Lardon3DTask *task = + lardon3d_project_create_feature_extract_task(state, image_id, parameters, task_id); + if (!task) { + return false; + } + if (!lardon3d_task_queue_add(state->task_queue, task, NULL)) { + lardon3d_task_destroy(task); + return false; + } + return true; +} diff --git a/src/project.c b/src/project.c index 800ebc8..1aee256 100644 --- a/src/project.c +++ b/src/project.c @@ -11,1180 +11,938 @@ #include #include -#include #include #include #include +#include #include #include #include enum { - MAX_CREATED_DIRECTORIES = 16, - INI_LINE_CAPACITY = 512, + MAX_CREATED_DIRECTORIES = 16, + INI_LINE_CAPACITY = 512, }; typedef struct { - char paths[MAX_CREATED_DIRECTORIES][PATH_MAX]; - size_t count; + char paths[MAX_CREATED_DIRECTORIES][PATH_MAX]; + size_t count; } CreatedDirectories; typedef struct { - char name[128]; - char stable_id[LARDON3D_PROJECT_DB_ID_CAPACITY]; - unsigned int version; + char name[128]; + char stable_id[LARDON3D_PROJECT_DB_ID_CAPACITY]; + unsigned int version; } ProjectMetadata; -static void -set_status(Lardon3DAppState *state, const char *message) -{ - (void)snprintf( - state->status_message, - sizeof(state->status_message), - "%s", - message - ); +static void set_status(Lardon3DAppState *state, const char *message) { + (void)snprintf(state->status_message, sizeof(state->status_message), "%s", message); } -static void -store_recovery_summary(Lardon3DAppState *state, - const Lardon3DProjectRecoverySummary *summary) -{ - state->recovery_inspected = summary ? summary->inspected : 0; - state->recovery_resumed = summary ? summary->resumed : 0; - state->recovery_skipped = summary ? summary->skipped : 0; - state->recovery_failed = summary ? summary->failed : 0; - state->recovery_published_not_durable = summary - ? summary->published_not_durable : 0; - state->recovery_queue_full = summary && summary->queue_full; +static void store_recovery_summary(Lardon3DAppState *state, + const Lardon3DProjectRecoverySummary *summary) { + state->recovery_inspected = summary ? summary->inspected : 0; + state->recovery_resumed = summary ? summary->resumed : 0; + state->recovery_skipped = summary ? summary->skipped : 0; + state->recovery_failed = summary ? summary->failed : 0; + state->recovery_published_not_durable = summary ? summary->published_not_durable : 0; + state->recovery_queue_full = summary && summary->queue_full; } -static void -set_database_status( - Lardon3DAppState *state, - Lardon3DProjectDb *database, - const char *open_error, - const char *fallback -) -{ - char detail[LARDON3D_PROJECT_DB_ERROR_CAPACITY] = ""; - if (database) { - (void)lardon3d_project_db_last_error(database, detail); - } else if (open_error) { - (void)snprintf(detail, sizeof(detail), "%s", open_error); - } - (void)snprintf( - state->status_message, - sizeof(state->status_message), - "Erreur project.db : %.220s", - detail[0] ? detail : fallback - ); +static void set_database_status(Lardon3DAppState *state, Lardon3DProjectDb *database, + const char *open_error, const char *fallback) { + char detail[LARDON3D_PROJECT_DB_ERROR_CAPACITY] = ""; + if (database) { + (void)lardon3d_project_db_last_error(database, detail); + } else if (open_error) { + (void)snprintf(detail, sizeof(detail), "%s", open_error); + } + (void)snprintf(state->status_message, sizeof(state->status_message), "Erreur project.db : %.220s", + detail[0] ? detail : fallback); } -static void -clear_catalog(Lardon3DAppState *state) -{ - lardon3d_image_view_destroy(state->image_view); - state->image_view = NULL; - lardon3d_image_catalog_destroy(state->image_catalog); - state->image_catalog = NULL; +static void clear_catalog(Lardon3DAppState *state) { + lardon3d_image_view_destroy(state->image_view); + state->image_view = NULL; + lardon3d_image_catalog_destroy(state->image_catalog); + state->image_catalog = NULL; } -static bool -normalize_name( - Lardon3DAppState *state, - const char *input, - char *output, - size_t output_size -) -{ - if (!input) { - set_status(state, "Erreur : le nom du projet est vide."); - return false; - } +static bool normalize_name(Lardon3DAppState *state, const char *input, char *output, + size_t output_size) { + if (!input) { + set_status(state, "Erreur : le nom du projet est vide."); + return false; + } - const char *start = input; - while (*start && isspace((unsigned char)*start)) { - ++start; - } + const char *start = input; + while (*start && isspace((unsigned char)*start)) { + ++start; + } - const char *end = input + strlen(input); - while (end > start && isspace((unsigned char)end[-1])) { - --end; - } + const char *end = input + strlen(input); + while (end > start && isspace((unsigned char)end[-1])) { + --end; + } - size_t length = (size_t)(end - start); - if (length == 0) { - set_status(state, "Erreur : le nom du projet est vide."); - return false; - } - if (length >= output_size) { - set_status(state, "Erreur : le nom du projet est trop long."); - return false; - } - if ((length == 1 && start[0] == '.') - || (length == 2 && start[0] == '.' && start[1] == '.')) { - set_status(state, "Erreur : nom de projet interdit."); - return false; - } + size_t length = (size_t)(end - start); + if (length == 0) { + set_status(state, "Erreur : le nom du projet est vide."); + return false; + } + if (length >= output_size) { + set_status(state, "Erreur : le nom du projet est trop long."); + return false; + } + if ((length == 1 && start[0] == '.') || (length == 2 && start[0] == '.' && start[1] == '.')) { + set_status(state, "Erreur : nom de projet interdit."); + return false; + } - for (const char *character = start; character < end; ++character) { - if (*character == '/' || *character == '\\' - || iscntrl((unsigned char)*character)) { - set_status(state, "Erreur : nom de projet interdit."); - return false; - } + for (const char *character = start; character < end; ++character) { + if (*character == '/' || *character == '\\' || iscntrl((unsigned char)*character)) { + set_status(state, "Erreur : nom de projet interdit."); + return false; } + } - (void)memcpy(output, start, length); - output[length] = '\0'; - return true; + (void)memcpy(output, start, length); + output[length] = '\0'; + return true; } -static bool -copy_path(char *destination, size_t size, const char *source) -{ - int written = snprintf(destination, size, "%s", source); - return written >= 0 && (size_t)written < size; +static bool copy_path(char *destination, size_t size, const char *source) { + int written = snprintf(destination, size, "%s", source); + return written >= 0 && (size_t)written < size; } -static bool -join_path( - char *destination, - size_t size, - const char *parent, - const char *child -) -{ - int written = snprintf(destination, size, "%s/%s", parent, child); - return written >= 0 && (size_t)written < size; +static bool join_path(char *destination, size_t size, const char *parent, const char *child) { + int written = snprintf(destination, size, "%s/%s", parent, child); + return written >= 0 && (size_t)written < size; } -static bool -generate_stable_id(char output[LARDON3D_PROJECT_DB_ID_CAPACITY]) -{ - unsigned char bytes[16]; - int descriptor = open("/dev/urandom", O_RDONLY | O_CLOEXEC); - if (descriptor < 0) { - return false; +static bool generate_stable_id(char output[LARDON3D_PROJECT_DB_ID_CAPACITY]) { + unsigned char bytes[16]; + int descriptor = open("/dev/urandom", O_RDONLY | O_CLOEXEC); + if (descriptor < 0) { + return false; + } + size_t used = 0; + while (used < sizeof(bytes)) { + ssize_t amount = read(descriptor, bytes + used, sizeof(bytes) - used); + if (amount < 0 && errno == EINTR) { + continue; } - size_t used = 0; - while (used < sizeof(bytes)) { - ssize_t amount = read(descriptor, bytes + used, sizeof(bytes) - used); - if (amount < 0 && errno == EINTR) { - continue; - } - if (amount <= 0) { - (void)close(descriptor); - return false; - } - used += (size_t)amount; + if (amount <= 0) { + (void)close(descriptor); + return false; } - if (close(descriptor) != 0) { - return false; + used += (size_t)amount; + } + if (close(descriptor) != 0) { + return false; + } + for (size_t index = 0; index < sizeof(bytes); ++index) { + (void)snprintf(output + index * 2, 3, "%02x", bytes[index]); + } + return true; +} + +static bool resolve_projects_root(Lardon3DAppState *state, char root[PATH_MAX]) { + const char *configured = getenv("LARDON3D_PROJECTS_ROOT"); + if (configured && configured[0]) { + if (configured[0] != '/' || !copy_path(root, PATH_MAX, configured)) { + set_status(state, "Erreur : répertoire racine invalide."); + return false; } - for (size_t index = 0; index < sizeof(bytes); ++index) { - (void)snprintf(output + index * 2, 3, "%02x", bytes[index]); + } else { + const char *home = getenv("HOME"); + if (!home || home[0] != '/') { + set_status(state, "Erreur : HOME est absent ou invalide."); + return false; + } + + int written = snprintf(root, PATH_MAX, "%s/Documents/Lardon/Projets3D", home); + if (written < 0 || (size_t)written >= PATH_MAX) { + set_status(state, "Erreur : chemin racine trop long."); + return false; + } + } + + size_t length = strlen(root); + while (length > 1 && root[length - 1] == '/') { + root[--length] = '\0'; + } + return true; +} + +static void cleanup_directories(CreatedDirectories *created) { + while (created->count > 0) { + --created->count; + (void)rmdir(created->paths[created->count]); + } +} + +static bool ensure_directory(Lardon3DAppState *state, const char *path, + CreatedDirectories *created) { + struct stat info; + if (lstat(path, &info) == 0) { + if (!S_ISDIR(info.st_mode)) { + set_status(state, "Erreur : un élément du chemin n'est pas un dossier."); + return false; } return true; + } + if (errno != ENOENT || created->count >= MAX_CREATED_DIRECTORIES) { + set_status(state, "Erreur : impossible de préparer le répertoire racine."); + return false; + } + if (mkdir(path, 0755) != 0) { + set_status(state, "Erreur : impossible de créer un dossier."); + return false; + } + if (!copy_path(created->paths[created->count], sizeof(created->paths[created->count]), path)) { + (void)rmdir(path); + set_status(state, "Erreur : chemin de dossier trop long."); + return false; + } + ++created->count; + return true; } -static bool -resolve_projects_root(Lardon3DAppState *state, char root[PATH_MAX]) -{ - const char *configured = getenv("LARDON3D_PROJECTS_ROOT"); - if (configured && configured[0]) { - if (configured[0] != '/' || !copy_path(root, PATH_MAX, configured)) { - set_status(state, "Erreur : répertoire racine invalide."); - return false; - } - } else { - const char *home = getenv("HOME"); - if (!home || home[0] != '/') { - set_status(state, "Erreur : HOME est absent ou invalide."); - return false; - } +static bool ensure_directory_tree(Lardon3DAppState *state, const char *path, + CreatedDirectories *created) { + char partial[PATH_MAX]; + if (!copy_path(partial, sizeof(partial), path)) { + set_status(state, "Erreur : chemin racine trop long."); + return false; + } - int written = snprintf( - root, - PATH_MAX, - "%s/Documents/Lardon/Projets3D", - home - ); - if (written < 0 || (size_t)written >= PATH_MAX) { - set_status(state, "Erreur : chemin racine trop long."); - return false; - } - } - - size_t length = strlen(root); - while (length > 1 && root[length - 1] == '/') { - root[--length] = '\0'; - } - return true; -} - -static void -cleanup_directories(CreatedDirectories *created) -{ - while (created->count > 0) { - --created->count; - (void)rmdir(created->paths[created->count]); - } -} - -static bool -ensure_directory( - Lardon3DAppState *state, - const char *path, - CreatedDirectories *created -) -{ - struct stat info; - if (lstat(path, &info) == 0) { - if (!S_ISDIR(info.st_mode)) { - set_status(state, "Erreur : un élément du chemin n'est pas un dossier."); - return false; - } - return true; - } - if (errno != ENOENT || created->count >= MAX_CREATED_DIRECTORIES) { - set_status(state, "Erreur : impossible de préparer le répertoire racine."); - return false; - } - if (mkdir(path, 0755) != 0) { - set_status(state, "Erreur : impossible de créer un dossier."); - return false; - } - if (!copy_path( - created->paths[created->count], - sizeof(created->paths[created->count]), - path - )) { - (void)rmdir(path); - set_status(state, "Erreur : chemin de dossier trop long."); - return false; - } - ++created->count; - return true; -} - -static bool -ensure_directory_tree( - Lardon3DAppState *state, - const char *path, - CreatedDirectories *created -) -{ - char partial[PATH_MAX]; - if (!copy_path(partial, sizeof(partial), path)) { - set_status(state, "Erreur : chemin racine trop long."); - return false; - } - - for (char *separator = partial + 1; *separator; ++separator) { - if (*separator != '/') { - continue; - } - *separator = '\0'; - bool success = ensure_directory(state, partial, created); - *separator = '/'; - if (!success) { - return false; - } - } - return ensure_directory(state, partial, created); -} - -static bool -write_project_ini( - Lardon3DAppState *state, - const char *project_path, - const char *project_name, - const char *stable_id -) -{ - char temporary_path[PATH_MAX]; - char final_path[PATH_MAX]; - if (!join_path(final_path, sizeof(final_path), project_path, "project.ini") - || !join_path( - temporary_path, - sizeof(temporary_path), - project_path, - ".project.ini.tmp.XXXXXX" - )) { - set_status(state, "Erreur : chemin de project.ini trop long."); - return false; - } - - int descriptor = mkstemp(temporary_path); - if (descriptor < 0) { - set_status(state, "Erreur : impossible de créer project.ini."); - return false; - } - - FILE *file = fdopen(descriptor, "w"); - if (!file) { - (void)close(descriptor); - (void)unlink(temporary_path); - set_status(state, "Erreur : impossible d'écrire project.ini."); - return false; - } - - bool success = fprintf( - file, - "[project]\nname=%s\nstable_id=%s\nversion=2\n", - project_name, - stable_id - ) >= 0; - if (success) { - success = fflush(file) == 0; - } - if (success) { - success = fsync(fileno(file)) == 0; - } - if (fclose(file) != 0) { - success = false; - } - if (success) { - success = rename(temporary_path, final_path) == 0; + for (char *separator = partial + 1; *separator; ++separator) { + if (*separator != '/') { + continue; } + *separator = '\0'; + bool success = ensure_directory(state, partial, created); + *separator = '/'; if (!success) { - (void)unlink(temporary_path); - set_status(state, "Erreur : impossible d'écrire project.ini."); + return false; } - return success; + } + return ensure_directory(state, partial, created); } -bool -lardon3d_project_create(Lardon3DAppState *state, const char *name) -{ - if (!state) { - return false; - } - if (state->project_loaded || state->project_db) { - set_status(state, "Erreur : un projet est déjà ouvert."); - return false; - } +static bool write_project_ini(Lardon3DAppState *state, const char *project_path, + const char *project_name, const char *stable_id) { + char temporary_path[PATH_MAX]; + char final_path[PATH_MAX]; + if (!join_path(final_path, sizeof(final_path), project_path, "project.ini") || + !join_path(temporary_path, sizeof(temporary_path), project_path, ".project.ini.tmp.XXXXXX")) { + set_status(state, "Erreur : chemin de project.ini trop long."); + return false; + } - char normalized_name[sizeof(state->project_name)]; - char root[PATH_MAX]; - char project_path[PATH_MAX]; - char stable_id[LARDON3D_PROJECT_DB_ID_CAPACITY]; - if (!normalize_name( - state, - name, - normalized_name, - sizeof(normalized_name) - )) { - return false; - } - if (!resolve_projects_root(state, root)) { - return false; - } - if (!generate_stable_id(stable_id)) { - set_status(state, "Erreur : impossible de créer l'identité du projet."); - return false; - } - if (!join_path( - project_path, - sizeof(project_path), - root, - normalized_name - )) { - set_status(state, "Erreur : chemin du projet trop long."); - return false; - } + int descriptor = mkstemp(temporary_path); + if (descriptor < 0) { + set_status(state, "Erreur : impossible de créer project.ini."); + return false; + } - CreatedDirectories created = {0}; - if (!ensure_directory_tree(state, root, &created)) { - cleanup_directories(&created); - return false; - } + FILE *file = fdopen(descriptor, "w"); + if (!file) { + (void)close(descriptor); + (void)unlink(temporary_path); + set_status(state, "Erreur : impossible d'écrire project.ini."); + return false; + } - struct stat info; - if (lstat(project_path, &info) == 0 || errno != ENOENT) { - cleanup_directories(&created); - set_status(state, "Erreur : ce projet existe déjà."); - return false; - } - if (!ensure_directory(state, project_path, &created)) { - cleanup_directories(&created); - return false; - } + bool success = + fprintf(file, "[project]\nname=%s\nstable_id=%s\nversion=2\n", project_name, stable_id) >= 0; + if (success) { + success = fflush(file) == 0; + } + if (success) { + success = fsync(fileno(file)) == 0; + } + if (fclose(file) != 0) { + success = false; + } + if (success) { + success = rename(temporary_path, final_path) == 0; + } + if (!success) { + (void)unlink(temporary_path); + set_status(state, "Erreur : impossible d'écrire project.ini."); + } + return success; +} - const char *subdirectories[] = { - "images", - "reconstruction", - "exports", - "logs", - ".lardon3d", - ".lardon3d/checkpoints", +bool lardon3d_project_create(Lardon3DAppState *state, const char *name) { + if (!state) { + return false; + } + if (state->project_loaded || state->project_db) { + set_status(state, "Erreur : un projet est déjà ouvert."); + return false; + } + + char normalized_name[sizeof(state->project_name)]; + char root[PATH_MAX]; + char project_path[PATH_MAX]; + char stable_id[LARDON3D_PROJECT_DB_ID_CAPACITY]; + if (!normalize_name(state, name, normalized_name, sizeof(normalized_name))) { + return false; + } + if (!resolve_projects_root(state, root)) { + return false; + } + if (!generate_stable_id(stable_id)) { + set_status(state, "Erreur : impossible de créer l'identité du projet."); + return false; + } + if (!join_path(project_path, sizeof(project_path), root, normalized_name)) { + set_status(state, "Erreur : chemin du projet trop long."); + return false; + } + + CreatedDirectories created = {0}; + if (!ensure_directory_tree(state, root, &created)) { + cleanup_directories(&created); + return false; + } + + struct stat info; + if (lstat(project_path, &info) == 0 || errno != ENOENT) { + cleanup_directories(&created); + set_status(state, "Erreur : ce projet existe déjà."); + return false; + } + if (!ensure_directory(state, project_path, &created)) { + cleanup_directories(&created); + return false; + } + + const char *subdirectories[] = { + "images", "reconstruction", "exports", "logs", ".lardon3d", ".lardon3d/checkpoints", + }; + for (size_t index = 0; index < sizeof(subdirectories) / sizeof(subdirectories[0]); ++index) { + char path[PATH_MAX]; + if (!join_path(path, sizeof(path), project_path, subdirectories[index])) { + set_status(state, "Erreur : chemin de dossier trop long."); + cleanup_directories(&created); + return false; + } + if (!ensure_directory(state, path, &created)) { + cleanup_directories(&created); + return false; + } + } + + if (!write_project_ini(state, project_path, normalized_name, stable_id)) { + cleanup_directories(&created); + return false; + } + + char database_path[PATH_MAX]; + char ini_path[PATH_MAX]; + if (!join_path(database_path, sizeof(database_path), project_path, "project.db") || + !join_path(ini_path, sizeof(ini_path), project_path, "project.ini")) { + cleanup_directories(&created); + set_status(state, "Erreur : chemin de base projet trop long."); + return false; + } + Lardon3DProjectDb *database = NULL; + char database_error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]; + Lardon3DProjectDbResult database_result = + lardon3d_project_db_open(database_path, &database, database_error); + struct timespec now; + if (database_result == LARDON3D_PROJECT_DB_OK && clock_gettime(CLOCK_REALTIME, &now) == 0) { + Lardon3DProjectDbProject project = { + .created_at = now.tv_sec, + .updated_at = now.tv_sec, }; - for (size_t index = 0; - index < sizeof(subdirectories) / sizeof(subdirectories[0]); - ++index) { - char path[PATH_MAX]; - if (!join_path( - path, - sizeof(path), - project_path, - subdirectories[index] - )) { - set_status(state, "Erreur : chemin de dossier trop long."); - cleanup_directories(&created); - return false; - } - if (!ensure_directory(state, path, &created)) { - cleanup_directories(&created); - return false; - } + (void)snprintf(project.stable_id, sizeof(project.stable_id), "%s", stable_id); + (void)snprintf(project.name, sizeof(project.name), "%s", normalized_name); + database_result = lardon3d_project_db_set_project(database, &project); + } else if (database_result == LARDON3D_PROJECT_DB_OK) { + database_result = LARDON3D_PROJECT_DB_IO_ERROR; + } + if (database_result != LARDON3D_PROJECT_DB_OK) { + set_database_status(state, database, database_error, "initialisation impossible"); + lardon3d_project_db_close(database); + (void)unlink(database_path); + (void)unlink(ini_path); + cleanup_directories(&created); + return false; + } + + clear_catalog(state); + state->project_loaded = true; + store_recovery_summary(state, NULL); + state->project_db = database; + (void)copy_path(state->project_name, sizeof(state->project_name), normalized_name); + (void)copy_path(state->project_path, sizeof(state->project_path), project_path); + (void)snprintf(state->project_stable_id, sizeof(state->project_stable_id), "%s", stable_id); + (void)snprintf(state->status_message, sizeof(state->status_message), "Projet créé : %s", + state->project_name); + return true; +} + +static bool read_project_ini(Lardon3DAppState *state, const char *path, ProjectMetadata *metadata) { + int descriptor = open(path, O_RDONLY | O_NOFOLLOW); + if (descriptor < 0) { + set_status(state, "Erreur : project.ini est absent ou inaccessible."); + return false; + } + + struct stat info; + if (fstat(descriptor, &info) != 0 || !S_ISREG(info.st_mode)) { + (void)close(descriptor); + set_status(state, "Erreur : project.ini n'est pas un fichier régulier."); + return false; + } + + FILE *file = fdopen(descriptor, "r"); + if (!file) { + (void)close(descriptor); + set_status(state, "Erreur : impossible de lire project.ini."); + return false; + } + + bool in_project_section = false; + bool section_found = false; + bool name_found = false; + bool stable_id_found = false; + bool version_found = false; + bool valid = true; + char line[INI_LINE_CAPACITY]; + + while (valid && fgets(line, sizeof(line), file)) { + size_t length = strlen(line); + if (length > 0 && line[length - 1] == '\n') { + line[--length] = '\0'; + } else if (!feof(file)) { + valid = false; + break; + } + if (length > 0 && line[length - 1] == '\r') { + line[--length] = '\0'; } - if (!write_project_ini(state, project_path, normalized_name, stable_id)) { - cleanup_directories(&created); - return false; + if (strcmp(line, "[project]") == 0) { + in_project_section = true; + section_found = true; + } else if (line[0] == '[') { + in_project_section = false; + } else if (in_project_section && strncmp(line, "name=", 5) == 0) { + if (name_found || + !normalize_name(state, line + 5, metadata->name, sizeof(state->project_name))) { + valid = false; + } + name_found = true; + } else if (in_project_section && strncmp(line, "stable_id=", 10) == 0) { + size_t id_length = strnlen(line + 10, sizeof(metadata->stable_id)); + if (stable_id_found || id_length != 32) { + valid = false; + } else { + for (size_t index = 0; index < id_length; ++index) { + char character = line[10 + index]; + if (!((character >= '0' && character <= '9') || (character >= 'a' && character <= 'f'))) { + valid = false; + } + } + if (valid) { + (void)snprintf(metadata->stable_id, sizeof(metadata->stable_id), "%s", line + 10); + } + } + stable_id_found = true; + } else if (in_project_section && strncmp(line, "version=", 8) == 0) { + if (version_found) { + valid = false; + } else if (strcmp(line, "version=1") == 0) { + metadata->version = 1; + } else if (strcmp(line, "version=2") == 0) { + metadata->version = 2; + } else { + valid = false; + } + version_found = true; } + } - char database_path[PATH_MAX]; - char ini_path[PATH_MAX]; - if (!join_path(database_path, sizeof(database_path), project_path, "project.db") - || !join_path(ini_path, sizeof(ini_path), project_path, "project.ini")) { - cleanup_directories(&created); - set_status(state, "Erreur : chemin de base projet trop long."); - return false; + if (ferror(file)) { + valid = false; + } + if (fclose(file) != 0) { + valid = false; + } + if (!valid || !section_found || !name_found || !version_found || + (metadata->version == 2 && !stable_id_found) || (metadata->version == 1 && stable_id_found)) { + set_status(state, "Erreur : project.ini invalide."); + return false; + } + return true; +} + +bool lardon3d_project_open(Lardon3DAppState *state, const char *directory_name) { + if (!state) { + return false; + } + if (state->project_loaded || state->project_db) { + set_status(state, "Erreur : un projet est déjà ouvert."); + return false; + } + + char normalized_directory[sizeof(state->project_name)]; + char root[PATH_MAX]; + char project_path[PATH_MAX]; + char ini_path[PATH_MAX]; + char database_path[PATH_MAX]; + if (!normalize_name(state, directory_name, normalized_directory, sizeof(normalized_directory))) { + return false; + } + if (!resolve_projects_root(state, root)) { + return false; + } + if (!join_path(project_path, sizeof(project_path), root, normalized_directory) || + !join_path(ini_path, sizeof(ini_path), project_path, "project.ini") || + !join_path(database_path, sizeof(database_path), project_path, "project.db")) { + set_status(state, "Erreur : chemin du projet trop long."); + return false; + } + + struct stat info; + if (lstat(project_path, &info) != 0 || !S_ISDIR(info.st_mode)) { + set_status(state, "Erreur : dossier projet absent ou invalide."); + return false; + } + + ProjectMetadata metadata = {0}; + if (!read_project_ini(state, ini_path, &metadata)) { + return false; + } + + bool database_existed = false; + if (lstat(database_path, &info) == 0) { + if (!S_ISREG(info.st_mode)) { + set_status(state, "Erreur : project.db n'est pas un fichier régulier."); + return false; } - Lardon3DProjectDb *database = NULL; - char database_error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]; - Lardon3DProjectDbResult database_result = lardon3d_project_db_open( - database_path, - &database, - database_error - ); - struct timespec now; - if (database_result == LARDON3D_PROJECT_DB_OK - && clock_gettime(CLOCK_REALTIME, &now) == 0) { - Lardon3DProjectDbProject project = { + database_existed = true; + } else if (errno != ENOENT) { + set_status(state, "Erreur : project.db est inaccessible."); + return false; + } + + CreatedDirectories created = {0}; + char internal_path[PATH_MAX]; + char checkpoint_path[PATH_MAX]; + if (!join_path(internal_path, sizeof(internal_path), project_path, ".lardon3d") || + !join_path(checkpoint_path, sizeof(checkpoint_path), internal_path, "checkpoints") || + !ensure_directory(state, internal_path, &created) || + !ensure_directory(state, checkpoint_path, &created)) { + cleanup_directories(&created); + return false; + } + + Lardon3DProjectDb *database = NULL; + char database_error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]; + Lardon3DProjectDbResult database_result = + lardon3d_project_db_open(database_path, &database, database_error); + if (database_result != LARDON3D_PROJECT_DB_OK) { + cleanup_directories(&created); + set_database_status(state, NULL, database_error, "ouverture impossible"); + return false; + } + + Lardon3DProjectDbProject db_project; + database_result = lardon3d_project_db_get_project(database, &db_project); + bool rewrite_metadata = metadata.version == 1; + struct timespec now; + if (clock_gettime(CLOCK_REALTIME, &now) != 0) { + database_result = LARDON3D_PROJECT_DB_IO_ERROR; + } else if (database_result == LARDON3D_PROJECT_DB_OK) { + if (metadata.stable_id[0] && strcmp(metadata.stable_id, db_project.stable_id) != 0) { + database_result = LARDON3D_PROJECT_DB_CONSTRAINT; + } else if (!metadata.stable_id[0]) { + (void)snprintf(metadata.stable_id, sizeof(metadata.stable_id), "%s", db_project.stable_id); + rewrite_metadata = true; + } + if (database_result == LARDON3D_PROJECT_DB_OK) { + db_project.updated_at = now.tv_sec; + (void)snprintf(db_project.name, sizeof(db_project.name), "%s", metadata.name); + database_result = lardon3d_project_db_set_project(database, &db_project); + } + } else if (database_result == LARDON3D_PROJECT_DB_NOT_FOUND) { + if (database_existed && !metadata.stable_id[0]) { + database_result = LARDON3D_PROJECT_DB_CONSTRAINT; + } else { + if (!metadata.stable_id[0]) { + if (!generate_stable_id(metadata.stable_id)) { + database_result = LARDON3D_PROJECT_DB_IO_ERROR; + } + rewrite_metadata = true; + } + if (database_result != LARDON3D_PROJECT_DB_IO_ERROR) { + Lardon3DProjectDbProject new_project = { .created_at = now.tv_sec, .updated_at = now.tv_sec, }; - (void)snprintf(project.stable_id, sizeof(project.stable_id), "%s", stable_id); - (void)snprintf(project.name, sizeof(project.name), "%s", normalized_name); - database_result = lardon3d_project_db_set_project(database, &project); - } else if (database_result == LARDON3D_PROJECT_DB_OK) { - database_result = LARDON3D_PROJECT_DB_IO_ERROR; + (void)snprintf(new_project.stable_id, sizeof(new_project.stable_id), "%s", + metadata.stable_id); + (void)snprintf(new_project.name, sizeof(new_project.name), "%s", metadata.name); + database_result = lardon3d_project_db_set_project(database, &new_project); + } } - if (database_result != LARDON3D_PROJECT_DB_OK) { - set_database_status( - state, - database, - database_error, - "initialisation impossible" - ); - lardon3d_project_db_close(database); - (void)unlink(database_path); - (void)unlink(ini_path); - cleanup_directories(&created); - return false; - } - - clear_catalog(state); - state->project_loaded = true; - store_recovery_summary(state, NULL); - state->project_db = database; - (void)copy_path( - state->project_name, - sizeof(state->project_name), - normalized_name - ); - (void)copy_path( - state->project_path, - sizeof(state->project_path), - project_path - ); - (void)snprintf( - state->project_stable_id, - sizeof(state->project_stable_id), - "%s", - stable_id - ); - (void)snprintf( - state->status_message, - sizeof(state->status_message), - "Projet créé : %s", - state->project_name - ); - return true; -} - -static bool -read_project_ini( - Lardon3DAppState *state, - const char *path, - ProjectMetadata *metadata -) -{ - int descriptor = open(path, O_RDONLY | O_NOFOLLOW); - if (descriptor < 0) { - set_status(state, "Erreur : project.ini est absent ou inaccessible."); - return false; - } - - struct stat info; - if (fstat(descriptor, &info) != 0 || !S_ISREG(info.st_mode)) { - (void)close(descriptor); - set_status(state, "Erreur : project.ini n'est pas un fichier régulier."); - return false; - } - - FILE *file = fdopen(descriptor, "r"); - if (!file) { - (void)close(descriptor); - set_status(state, "Erreur : impossible de lire project.ini."); - return false; - } - - bool in_project_section = false; - bool section_found = false; - bool name_found = false; - bool stable_id_found = false; - bool version_found = false; - bool valid = true; - char line[INI_LINE_CAPACITY]; - - while (valid && fgets(line, sizeof(line), file)) { - size_t length = strlen(line); - if (length > 0 && line[length - 1] == '\n') { - line[--length] = '\0'; - } else if (!feof(file)) { - valid = false; - break; - } - if (length > 0 && line[length - 1] == '\r') { - line[--length] = '\0'; - } - - if (strcmp(line, "[project]") == 0) { - in_project_section = true; - section_found = true; - } else if (line[0] == '[') { - in_project_section = false; - } else if (in_project_section && strncmp(line, "name=", 5) == 0) { - if (name_found || !normalize_name( - state, - line + 5, - metadata->name, - sizeof(state->project_name) - )) { - valid = false; - } - name_found = true; - } else if (in_project_section && strncmp(line, "stable_id=", 10) == 0) { - size_t id_length = strnlen( - line + 10, - sizeof(metadata->stable_id) - ); - if (stable_id_found || id_length != 32) { - valid = false; - } else { - for (size_t index = 0; index < id_length; ++index) { - char character = line[10 + index]; - if (!((character >= '0' && character <= '9') - || (character >= 'a' && character <= 'f'))) { - valid = false; - } - } - if (valid) { - (void)snprintf( - metadata->stable_id, - sizeof(metadata->stable_id), - "%s", - line + 10 - ); - } - } - stable_id_found = true; - } else if (in_project_section && strncmp(line, "version=", 8) == 0) { - if (version_found) { - valid = false; - } else if (strcmp(line, "version=1") == 0) { - metadata->version = 1; - } else if (strcmp(line, "version=2") == 0) { - metadata->version = 2; - } else { - valid = false; - } - version_found = true; - } - } - - if (ferror(file)) { - valid = false; - } - if (fclose(file) != 0) { - valid = false; - } - if (!valid || !section_found || !name_found || !version_found - || (metadata->version == 2 && !stable_id_found) - || (metadata->version == 1 && stable_id_found)) { - set_status(state, "Erreur : project.ini invalide."); - return false; - } - return true; -} - -bool -lardon3d_project_open( - Lardon3DAppState *state, - const char *directory_name -) -{ - if (!state) { - return false; - } - if (state->project_loaded || state->project_db) { - set_status(state, "Erreur : un projet est déjà ouvert."); - return false; - } - - char normalized_directory[sizeof(state->project_name)]; - char root[PATH_MAX]; - char project_path[PATH_MAX]; - char ini_path[PATH_MAX]; - char database_path[PATH_MAX]; - if (!normalize_name( - state, - directory_name, - normalized_directory, - sizeof(normalized_directory) - )) { - return false; - } - if (!resolve_projects_root(state, root)) { - return false; - } - if (!join_path( - project_path, - sizeof(project_path), - root, - normalized_directory - ) - || !join_path(ini_path, sizeof(ini_path), project_path, "project.ini") - || !join_path(database_path, sizeof(database_path), project_path, "project.db")) { - set_status(state, "Erreur : chemin du projet trop long."); - return false; - } - - struct stat info; - if (lstat(project_path, &info) != 0 || !S_ISDIR(info.st_mode)) { - set_status(state, "Erreur : dossier projet absent ou invalide."); - return false; - } - - ProjectMetadata metadata = {0}; - if (!read_project_ini(state, ini_path, &metadata)) { - return false; - } - - bool database_existed = false; - if (lstat(database_path, &info) == 0) { - if (!S_ISREG(info.st_mode)) { - set_status(state, "Erreur : project.db n'est pas un fichier régulier."); - return false; - } - database_existed = true; - } else if (errno != ENOENT) { - set_status(state, "Erreur : project.db est inaccessible."); - return false; - } - - CreatedDirectories created = {0}; - char internal_path[PATH_MAX]; - char checkpoint_path[PATH_MAX]; - if (!join_path(internal_path, sizeof(internal_path), project_path, ".lardon3d") - || !join_path(checkpoint_path, sizeof(checkpoint_path), internal_path, "checkpoints") - || !ensure_directory(state, internal_path, &created) - || !ensure_directory(state, checkpoint_path, &created)) { - cleanup_directories(&created); - return false; - } - - Lardon3DProjectDb *database = NULL; - char database_error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]; - Lardon3DProjectDbResult database_result = lardon3d_project_db_open( - database_path, - &database, - database_error - ); - if (database_result != LARDON3D_PROJECT_DB_OK) { - cleanup_directories(&created); - set_database_status( - state, - NULL, - database_error, - "ouverture impossible" - ); - return false; - } - - Lardon3DProjectDbProject db_project; - database_result = lardon3d_project_db_get_project(database, &db_project); - bool rewrite_metadata = metadata.version == 1; - struct timespec now; - if (clock_gettime(CLOCK_REALTIME, &now) != 0) { - database_result = LARDON3D_PROJECT_DB_IO_ERROR; - } else if (database_result == LARDON3D_PROJECT_DB_OK) { - if (metadata.stable_id[0] - && strcmp(metadata.stable_id, db_project.stable_id) != 0) { - database_result = LARDON3D_PROJECT_DB_CONSTRAINT; - } else if (!metadata.stable_id[0]) { - (void)snprintf( - metadata.stable_id, - sizeof(metadata.stable_id), - "%s", - db_project.stable_id - ); - rewrite_metadata = true; - } - if (database_result == LARDON3D_PROJECT_DB_OK) { - db_project.updated_at = now.tv_sec; - (void)snprintf(db_project.name, sizeof(db_project.name), "%s", metadata.name); - database_result = lardon3d_project_db_set_project(database, &db_project); - } - } else if (database_result == LARDON3D_PROJECT_DB_NOT_FOUND) { - if (database_existed && !metadata.stable_id[0]) { - database_result = LARDON3D_PROJECT_DB_CONSTRAINT; - } else { - if (!metadata.stable_id[0]) { - if (!generate_stable_id(metadata.stable_id)) { - database_result = LARDON3D_PROJECT_DB_IO_ERROR; - } - rewrite_metadata = true; - } - if (database_result != LARDON3D_PROJECT_DB_IO_ERROR) { - Lardon3DProjectDbProject new_project = { - .created_at = now.tv_sec, - .updated_at = now.tv_sec, - }; - (void)snprintf(new_project.stable_id, sizeof(new_project.stable_id), "%s", metadata.stable_id); - (void)snprintf(new_project.name, sizeof(new_project.name), "%s", metadata.name); - database_result = lardon3d_project_db_set_project(database, &new_project); - } - } - } - if (database_result == LARDON3D_PROJECT_DB_OK && rewrite_metadata - && !write_project_ini(state, project_path, metadata.name, metadata.stable_id)) { - database_result = LARDON3D_PROJECT_DB_IO_ERROR; - } - if (database_result != LARDON3D_PROJECT_DB_OK) { - if (database_result == LARDON3D_PROJECT_DB_CONSTRAINT) { - set_status(state, "Erreur : identité project.ini/project.db incohérente."); - } else { - set_database_status( - state, - database, - database_error, - "initialisation impossible" - ); - } - lardon3d_project_db_close(database); - if (!database_existed) { - (void)unlink(database_path); - } - cleanup_directories(&created); - return false; - } - - clear_catalog(state); - state->project_loaded = true; - state->project_db = database; - (void)copy_path( - state->project_name, - sizeof(state->project_name), - metadata.name - ); - (void)snprintf( - state->project_stable_id, - sizeof(state->project_stable_id), - "%s", - metadata.stable_id - ); - (void)copy_path( - state->project_path, - sizeof(state->project_path), - project_path - ); - store_recovery_summary(state, NULL); - if (state->task_queue && state->resource_governor) { - Lardon3DProjectRecoverySummary summary; - (void)lardon3d_project_resume_recoverable_tasks(state, - lardon3d_task_kind_registry_production(), &summary); - (void)snprintf(state->status_message, sizeof(state->status_message), - "Projet ouvert — %zu tâche(s) reprise(s), %zu ignorée(s), %zu en échec%s.", - summary.resumed, summary.skipped, summary.failed, - summary.queue_full ? ", fenêtre de reprise saturée" : ""); + } + if (database_result == LARDON3D_PROJECT_DB_OK && rewrite_metadata && + !write_project_ini(state, project_path, metadata.name, metadata.stable_id)) { + database_result = LARDON3D_PROJECT_DB_IO_ERROR; + } + if (database_result != LARDON3D_PROJECT_DB_OK) { + if (database_result == LARDON3D_PROJECT_DB_CONSTRAINT) { + set_status(state, "Erreur : identité project.ini/project.db incohérente."); } else { - (void)snprintf( - state->status_message, - sizeof(state->status_message), - "Projet ouvert : %s", - state->project_name - ); + set_database_status(state, database, database_error, "initialisation impossible"); } - return true; + lardon3d_project_db_close(database); + if (!database_existed) { + (void)unlink(database_path); + } + cleanup_directories(&created); + return false; + } + + clear_catalog(state); + state->project_loaded = true; + state->project_db = database; + (void)copy_path(state->project_name, sizeof(state->project_name), metadata.name); + (void)snprintf(state->project_stable_id, sizeof(state->project_stable_id), "%s", + metadata.stable_id); + (void)copy_path(state->project_path, sizeof(state->project_path), project_path); + store_recovery_summary(state, NULL); + if (state->task_queue && state->resource_governor) { + Lardon3DProjectRecoverySummary summary; + (void)lardon3d_project_resume_recoverable_tasks(state, lardon3d_task_kind_registry_production(), + &summary); + (void)snprintf(state->status_message, sizeof(state->status_message), + "Projet ouvert — %zu tâche(s) reprise(s), %zu ignorée(s), %zu en échec%s.", + summary.resumed, summary.skipped, summary.failed, + summary.queue_full ? ", fenêtre de reprise saturée" : ""); + } else { + (void)snprintf(state->status_message, sizeof(state->status_message), "Projet ouvert : %s", + state->project_name); + } + return true; } -void -lardon3d_project_close(Lardon3DAppState *state) -{ - if (!state) { - return; - } +void lardon3d_project_close(Lardon3DAppState *state) { + if (!state) { + return; + } - if (!state->project_loaded) { - set_status(state, "Aucun projet à fermer."); - return; - } + if (!state->project_loaded) { + set_status(state, "Aucun projet à fermer."); + return; + } - clear_catalog(state); - lardon3d_project_db_close(state->project_db); - state->project_db = NULL; - state->project_loaded = false; - state->project_name[0] = '\0'; - state->project_path[0] = '\0'; - state->project_stable_id[0] = '\0'; - store_recovery_summary(state, NULL); - set_status(state, "Projet fermé."); + clear_catalog(state); + lardon3d_project_db_close(state->project_db); + state->project_db = NULL; + state->project_loaded = false; + state->project_name[0] = '\0'; + state->project_path[0] = '\0'; + state->project_stable_id[0] = '\0'; + store_recovery_summary(state, NULL); + set_status(state, "Projet fermé."); } -static bool -checkpoint_paths( - const Lardon3DAppState *state, - uint64_t task_id, - char relative[LARDON3D_PROJECT_DB_PATH_CAPACITY], - char absolute[PATH_MAX] -) -{ - int written = snprintf( - relative, - LARDON3D_PROJECT_DB_PATH_CAPACITY, - ".lardon3d/checkpoints/%" PRIu64 ".chk", - task_id - ); - return written > 0 - && (size_t)written < LARDON3D_PROJECT_DB_PATH_CAPACITY - && join_path(absolute, PATH_MAX, state->project_path, relative); +static bool checkpoint_paths(const Lardon3DAppState *state, uint64_t task_id, + char relative[LARDON3D_PROJECT_DB_PATH_CAPACITY], + char absolute[PATH_MAX]) { + int written = snprintf(relative, LARDON3D_PROJECT_DB_PATH_CAPACITY, + ".lardon3d/checkpoints/%" PRIu64 ".chk", task_id); + return written > 0 && (size_t)written < LARDON3D_PROJECT_DB_PATH_CAPACITY && + join_path(absolute, PATH_MAX, state->project_path, relative); } static Lardon3DProjectTaskCheckpointResult -checkpoint_task_internal( - Lardon3DAppState *state, - const Lardon3DTask *task, - const char *image_import_source, - uint64_t image_import_scanset_id -) -{ - if (!state || !state->project_loaded || !state->project_db) { - return LARDON3D_PROJECT_TASK_CHECKPOINT_NO_PROJECT; - } - Lardon3DTaskDurableSnapshot snapshot; - if (!lardon3d_task_durable_snapshot(task, &snapshot) || snapshot.id == 0) { - return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; - } - char task_kind[LARDON3D_TASK_KIND_CAPACITY]; - uint32_t task_kind_version = 0; - if (!lardon3d_task_kind(task, task_kind, &task_kind_version)) { - return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; - } - char relative[LARDON3D_PROJECT_DB_PATH_CAPACITY]; - char absolute[PATH_MAX]; - if (!checkpoint_paths(state, snapshot.id, relative, absolute)) { - return LARDON3D_PROJECT_TASK_CHECKPOINT_IO_ERROR; - } - Lardon3DTaskCheckpointResult saved = lardon3d_task_checkpoint_save( - absolute, - &snapshot - ); - if (saved == LARDON3D_TASK_CHECKPOINT_INVALID) { - return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; - } - if (saved != LARDON3D_TASK_CHECKPOINT_OK - && saved != LARDON3D_TASK_CHECKPOINT_PUBLISHED_NOT_DURABLE) { - return LARDON3D_PROJECT_TASK_CHECKPOINT_IO_ERROR; - } - struct timespec now; - if (clock_gettime(CLOCK_REALTIME, &now) != 0) { - return LARDON3D_PROJECT_TASK_CHECKPOINT_DB_ERROR; - } - Lardon3DProjectDbCheckpoint checkpoint = { - .format_version = LARDON3D_TASK_CHECKPOINT_VERSION, - .durability = saved == LARDON3D_TASK_CHECKPOINT_OK - ? LARDON3D_DB_CHECKPOINT_DURABLE - : LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE, - .updated_at = now.tv_sec, - }; - (void)snprintf(checkpoint.path, sizeof(checkpoint.path), "%s", relative); - Lardon3DProjectDbResult recorded = image_import_source - ? lardon3d_project_db_record_image_import_task( - state->project_db, &snapshot, task_kind, task_kind_version, - &checkpoint, image_import_source, image_import_scanset_id, - now.tv_sec) - : lardon3d_project_db_record_task( - state->project_db, &snapshot, task_kind, task_kind_version, - &checkpoint, now.tv_sec); - if (recorded == LARDON3D_PROJECT_DB_BUSY) { - return LARDON3D_PROJECT_TASK_CHECKPOINT_DB_BUSY; - } - if (recorded != LARDON3D_PROJECT_DB_OK) { - return LARDON3D_PROJECT_TASK_CHECKPOINT_DB_ERROR; - } - return saved == LARDON3D_TASK_CHECKPOINT_OK - ? LARDON3D_PROJECT_TASK_CHECKPOINT_OK - : LARDON3D_PROJECT_TASK_CHECKPOINT_PUBLISHED_NOT_DURABLE; +checkpoint_task_internal(Lardon3DAppState *state, const Lardon3DTask *task, + const char *image_import_source, uint64_t image_import_scanset_id, + const Lardon3DProjectDbFeatureExtractTask *feature_parameters) { + if (!state || !state->project_loaded || !state->project_db) { + return LARDON3D_PROJECT_TASK_CHECKPOINT_NO_PROJECT; + } + Lardon3DTaskDurableSnapshot snapshot; + if (!lardon3d_task_durable_snapshot(task, &snapshot) || snapshot.id == 0) { + return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; + } + char task_kind[LARDON3D_TASK_KIND_CAPACITY]; + uint32_t task_kind_version = 0; + if (!lardon3d_task_kind(task, task_kind, &task_kind_version)) { + return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; + } + char relative[LARDON3D_PROJECT_DB_PATH_CAPACITY]; + char absolute[PATH_MAX]; + if (!checkpoint_paths(state, snapshot.id, relative, absolute)) { + return LARDON3D_PROJECT_TASK_CHECKPOINT_IO_ERROR; + } + Lardon3DTaskCheckpointResult saved = lardon3d_task_checkpoint_save(absolute, &snapshot); + if (saved == LARDON3D_TASK_CHECKPOINT_INVALID) { + return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; + } + if (saved != LARDON3D_TASK_CHECKPOINT_OK && + saved != LARDON3D_TASK_CHECKPOINT_PUBLISHED_NOT_DURABLE) { + return LARDON3D_PROJECT_TASK_CHECKPOINT_IO_ERROR; + } + struct timespec now; + if (clock_gettime(CLOCK_REALTIME, &now) != 0) { + return LARDON3D_PROJECT_TASK_CHECKPOINT_DB_ERROR; + } + Lardon3DProjectDbCheckpoint checkpoint = { + .format_version = LARDON3D_TASK_CHECKPOINT_VERSION, + .durability = saved == LARDON3D_TASK_CHECKPOINT_OK + ? LARDON3D_DB_CHECKPOINT_DURABLE + : LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE, + .updated_at = now.tv_sec, + }; + (void)snprintf(checkpoint.path, sizeof(checkpoint.path), "%s", relative); + Lardon3DProjectDbResult recorded = + image_import_source + ? lardon3d_project_db_record_image_import_task( + state->project_db, &snapshot, task_kind, task_kind_version, &checkpoint, + image_import_source, image_import_scanset_id, now.tv_sec) + : feature_parameters + ? lardon3d_project_db_record_feature_extract_task(state->project_db, &snapshot, task_kind, + task_kind_version, &checkpoint, + feature_parameters, now.tv_sec) + : lardon3d_project_db_record_task(state->project_db, &snapshot, task_kind, + task_kind_version, &checkpoint, now.tv_sec); + if (recorded == LARDON3D_PROJECT_DB_BUSY) { + return LARDON3D_PROJECT_TASK_CHECKPOINT_DB_BUSY; + } + if (recorded != LARDON3D_PROJECT_DB_OK) { + return LARDON3D_PROJECT_TASK_CHECKPOINT_DB_ERROR; + } + return saved == LARDON3D_TASK_CHECKPOINT_OK + ? LARDON3D_PROJECT_TASK_CHECKPOINT_OK + : LARDON3D_PROJECT_TASK_CHECKPOINT_PUBLISHED_NOT_DURABLE; +} + +Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_task(Lardon3DAppState *state, + const Lardon3DTask *task) { + return checkpoint_task_internal(state, task, NULL, 0, NULL); } Lardon3DProjectTaskCheckpointResult -lardon3d_project_checkpoint_task( - Lardon3DAppState *state, - const Lardon3DTask *task -) -{ - return checkpoint_task_internal(state, task, NULL, 0); +lardon3d_project_checkpoint_image_import_task(Lardon3DAppState *state, const Lardon3DTask *task, + const char *source_path, uint64_t scanset_id) { + if (!source_path || !source_path[0] || scanset_id == 0) { + return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; + } + return checkpoint_task_internal(state, task, source_path, scanset_id, NULL); } -Lardon3DProjectTaskCheckpointResult -lardon3d_project_checkpoint_image_import_task( - Lardon3DAppState *state, - const Lardon3DTask *task, - const char *source_path, - uint64_t scanset_id -) -{ - if (!source_path || !source_path[0] || scanset_id == 0) { - return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; +Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_task( + Lardon3DAppState *state, const Lardon3DTask *task, + const Lardon3DProjectDbFeatureExtractTask *parameters) { + if (!parameters) { + return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; + } + return checkpoint_task_internal(state, task, NULL, 0, parameters); +} + +static bool coherent_recovery(const Lardon3DProjectDbTask *database_task, + const Lardon3DTaskDurableSnapshot *snapshot) { + return snapshot->id == database_task->task_id && + strcmp(snapshot->name, database_task->name) == 0 && + snapshot->saved_state == database_task->saved_state && + snapshot->recovery_state == database_task->recovery_state && + snapshot->progress == database_task->progress && + snapshot->sequence_count == database_task->sequence_count; +} + +Lardon3DProjectDbResult lardon3d_project_list_recoverable(Lardon3DAppState *state, + const Lardon3DTaskKindRegistry *registry, + uint64_t after_task_id, + Lardon3DProjectRecoveryEntry *entries, + size_t capacity, size_t *count) { + if (count) { + *count = 0; + } + if (!state || !state->project_loaded || !state->project_db || !registry || !entries || !count || + capacity == 0 || capacity > LARDON3D_PROJECT_DB_RECOVERY_PAGE_MAX) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + enum { RECOVERY_CHUNK = 8 }; + Lardon3DProjectDbTask tasks[RECOVERY_CHUNK]; + uint64_t cursor = after_task_id; + while (*count < capacity) { + size_t requested = capacity - *count; + if (requested > RECOVERY_CHUNK) { + requested = RECOVERY_CHUNK; } - return checkpoint_task_internal(state, task, source_path, scanset_id); -} - -static bool -coherent_recovery( - const Lardon3DProjectDbTask *database_task, - const Lardon3DTaskDurableSnapshot *snapshot -) -{ - return snapshot->id == database_task->task_id - && strcmp(snapshot->name, database_task->name) == 0 - && snapshot->saved_state == database_task->saved_state - && snapshot->recovery_state == database_task->recovery_state - && snapshot->progress == database_task->progress - && snapshot->sequence_count == database_task->sequence_count; + size_t task_count = 0; + Lardon3DProjectDbResult result = lardon3d_project_db_list_recoverable( + state->project_db, cursor, tasks, requested, &task_count); + if (result != LARDON3D_PROJECT_DB_OK) { + return result; + } + for (size_t index = 0; index < task_count; ++index) { + Lardon3DProjectRecoveryEntry *entry = &entries[*count]; + memset(entry, 0, sizeof(*entry)); + entry->task_id = tasks[index].task_id; + entry->durability = tasks[index].checkpoint.durability; + (void)snprintf(entry->name, sizeof(entry->name), "%s", tasks[index].name); + if (tasks[index].has_task_kind) { + (void)snprintf(entry->task_kind, sizeof(entry->task_kind), "%s", tasks[index].task_kind); + entry->task_kind_version = tasks[index].task_kind_version; + } + char relative[LARDON3D_PROJECT_DB_PATH_CAPACITY]; + char absolute[PATH_MAX]; + const Lardon3DTaskKindDescriptor *descriptor = NULL; + Lardon3DTaskKindResult kind_result = + tasks[index].has_task_kind + ? lardon3d_task_kind_registry_lookup(registry, tasks[index].task_kind, + tasks[index].task_kind_version, &descriptor) + : LARDON3D_TASK_KIND_UNKNOWN; + if (!tasks[index].has_task_kind) { + entry->status = LARDON3D_PROJECT_RECOVERY_LEGACY_UNTYPED; + } else if (kind_result == LARDON3D_TASK_KIND_UNKNOWN) { + entry->status = LARDON3D_PROJECT_RECOVERY_UNKNOWN_TASK_KIND; + } else if (kind_result == LARDON3D_TASK_KIND_UNSUPPORTED_VERSION) { + entry->status = LARDON3D_PROJECT_RECOVERY_UNSUPPORTED_TASK_KIND_VERSION; + } else if (kind_result != LARDON3D_TASK_KIND_OK) { + entry->status = LARDON3D_PROJECT_RECOVERY_UNKNOWN_TASK_KIND; + } else if (!checkpoint_paths(state, entry->task_id, relative, absolute) || + strcmp(relative, tasks[index].checkpoint.path) != 0) { + entry->status = LARDON3D_PROJECT_RECOVERY_INVALID_CHECKPOINT; + } else { + uint32_t version = 0; + Lardon3DTaskCheckpointResult loaded = + lardon3d_task_checkpoint_load(absolute, &entry->snapshot, &version); + if (loaded == LARDON3D_TASK_CHECKPOINT_NOT_FOUND) { + entry->status = LARDON3D_PROJECT_RECOVERY_MISSING_CHECKPOINT; + } else if (loaded == LARDON3D_TASK_CHECKPOINT_UNSUPPORTED_VERSION) { + entry->status = LARDON3D_PROJECT_RECOVERY_UNSUPPORTED_CHECKPOINT; + } else if (loaded == LARDON3D_TASK_CHECKPOINT_INVALID) { + entry->status = LARDON3D_PROJECT_RECOVERY_INVALID_CHECKPOINT; + } else if (loaded != LARDON3D_TASK_CHECKPOINT_OK || + version != tasks[index].checkpoint.format_version || + !coherent_recovery(&tasks[index], &entry->snapshot)) { + entry->status = loaded == LARDON3D_TASK_CHECKPOINT_IO_ERROR + ? LARDON3D_PROJECT_RECOVERY_CHECKPOINT_IO_ERROR + : LARDON3D_PROJECT_RECOVERY_INVALID_CHECKPOINT; + } else { + entry->status = entry->durability == LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE + ? LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE + : LARDON3D_PROJECT_RECOVERABLE; + } + } + cursor = entry->task_id; + ++*count; + } + if (task_count < requested) { + break; + } + } + return LARDON3D_PROJECT_DB_OK; } Lardon3DProjectDbResult -lardon3d_project_list_recoverable( - Lardon3DAppState *state, - const Lardon3DTaskKindRegistry *registry, - uint64_t after_task_id, - Lardon3DProjectRecoveryEntry *entries, - size_t capacity, - size_t *count -) -{ - if (count) { - *count = 0; +lardon3d_project_resume_recoverable_tasks(Lardon3DAppState *state, + const Lardon3DTaskKindRegistry *registry, + Lardon3DProjectRecoverySummary *summary) { + if (summary) { + memset(summary, 0, sizeof(*summary)); + } + if (!state || !state->project_loaded || !state->project_db || !state->task_queue || + !state->resource_governor || !registry || !summary) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + enum { RECOVERY_PAGE_SIZE = 8 }; + uint64_t cursor = 0; + Lardon3DProjectDbResult result = LARDON3D_PROJECT_DB_OK; + bool stop = false; + do { + Lardon3DProjectRecoveryEntry entries[RECOVERY_PAGE_SIZE]; + size_t count = 0; + result = lardon3d_project_list_recoverable(state, registry, cursor, entries, RECOVERY_PAGE_SIZE, + &count); + if (result != LARDON3D_PROJECT_DB_OK) { + ++summary->failed; + break; } - if (!state || !state->project_loaded || !state->project_db || !registry || !entries - || !count || capacity == 0 - || capacity > LARDON3D_PROJECT_DB_RECOVERY_PAGE_MAX) { - return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + for (size_t index = 0; index < count; ++index) { + Lardon3DProjectRecoveryEntry *entry = &entries[index]; + cursor = entry->task_id; + ++summary->inspected; + bool recoverable = entry->status == LARDON3D_PROJECT_RECOVERABLE || + entry->status == LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE; + if (!recoverable || entry->snapshot.recovery_state != TASK_PENDING) { + ++summary->skipped; + continue; + } + if (entry->status == LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE) { + ++summary->published_not_durable; + } + Lardon3DTaskSnapshot existing; + if (lardon3d_task_queue_get(state->task_queue, entry->task_id, &existing)) { + ++summary->skipped; + continue; + } + Lardon3DImageImportReconstructionContext context = { + .project_path = state->project_path, + .project_db = state->project_db, + .resource_governor = state->resource_governor, + }; + Lardon3DTask *task = NULL; + Lardon3DTaskKindResult restored = lardon3d_task_kind_registry_restore( + registry, entry->task_kind, entry->task_kind_version, &entry->snapshot, &context, &task); + if (restored != LARDON3D_TASK_KIND_OK || !task) { + ++summary->failed; + continue; + } + Lardon3DTaskQueueAddResult added = + lardon3d_task_queue_try_add_ex(state->task_queue, task, NULL); + if (added == LARDON3D_TASK_QUEUE_ADD_OK) { + ++summary->resumed; + continue; + } + lardon3d_task_destroy(task); + if (added == LARDON3D_TASK_QUEUE_ADD_DUPLICATE_ID) { + ++summary->skipped; + } else if (added == LARDON3D_TASK_QUEUE_ADD_FULL) { + ++summary->skipped; + } else { + ++summary->failed; + } + if (added == LARDON3D_TASK_QUEUE_ADD_FULL || added == LARDON3D_TASK_QUEUE_ADD_STOPPING) { + summary->queue_full = added == LARDON3D_TASK_QUEUE_ADD_FULL; + stop = true; + break; + } } - enum { RECOVERY_CHUNK = 8 }; - Lardon3DProjectDbTask tasks[RECOVERY_CHUNK]; - uint64_t cursor = after_task_id; - while (*count < capacity) { - size_t requested = capacity - *count; - if (requested > RECOVERY_CHUNK) { - requested = RECOVERY_CHUNK; - } - size_t task_count = 0; - Lardon3DProjectDbResult result = lardon3d_project_db_list_recoverable( - state->project_db, - cursor, - tasks, - requested, - &task_count - ); - if (result != LARDON3D_PROJECT_DB_OK) { - return result; - } - for (size_t index = 0; index < task_count; ++index) { - Lardon3DProjectRecoveryEntry *entry = &entries[*count]; - memset(entry, 0, sizeof(*entry)); - entry->task_id = tasks[index].task_id; - entry->durability = tasks[index].checkpoint.durability; - (void)snprintf(entry->name, sizeof(entry->name), "%s", tasks[index].name); - if (tasks[index].has_task_kind) { - (void)snprintf(entry->task_kind, sizeof(entry->task_kind), "%s", - tasks[index].task_kind); - entry->task_kind_version = tasks[index].task_kind_version; - } - char relative[LARDON3D_PROJECT_DB_PATH_CAPACITY]; - char absolute[PATH_MAX]; - const Lardon3DTaskKindDescriptor *descriptor = NULL; - Lardon3DTaskKindResult kind_result = tasks[index].has_task_kind - ? lardon3d_task_kind_registry_lookup( - registry, tasks[index].task_kind, - tasks[index].task_kind_version, &descriptor) - : LARDON3D_TASK_KIND_UNKNOWN; - if (!tasks[index].has_task_kind) { - entry->status = LARDON3D_PROJECT_RECOVERY_LEGACY_UNTYPED; - } else if (kind_result == LARDON3D_TASK_KIND_UNKNOWN) { - entry->status = LARDON3D_PROJECT_RECOVERY_UNKNOWN_TASK_KIND; - } else if (kind_result == LARDON3D_TASK_KIND_UNSUPPORTED_VERSION) { - entry->status = LARDON3D_PROJECT_RECOVERY_UNSUPPORTED_TASK_KIND_VERSION; - } else if (kind_result != LARDON3D_TASK_KIND_OK) { - entry->status = LARDON3D_PROJECT_RECOVERY_UNKNOWN_TASK_KIND; - } else if (!checkpoint_paths(state, entry->task_id, relative, absolute) - || strcmp(relative, tasks[index].checkpoint.path) != 0) { - entry->status = LARDON3D_PROJECT_RECOVERY_INVALID_CHECKPOINT; - } else { - uint32_t version = 0; - Lardon3DTaskCheckpointResult loaded = lardon3d_task_checkpoint_load( - absolute, - &entry->snapshot, - &version - ); - if (loaded == LARDON3D_TASK_CHECKPOINT_NOT_FOUND) { - entry->status = LARDON3D_PROJECT_RECOVERY_MISSING_CHECKPOINT; - } else if (loaded == LARDON3D_TASK_CHECKPOINT_UNSUPPORTED_VERSION) { - entry->status = LARDON3D_PROJECT_RECOVERY_UNSUPPORTED_CHECKPOINT; - } else if (loaded == LARDON3D_TASK_CHECKPOINT_INVALID) { - entry->status = LARDON3D_PROJECT_RECOVERY_INVALID_CHECKPOINT; - } else if (loaded != LARDON3D_TASK_CHECKPOINT_OK - || version != tasks[index].checkpoint.format_version - || !coherent_recovery(&tasks[index], &entry->snapshot)) { - entry->status = loaded == LARDON3D_TASK_CHECKPOINT_IO_ERROR - ? LARDON3D_PROJECT_RECOVERY_CHECKPOINT_IO_ERROR - : LARDON3D_PROJECT_RECOVERY_INVALID_CHECKPOINT; - } else { - entry->status = entry->durability - == LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE - ? LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE - : LARDON3D_PROJECT_RECOVERABLE; - } - } - cursor = entry->task_id; - ++*count; - } - if (task_count < requested) { - break; - } + if (stop || count < RECOVERY_PAGE_SIZE) { + break; } - return LARDON3D_PROJECT_DB_OK; + } while (cursor > 0); + store_recovery_summary(state, summary); + return result; } -Lardon3DProjectDbResult -lardon3d_project_resume_recoverable_tasks( - Lardon3DAppState *state, - const Lardon3DTaskKindRegistry *registry, - Lardon3DProjectRecoverySummary *summary -) -{ - if (summary) memset(summary, 0, sizeof(*summary)); - if (!state || !state->project_loaded || !state->project_db - || !state->task_queue || !state->resource_governor || !registry - || !summary) { - return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; - } - enum { RECOVERY_PAGE_SIZE = 8 }; - uint64_t cursor = 0; - Lardon3DProjectDbResult result = LARDON3D_PROJECT_DB_OK; - bool stop = false; - do { - Lardon3DProjectRecoveryEntry entries[RECOVERY_PAGE_SIZE]; - size_t count = 0; - result = lardon3d_project_list_recoverable(state, registry, cursor, - entries, RECOVERY_PAGE_SIZE, &count); - if (result != LARDON3D_PROJECT_DB_OK) { - ++summary->failed; - break; - } - for (size_t index = 0; index < count; ++index) { - Lardon3DProjectRecoveryEntry *entry = &entries[index]; - cursor = entry->task_id; - ++summary->inspected; - bool recoverable = entry->status == LARDON3D_PROJECT_RECOVERABLE - || entry->status - == LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE; - if (!recoverable || entry->snapshot.recovery_state != TASK_PENDING) { - ++summary->skipped; - continue; - } - if (entry->status - == LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE) { - ++summary->published_not_durable; - } - Lardon3DTaskSnapshot existing; - if (lardon3d_task_queue_get(state->task_queue, entry->task_id, - &existing)) { - ++summary->skipped; - continue; - } - Lardon3DImageImportReconstructionContext context = { - .project_path = state->project_path, - .project_db = state->project_db, - .resource_governor = state->resource_governor, - }; - Lardon3DTask *task = NULL; - Lardon3DTaskKindResult restored = - lardon3d_task_kind_registry_restore(registry, - entry->task_kind, entry->task_kind_version, - &entry->snapshot, &context, &task); - if (restored != LARDON3D_TASK_KIND_OK || !task) { - ++summary->failed; - continue; - } - Lardon3DTaskQueueAddResult added = - lardon3d_task_queue_try_add_ex(state->task_queue, task, NULL); - if (added == LARDON3D_TASK_QUEUE_ADD_OK) { - ++summary->resumed; - continue; - } - lardon3d_task_destroy(task); - if (added == LARDON3D_TASK_QUEUE_ADD_DUPLICATE_ID) { - ++summary->skipped; - } else if (added == LARDON3D_TASK_QUEUE_ADD_FULL) { - ++summary->skipped; - } else { - ++summary->failed; - } - if (added == LARDON3D_TASK_QUEUE_ADD_FULL - || added == LARDON3D_TASK_QUEUE_ADD_STOPPING) { - summary->queue_full = added == LARDON3D_TASK_QUEUE_ADD_FULL; - stop = true; - break; - } - } - if (stop || count < RECOVERY_PAGE_SIZE) break; - } while (cursor > 0); - store_recovery_summary(state, summary); - return result; -} - -bool -lardon3d_project_last_recovery_summary( - const Lardon3DAppState *state, - Lardon3DProjectRecoverySummary *summary -) -{ - if (!state || !summary) return false; - *summary = (Lardon3DProjectRecoverySummary) { - .inspected = state->recovery_inspected, - .resumed = state->recovery_resumed, - .skipped = state->recovery_skipped, - .failed = state->recovery_failed, - .published_not_durable = state->recovery_published_not_durable, - .queue_full = state->recovery_queue_full, - }; - return true; +bool lardon3d_project_last_recovery_summary(const Lardon3DAppState *state, + Lardon3DProjectRecoverySummary *summary) { + if (!state || !summary) { + return false; + } + *summary = (Lardon3DProjectRecoverySummary){ + .inspected = state->recovery_inspected, + .resumed = state->recovery_resumed, + .skipped = state->recovery_skipped, + .failed = state->recovery_failed, + .published_not_durable = state->recovery_published_not_durable, + .queue_full = state->recovery_queue_full, + }; + return true; } diff --git a/src/project_db.c b/src/project_db.c index f4d88af..b3610da 100644 --- a/src/project_db.c +++ b/src/project_db.c @@ -12,1196 +12,2019 @@ #include struct Lardon3DProjectDb { - sqlite3 *connection; - pthread_mutex_t mutex; - char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]; + sqlite3 *connection; + pthread_mutex_t mutex; + char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]; }; -static const char schema_v4[] = +static const char schema_v5[] = "CREATE TABLE metadata(key TEXT PRIMARY KEY,value INTEGER NOT NULL);" - "INSERT INTO metadata(key,value) VALUES('schema_version',4);" + "INSERT INTO metadata(key,value) VALUES('schema_version',5);" "INSERT INTO metadata(key,value) VALUES('next_task_id',1);" "INSERT INTO metadata(key,value) VALUES('legacy_image_catalog_pending',0);" "CREATE TABLE project(singleton INTEGER PRIMARY KEY CHECK(singleton=1)," - "stable_id TEXT NOT NULL UNIQUE,name TEXT NOT NULL,created_at INTEGER NOT NULL,updated_at INTEGER NOT NULL);" + "stable_id TEXT NOT NULL UNIQUE,name TEXT NOT NULL,created_at INTEGER NOT NULL,updated_at " + "INTEGER NOT NULL);" "CREATE TABLE tasks(task_id INTEGER PRIMARY KEY CHECK(task_id>0),name TEXT NOT NULL," "task_kind TEXT,task_kind_version INTEGER," "saved_state INTEGER NOT NULL CHECK(saved_state BETWEEN 0 AND 5)," "recovery_state INTEGER NOT NULL CHECK(recovery_state BETWEEN 0 AND 5)," - "progress INTEGER NOT NULL CHECK(progress BETWEEN 0 AND 100),sequence_count INTEGER NOT NULL CHECK(sequence_count>=0)," - "started_sec INTEGER NOT NULL,started_nsec INTEGER NOT NULL CHECK(started_nsec BETWEEN 0 AND 999999999)," - "finished_sec INTEGER NOT NULL,finished_nsec INTEGER NOT NULL CHECK(finished_nsec BETWEEN 0 AND 999999999),updated_at INTEGER NOT NULL," - "CHECK((task_kind IS NULL AND task_kind_version IS NULL) OR (task_kind IS NOT NULL AND task_kind_version>0)));" + "progress INTEGER NOT NULL CHECK(progress BETWEEN 0 AND 100),sequence_count INTEGER NOT NULL " + "CHECK(sequence_count>=0)," + "started_sec INTEGER NOT NULL,started_nsec INTEGER NOT NULL CHECK(started_nsec BETWEEN 0 AND " + "999999999)," + "finished_sec INTEGER NOT NULL,finished_nsec INTEGER NOT NULL CHECK(finished_nsec BETWEEN 0 " + "AND 999999999),updated_at INTEGER NOT NULL," + "CHECK((task_kind IS NULL AND task_kind_version IS NULL) OR (task_kind IS NOT NULL AND " + "task_kind_version>0)));" "CREATE INDEX tasks_recovery_state_idx ON tasks(recovery_state,task_id);" - "CREATE TABLE checkpoints(task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE CASCADE," + "CREATE TABLE checkpoints(task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE " + "CASCADE," "path TEXT NOT NULL,format_version INTEGER NOT NULL CHECK(format_version>0)," "durability INTEGER NOT NULL CHECK(durability BETWEEN 0 AND 1),updated_at INTEGER NOT NULL);" "CREATE TABLE artifacts(artifact_id TEXT PRIMARY KEY,kind TEXT NOT NULL,path TEXT NOT NULL," - "state INTEGER NOT NULL CHECK(state BETWEEN 0 AND 1),size_bytes INTEGER NOT NULL CHECK(size_bytes>=0)," - "producer_task_id INTEGER REFERENCES tasks(task_id),created_at INTEGER NOT NULL,updated_at INTEGER NOT NULL);" + "state INTEGER NOT NULL CHECK(state BETWEEN 0 AND 1),size_bytes INTEGER NOT NULL " + "CHECK(size_bytes>=0)," + "producer_task_id INTEGER REFERENCES tasks(task_id),created_at INTEGER NOT NULL,updated_at " + "INTEGER NOT NULL);" "CREATE INDEX artifacts_state_idx ON artifacts(state,artifact_id);" "CREATE INDEX artifacts_producer_idx ON artifacts(producer_task_id);" "CREATE TABLE scansets(scanset_id INTEGER PRIMARY KEY AUTOINCREMENT CHECK(scanset_id>0)," - "name TEXT NOT NULL CHECK(length(name)>0 AND length(name)<256),created_at INTEGER NOT NULL CHECK(created_at>=0)," + "name TEXT NOT NULL CHECK(length(name)>0 AND length(name)<256),created_at INTEGER NOT NULL " + "CHECK(created_at>=0)," "updated_at INTEGER NOT NULL CHECK(updated_at>=created_at));" - "CREATE TABLE image_assets(asset_id INTEGER PRIMARY KEY AUTOINCREMENT CHECK(asset_id>0),sha256 BLOB NOT NULL UNIQUE CHECK(length(sha256)=32)," - "path TEXT NOT NULL UNIQUE CHECK(length(path)>0 AND length(path)<4096),size_bytes INTEGER NOT NULL CHECK(size_bytes>=0)," + "CREATE TABLE image_assets(asset_id INTEGER PRIMARY KEY AUTOINCREMENT CHECK(asset_id>0),sha256 " + "BLOB NOT NULL UNIQUE CHECK(length(sha256)=32)," + "path TEXT NOT NULL UNIQUE CHECK(length(path)>0 AND length(path)<4096),size_bytes INTEGER NOT " + "NULL CHECK(size_bytes>=0)," "state INTEGER NOT NULL CHECK(state=1),created_at INTEGER NOT NULL CHECK(created_at>=0));" - "CREATE TABLE images(image_id INTEGER PRIMARY KEY AUTOINCREMENT CHECK(image_id>0),scanset_id INTEGER NOT NULL REFERENCES scansets(scanset_id)," - "asset_id INTEGER NOT NULL REFERENCES image_assets(asset_id),original_name TEXT NOT NULL CHECK(length(original_name)>0 AND length(original_name)<256)," + "CREATE TABLE images(image_id INTEGER PRIMARY KEY AUTOINCREMENT CHECK(image_id>0),scanset_id " + "INTEGER NOT NULL REFERENCES scansets(scanset_id)," + "asset_id INTEGER NOT NULL REFERENCES image_assets(asset_id),original_name TEXT NOT NULL " + "CHECK(length(original_name)>0 AND length(original_name)<256)," "source_path TEXT NOT NULL CHECK(length(source_path)>0 AND length(source_path)<4096)," - "producer_task_id INTEGER REFERENCES tasks(task_id),imported_at INTEGER NOT NULL CHECK(imported_at>=0)," + "producer_task_id INTEGER REFERENCES tasks(task_id),imported_at INTEGER NOT NULL " + "CHECK(imported_at>=0)," "UNIQUE(scanset_id,asset_id));" "CREATE INDEX images_scanset_idx ON images(scanset_id,image_id);" "CREATE INDEX images_producer_idx ON images(producer_task_id,image_id);" - "CREATE TABLE image_import_tasks(task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE CASCADE," + "CREATE TABLE image_import_tasks(task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON " + "DELETE CASCADE," "source_path TEXT NOT NULL,scanset_id INTEGER NOT NULL REFERENCES scansets(scanset_id));"; -static void -copy_error(char destination[LARDON3D_PROJECT_DB_ERROR_CAPACITY], const char *text) -{ - if (destination) { - (void)snprintf(destination, LARDON3D_PROJECT_DB_ERROR_CAPACITY, "%s", text ? text : ""); - } +static const char schema_feature_v5[] = + "CREATE TABLE feature_assets(feature_asset_id INTEGER PRIMARY KEY AUTOINCREMENT " + "CHECK(feature_asset_id>0),sha256 BLOB NOT NULL UNIQUE CHECK(length(sha256)=32),path TEXT NOT " + "NULL UNIQUE CHECK(length(path)>0 AND length(path)<4096),size_bytes INTEGER NOT NULL " + "CHECK(size_bytes>=0),durability INTEGER NOT NULL CHECK(durability BETWEEN 0 AND 1),created_at " + "INTEGER NOT NULL CHECK(created_at>=0));" + "CREATE TABLE feature_sets(feature_set_id INTEGER PRIMARY KEY AUTOINCREMENT " + "CHECK(feature_set_id>0),image_id INTEGER NOT NULL REFERENCES " + "images(image_id),feature_asset_id INTEGER NOT NULL REFERENCES " + "feature_assets(feature_asset_id),extractor_kind TEXT NOT NULL CHECK(length(extractor_kind)>0 " + "AND length(extractor_kind)<65),extractor_version INTEGER NOT NULL " + "CHECK(extractor_version>0),parameter_fingerprint BLOB NOT NULL " + "CHECK(length(parameter_fingerprint)=32),source_image_sha256 BLOB NOT NULL " + "CHECK(length(source_image_sha256)=32),feature_count INTEGER NOT NULL " + "CHECK(feature_count BETWEEN 0 AND 8192),descriptor_type INTEGER NOT NULL " + "CHECK(descriptor_type " + "IN(1,2)),descriptor_dimension INTEGER NOT NULL " + "CHECK(descriptor_dimension BETWEEN 1 AND 4096),producer_task_id " + "INTEGER REFERENCES tasks(task_id),created_at INTEGER NOT NULL " + "CHECK(created_at>=0),UNIQUE(image_id,extractor_kind,extractor_version,parameter_fingerprint));" + "CREATE INDEX feature_sets_image_idx ON feature_sets(image_id,feature_set_id);" + "CREATE INDEX feature_sets_producer_idx ON feature_sets(producer_task_id,feature_set_id);" + "CREATE TABLE feature_extract_tasks(task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON " + "DELETE CASCADE,image_id INTEGER NOT NULL REFERENCES images(image_id),extractor_kind TEXT NOT " + "NULL,extractor_version INTEGER NOT NULL CHECK(extractor_version>0),max_features INTEGER NOT " + "NULL CHECK(max_features BETWEEN 1 AND 8192),pyramid_levels INTEGER NOT NULL " + "CHECK(pyramid_levels BETWEEN 1 AND 16),fast_threshold INTEGER NOT NULL " + "CHECK(fast_threshold BETWEEN 1 AND 255),parameter_fingerprint BLOB NOT NULL " + "CHECK(length(parameter_fingerprint)=32));"; + +static void copy_error(char destination[LARDON3D_PROJECT_DB_ERROR_CAPACITY], const char *text) { + if (destination) { + (void)snprintf(destination, LARDON3D_PROJECT_DB_ERROR_CAPACITY, "%s", text ? text : ""); + } } -static Lardon3DProjectDbResult -sqlite_result(Lardon3DProjectDb *database, int code, const char *context) -{ - (void)snprintf(database->error, sizeof(database->error), "%s: %s", context, sqlite3_errmsg(database->connection)); - if (code == SQLITE_BUSY || code == SQLITE_LOCKED) return LARDON3D_PROJECT_DB_BUSY; - if (code == SQLITE_CONSTRAINT) return LARDON3D_PROJECT_DB_CONSTRAINT; - if (code == SQLITE_CORRUPT || code == SQLITE_NOTADB) return LARDON3D_PROJECT_DB_CORRUPT; - return LARDON3D_PROJECT_DB_IO_ERROR; +static Lardon3DProjectDbResult sqlite_result(Lardon3DProjectDb *database, int code, + const char *context) { + (void)snprintf(database->error, sizeof(database->error), "%s: %s", context, + sqlite3_errmsg(database->connection)); + if (code == SQLITE_BUSY || code == SQLITE_LOCKED) { + return LARDON3D_PROJECT_DB_BUSY; + } + if (code == SQLITE_CONSTRAINT) { + return LARDON3D_PROJECT_DB_CONSTRAINT; + } + if (code == SQLITE_CORRUPT || code == SQLITE_NOTADB) { + return LARDON3D_PROJECT_DB_CORRUPT; + } + return LARDON3D_PROJECT_DB_IO_ERROR; } -static bool -bounded_text(const char *text, size_t capacity, bool allow_empty) -{ - if (!text) return false; - size_t length = strnlen(text, capacity); - return length < capacity && (allow_empty || length > 0); +static bool bounded_text(const char *text, size_t capacity, bool allow_empty) { + if (!text) { + return false; + } + size_t length = strnlen(text, capacity); + return length < capacity && (allow_empty || length > 0); } -static bool -valid_task_id(uint64_t id) -{ - return id > 0 && id <= INT64_MAX; +static bool valid_task_id(uint64_t id) { return id > 0 && id <= INT64_MAX; } + +static bool valid_state(Lardon3DTaskState state) { + return state >= TASK_PENDING && state <= TASK_COMPLETED; } -static bool -valid_state(Lardon3DTaskState state) -{ - return state >= TASK_PENDING && state <= TASK_COMPLETED; +static bool database_time(sqlite3_int64 value, time_t *output) { + if (value < 0) { + return false; + } + time_t converted = (time_t)value; + if (converted < 0 || (uint64_t)converted != (uint64_t)value) { + return false; + } + *output = converted; + return true; } -static bool -database_time(sqlite3_int64 value, time_t *output) -{ - if (value < 0) return false; - time_t converted = (time_t)value; - if (converted < 0 || (uint64_t)converted != (uint64_t)value) return false; - *output = converted; - return true; -} - -static Lardon3DProjectDbResult -execute(Lardon3DProjectDb *database, const char *sql, const char *context) -{ - char *message = NULL; - int code = sqlite3_exec(database->connection, sql, NULL, NULL, &message); - if (code == SQLITE_OK) return LARDON3D_PROJECT_DB_OK; - (void)snprintf(database->error, sizeof(database->error), "%s: %s", context, message ? message : sqlite3_errmsg(database->connection)); - sqlite3_free(message); - return sqlite_result(database, code, context); -} - -static Lardon3DProjectDbResult -prepare(Lardon3DProjectDb *database, const char *sql, sqlite3_stmt **statement) -{ - int code = sqlite3_prepare_v2(database->connection, sql, -1, statement, NULL); - return code == SQLITE_OK ? LARDON3D_PROJECT_DB_OK : sqlite_result(database, code, "prepare"); -} - -static bool -table_exists(sqlite3 *connection, const char *name) -{ - sqlite3_stmt *statement = NULL; - if (sqlite3_prepare_v2(connection, "SELECT 1 FROM sqlite_master WHERE type='table' AND name=?1", -1, &statement, NULL) != SQLITE_OK) return false; - (void)sqlite3_bind_text(statement, 1, name, -1, SQLITE_STATIC); - bool exists = sqlite3_step(statement) == SQLITE_ROW; - (void)sqlite3_finalize(statement); - return exists; -} - -static Lardon3DProjectDbResult -migrate(Lardon3DProjectDb *database, unsigned int from_version) -{ - if (from_version > LARDON3D_PROJECT_DB_SCHEMA_VERSION) { - copy_error(database->error, "Version de schéma future non supportée."); - return LARDON3D_PROJECT_DB_UNSUPPORTED_SCHEMA; - } - if (from_version == LARDON3D_PROJECT_DB_SCHEMA_VERSION) { - return LARDON3D_PROJECT_DB_OK; - } - if (from_version != 0 && from_version != 1 && from_version != 2 - && from_version != 3) { - return LARDON3D_PROJECT_DB_CORRUPT; - } - Lardon3DProjectDbResult result = execute(database, "BEGIN IMMEDIATE", "begin migration"); - if (result == LARDON3D_PROJECT_DB_OK && from_version == 0) { - result = execute(database, schema_v4, "create schema v4"); - } - if (result == LARDON3D_PROJECT_DB_OK && from_version == 1) { - result = execute(database, - "ALTER TABLE tasks ADD COLUMN task_kind TEXT;" - "ALTER TABLE tasks ADD COLUMN task_kind_version INTEGER CHECK(task_kind_version IS NULL OR task_kind_version>0)", - "migrate schema v1 to v2"); -#ifdef LARDON3D_PROJECT_DB_TESTING - const char *forced_failure = getenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V2"); - if (result == LARDON3D_PROJECT_DB_OK && forced_failure - && strcmp(forced_failure, "1") == 0) { - result = execute(database, "INSERT INTO missing_test_table VALUES(1)", - "forced migration failure"); - } -#endif - if (result == LARDON3D_PROJECT_DB_OK) { - result = execute(database, - "UPDATE metadata SET value=2 WHERE key='schema_version' AND value=1", - "finish schema v2 migration"); - } - } - if (result == LARDON3D_PROJECT_DB_OK && from_version != 0 - && from_version < 3) { - result = execute(database, - "CREATE TABLE image_import_tasks(task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE CASCADE,source_path TEXT NOT NULL);" - "INSERT INTO metadata(key,value) VALUES('next_task_id',(SELECT CASE WHEN COALESCE(MAX(task_id),0)>=9223372036854775807 THEN 0 ELSE COALESCE(MAX(task_id),0)+1 END FROM tasks))", - "migrate schema v2 to v3"); -#ifdef LARDON3D_PROJECT_DB_TESTING - const char *forced_failure = getenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V3"); - if (result == LARDON3D_PROJECT_DB_OK && forced_failure - && strcmp(forced_failure, "1") == 0) { - result = execute(database, "INSERT INTO missing_test_table VALUES(1)", - "forced migration failure"); - } -#endif - if (result == LARDON3D_PROJECT_DB_OK) { - result = execute(database, - "UPDATE metadata SET value=3 WHERE key='schema_version' AND value=2", - "finish schema v3 migration"); - } - } - if (result == LARDON3D_PROJECT_DB_OK && from_version != 0) { - result = execute(database, - "CREATE TABLE scansets(scanset_id INTEGER PRIMARY KEY AUTOINCREMENT CHECK(scanset_id>0),name TEXT NOT NULL CHECK(length(name)>0 AND length(name)<256),created_at INTEGER NOT NULL CHECK(created_at>=0),updated_at INTEGER NOT NULL CHECK(updated_at>=created_at));" - "CREATE TABLE image_assets(asset_id INTEGER PRIMARY KEY AUTOINCREMENT CHECK(asset_id>0),sha256 BLOB NOT NULL UNIQUE CHECK(length(sha256)=32),path TEXT NOT NULL UNIQUE CHECK(length(path)>0 AND length(path)<4096),size_bytes INTEGER NOT NULL CHECK(size_bytes>=0),state INTEGER NOT NULL CHECK(state=1),created_at INTEGER NOT NULL CHECK(created_at>=0));" - "CREATE TABLE images(image_id INTEGER PRIMARY KEY AUTOINCREMENT CHECK(image_id>0),scanset_id INTEGER NOT NULL REFERENCES scansets(scanset_id),asset_id INTEGER NOT NULL REFERENCES image_assets(asset_id),original_name TEXT NOT NULL CHECK(length(original_name)>0 AND length(original_name)<256),source_path TEXT NOT NULL CHECK(length(source_path)>0 AND length(source_path)<4096),producer_task_id INTEGER REFERENCES tasks(task_id),imported_at INTEGER NOT NULL CHECK(imported_at>=0),UNIQUE(scanset_id,asset_id));" - "CREATE INDEX images_scanset_idx ON images(scanset_id,image_id);" - "CREATE INDEX images_producer_idx ON images(producer_task_id,image_id);" - "ALTER TABLE image_import_tasks ADD COLUMN scanset_id INTEGER REFERENCES scansets(scanset_id);" - "INSERT INTO scansets(name,created_at,updated_at) SELECT 'Imports antérieurs à ScanSet v1',0,0 WHERE EXISTS(SELECT 1 FROM image_import_tasks);" - "UPDATE image_import_tasks SET scanset_id=(SELECT scanset_id FROM scansets WHERE name='Imports antérieurs à ScanSet v1' ORDER BY scanset_id LIMIT 1) WHERE scanset_id IS NULL;" - "INSERT INTO metadata(key,value) VALUES('legacy_image_catalog_pending',CASE WHEN EXISTS(SELECT 1 FROM image_import_tasks) THEN 1 ELSE 0 END)", - "migrate schema v3 to v4"); -#ifdef LARDON3D_PROJECT_DB_TESTING - const char *forced_failure = getenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V4"); - if (result == LARDON3D_PROJECT_DB_OK && forced_failure - && strcmp(forced_failure, "1") == 0) { - result = execute(database, "INSERT INTO missing_test_table VALUES(1)", - "forced migration failure"); - } -#endif - if (result == LARDON3D_PROJECT_DB_OK) { - result = execute(database, - "UPDATE metadata SET value=4 WHERE key='schema_version' AND value=3", - "finish schema v4 migration"); - } - } - if (result == LARDON3D_PROJECT_DB_OK) result = execute(database, "COMMIT", "commit migration"); - if (result != LARDON3D_PROJECT_DB_OK) (void)execute(database, "ROLLBACK", "rollback migration"); - return result; -} - -static Lardon3DProjectDbResult -read_schema_version(Lardon3DProjectDb *database, unsigned int *version) -{ - if (!table_exists(database->connection, "metadata")) { - sqlite3_stmt *statement = NULL; - int code = sqlite3_prepare_v2(database->connection, "SELECT 1 FROM sqlite_master WHERE type='table' LIMIT 1", -1, &statement, NULL); - if (code != SQLITE_OK) return sqlite_result(database, code, "inspect schema"); - code = sqlite3_step(statement); - bool empty = code == SQLITE_DONE; - (void)sqlite3_finalize(statement); - if (!empty) { - copy_error(database->error, "Base incohérente sans metadata."); - return LARDON3D_PROJECT_DB_CORRUPT; - } - *version = 0; - return LARDON3D_PROJECT_DB_OK; - } - sqlite3_stmt *statement = NULL; - Lardon3DProjectDbResult result = prepare(database, "SELECT value FROM metadata WHERE key='schema_version'", &statement); - if (result != LARDON3D_PROJECT_DB_OK) return result; - int code = sqlite3_step(statement); - if (code != SQLITE_ROW || sqlite3_column_type(statement, 0) != SQLITE_INTEGER) { - (void)sqlite3_finalize(statement); - copy_error(database->error, "schema_version absente ou invalide."); - return LARDON3D_PROJECT_DB_CORRUPT; - } - sqlite3_int64 value = sqlite3_column_int64(statement, 0); - code = sqlite3_step(statement); - (void)sqlite3_finalize(statement); - if (code != SQLITE_DONE || value < 1 || value > UINT_MAX) return LARDON3D_PROJECT_DB_CORRUPT; - *version = (unsigned int)value; +static Lardon3DProjectDbResult execute(Lardon3DProjectDb *database, const char *sql, + const char *context) { + char *message = NULL; + int code = sqlite3_exec(database->connection, sql, NULL, NULL, &message); + if (code == SQLITE_OK) { return LARDON3D_PROJECT_DB_OK; + } + (void)snprintf(database->error, sizeof(database->error), "%s: %s", context, + message ? message : sqlite3_errmsg(database->connection)); + sqlite3_free(message); + return sqlite_result(database, code, context); } -Lardon3DProjectDbResult -lardon3d_project_db_open(const char *path, Lardon3DProjectDb **output, char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]) -{ - if (output) *output = NULL; - if (!path || !path[0] || !output || strnlen(path, LARDON3D_PROJECT_DB_PATH_CAPACITY) >= LARDON3D_PROJECT_DB_PATH_CAPACITY) { - copy_error(error, "Chemin de base invalide."); - return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; - } - Lardon3DProjectDb *database = calloc(1, sizeof(*database)); - if (!database) { copy_error(error, "Allocation impossible."); return LARDON3D_PROJECT_DB_IO_ERROR; } - if (pthread_mutex_init(&database->mutex, NULL) != 0) { free(database); copy_error(error, "Mutex impossible."); return LARDON3D_PROJECT_DB_IO_ERROR; } - int code = sqlite3_open_v2(path, &database->connection, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_NOMUTEX, NULL); - if (code != SQLITE_OK) { - copy_error(error, database->connection ? sqlite3_errmsg(database->connection) : "Ouverture SQLite impossible."); - if (database->connection) (void)sqlite3_close(database->connection); - (void)pthread_mutex_destroy(&database->mutex); free(database); - return code == SQLITE_CANTOPEN ? LARDON3D_PROJECT_DB_IO_ERROR : LARDON3D_PROJECT_DB_CORRUPT; - } - Lardon3DProjectDbResult result = execute(database, "PRAGMA foreign_keys=ON;PRAGMA journal_mode=DELETE;PRAGMA synchronous=FULL;PRAGMA busy_timeout=5000", "configure SQLite"); - unsigned int version = 0; - if (result == LARDON3D_PROJECT_DB_OK) result = read_schema_version(database, &version); - if (result == LARDON3D_PROJECT_DB_OK) result = migrate(database, version); +static Lardon3DProjectDbResult prepare(Lardon3DProjectDb *database, const char *sql, + sqlite3_stmt **statement) { + int code = sqlite3_prepare_v2(database->connection, sql, -1, statement, NULL); + return code == SQLITE_OK ? LARDON3D_PROJECT_DB_OK : sqlite_result(database, code, "prepare"); +} + +static bool table_exists(sqlite3 *connection, const char *name) { + sqlite3_stmt *statement = NULL; + if (sqlite3_prepare_v2(connection, "SELECT 1 FROM sqlite_master WHERE type='table' AND name=?1", + -1, &statement, NULL) != SQLITE_OK) { + return false; + } + (void)sqlite3_bind_text(statement, 1, name, -1, SQLITE_STATIC); + bool exists = sqlite3_step(statement) == SQLITE_ROW; + (void)sqlite3_finalize(statement); + return exists; +} + +static Lardon3DProjectDbResult migrate(Lardon3DProjectDb *database, unsigned int from_version) { + if (from_version > LARDON3D_PROJECT_DB_SCHEMA_VERSION) { + copy_error(database->error, "Version de schéma future non supportée."); + return LARDON3D_PROJECT_DB_UNSUPPORTED_SCHEMA; + } + if (from_version == LARDON3D_PROJECT_DB_SCHEMA_VERSION) { + return LARDON3D_PROJECT_DB_OK; + } + if (from_version != 0 && from_version != 1 && from_version != 2 && from_version != 3 && + from_version != 4) { + return LARDON3D_PROJECT_DB_CORRUPT; + } + Lardon3DProjectDbResult result = execute(database, "BEGIN IMMEDIATE", "begin migration"); + if (result == LARDON3D_PROJECT_DB_OK && from_version == 0) { + result = execute(database, schema_v5, "create schema v5"); if (result == LARDON3D_PROJECT_DB_OK) { - const char *required[] = {"project", "tasks", "checkpoints", "artifacts", "image_import_tasks", "scansets", "image_assets", "images"}; - for (size_t index = 0; index < 8 && result == LARDON3D_PROJECT_DB_OK; ++index) { - if (!table_exists(database->connection, required[index])) { - copy_error(database->error, "Schéma v1 incomplet."); - result = LARDON3D_PROJECT_DB_CORRUPT; - } - } + result = execute(database, schema_feature_v5, "create feature schema v5"); } - if (result != LARDON3D_PROJECT_DB_OK) { - copy_error(error, database->error); (void)sqlite3_close(database->connection); - (void)pthread_mutex_destroy(&database->mutex); free(database); return result; + } + if (result == LARDON3D_PROJECT_DB_OK && from_version == 1) { + result = execute(database, + "ALTER TABLE tasks ADD COLUMN task_kind TEXT;" + "ALTER TABLE tasks ADD COLUMN task_kind_version INTEGER " + "CHECK(task_kind_version IS NULL OR task_kind_version>0)", + "migrate schema v1 to v2"); +#ifdef LARDON3D_PROJECT_DB_TESTING + const char *forced_failure = getenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V2"); + if (result == LARDON3D_PROJECT_DB_OK && forced_failure && strcmp(forced_failure, "1") == 0) { + result = + execute(database, "INSERT INTO missing_test_table VALUES(1)", "forced migration failure"); } - database->error[0] = '\0'; copy_error(error, ""); *output = database; - return LARDON3D_PROJECT_DB_OK; +#endif + if (result == LARDON3D_PROJECT_DB_OK) { + result = + execute(database, "UPDATE metadata SET value=2 WHERE key='schema_version' AND value=1", + "finish schema v2 migration"); + } + } + if (result == LARDON3D_PROJECT_DB_OK && from_version != 0 && from_version < 3) { + result = execute(database, + "CREATE TABLE image_import_tasks(task_id INTEGER PRIMARY KEY REFERENCES " + "tasks(task_id) ON DELETE CASCADE,source_path TEXT NOT NULL);" + "INSERT INTO metadata(key,value) VALUES('next_task_id',(SELECT CASE WHEN " + "COALESCE(MAX(task_id),0)>=9223372036854775807 THEN 0 ELSE " + "COALESCE(MAX(task_id),0)+1 END FROM tasks))", + "migrate schema v2 to v3"); +#ifdef LARDON3D_PROJECT_DB_TESTING + const char *forced_failure = getenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V3"); + if (result == LARDON3D_PROJECT_DB_OK && forced_failure && strcmp(forced_failure, "1") == 0) { + result = + execute(database, "INSERT INTO missing_test_table VALUES(1)", "forced migration failure"); + } +#endif + if (result == LARDON3D_PROJECT_DB_OK) { + result = + execute(database, "UPDATE metadata SET value=3 WHERE key='schema_version' AND value=2", + "finish schema v3 migration"); + } + } + if (result == LARDON3D_PROJECT_DB_OK && from_version != 0 && from_version < 4) { + result = execute( + database, + "CREATE TABLE scansets(scanset_id INTEGER PRIMARY KEY AUTOINCREMENT " + "CHECK(scanset_id>0),name TEXT NOT NULL CHECK(length(name)>0 AND " + "length(name)<256),created_at INTEGER NOT NULL CHECK(created_at>=0),updated_at INTEGER " + "NOT NULL CHECK(updated_at>=created_at));" + "CREATE TABLE image_assets(asset_id INTEGER PRIMARY KEY AUTOINCREMENT " + "CHECK(asset_id>0),sha256 BLOB NOT NULL UNIQUE CHECK(length(sha256)=32),path TEXT NOT " + "NULL UNIQUE CHECK(length(path)>0 AND length(path)<4096),size_bytes INTEGER NOT NULL " + "CHECK(size_bytes>=0),state INTEGER NOT NULL CHECK(state=1),created_at INTEGER NOT " + "NULL CHECK(created_at>=0));" + "CREATE TABLE images(image_id INTEGER PRIMARY KEY AUTOINCREMENT " + "CHECK(image_id>0),scanset_id INTEGER NOT NULL REFERENCES " + "scansets(scanset_id),asset_id INTEGER NOT NULL REFERENCES " + "image_assets(asset_id),original_name TEXT NOT NULL CHECK(length(original_name)>0 AND " + "length(original_name)<256),source_path TEXT NOT NULL CHECK(length(source_path)>0 AND " + "length(source_path)<4096),producer_task_id INTEGER REFERENCES " + "tasks(task_id),imported_at INTEGER NOT NULL " + "CHECK(imported_at>=0),UNIQUE(scanset_id,asset_id));" + "CREATE INDEX images_scanset_idx ON images(scanset_id,image_id);" + "CREATE INDEX images_producer_idx ON images(producer_task_id,image_id);" + "ALTER TABLE image_import_tasks ADD COLUMN scanset_id INTEGER REFERENCES " + "scansets(scanset_id);" + "INSERT INTO scansets(name,created_at,updated_at) SELECT 'Imports antérieurs à ScanSet " + "v1',0,0 WHERE EXISTS(SELECT 1 FROM image_import_tasks);" + "UPDATE image_import_tasks SET scanset_id=(SELECT scanset_id FROM scansets WHERE " + "name='Imports antérieurs à ScanSet v1' ORDER BY scanset_id LIMIT 1) WHERE scanset_id " + "IS NULL;" + "INSERT INTO metadata(key,value) VALUES('legacy_image_catalog_pending',CASE WHEN " + "EXISTS(SELECT 1 FROM image_import_tasks) THEN 1 ELSE 0 END)", + "migrate schema v3 to v4"); +#ifdef LARDON3D_PROJECT_DB_TESTING + const char *forced_failure = getenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V4"); + if (result == LARDON3D_PROJECT_DB_OK && forced_failure && strcmp(forced_failure, "1") == 0) { + result = + execute(database, "INSERT INTO missing_test_table VALUES(1)", "forced migration failure"); + } +#endif + if (result == LARDON3D_PROJECT_DB_OK) { + result = + execute(database, "UPDATE metadata SET value=4 WHERE key='schema_version' AND value=3", + "finish schema v4 migration"); + } + } + if (result == LARDON3D_PROJECT_DB_OK && from_version != 0) { + result = execute( + database, + "CREATE TABLE feature_assets(feature_asset_id INTEGER PRIMARY KEY AUTOINCREMENT " + "CHECK(feature_asset_id>0),sha256 BLOB NOT NULL UNIQUE CHECK(length(sha256)=32),path " + "TEXT NOT NULL UNIQUE CHECK(length(path)>0 AND length(path)<4096),size_bytes INTEGER " + "NOT NULL CHECK(size_bytes>=0),durability INTEGER NOT NULL CHECK(durability BETWEEN 0 " + "AND 1),created_at INTEGER NOT NULL CHECK(created_at>=0));" + "CREATE TABLE feature_sets(feature_set_id INTEGER PRIMARY KEY AUTOINCREMENT " + "CHECK(feature_set_id>0),image_id INTEGER NOT NULL REFERENCES " + "images(image_id),feature_asset_id INTEGER NOT NULL REFERENCES " + "feature_assets(feature_asset_id),extractor_kind TEXT NOT NULL " + "CHECK(length(extractor_kind)>0 AND length(extractor_kind)<65),extractor_version " + "INTEGER NOT NULL CHECK(extractor_version>0),parameter_fingerprint BLOB NOT NULL " + "CHECK(length(parameter_fingerprint)=32),source_image_sha256 BLOB NOT NULL " + "CHECK(length(source_image_sha256)=32),feature_count INTEGER NOT NULL " + "CHECK(feature_count BETWEEN 0 AND 8192),descriptor_type INTEGER NOT NULL " + "CHECK(descriptor_type " + "IN(1,2)),descriptor_dimension INTEGER NOT NULL " + "CHECK(descriptor_dimension BETWEEN 1 AND 4096),producer_task_id INTEGER REFERENCES " + "tasks(task_id),created_at INTEGER NOT NULL " + "CHECK(created_at>=0),UNIQUE(image_id,extractor_kind,extractor_version,parameter_" + "fingerprint));" + "CREATE INDEX feature_sets_image_idx ON feature_sets(image_id,feature_set_id);" + "CREATE INDEX feature_sets_producer_idx ON " + "feature_sets(producer_task_id,feature_set_id);" + "CREATE TABLE feature_extract_tasks(task_id INTEGER PRIMARY KEY REFERENCES " + "tasks(task_id) ON DELETE CASCADE,image_id INTEGER NOT NULL REFERENCES " + "images(image_id),extractor_kind TEXT NOT NULL,extractor_version INTEGER NOT NULL " + "CHECK(extractor_version>0),max_features INTEGER NOT NULL " + "CHECK(max_features BETWEEN 1 AND 8192),pyramid_levels INTEGER NOT NULL " + "CHECK(pyramid_levels BETWEEN 1 AND 16),fast_threshold INTEGER NOT NULL " + "CHECK(fast_threshold BETWEEN 1 AND 255),parameter_fingerprint BLOB NOT NULL " + "CHECK(length(parameter_fingerprint)=32));", + "migrate schema v4 to v5"); +#ifdef LARDON3D_PROJECT_DB_TESTING + const char *forced_failure = getenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V5"); + if (result == LARDON3D_PROJECT_DB_OK && forced_failure && strcmp(forced_failure, "1") == 0) { + result = execute(database, "INSERT INTO missing_test_table VALUES(1)", + "forced migration v5 failure"); + } +#endif + if (result == LARDON3D_PROJECT_DB_OK) { + result = + execute(database, "UPDATE metadata SET value=5 WHERE key='schema_version' AND value=4", + "finish schema v5 migration"); + } + } + if (result == LARDON3D_PROJECT_DB_OK) { + result = execute(database, "COMMIT", "commit migration"); + } + if (result != LARDON3D_PROJECT_DB_OK) { + (void)execute(database, "ROLLBACK", "rollback migration"); + } + return result; } -void lardon3d_project_db_close(Lardon3DProjectDb *database) -{ - if (!database) return; - (void)pthread_mutex_lock(&database->mutex); - (void)sqlite3_close(database->connection); - (void)pthread_mutex_unlock(&database->mutex); +static Lardon3DProjectDbResult read_schema_version(Lardon3DProjectDb *database, + unsigned int *version) { + if (!table_exists(database->connection, "metadata")) { + sqlite3_stmt *statement = NULL; + int code = sqlite3_prepare_v2(database->connection, + "SELECT 1 FROM sqlite_master WHERE type='table' LIMIT 1", -1, + &statement, NULL); + if (code != SQLITE_OK) { + return sqlite_result(database, code, "inspect schema"); + } + code = sqlite3_step(statement); + bool empty = code == SQLITE_DONE; + (void)sqlite3_finalize(statement); + if (!empty) { + copy_error(database->error, "Base incohérente sans metadata."); + return LARDON3D_PROJECT_DB_CORRUPT; + } + *version = 0; + return LARDON3D_PROJECT_DB_OK; + } + sqlite3_stmt *statement = NULL; + Lardon3DProjectDbResult result = + prepare(database, "SELECT value FROM metadata WHERE key='schema_version'", &statement); + if (result != LARDON3D_PROJECT_DB_OK) { + return result; + } + int code = sqlite3_step(statement); + if (code != SQLITE_ROW || sqlite3_column_type(statement, 0) != SQLITE_INTEGER) { + (void)sqlite3_finalize(statement); + copy_error(database->error, "schema_version absente ou invalide."); + return LARDON3D_PROJECT_DB_CORRUPT; + } + sqlite3_int64 value = sqlite3_column_int64(statement, 0); + code = sqlite3_step(statement); + (void)sqlite3_finalize(statement); + if (code != SQLITE_DONE || value < 1 || value > UINT_MAX) { + return LARDON3D_PROJECT_DB_CORRUPT; + } + *version = (unsigned int)value; + return LARDON3D_PROJECT_DB_OK; +} + +Lardon3DProjectDbResult lardon3d_project_db_open(const char *path, Lardon3DProjectDb **output, + char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]) { + if (output) { + *output = NULL; + } + if (!path || !path[0] || !output || + strnlen(path, LARDON3D_PROJECT_DB_PATH_CAPACITY) >= LARDON3D_PROJECT_DB_PATH_CAPACITY) { + copy_error(error, "Chemin de base invalide."); + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + Lardon3DProjectDb *database = calloc(1, sizeof(*database)); + if (!database) { + copy_error(error, "Allocation impossible."); + return LARDON3D_PROJECT_DB_IO_ERROR; + } + if (pthread_mutex_init(&database->mutex, NULL) != 0) { + free(database); + copy_error(error, "Mutex impossible."); + return LARDON3D_PROJECT_DB_IO_ERROR; + } + int code = + sqlite3_open_v2(path, &database->connection, + SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_NOMUTEX, NULL); + if (code != SQLITE_OK) { + copy_error(error, database->connection ? sqlite3_errmsg(database->connection) + : "Ouverture SQLite impossible."); + if (database->connection) { + (void)sqlite3_close(database->connection); + } (void)pthread_mutex_destroy(&database->mutex); free(database); -} - -bool -lardon3d_project_db_last_error(Lardon3DProjectDb *database, char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]) -{ - if (!database || !error) return false; - (void)pthread_mutex_lock(&database->mutex); + return code == SQLITE_CANTOPEN ? LARDON3D_PROJECT_DB_IO_ERROR : LARDON3D_PROJECT_DB_CORRUPT; + } + Lardon3DProjectDbResult result = + execute(database, + "PRAGMA foreign_keys=ON;PRAGMA journal_mode=DELETE;PRAGMA synchronous=FULL;PRAGMA " + "busy_timeout=5000", + "configure SQLite"); + unsigned int version = 0; + if (result == LARDON3D_PROJECT_DB_OK) { + result = read_schema_version(database, &version); + } + if (result == LARDON3D_PROJECT_DB_OK) { + result = migrate(database, version); + } + if (result == LARDON3D_PROJECT_DB_OK) { + const char *required[] = {"project", + "tasks", + "checkpoints", + "artifacts", + "image_import_tasks", + "scansets", + "image_assets", + "images", + "feature_assets", + "feature_sets", + "feature_extract_tasks"}; + for (size_t index = 0; index < 11 && result == LARDON3D_PROJECT_DB_OK; ++index) { + if (!table_exists(database->connection, required[index])) { + copy_error(database->error, "Schéma v1 incomplet."); + result = LARDON3D_PROJECT_DB_CORRUPT; + } + } + } + if (result != LARDON3D_PROJECT_DB_OK) { copy_error(error, database->error); - (void)pthread_mutex_unlock(&database->mutex); - return true; -} -unsigned int lardon3d_project_db_schema_version(Lardon3DProjectDb *database) { return database ? LARDON3D_PROJECT_DB_SCHEMA_VERSION : 0U; } - -Lardon3DProjectDbResult -lardon3d_project_db_legacy_catalog_pending(Lardon3DProjectDb *database, - bool *pending) -{ - if (pending) *pending = false; - if (!database || !pending) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; - (void)pthread_mutex_lock(&database->mutex); - sqlite3_stmt *statement = NULL; - Lardon3DProjectDbResult result = prepare(database, - "SELECT value FROM metadata WHERE key='legacy_image_catalog_pending'", - &statement); - if (result == LARDON3D_PROJECT_DB_OK) { - int code = sqlite3_step(statement); - sqlite3_int64 value = sqlite3_column_int64(statement, 0); - if (code != SQLITE_ROW || (value != 0 && value != 1) - || sqlite3_step(statement) != SQLITE_DONE) { - result = LARDON3D_PROJECT_DB_CORRUPT; - } else { - *pending = value == 1; - } - (void)sqlite3_finalize(statement); - } - (void)pthread_mutex_unlock(&database->mutex); + (void)sqlite3_close(database->connection); + (void)pthread_mutex_destroy(&database->mutex); + free(database); return result; + } + database->error[0] = '\0'; + copy_error(error, ""); + *output = database; + return LARDON3D_PROJECT_DB_OK; } -static Lardon3DProjectDbResult -step_done(Lardon3DProjectDb *database, sqlite3_stmt *statement, const char *context) -{ +void lardon3d_project_db_close(Lardon3DProjectDb *database) { + if (!database) { + return; + } + (void)pthread_mutex_lock(&database->mutex); + (void)sqlite3_close(database->connection); + (void)pthread_mutex_unlock(&database->mutex); + (void)pthread_mutex_destroy(&database->mutex); + free(database); +} + +bool lardon3d_project_db_last_error(Lardon3DProjectDb *database, + char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]) { + if (!database || !error) { + return false; + } + (void)pthread_mutex_lock(&database->mutex); + copy_error(error, database->error); + (void)pthread_mutex_unlock(&database->mutex); + return true; +} +unsigned int lardon3d_project_db_schema_version(Lardon3DProjectDb *database) { + return database ? LARDON3D_PROJECT_DB_SCHEMA_VERSION : 0U; +} + +Lardon3DProjectDbResult lardon3d_project_db_legacy_catalog_pending(Lardon3DProjectDb *database, + bool *pending) { + if (pending) { + *pending = false; + } + if (!database || !pending) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + (void)pthread_mutex_lock(&database->mutex); + sqlite3_stmt *statement = NULL; + Lardon3DProjectDbResult result = prepare( + database, "SELECT value FROM metadata WHERE key='legacy_image_catalog_pending'", &statement); + if (result == LARDON3D_PROJECT_DB_OK) { int code = sqlite3_step(statement); - if (code != SQLITE_DONE) { - Lardon3DProjectDbResult result = sqlite_result(database, code, context); - (void)sqlite3_finalize(statement); - return result; + sqlite3_int64 value = sqlite3_column_int64(statement, 0); + if (code != SQLITE_ROW || (value != 0 && value != 1) || + sqlite3_step(statement) != SQLITE_DONE) { + result = LARDON3D_PROJECT_DB_CORRUPT; + } else { + *pending = value == 1; } - code = sqlite3_finalize(statement); - return code == SQLITE_OK ? LARDON3D_PROJECT_DB_OK : sqlite_result(database, code, context); + (void)sqlite3_finalize(statement); + } + (void)pthread_mutex_unlock(&database->mutex); + return result; } -static bool -copy_column(sqlite3_stmt *statement, int column, char *destination, size_t capacity) -{ - if (sqlite3_column_type(statement, column) != SQLITE_TEXT) return false; - int bytes = sqlite3_column_bytes(statement, column); - const unsigned char *text = sqlite3_column_text(statement, column); - if (!text || bytes < 0 || (size_t)bytes >= capacity) return false; - memcpy(destination, text, (size_t)bytes); - destination[bytes] = '\0'; - return true; -} - -Lardon3DProjectDbResult -lardon3d_project_db_set_project(Lardon3DProjectDb *database, const Lardon3DProjectDbProject *project) -{ - if (!database || !project - || !bounded_text(project->stable_id, sizeof(project->stable_id), false) - || !bounded_text(project->name, sizeof(project->name), false) - || project->created_at < 0 || project->updated_at < project->created_at) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; - (void)pthread_mutex_lock(&database->mutex); - sqlite3_stmt *statement = NULL; - Lardon3DProjectDbResult result = prepare(database, - "INSERT INTO project(singleton,stable_id,name,created_at,updated_at) VALUES(1,?1,?2,?3,?4) " - "ON CONFLICT(singleton) DO UPDATE SET name=excluded.name,updated_at=excluded.updated_at " - "WHERE project.stable_id=excluded.stable_id AND project.created_at=excluded.created_at", &statement); - if (result == LARDON3D_PROJECT_DB_OK) { - (void)sqlite3_bind_text(statement, 1, project->stable_id, -1, SQLITE_TRANSIENT); - (void)sqlite3_bind_text(statement, 2, project->name, -1, SQLITE_TRANSIENT); - (void)sqlite3_bind_int64(statement, 3, project->created_at); - (void)sqlite3_bind_int64(statement, 4, project->updated_at); - result = step_done(database, statement, "set project"); - if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(database->connection) != 1) { - copy_error(database->error, "Identité de projet contradictoire."); - result = LARDON3D_PROJECT_DB_CONSTRAINT; - } - } - (void)pthread_mutex_unlock(&database->mutex); +static Lardon3DProjectDbResult step_done(Lardon3DProjectDb *database, sqlite3_stmt *statement, + const char *context) { + int code = sqlite3_step(statement); + if (code != SQLITE_DONE) { + Lardon3DProjectDbResult result = sqlite_result(database, code, context); + (void)sqlite3_finalize(statement); return result; + } + code = sqlite3_finalize(statement); + return code == SQLITE_OK ? LARDON3D_PROJECT_DB_OK : sqlite_result(database, code, context); } -Lardon3DProjectDbResult -lardon3d_project_db_get_project(Lardon3DProjectDb *database, Lardon3DProjectDbProject *project) -{ - if (!database || !project) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; - (void)pthread_mutex_lock(&database->mutex); - sqlite3_stmt *statement = NULL; - Lardon3DProjectDbResult result = prepare(database, "SELECT stable_id,name,created_at,updated_at FROM project WHERE singleton=1", &statement); - if (result == LARDON3D_PROJECT_DB_OK) { - int code = sqlite3_step(statement); - if (code == SQLITE_DONE) result = LARDON3D_PROJECT_DB_NOT_FOUND; - else if (code != SQLITE_ROW || !copy_column(statement, 0, project->stable_id, sizeof(project->stable_id)) - || !copy_column(statement, 1, project->name, sizeof(project->name))) result = LARDON3D_PROJECT_DB_CORRUPT; - else { project->created_at = sqlite3_column_int64(statement, 2); project->updated_at = sqlite3_column_int64(statement, 3); } - (void)sqlite3_finalize(statement); +static bool copy_column(sqlite3_stmt *statement, int column, char *destination, size_t capacity) { + if (sqlite3_column_type(statement, column) != SQLITE_TEXT) { + return false; + } + int bytes = sqlite3_column_bytes(statement, column); + const unsigned char *text = sqlite3_column_text(statement, column); + if (!text || bytes < 0 || (size_t)bytes >= capacity) { + return false; + } + memcpy(destination, text, (size_t)bytes); + destination[bytes] = '\0'; + return true; +} + +Lardon3DProjectDbResult lardon3d_project_db_set_project(Lardon3DProjectDb *database, + const Lardon3DProjectDbProject *project) { + if (!database || !project || + !bounded_text(project->stable_id, sizeof(project->stable_id), false) || + !bounded_text(project->name, sizeof(project->name), false) || project->created_at < 0 || + project->updated_at < project->created_at) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + (void)pthread_mutex_lock(&database->mutex); + sqlite3_stmt *statement = NULL; + Lardon3DProjectDbResult result = prepare( + database, + "INSERT INTO project(singleton,stable_id,name,created_at,updated_at) VALUES(1,?1,?2,?3,?4) " + "ON CONFLICT(singleton) DO UPDATE SET name=excluded.name,updated_at=excluded.updated_at " + "WHERE project.stable_id=excluded.stable_id AND project.created_at=excluded.created_at", + &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_text(statement, 1, project->stable_id, -1, SQLITE_TRANSIENT); + (void)sqlite3_bind_text(statement, 2, project->name, -1, SQLITE_TRANSIENT); + (void)sqlite3_bind_int64(statement, 3, project->created_at); + (void)sqlite3_bind_int64(statement, 4, project->updated_at); + result = step_done(database, statement, "set project"); + if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(database->connection) != 1) { + copy_error(database->error, "Identité de projet contradictoire."); + result = LARDON3D_PROJECT_DB_CONSTRAINT; } - (void)pthread_mutex_unlock(&database->mutex); - return result; + } + (void)pthread_mutex_unlock(&database->mutex); + return result; } -static bool -valid_checkpoint(const Lardon3DProjectDbCheckpoint *checkpoint) -{ - return checkpoint && bounded_text(checkpoint->path, sizeof(checkpoint->path), false) - && checkpoint->format_version > 0 && checkpoint->updated_at >= 0 - && checkpoint->durability >= LARDON3D_DB_CHECKPOINT_DURABLE - && checkpoint->durability <= LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE; +Lardon3DProjectDbResult lardon3d_project_db_get_project(Lardon3DProjectDb *database, + Lardon3DProjectDbProject *project) { + if (!database || !project) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + (void)pthread_mutex_lock(&database->mutex); + sqlite3_stmt *statement = NULL; + Lardon3DProjectDbResult result = prepare( + database, "SELECT stable_id,name,created_at,updated_at FROM project WHERE singleton=1", + &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + int code = sqlite3_step(statement); + if (code == SQLITE_DONE) { + result = LARDON3D_PROJECT_DB_NOT_FOUND; + } else if (code != SQLITE_ROW || + !copy_column(statement, 0, project->stable_id, sizeof(project->stable_id)) || + !copy_column(statement, 1, project->name, sizeof(project->name))) { + result = LARDON3D_PROJECT_DB_CORRUPT; + } else { + project->created_at = sqlite3_column_int64(statement, 2); + project->updated_at = sqlite3_column_int64(statement, 3); + } + (void)sqlite3_finalize(statement); + } + (void)pthread_mutex_unlock(&database->mutex); + return result; } -static bool -valid_durable_task(const Lardon3DTaskDurableSnapshot *snapshot, int64_t updated_at) -{ - return snapshot && valid_task_id(snapshot->id) - && bounded_text(snapshot->name, sizeof(snapshot->name), false) - && valid_state(snapshot->saved_state) && valid_state(snapshot->recovery_state) - && snapshot->recovery_state == ((snapshot->saved_state == TASK_RUNNING || snapshot->saved_state == TASK_PAUSED) ? TASK_PENDING : snapshot->saved_state) - && snapshot->progress <= 100 && snapshot->sequence_count <= INT_MAX - && snapshot->started_at.tv_sec >= 0 && snapshot->started_at.tv_sec <= INT64_MAX - && snapshot->started_at.tv_nsec >= 0 && snapshot->started_at.tv_nsec < 1000000000L - && snapshot->finished_at.tv_sec >= 0 && snapshot->finished_at.tv_sec <= INT64_MAX - && snapshot->finished_at.tv_nsec >= 0 && snapshot->finished_at.tv_nsec < 1000000000L - && updated_at >= 0; +static bool valid_checkpoint(const Lardon3DProjectDbCheckpoint *checkpoint) { + return checkpoint && bounded_text(checkpoint->path, sizeof(checkpoint->path), false) && + checkpoint->format_version > 0 && checkpoint->updated_at >= 0 && + checkpoint->durability >= LARDON3D_DB_CHECKPOINT_DURABLE && + checkpoint->durability <= LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE; +} + +static bool valid_durable_task(const Lardon3DTaskDurableSnapshot *snapshot, int64_t updated_at) { + return snapshot && valid_task_id(snapshot->id) && + bounded_text(snapshot->name, sizeof(snapshot->name), false) && + valid_state(snapshot->saved_state) && valid_state(snapshot->recovery_state) && + snapshot->recovery_state == + ((snapshot->saved_state == TASK_RUNNING || snapshot->saved_state == TASK_PAUSED) + ? TASK_PENDING + : snapshot->saved_state) && + snapshot->progress <= 100 && snapshot->sequence_count <= INT_MAX && + snapshot->started_at.tv_sec >= 0 && snapshot->started_at.tv_sec <= INT64_MAX && + snapshot->started_at.tv_nsec >= 0 && snapshot->started_at.tv_nsec < 1000000000L && + snapshot->finished_at.tv_sec >= 0 && snapshot->finished_at.tv_sec <= INT64_MAX && + snapshot->finished_at.tv_nsec >= 0 && snapshot->finished_at.tv_nsec < 1000000000L && + updated_at >= 0; } static Lardon3DProjectDbResult record_task_internal(Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, - const char *task_kind, uint32_t task_kind_version, - const Lardon3DProjectDbCheckpoint *checkpoint, const char *source_path, - uint64_t scanset_id, int64_t updated_at) -{ - bool typed = task_kind != NULL; - if (!database || !valid_durable_task(snapshot, updated_at) - || (typed && (!lardon3d_task_kind_is_valid(task_kind) - || task_kind_version == 0)) - || (!typed && task_kind_version != 0) - || (source_path && !bounded_text(source_path, - LARDON3D_PROJECT_DB_PATH_CAPACITY, false)) - || (source_path && !valid_task_id(scanset_id)) - || (checkpoint && !valid_checkpoint(checkpoint))) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + const char *task_kind, uint32_t task_kind_version, + const Lardon3DProjectDbCheckpoint *checkpoint, const char *source_path, + uint64_t scanset_id, const Lardon3DProjectDbFeatureExtractTask *feature, + int64_t updated_at) { + bool typed = task_kind != NULL; + if (!database || !valid_durable_task(snapshot, updated_at) || + (typed && (!lardon3d_task_kind_is_valid(task_kind) || task_kind_version == 0)) || + (!typed && task_kind_version != 0) || + (source_path && !bounded_text(source_path, LARDON3D_PROJECT_DB_PATH_CAPACITY, false)) || + (source_path && !valid_task_id(scanset_id)) || + (feature && + (!valid_task_id(feature->task_id) || feature->task_id != snapshot->id || + !valid_task_id(feature->image_id) || + !lardon3d_task_kind_is_valid(feature->extractor_kind) || feature->extractor_version == 0 || + feature->max_features == 0 || feature->max_features > 8192 || + feature->pyramid_levels == 0 || feature->pyramid_levels > 16 || + feature->fast_threshold == 0 || feature->fast_threshold > 255)) || + (checkpoint && !valid_checkpoint(checkpoint))) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } #ifdef LARDON3D_PROJECT_DB_TESTING - const char *forced_busy = getenv("LARDON3D_TEST_PROJECT_DB_BUSY_CHECKPOINT"); - if (forced_busy && checkpoint && strcmp(forced_busy, "1") == 0) { - copy_error(database->error, "Verrou DB injecté pour test."); - return LARDON3D_PROJECT_DB_BUSY; - } + const char *forced_busy = getenv("LARDON3D_TEST_PROJECT_DB_BUSY_CHECKPOINT"); + if (forced_busy && checkpoint && strcmp(forced_busy, "1") == 0) { + copy_error(database->error, "Verrou DB injecté pour test."); + return LARDON3D_PROJECT_DB_BUSY; + } #endif - (void)pthread_mutex_lock(&database->mutex); - Lardon3DProjectDbResult result = execute(database, "BEGIN IMMEDIATE", "begin task record"); - sqlite3_stmt *statement = NULL; - if (result == LARDON3D_PROJECT_DB_OK) result = prepare(database, - "INSERT INTO tasks(task_id,name,task_kind,task_kind_version,saved_state,recovery_state,progress,sequence_count,started_sec,started_nsec,finished_sec,finished_nsec,updated_at)" - "VALUES(?1,?2,?3,?4,?5,?6,?7,?8,?9,?10,?11,?12,?13) ON CONFLICT(task_id) DO UPDATE SET name=excluded.name,saved_state=excluded.saved_state," - "recovery_state=excluded.recovery_state,progress=excluded.progress,sequence_count=excluded.sequence_count,started_sec=excluded.started_sec," - "started_nsec=excluded.started_nsec,finished_sec=excluded.finished_sec,finished_nsec=excluded.finished_nsec,updated_at=excluded.updated_at " - "WHERE (tasks.task_kind IS NULL AND excluded.task_kind IS NULL) OR (tasks.task_kind=excluded.task_kind AND tasks.task_kind_version=excluded.task_kind_version)", &statement); - if (result == LARDON3D_PROJECT_DB_OK) { - (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)snapshot->id); (void)sqlite3_bind_text(statement, 2, snapshot->name, -1, SQLITE_TRANSIENT); - if (typed) { - (void)sqlite3_bind_text(statement, 3, task_kind, -1, SQLITE_TRANSIENT); - (void)sqlite3_bind_int64(statement, 4, task_kind_version); - } else { - (void)sqlite3_bind_null(statement, 3); (void)sqlite3_bind_null(statement, 4); - } - (void)sqlite3_bind_int(statement, 5, (int)snapshot->saved_state); (void)sqlite3_bind_int(statement, 6, (int)snapshot->recovery_state); - (void)sqlite3_bind_int(statement, 7, (int)snapshot->progress); (void)sqlite3_bind_int(statement, 8, (int)snapshot->sequence_count); - (void)sqlite3_bind_int64(statement, 9, snapshot->started_at.tv_sec); (void)sqlite3_bind_int64(statement, 10, snapshot->started_at.tv_nsec); - (void)sqlite3_bind_int64(statement, 11, snapshot->finished_at.tv_sec); (void)sqlite3_bind_int64(statement, 12, snapshot->finished_at.tv_nsec); - (void)sqlite3_bind_int64(statement, 13, updated_at); result = step_done(database, statement, "upsert task"); - if (result == LARDON3D_PROJECT_DB_OK - && sqlite3_changes(database->connection) != 1) { - copy_error(database->error, "Type métier de tâche immuable."); - result = LARDON3D_PROJECT_DB_CONSTRAINT; - } - } - if (result == LARDON3D_PROJECT_DB_OK && checkpoint) { -#ifdef LARDON3D_PROJECT_DB_TESTING - const char *forced_failure = getenv("LARDON3D_TEST_PROJECT_DB_FAIL_CHECKPOINT"); - if (forced_failure && strcmp(forced_failure, "1") == 0) { - result = execute(database, "INSERT INTO missing_test_table VALUES(1)", "forced checkpoint failure"); - } -#endif - } - if (result == LARDON3D_PROJECT_DB_OK) { - result = prepare(database, - "UPDATE metadata SET value=CASE WHEN ?1=9223372036854775807 THEN 0 ELSE ?1+1 END " - "WHERE key='next_task_id' AND value>0 AND value<=?1", - &statement); - if (result == LARDON3D_PROJECT_DB_OK) { - (void)sqlite3_bind_int64(statement, 1, - (sqlite3_int64)snapshot->id); - result = step_done(database, statement, "advance recorded task id"); - } - } - if (result == LARDON3D_PROJECT_DB_OK && source_path) { - result = prepare(database, - "INSERT INTO image_import_tasks(task_id,source_path,scanset_id) VALUES(?1,?2,?3) " - "ON CONFLICT(task_id) DO UPDATE SET source_path=excluded.source_path " - "WHERE image_import_tasks.source_path=excluded.source_path AND image_import_tasks.scanset_id=excluded.scanset_id", - &statement); - if (result == LARDON3D_PROJECT_DB_OK) { - (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)snapshot->id); - (void)sqlite3_bind_text(statement, 2, source_path, -1, - SQLITE_TRANSIENT); - (void)sqlite3_bind_int64(statement, 3, (sqlite3_int64)scanset_id); - result = step_done(database, statement, "upsert image import"); - if (result == LARDON3D_PROJECT_DB_OK - && sqlite3_changes(database->connection) != 1) { - copy_error(database->error, "Source d'import immuable."); - result = LARDON3D_PROJECT_DB_CONSTRAINT; - } - } - } - if (result == LARDON3D_PROJECT_DB_OK && checkpoint) { - result = prepare(database, "INSERT INTO checkpoints(task_id,path,format_version,durability,updated_at) VALUES(?1,?2,?3,?4,?5) " - "ON CONFLICT(task_id) DO UPDATE SET path=excluded.path,format_version=excluded.format_version,durability=excluded.durability,updated_at=excluded.updated_at", &statement); - if (result == LARDON3D_PROJECT_DB_OK) { - (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)snapshot->id); (void)sqlite3_bind_text(statement, 2, checkpoint->path, -1, SQLITE_TRANSIENT); - (void)sqlite3_bind_int64(statement, 3, checkpoint->format_version); (void)sqlite3_bind_int(statement, 4, (int)checkpoint->durability); - (void)sqlite3_bind_int64(statement, 5, checkpoint->updated_at); result = step_done(database, statement, "upsert checkpoint"); - } - } - if (result == LARDON3D_PROJECT_DB_OK) result = execute(database, "COMMIT", "commit task record"); - if (result != LARDON3D_PROJECT_DB_OK) (void)execute(database, "ROLLBACK", "rollback task record"); - (void)pthread_mutex_unlock(&database->mutex); - return result; -} - -Lardon3DProjectDbResult -lardon3d_project_db_record_task( - Lardon3DProjectDb *database, - const Lardon3DTaskDurableSnapshot *snapshot, - const char *task_kind, - uint32_t task_kind_version, - const Lardon3DProjectDbCheckpoint *checkpoint, - int64_t updated_at -) -{ - return record_task_internal(database, snapshot, task_kind, - task_kind_version, checkpoint, NULL, 0, updated_at); -} - -Lardon3DProjectDbResult -lardon3d_project_db_record_image_import_task( - Lardon3DProjectDb *database, - const Lardon3DTaskDurableSnapshot *snapshot, - const char *task_kind, - uint32_t task_kind_version, - const Lardon3DProjectDbCheckpoint *checkpoint, - const char *source_path, - uint64_t scanset_id, - int64_t updated_at -) -{ - if (!source_path) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; - return record_task_internal(database, snapshot, task_kind, - task_kind_version, checkpoint, source_path, scanset_id, updated_at); -} - -static bool -read_task(sqlite3_stmt *statement, Lardon3DProjectDbTask *task) -{ - memset(task, 0, sizeof(*task)); - sqlite3_int64 id = sqlite3_column_int64(statement, 0); - bool has_kind = sqlite3_column_type(statement, 2) != SQLITE_NULL; - bool has_kind_version = sqlite3_column_type(statement, 3) != SQLITE_NULL; - int progress = sqlite3_column_int(statement, 6); - sqlite3_int64 sequence_count = sqlite3_column_int64(statement, 7); - sqlite3_int64 started_nsec = sqlite3_column_int64(statement, 9); - sqlite3_int64 finished_nsec = sqlite3_column_int64(statement, 11); - if (id <= 0 || !copy_column(statement, 1, task->name, sizeof(task->name))) return false; - if (has_kind != has_kind_version) return false; - if (has_kind) { - sqlite3_int64 version = sqlite3_column_int64(statement, 3); - if (!copy_column(statement, 2, task->task_kind, sizeof(task->task_kind)) - || !lardon3d_task_kind_is_valid(task->task_kind) - || version <= 0 || version > UINT32_MAX) return false; - task->has_task_kind = true; - task->task_kind_version = (uint32_t)version; - } - task->task_id = (uint64_t)id; task->saved_state = (Lardon3DTaskState)sqlite3_column_int(statement, 4); - task->recovery_state = (Lardon3DTaskState)sqlite3_column_int(statement, 5); - if (progress < 0 || progress > 100 || sequence_count < 0 || sequence_count > UINT_MAX - || started_nsec < 0 || started_nsec >= 1000000000 || finished_nsec < 0 || finished_nsec >= 1000000000 - || !database_time(sqlite3_column_int64(statement, 8), &task->started_at.tv_sec) - || !database_time(sqlite3_column_int64(statement, 10), &task->finished_at.tv_sec)) return false; - task->progress = (unsigned int)progress; task->sequence_count = (unsigned int)sequence_count; - task->started_at.tv_nsec = (long)started_nsec; task->finished_at.tv_nsec = (long)finished_nsec; task->updated_at = sqlite3_column_int64(statement, 12); - if (sqlite3_column_type(statement, 13) != SQLITE_NULL) { - task->has_checkpoint = true; - if (!copy_column(statement, 13, task->checkpoint.path, sizeof(task->checkpoint.path))) return false; - sqlite3_int64 format_version = sqlite3_column_int64(statement, 14); - if (format_version <= 0 || format_version > UINT32_MAX) return false; - task->checkpoint.format_version = (uint32_t)format_version; - task->checkpoint.durability = (Lardon3DProjectDbCheckpointDurability)sqlite3_column_int(statement, 15); - task->checkpoint.updated_at = sqlite3_column_int64(statement, 16); - if (task->checkpoint.durability < LARDON3D_DB_CHECKPOINT_DURABLE - || task->checkpoint.durability > LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE) return false; - } - return valid_state(task->saved_state) && valid_state(task->recovery_state) && task->progress <= 100; -} - -static const char task_select[] = "SELECT t.task_id,t.name,t.task_kind,t.task_kind_version,t.saved_state,t.recovery_state,t.progress,t.sequence_count,t.started_sec,t.started_nsec," - "t.finished_sec,t.finished_nsec,t.updated_at,c.path,c.format_version,c.durability,c.updated_at FROM tasks t LEFT JOIN checkpoints c ON c.task_id=t.task_id "; - -Lardon3DProjectDbResult -lardon3d_project_db_load_task(Lardon3DProjectDb *database, uint64_t task_id, Lardon3DProjectDbTask *task) -{ - if (!database || !valid_task_id(task_id) || !task) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; - (void)pthread_mutex_lock(&database->mutex); sqlite3_stmt *statement = NULL; - char sql[768]; (void)snprintf(sql, sizeof(sql), "%s WHERE t.task_id=?1", task_select); - Lardon3DProjectDbResult result = prepare(database, sql, &statement); - if (result == LARDON3D_PROJECT_DB_OK) { (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)task_id); int code = sqlite3_step(statement); - if (code == SQLITE_DONE) result = LARDON3D_PROJECT_DB_NOT_FOUND; else if (code != SQLITE_ROW || !read_task(statement, task)) result = LARDON3D_PROJECT_DB_CORRUPT; - (void)sqlite3_finalize(statement); } - (void)pthread_mutex_unlock(&database->mutex); return result; -} - -Lardon3DProjectDbResult -lardon3d_project_db_list_recoverable(Lardon3DProjectDb *database, uint64_t after_task_id, Lardon3DProjectDbTask *tasks, size_t capacity, size_t *count) -{ - if (count) *count = 0; - if (!database || !tasks || !count || after_task_id > INT64_MAX || capacity == 0 || capacity > LARDON3D_PROJECT_DB_RECOVERY_PAGE_MAX) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; -#ifdef LARDON3D_PROJECT_DB_TESTING - const char *forced_busy = getenv("LARDON3D_TEST_PROJECT_DB_BUSY_RECOVERY"); - if (forced_busy && strcmp(forced_busy, "1") == 0) { - return LARDON3D_PROJECT_DB_BUSY; - } -#endif - (void)pthread_mutex_lock(&database->mutex); sqlite3_stmt *statement = NULL; char sql[896]; - (void)snprintf(sql, sizeof(sql), "%s WHERE t.recovery_state=?1 AND c.task_id IS NOT NULL AND t.task_id>?2 ORDER BY t.task_id LIMIT ?3", task_select); - Lardon3DProjectDbResult result = prepare(database, sql, &statement); - if (result == LARDON3D_PROJECT_DB_OK) { - (void)sqlite3_bind_int(statement, 1, TASK_PENDING); (void)sqlite3_bind_int64(statement, 2, (sqlite3_int64)after_task_id); (void)sqlite3_bind_int64(statement, 3, (sqlite3_int64)capacity); - int code = SQLITE_DONE; while (*count < capacity && (code = sqlite3_step(statement)) == SQLITE_ROW) { if (!read_task(statement, &tasks[*count])) { result = LARDON3D_PROJECT_DB_CORRUPT; break; } ++*count; } - if (result == LARDON3D_PROJECT_DB_OK && *count < capacity && code != SQLITE_DONE) result = sqlite_result(database, code, "list recoverable"); - (void)sqlite3_finalize(statement); - } - (void)pthread_mutex_unlock(&database->mutex); return result; -} - -Lardon3DProjectDbResult -lardon3d_project_db_allocate_task_id( - Lardon3DProjectDb *database, - uint64_t *task_id -) -{ - if (task_id) *task_id = 0; - if (!database || !task_id) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; - (void)pthread_mutex_lock(&database->mutex); - Lardon3DProjectDbResult result = execute(database, "BEGIN IMMEDIATE", - "begin task id allocation"); - sqlite3_stmt *statement = NULL; - sqlite3_int64 next = 0; - if (result == LARDON3D_PROJECT_DB_OK) { - result = prepare(database, - "SELECT value FROM metadata WHERE key='next_task_id'", &statement); - } - if (result == LARDON3D_PROJECT_DB_OK) { - int code = sqlite3_step(statement); - if (code != SQLITE_ROW || sqlite3_column_type(statement, 0) != SQLITE_INTEGER) { - result = LARDON3D_PROJECT_DB_CORRUPT; - } else { - next = sqlite3_column_int64(statement, 0); - if (next <= 0 || next > INT64_MAX) result = LARDON3D_PROJECT_DB_CONSTRAINT; - } - (void)sqlite3_finalize(statement); statement = NULL; - } - if (result == LARDON3D_PROJECT_DB_OK) { - result = prepare(database, - "UPDATE metadata SET value=CASE WHEN value=9223372036854775807 THEN 0 ELSE value+1 END WHERE key='next_task_id' AND value=?1", - &statement); - } - if (result == LARDON3D_PROJECT_DB_OK) { - (void)sqlite3_bind_int64(statement, 1, next); - result = step_done(database, statement, "advance task id"); - statement = NULL; - if (result == LARDON3D_PROJECT_DB_OK - && sqlite3_changes(database->connection) != 1) { - result = LARDON3D_PROJECT_DB_CONSTRAINT; - } - } - if (result == LARDON3D_PROJECT_DB_OK) { - result = execute(database, "COMMIT", "commit task id allocation"); - } - if (result != LARDON3D_PROJECT_DB_OK) { - (void)execute(database, "ROLLBACK", "rollback task id allocation"); + (void)pthread_mutex_lock(&database->mutex); + Lardon3DProjectDbResult result = execute(database, "BEGIN IMMEDIATE", "begin task record"); + sqlite3_stmt *statement = NULL; + if (result == LARDON3D_PROJECT_DB_OK) { + result = prepare( + database, + "INSERT INTO " + "tasks(task_id,name,task_kind,task_kind_version,saved_state,recovery_state,progress," + "sequence_count,started_sec,started_nsec,finished_sec,finished_nsec,updated_at)" + "VALUES(?1,?2,?3,?4,?5,?6,?7,?8,?9,?10,?11,?12,?13) ON CONFLICT(task_id) DO UPDATE SET " + "name=excluded.name,saved_state=excluded.saved_state," + "recovery_state=excluded.recovery_state,progress=excluded.progress,sequence_count=" + "excluded.sequence_count,started_sec=excluded.started_sec," + "started_nsec=excluded.started_nsec,finished_sec=excluded.finished_sec,finished_nsec=" + "excluded.finished_nsec,updated_at=excluded.updated_at " + "WHERE (tasks.task_kind IS NULL AND excluded.task_kind IS NULL) OR " + "(tasks.task_kind=excluded.task_kind AND " + "tasks.task_kind_version=excluded.task_kind_version)", + &statement); + } + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)snapshot->id); + (void)sqlite3_bind_text(statement, 2, snapshot->name, -1, SQLITE_TRANSIENT); + if (typed) { + (void)sqlite3_bind_text(statement, 3, task_kind, -1, SQLITE_TRANSIENT); + (void)sqlite3_bind_int64(statement, 4, task_kind_version); } else { - *task_id = (uint64_t)next; + (void)sqlite3_bind_null(statement, 3); + (void)sqlite3_bind_null(statement, 4); } - (void)pthread_mutex_unlock(&database->mutex); - return result; -} - -Lardon3DProjectDbResult -lardon3d_project_db_load_image_import( - Lardon3DProjectDb *database, - uint64_t task_id, - Lardon3DProjectDbImageImport *parameters -) -{ - if (!database || !valid_task_id(task_id) || !parameters) { - return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + (void)sqlite3_bind_int(statement, 5, (int)snapshot->saved_state); + (void)sqlite3_bind_int(statement, 6, (int)snapshot->recovery_state); + (void)sqlite3_bind_int(statement, 7, (int)snapshot->progress); + (void)sqlite3_bind_int(statement, 8, (int)snapshot->sequence_count); + (void)sqlite3_bind_int64(statement, 9, snapshot->started_at.tv_sec); + (void)sqlite3_bind_int64(statement, 10, snapshot->started_at.tv_nsec); + (void)sqlite3_bind_int64(statement, 11, snapshot->finished_at.tv_sec); + (void)sqlite3_bind_int64(statement, 12, snapshot->finished_at.tv_nsec); + (void)sqlite3_bind_int64(statement, 13, updated_at); + result = step_done(database, statement, "upsert task"); + if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(database->connection) != 1) { + copy_error(database->error, "Type métier de tâche immuable."); + result = LARDON3D_PROJECT_DB_CONSTRAINT; } - memset(parameters, 0, sizeof(*parameters)); - (void)pthread_mutex_lock(&database->mutex); - sqlite3_stmt *statement = NULL; - Lardon3DProjectDbResult result = prepare(database, - "SELECT source_path,scanset_id FROM image_import_tasks WHERE task_id=?1", - &statement); + } + if (result == LARDON3D_PROJECT_DB_OK && checkpoint) { +#ifdef LARDON3D_PROJECT_DB_TESTING + const char *forced_failure = getenv("LARDON3D_TEST_PROJECT_DB_FAIL_CHECKPOINT"); + if (forced_failure && strcmp(forced_failure, "1") == 0) { + result = execute(database, "INSERT INTO missing_test_table VALUES(1)", + "forced checkpoint failure"); + } +#endif + } + if (result == LARDON3D_PROJECT_DB_OK) { + result = + prepare(database, + "UPDATE metadata SET value=CASE WHEN ?1=9223372036854775807 THEN 0 ELSE ?1+1 END " + "WHERE key='next_task_id' AND value>0 AND value<=?1", + &statement); if (result == LARDON3D_PROJECT_DB_OK) { - (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)task_id); - int code = sqlite3_step(statement); - if (code == SQLITE_DONE) result = LARDON3D_PROJECT_DB_NOT_FOUND; - else if (code != SQLITE_ROW || !copy_column(statement, 0, - parameters->source_path, sizeof(parameters->source_path))) { - result = LARDON3D_PROJECT_DB_CORRUPT; - } else { - sqlite3_int64 scanset = sqlite3_column_int64(statement, 1); - if (scanset <= 0) { - result = LARDON3D_PROJECT_DB_CORRUPT; - } else { - parameters->task_id = task_id; - parameters->scanset_id = (uint64_t)scanset; - } - } - (void)sqlite3_finalize(statement); + (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)snapshot->id); + result = step_done(database, statement, "advance recorded task id"); } - (void)pthread_mutex_unlock(&database->mutex); - return result; -} - -static bool -valid_catalog_id(uint64_t value) -{ - return value > 0 && value <= INT64_MAX; -} - -static bool -valid_original_name(const char *name) -{ - return bounded_text(name, LARDON3D_PROJECT_DB_IMAGE_NAME_CAPACITY, false) - && !strchr(name, '/') && !strchr(name, '\\') - && !strchr(name, '\t') && !strchr(name, '\r') && !strchr(name, '\n'); -} - -static bool -valid_relative_asset_path(const char *path) -{ - return bounded_text(path, LARDON3D_PROJECT_DB_PATH_CAPACITY, false) - && path[0] != '/' && !strchr(path, '\\') - && strncmp(path, "../", 3) != 0 && !strstr(path, "/../") - && strcmp(path, "..") != 0; -} - -static bool -canonical_asset_path( - const unsigned char hash[LARDON3D_PROJECT_DB_SHA256_SIZE], - const char *path -) -{ - static const char digits[] = "0123456789abcdef"; - char hex[65]; - for (size_t index = 0; index < LARDON3D_PROJECT_DB_SHA256_SIZE; ++index) { - hex[index * 2] = digits[hash[index] >> 4]; - hex[index * 2 + 1] = digits[hash[index] & 15U]; - } - hex[64] = '\0'; - char expected[LARDON3D_PROJECT_DB_PATH_CAPACITY]; - int written = snprintf(expected, sizeof(expected), - "assets/images/%c%c/%s", hex[0], hex[1], hex); - return written > 0 && (size_t)written < sizeof(expected) - && strcmp(path, expected) == 0; -} - -Lardon3DProjectDbResult -lardon3d_project_db_create_scanset(Lardon3DProjectDb *database, - const char *name, Lardon3DProjectDbScanSet *scanset) -{ - if (!database || !scanset - || !bounded_text(name, LARDON3D_PROJECT_DB_SCANSET_NAME_CAPACITY, false)) { - return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; - } - int64_t now = (int64_t)time(NULL); - if (now < 0) return LARDON3D_PROJECT_DB_IO_ERROR; - memset(scanset, 0, sizeof(*scanset)); - (void)pthread_mutex_lock(&database->mutex); - sqlite3_stmt *statement = NULL; - Lardon3DProjectDbResult result = prepare(database, - "INSERT INTO scansets(name,created_at,updated_at) VALUES(?1,?2,?2)", - &statement); + } + if (result == LARDON3D_PROJECT_DB_OK && source_path) { + result = + prepare(database, + "INSERT INTO image_import_tasks(task_id,source_path,scanset_id) VALUES(?1,?2,?3) " + "ON CONFLICT(task_id) DO UPDATE SET source_path=excluded.source_path " + "WHERE image_import_tasks.source_path=excluded.source_path AND " + "image_import_tasks.scanset_id=excluded.scanset_id", + &statement); if (result == LARDON3D_PROJECT_DB_OK) { - (void)sqlite3_bind_text(statement, 1, name, -1, SQLITE_TRANSIENT); - (void)sqlite3_bind_int64(statement, 2, now); - result = step_done(database, statement, "create scanset"); - } - sqlite3_int64 id = sqlite3_last_insert_rowid(database->connection); - if (result == LARDON3D_PROJECT_DB_OK && id <= 0) { + (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)snapshot->id); + (void)sqlite3_bind_text(statement, 2, source_path, -1, SQLITE_TRANSIENT); + (void)sqlite3_bind_int64(statement, 3, (sqlite3_int64)scanset_id); + result = step_done(database, statement, "upsert image import"); + if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(database->connection) != 1) { + copy_error(database->error, "Source d'import immuable."); result = LARDON3D_PROJECT_DB_CONSTRAINT; + } } + } + if (result == LARDON3D_PROJECT_DB_OK && feature) { + result = prepare( + database, + "INSERT INTO " + "feature_extract_tasks(task_id,image_id,extractor_kind,extractor_version,max_features," + "pyramid_levels,fast_threshold,parameter_fingerprint) VALUES(?1,?2,?3,?4,?5,?6,?7,?8) " + "ON CONFLICT(task_id) DO UPDATE SET image_id=excluded.image_id WHERE " + "feature_extract_tasks.image_id=excluded.image_id AND " + "feature_extract_tasks.extractor_kind=excluded.extractor_kind AND " + "feature_extract_tasks.extractor_version=excluded.extractor_version AND " + "feature_extract_tasks.max_features=excluded.max_features AND " + "feature_extract_tasks.pyramid_levels=excluded.pyramid_levels AND " + "feature_extract_tasks.fast_threshold=excluded.fast_threshold AND " + "feature_extract_tasks.parameter_fingerprint=excluded.parameter_fingerprint", + &statement); if (result == LARDON3D_PROJECT_DB_OK) { - scanset->scanset_id = (uint64_t)id; - scanset->created_at = now; - scanset->updated_at = now; - (void)snprintf(scanset->name, sizeof(scanset->name), "%s", name); + (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)feature->task_id); + (void)sqlite3_bind_int64(statement, 2, (sqlite3_int64)feature->image_id); + (void)sqlite3_bind_text(statement, 3, feature->extractor_kind, -1, SQLITE_TRANSIENT); + (void)sqlite3_bind_int64(statement, 4, feature->extractor_version); + (void)sqlite3_bind_int64(statement, 5, feature->max_features); + (void)sqlite3_bind_int64(statement, 6, feature->pyramid_levels); + (void)sqlite3_bind_int64(statement, 7, feature->fast_threshold); + (void)sqlite3_bind_blob(statement, 8, feature->parameter_fingerprint, 32, SQLITE_TRANSIENT); + result = step_done(database, statement, "upsert feature extract"); + statement = NULL; + if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(database->connection) != 1) { + copy_error(database->error, "Paramètres feature immuables."); + result = LARDON3D_PROJECT_DB_CONSTRAINT; + } } - (void)pthread_mutex_unlock(&database->mutex); - return result; + } + if (result == LARDON3D_PROJECT_DB_OK && checkpoint) { + result = + prepare(database, + "INSERT INTO checkpoints(task_id,path,format_version,durability,updated_at) " + "VALUES(?1,?2,?3,?4,?5) " + "ON CONFLICT(task_id) DO UPDATE SET " + "path=excluded.path,format_version=excluded.format_version,durability=excluded." + "durability,updated_at=excluded.updated_at", + &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)snapshot->id); + (void)sqlite3_bind_text(statement, 2, checkpoint->path, -1, SQLITE_TRANSIENT); + (void)sqlite3_bind_int64(statement, 3, checkpoint->format_version); + (void)sqlite3_bind_int(statement, 4, (int)checkpoint->durability); + (void)sqlite3_bind_int64(statement, 5, checkpoint->updated_at); + result = step_done(database, statement, "upsert checkpoint"); + } + } + if (result == LARDON3D_PROJECT_DB_OK) { + result = execute(database, "COMMIT", "commit task record"); + } + if (result != LARDON3D_PROJECT_DB_OK) { + (void)execute(database, "ROLLBACK", "rollback task record"); + } + (void)pthread_mutex_unlock(&database->mutex); + return result; } -static bool -read_scanset(sqlite3_stmt *statement, Lardon3DProjectDbScanSet *scanset) -{ - sqlite3_int64 id = sqlite3_column_int64(statement, 0); - sqlite3_int64 created = sqlite3_column_int64(statement, 2); - sqlite3_int64 updated = sqlite3_column_int64(statement, 3); - if (id <= 0 || created < 0 || updated < created - || !copy_column(statement, 1, scanset->name, sizeof(scanset->name))) { - return false; +Lardon3DProjectDbResult lardon3d_project_db_record_task( + Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, const char *task_kind, + uint32_t task_kind_version, const Lardon3DProjectDbCheckpoint *checkpoint, int64_t updated_at) { + return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0, + NULL, updated_at); +} + +Lardon3DProjectDbResult lardon3d_project_db_record_image_import_task( + Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, const char *task_kind, + uint32_t task_kind_version, const Lardon3DProjectDbCheckpoint *checkpoint, + const char *source_path, uint64_t scanset_id, int64_t updated_at) { + if (!source_path) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, + source_path, scanset_id, NULL, updated_at); +} + +Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_task( + Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, const char *task_kind, + uint32_t task_kind_version, const Lardon3DProjectDbCheckpoint *checkpoint, + const Lardon3DProjectDbFeatureExtractTask *parameters, int64_t updated_at) { + if (!parameters) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0, + parameters, updated_at); +} + +static bool read_task(sqlite3_stmt *statement, Lardon3DProjectDbTask *task) { + memset(task, 0, sizeof(*task)); + sqlite3_int64 id = sqlite3_column_int64(statement, 0); + bool has_kind = sqlite3_column_type(statement, 2) != SQLITE_NULL; + bool has_kind_version = sqlite3_column_type(statement, 3) != SQLITE_NULL; + int progress = sqlite3_column_int(statement, 6); + sqlite3_int64 sequence_count = sqlite3_column_int64(statement, 7); + sqlite3_int64 started_nsec = sqlite3_column_int64(statement, 9); + sqlite3_int64 finished_nsec = sqlite3_column_int64(statement, 11); + if (id <= 0 || !copy_column(statement, 1, task->name, sizeof(task->name))) { + return false; + } + if (has_kind != has_kind_version) { + return false; + } + if (has_kind) { + sqlite3_int64 version = sqlite3_column_int64(statement, 3); + if (!copy_column(statement, 2, task->task_kind, sizeof(task->task_kind)) || + !lardon3d_task_kind_is_valid(task->task_kind) || version <= 0 || version > UINT32_MAX) { + return false; } + task->has_task_kind = true; + task->task_kind_version = (uint32_t)version; + } + task->task_id = (uint64_t)id; + task->saved_state = (Lardon3DTaskState)sqlite3_column_int(statement, 4); + task->recovery_state = (Lardon3DTaskState)sqlite3_column_int(statement, 5); + if (progress < 0 || progress > 100 || sequence_count < 0 || sequence_count > UINT_MAX || + started_nsec < 0 || started_nsec >= 1000000000 || finished_nsec < 0 || + finished_nsec >= 1000000000 || + !database_time(sqlite3_column_int64(statement, 8), &task->started_at.tv_sec) || + !database_time(sqlite3_column_int64(statement, 10), &task->finished_at.tv_sec)) { + return false; + } + task->progress = (unsigned int)progress; + task->sequence_count = (unsigned int)sequence_count; + task->started_at.tv_nsec = (long)started_nsec; + task->finished_at.tv_nsec = (long)finished_nsec; + task->updated_at = sqlite3_column_int64(statement, 12); + if (sqlite3_column_type(statement, 13) != SQLITE_NULL) { + task->has_checkpoint = true; + if (!copy_column(statement, 13, task->checkpoint.path, sizeof(task->checkpoint.path))) { + return false; + } + sqlite3_int64 format_version = sqlite3_column_int64(statement, 14); + if (format_version <= 0 || format_version > UINT32_MAX) { + return false; + } + task->checkpoint.format_version = (uint32_t)format_version; + task->checkpoint.durability = + (Lardon3DProjectDbCheckpointDurability)sqlite3_column_int(statement, 15); + task->checkpoint.updated_at = sqlite3_column_int64(statement, 16); + if (task->checkpoint.durability < LARDON3D_DB_CHECKPOINT_DURABLE || + task->checkpoint.durability > LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE) { + return false; + } + } + return valid_state(task->saved_state) && valid_state(task->recovery_state) && + task->progress <= 100; +} + +static const char task_select[] = + "SELECT " + "t.task_id,t.name,t.task_kind,t.task_kind_version,t.saved_state,t.recovery_state,t.progress,t." + "sequence_count,t.started_sec,t.started_nsec," + "t.finished_sec,t.finished_nsec,t.updated_at,c.path,c.format_version,c.durability,c.updated_at " + "FROM tasks t LEFT JOIN checkpoints c ON c.task_id=t.task_id "; + +Lardon3DProjectDbResult lardon3d_project_db_load_task(Lardon3DProjectDb *database, uint64_t task_id, + Lardon3DProjectDbTask *task) { + if (!database || !valid_task_id(task_id) || !task) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + (void)pthread_mutex_lock(&database->mutex); + sqlite3_stmt *statement = NULL; + char sql[768]; + (void)snprintf(sql, sizeof(sql), "%s WHERE t.task_id=?1", task_select); + Lardon3DProjectDbResult result = prepare(database, sql, &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)task_id); + int code = sqlite3_step(statement); + if (code == SQLITE_DONE) { + result = LARDON3D_PROJECT_DB_NOT_FOUND; + } else if (code != SQLITE_ROW || !read_task(statement, task)) { + result = LARDON3D_PROJECT_DB_CORRUPT; + } + (void)sqlite3_finalize(statement); + } + (void)pthread_mutex_unlock(&database->mutex); + return result; +} + +Lardon3DProjectDbResult lardon3d_project_db_list_recoverable(Lardon3DProjectDb *database, + uint64_t after_task_id, + Lardon3DProjectDbTask *tasks, + size_t capacity, size_t *count) { + if (count) { + *count = 0; + } + if (!database || !tasks || !count || after_task_id > INT64_MAX || capacity == 0 || + capacity > LARDON3D_PROJECT_DB_RECOVERY_PAGE_MAX) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } +#ifdef LARDON3D_PROJECT_DB_TESTING + const char *forced_busy = getenv("LARDON3D_TEST_PROJECT_DB_BUSY_RECOVERY"); + if (forced_busy && strcmp(forced_busy, "1") == 0) { + return LARDON3D_PROJECT_DB_BUSY; + } +#endif + (void)pthread_mutex_lock(&database->mutex); + sqlite3_stmt *statement = NULL; + char sql[896]; + (void)snprintf(sql, sizeof(sql), + "%s WHERE t.recovery_state=?1 AND c.task_id IS NOT NULL AND t.task_id>?2 ORDER " + "BY t.task_id LIMIT ?3", + task_select); + Lardon3DProjectDbResult result = prepare(database, sql, &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_int(statement, 1, TASK_PENDING); + (void)sqlite3_bind_int64(statement, 2, (sqlite3_int64)after_task_id); + (void)sqlite3_bind_int64(statement, 3, (sqlite3_int64)capacity); + int code = SQLITE_DONE; + while (*count < capacity && (code = sqlite3_step(statement)) == SQLITE_ROW) { + if (!read_task(statement, &tasks[*count])) { + result = LARDON3D_PROJECT_DB_CORRUPT; + break; + } + ++*count; + } + if (result == LARDON3D_PROJECT_DB_OK && *count < capacity && code != SQLITE_DONE) { + result = sqlite_result(database, code, "list recoverable"); + } + (void)sqlite3_finalize(statement); + } + (void)pthread_mutex_unlock(&database->mutex); + return result; +} + +Lardon3DProjectDbResult lardon3d_project_db_allocate_task_id(Lardon3DProjectDb *database, + uint64_t *task_id) { + if (task_id) { + *task_id = 0; + } + if (!database || !task_id) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + (void)pthread_mutex_lock(&database->mutex); + Lardon3DProjectDbResult result = execute(database, "BEGIN IMMEDIATE", "begin task id allocation"); + sqlite3_stmt *statement = NULL; + sqlite3_int64 next = 0; + if (result == LARDON3D_PROJECT_DB_OK) { + result = prepare(database, "SELECT value FROM metadata WHERE key='next_task_id'", &statement); + } + if (result == LARDON3D_PROJECT_DB_OK) { + int code = sqlite3_step(statement); + if (code != SQLITE_ROW || sqlite3_column_type(statement, 0) != SQLITE_INTEGER) { + result = LARDON3D_PROJECT_DB_CORRUPT; + } else { + next = sqlite3_column_int64(statement, 0); + if (next <= 0 || next > INT64_MAX) { + result = LARDON3D_PROJECT_DB_CONSTRAINT; + } + } + (void)sqlite3_finalize(statement); + statement = NULL; + } + if (result == LARDON3D_PROJECT_DB_OK) { + result = prepare(database, + "UPDATE metadata SET value=CASE WHEN value=9223372036854775807 THEN 0 " + "ELSE value+1 END WHERE key='next_task_id' AND value=?1", + &statement); + } + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_int64(statement, 1, next); + result = step_done(database, statement, "advance task id"); + statement = NULL; + if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(database->connection) != 1) { + result = LARDON3D_PROJECT_DB_CONSTRAINT; + } + } + if (result == LARDON3D_PROJECT_DB_OK) { + result = execute(database, "COMMIT", "commit task id allocation"); + } + if (result != LARDON3D_PROJECT_DB_OK) { + (void)execute(database, "ROLLBACK", "rollback task id allocation"); + } else { + *task_id = (uint64_t)next; + } + (void)pthread_mutex_unlock(&database->mutex); + return result; +} + +Lardon3DProjectDbResult +lardon3d_project_db_load_image_import(Lardon3DProjectDb *database, uint64_t task_id, + Lardon3DProjectDbImageImport *parameters) { + if (!database || !valid_task_id(task_id) || !parameters) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + memset(parameters, 0, sizeof(*parameters)); + (void)pthread_mutex_lock(&database->mutex); + sqlite3_stmt *statement = NULL; + Lardon3DProjectDbResult result = + prepare(database, "SELECT source_path,scanset_id FROM image_import_tasks WHERE task_id=?1", + &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)task_id); + int code = sqlite3_step(statement); + if (code == SQLITE_DONE) { + result = LARDON3D_PROJECT_DB_NOT_FOUND; + } else if (code != SQLITE_ROW || !copy_column(statement, 0, parameters->source_path, + sizeof(parameters->source_path))) { + result = LARDON3D_PROJECT_DB_CORRUPT; + } else { + sqlite3_int64 scanset = sqlite3_column_int64(statement, 1); + if (scanset <= 0) { + result = LARDON3D_PROJECT_DB_CORRUPT; + } else { + parameters->task_id = task_id; + parameters->scanset_id = (uint64_t)scanset; + } + } + (void)sqlite3_finalize(statement); + } + (void)pthread_mutex_unlock(&database->mutex); + return result; +} + +static bool valid_catalog_id(uint64_t value) { return value > 0 && value <= INT64_MAX; } + +static bool valid_original_name(const char *name) { + return bounded_text(name, LARDON3D_PROJECT_DB_IMAGE_NAME_CAPACITY, false) && !strchr(name, '/') && + !strchr(name, '\\') && !strchr(name, '\t') && !strchr(name, '\r') && !strchr(name, '\n'); +} + +static bool valid_relative_asset_path(const char *path) { + return bounded_text(path, LARDON3D_PROJECT_DB_PATH_CAPACITY, false) && path[0] != '/' && + !strchr(path, '\\') && strncmp(path, "../", 3) != 0 && !strstr(path, "/../") && + strcmp(path, "..") != 0; +} + +static bool canonical_asset_path(const unsigned char hash[LARDON3D_PROJECT_DB_SHA256_SIZE], + const char *path) { + static const char digits[] = "0123456789abcdef"; + char hex[65]; + for (size_t index = 0; index < LARDON3D_PROJECT_DB_SHA256_SIZE; ++index) { + hex[index * 2] = digits[hash[index] >> 4]; + hex[index * 2 + 1] = digits[hash[index] & 15U]; + } + hex[64] = '\0'; + char expected[LARDON3D_PROJECT_DB_PATH_CAPACITY]; + int written = snprintf(expected, sizeof(expected), "assets/images/%c%c/%s", hex[0], hex[1], hex); + return written > 0 && (size_t)written < sizeof(expected) && strcmp(path, expected) == 0; +} + +Lardon3DProjectDbResult lardon3d_project_db_create_scanset(Lardon3DProjectDb *database, + const char *name, + Lardon3DProjectDbScanSet *scanset) { + if (!database || !scanset || + !bounded_text(name, LARDON3D_PROJECT_DB_SCANSET_NAME_CAPACITY, false)) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + int64_t now = (int64_t)time(NULL); + if (now < 0) { + return LARDON3D_PROJECT_DB_IO_ERROR; + } + memset(scanset, 0, sizeof(*scanset)); + (void)pthread_mutex_lock(&database->mutex); + sqlite3_stmt *statement = NULL; + Lardon3DProjectDbResult result = prepare( + database, "INSERT INTO scansets(name,created_at,updated_at) VALUES(?1,?2,?2)", &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_text(statement, 1, name, -1, SQLITE_TRANSIENT); + (void)sqlite3_bind_int64(statement, 2, now); + result = step_done(database, statement, "create scanset"); + } + sqlite3_int64 id = sqlite3_last_insert_rowid(database->connection); + if (result == LARDON3D_PROJECT_DB_OK && id <= 0) { + result = LARDON3D_PROJECT_DB_CONSTRAINT; + } + if (result == LARDON3D_PROJECT_DB_OK) { scanset->scanset_id = (uint64_t)id; - scanset->created_at = created; - scanset->updated_at = updated; - return true; + scanset->created_at = now; + scanset->updated_at = now; + (void)snprintf(scanset->name, sizeof(scanset->name), "%s", name); + } + (void)pthread_mutex_unlock(&database->mutex); + return result; } -Lardon3DProjectDbResult -lardon3d_project_db_load_scanset(Lardon3DProjectDb *database, - uint64_t scanset_id, Lardon3DProjectDbScanSet *scanset) -{ - if (!database || !valid_catalog_id(scanset_id) || !scanset) - return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; - memset(scanset, 0, sizeof(*scanset)); - (void)pthread_mutex_lock(&database->mutex); - sqlite3_stmt *statement = NULL; - Lardon3DProjectDbResult result = prepare(database, - "SELECT scanset_id,name,created_at,updated_at FROM scansets WHERE scanset_id=?1", - &statement); - if (result == LARDON3D_PROJECT_DB_OK) { - (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)scanset_id); - int code = sqlite3_step(statement); - if (code == SQLITE_DONE) result = LARDON3D_PROJECT_DB_NOT_FOUND; - else if (code != SQLITE_ROW || !read_scanset(statement, scanset)) - result = LARDON3D_PROJECT_DB_CORRUPT; - (void)sqlite3_finalize(statement); +static bool read_scanset(sqlite3_stmt *statement, Lardon3DProjectDbScanSet *scanset) { + sqlite3_int64 id = sqlite3_column_int64(statement, 0); + sqlite3_int64 created = sqlite3_column_int64(statement, 2); + sqlite3_int64 updated = sqlite3_column_int64(statement, 3); + if (id <= 0 || created < 0 || updated < created || + !copy_column(statement, 1, scanset->name, sizeof(scanset->name))) { + return false; + } + scanset->scanset_id = (uint64_t)id; + scanset->created_at = created; + scanset->updated_at = updated; + return true; +} + +Lardon3DProjectDbResult lardon3d_project_db_load_scanset(Lardon3DProjectDb *database, + uint64_t scanset_id, + Lardon3DProjectDbScanSet *scanset) { + if (!database || !valid_catalog_id(scanset_id) || !scanset) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + memset(scanset, 0, sizeof(*scanset)); + (void)pthread_mutex_lock(&database->mutex); + sqlite3_stmt *statement = NULL; + Lardon3DProjectDbResult result = prepare( + database, "SELECT scanset_id,name,created_at,updated_at FROM scansets WHERE scanset_id=?1", + &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)scanset_id); + int code = sqlite3_step(statement); + if (code == SQLITE_DONE) { + result = LARDON3D_PROJECT_DB_NOT_FOUND; + } else if (code != SQLITE_ROW || !read_scanset(statement, scanset)) { + result = LARDON3D_PROJECT_DB_CORRUPT; } - (void)pthread_mutex_unlock(&database->mutex); - return result; + (void)sqlite3_finalize(statement); + } + (void)pthread_mutex_unlock(&database->mutex); + return result; } -Lardon3DProjectDbResult -lardon3d_project_db_list_scansets(Lardon3DProjectDb *database, - uint64_t after_scanset_id, Lardon3DProjectDbScanSet *scansets, - size_t capacity, size_t *count) -{ - if (count) *count = 0; - if (!database || !scansets || !count || after_scanset_id > INT64_MAX - || capacity == 0 || capacity > LARDON3D_PROJECT_DB_CATALOG_PAGE_MAX) - return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; - (void)pthread_mutex_lock(&database->mutex); - sqlite3_stmt *statement = NULL; - Lardon3DProjectDbResult result = prepare(database, - "SELECT scanset_id,name,created_at,updated_at FROM scansets WHERE scanset_id>?1 ORDER BY scanset_id LIMIT ?2", - &statement); - if (result == LARDON3D_PROJECT_DB_OK) { - (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)after_scanset_id); - (void)sqlite3_bind_int64(statement, 2, (sqlite3_int64)capacity); - int code = SQLITE_DONE; - while (*count < capacity && (code = sqlite3_step(statement)) == SQLITE_ROW) { - if (!read_scanset(statement, &scansets[*count])) { - result = LARDON3D_PROJECT_DB_CORRUPT; break; - } - ++*count; - } - if (result == LARDON3D_PROJECT_DB_OK && *count < capacity - && code != SQLITE_DONE) result = sqlite_result(database, code, "list scansets"); - (void)sqlite3_finalize(statement); +Lardon3DProjectDbResult lardon3d_project_db_list_scansets(Lardon3DProjectDb *database, + uint64_t after_scanset_id, + Lardon3DProjectDbScanSet *scansets, + size_t capacity, size_t *count) { + if (count) { + *count = 0; + } + if (!database || !scansets || !count || after_scanset_id > INT64_MAX || capacity == 0 || + capacity > LARDON3D_PROJECT_DB_CATALOG_PAGE_MAX) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + (void)pthread_mutex_lock(&database->mutex); + sqlite3_stmt *statement = NULL; + Lardon3DProjectDbResult result = + prepare(database, + "SELECT scanset_id,name,created_at,updated_at FROM scansets WHERE scanset_id>?1 " + "ORDER BY scanset_id LIMIT ?2", + &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)after_scanset_id); + (void)sqlite3_bind_int64(statement, 2, (sqlite3_int64)capacity); + int code = SQLITE_DONE; + while (*count < capacity && (code = sqlite3_step(statement)) == SQLITE_ROW) { + if (!read_scanset(statement, &scansets[*count])) { + result = LARDON3D_PROJECT_DB_CORRUPT; + break; + } + ++*count; } - (void)pthread_mutex_unlock(&database->mutex); - return result; + if (result == LARDON3D_PROJECT_DB_OK && *count < capacity && code != SQLITE_DONE) { + result = sqlite_result(database, code, "list scansets"); + } + (void)sqlite3_finalize(statement); + } + (void)pthread_mutex_unlock(&database->mutex); + return result; } -static bool -read_image(sqlite3_stmt *statement, Lardon3DProjectDbImage *image, - Lardon3DProjectDbImageAsset *asset) -{ - sqlite3_int64 image_id = sqlite3_column_int64(statement, 0); - sqlite3_int64 scanset_id = sqlite3_column_int64(statement, 1); - sqlite3_int64 asset_id = sqlite3_column_int64(statement, 2); - sqlite3_int64 size = sqlite3_column_int64(statement, 7); - int hash_size = sqlite3_column_bytes(statement, 5); - const void *hash = sqlite3_column_blob(statement, 5); - if (image_id <= 0 || scanset_id <= 0 || asset_id <= 0 || size < 0 - || hash_size != LARDON3D_PROJECT_DB_SHA256_SIZE || !hash - || !copy_column(statement, 3, image->original_name, sizeof(image->original_name)) - || !copy_column(statement, 4, image->source_path, sizeof(image->source_path)) - || !copy_column(statement, 6, asset->path, sizeof(asset->path)) - || !valid_original_name(image->original_name) - || !valid_relative_asset_path(asset->path)) return false; - image->image_id = (uint64_t)image_id; - image->scanset_id = (uint64_t)scanset_id; - image->asset_id = (uint64_t)asset_id; - asset->asset_id = (uint64_t)asset_id; - memcpy(asset->sha256, hash, LARDON3D_PROJECT_DB_SHA256_SIZE); - if (!canonical_asset_path(asset->sha256, asset->path)) return false; - asset->size_bytes = (uint64_t)size; - asset->state = (Lardon3DProjectDbImageAssetState)sqlite3_column_int(statement, 8); - asset->created_at = sqlite3_column_int64(statement, 9); - image->has_producer_task = sqlite3_column_type(statement, 10) != SQLITE_NULL; - sqlite3_int64 producer = sqlite3_column_int64(statement, 10); - if (image->has_producer_task && producer <= 0) return false; - image->producer_task_id = image->has_producer_task ? (uint64_t)producer : 0; - image->imported_at = sqlite3_column_int64(statement, 11); - return asset->state == LARDON3D_DB_IMAGE_ASSET_READY - && asset->created_at >= 0 && image->imported_at >= 0; +static bool read_image(sqlite3_stmt *statement, Lardon3DProjectDbImage *image, + Lardon3DProjectDbImageAsset *asset) { + sqlite3_int64 image_id = sqlite3_column_int64(statement, 0); + sqlite3_int64 scanset_id = sqlite3_column_int64(statement, 1); + sqlite3_int64 asset_id = sqlite3_column_int64(statement, 2); + sqlite3_int64 size = sqlite3_column_int64(statement, 7); + int hash_size = sqlite3_column_bytes(statement, 5); + const void *hash = sqlite3_column_blob(statement, 5); + if (image_id <= 0 || scanset_id <= 0 || asset_id <= 0 || size < 0 || + hash_size != LARDON3D_PROJECT_DB_SHA256_SIZE || !hash || + !copy_column(statement, 3, image->original_name, sizeof(image->original_name)) || + !copy_column(statement, 4, image->source_path, sizeof(image->source_path)) || + !copy_column(statement, 6, asset->path, sizeof(asset->path)) || + !valid_original_name(image->original_name) || !valid_relative_asset_path(asset->path)) { + return false; + } + image->image_id = (uint64_t)image_id; + image->scanset_id = (uint64_t)scanset_id; + image->asset_id = (uint64_t)asset_id; + asset->asset_id = (uint64_t)asset_id; + memcpy(asset->sha256, hash, LARDON3D_PROJECT_DB_SHA256_SIZE); + if (!canonical_asset_path(asset->sha256, asset->path)) { + return false; + } + asset->size_bytes = (uint64_t)size; + asset->state = (Lardon3DProjectDbImageAssetState)sqlite3_column_int(statement, 8); + asset->created_at = sqlite3_column_int64(statement, 9); + image->has_producer_task = sqlite3_column_type(statement, 10) != SQLITE_NULL; + sqlite3_int64 producer = sqlite3_column_int64(statement, 10); + if (image->has_producer_task && producer <= 0) { + return false; + } + image->producer_task_id = image->has_producer_task ? (uint64_t)producer : 0; + image->imported_at = sqlite3_column_int64(statement, 11); + return asset->state == LARDON3D_DB_IMAGE_ASSET_READY && asset->created_at >= 0 && + image->imported_at >= 0; } static const char image_select[] = - "SELECT i.image_id,i.scanset_id,i.asset_id,i.original_name,i.source_path,a.sha256,a.path,a.size_bytes,a.state,a.created_at,i.producer_task_id,i.imported_at FROM images i JOIN image_assets a ON a.asset_id=i.asset_id "; + "SELECT " + "i.image_id,i.scanset_id,i.asset_id,i.original_name,i.source_path,a.sha256,a.path,a.size_bytes," + "a.state,a.created_at,i.producer_task_id,i.imported_at FROM images i JOIN image_assets a ON " + "a.asset_id=i.asset_id "; -Lardon3DProjectDbResult -lardon3d_project_db_register_image(Lardon3DProjectDb *database, - uint64_t scanset_id, const unsigned char sha256[LARDON3D_PROJECT_DB_SHA256_SIZE], - const char *asset_path, uint64_t size_bytes, const char *original_name, - const char *source_path, uint64_t producer_task_id, int64_t imported_at, - Lardon3DProjectDbImageRegisterStatus *status, Lardon3DProjectDbImage *image) -{ - if (!database || !valid_catalog_id(scanset_id) || !sha256 - || !valid_relative_asset_path(asset_path) - || !canonical_asset_path(sha256, asset_path) || size_bytes > INT64_MAX - || !valid_original_name(original_name) - || !bounded_text(source_path, LARDON3D_PROJECT_DB_PATH_CAPACITY, false) - || (producer_task_id != 0 && !valid_task_id(producer_task_id)) - || imported_at < 0 || !status || !image) - return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; +Lardon3DProjectDbResult lardon3d_project_db_register_image( + Lardon3DProjectDb *database, uint64_t scanset_id, + const unsigned char sha256[LARDON3D_PROJECT_DB_SHA256_SIZE], const char *asset_path, + uint64_t size_bytes, const char *original_name, const char *source_path, + uint64_t producer_task_id, int64_t imported_at, Lardon3DProjectDbImageRegisterStatus *status, + Lardon3DProjectDbImage *image) { + if (!database || !valid_catalog_id(scanset_id) || !sha256 || + !valid_relative_asset_path(asset_path) || !canonical_asset_path(sha256, asset_path) || + size_bytes > INT64_MAX || !valid_original_name(original_name) || + !bounded_text(source_path, LARDON3D_PROJECT_DB_PATH_CAPACITY, false) || + (producer_task_id != 0 && !valid_task_id(producer_task_id)) || imported_at < 0 || !status || + !image) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } #ifdef LARDON3D_PROJECT_DB_TESTING - const char *failure = getenv("LARDON3D_TEST_PROJECT_DB_FAIL_IMAGE_REGISTER"); - if (failure && strcmp(failure, "1") == 0) return LARDON3D_PROJECT_DB_BUSY; + const char *failure = getenv("LARDON3D_TEST_PROJECT_DB_FAIL_IMAGE_REGISTER"); + if (failure && strcmp(failure, "1") == 0) { + return LARDON3D_PROJECT_DB_BUSY; + } #endif - memset(image, 0, sizeof(*image)); - *status = LARDON3D_PROJECT_DB_IMAGE_REGISTERED; - (void)pthread_mutex_lock(&database->mutex); - Lardon3DProjectDbResult result = execute(database, "BEGIN IMMEDIATE", "begin image register"); - sqlite3_stmt *statement = NULL; - if (result == LARDON3D_PROJECT_DB_OK) result = prepare(database, - "INSERT INTO image_assets(sha256,path,size_bytes,state,created_at) VALUES(?1,?2,?3,1,?4) ON CONFLICT(sha256) DO NOTHING", &statement); - if (result == LARDON3D_PROJECT_DB_OK) { - (void)sqlite3_bind_blob(statement, 1, sha256, LARDON3D_PROJECT_DB_SHA256_SIZE, SQLITE_TRANSIENT); - (void)sqlite3_bind_text(statement, 2, asset_path, -1, SQLITE_TRANSIENT); - (void)sqlite3_bind_int64(statement, 3, (sqlite3_int64)size_bytes); - (void)sqlite3_bind_int64(statement, 4, imported_at); - result = step_done(database, statement, "insert image asset"); statement = NULL; + memset(image, 0, sizeof(*image)); + *status = LARDON3D_PROJECT_DB_IMAGE_REGISTERED; + (void)pthread_mutex_lock(&database->mutex); + Lardon3DProjectDbResult result = execute(database, "BEGIN IMMEDIATE", "begin image register"); + sqlite3_stmt *statement = NULL; + if (result == LARDON3D_PROJECT_DB_OK) { + result = prepare(database, + "INSERT INTO image_assets(sha256,path,size_bytes,state,created_at) " + "VALUES(?1,?2,?3,1,?4) ON CONFLICT(sha256) DO NOTHING", + &statement); + } + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_blob(statement, 1, sha256, LARDON3D_PROJECT_DB_SHA256_SIZE, + SQLITE_TRANSIENT); + (void)sqlite3_bind_text(statement, 2, asset_path, -1, SQLITE_TRANSIENT); + (void)sqlite3_bind_int64(statement, 3, (sqlite3_int64)size_bytes); + (void)sqlite3_bind_int64(statement, 4, imported_at); + result = step_done(database, statement, "insert image asset"); + statement = NULL; + } + sqlite3_int64 asset_id = 0; + if (result == LARDON3D_PROJECT_DB_OK) { + result = + prepare(database, "SELECT asset_id,path,size_bytes,state FROM image_assets WHERE sha256=?1", + &statement); + } + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_blob(statement, 1, sha256, LARDON3D_PROJECT_DB_SHA256_SIZE, + SQLITE_TRANSIENT); + int code = sqlite3_step(statement); + char stored_path[LARDON3D_PROJECT_DB_PATH_CAPACITY]; + if (code != SQLITE_ROW || !copy_column(statement, 1, stored_path, sizeof(stored_path)) || + strcmp(stored_path, asset_path) != 0 || + sqlite3_column_int64(statement, 2) != (sqlite3_int64)size_bytes || + sqlite3_column_int(statement, 3) != LARDON3D_DB_IMAGE_ASSET_READY) { + result = LARDON3D_PROJECT_DB_CORRUPT; + } else { + asset_id = sqlite3_column_int64(statement, 0); } - sqlite3_int64 asset_id = 0; - if (result == LARDON3D_PROJECT_DB_OK) result = prepare(database, - "SELECT asset_id,path,size_bytes,state FROM image_assets WHERE sha256=?1", &statement); - if (result == LARDON3D_PROJECT_DB_OK) { - (void)sqlite3_bind_blob(statement, 1, sha256, LARDON3D_PROJECT_DB_SHA256_SIZE, SQLITE_TRANSIENT); - int code = sqlite3_step(statement); - char stored_path[LARDON3D_PROJECT_DB_PATH_CAPACITY]; - if (code != SQLITE_ROW || !copy_column(statement, 1, stored_path, sizeof(stored_path)) - || strcmp(stored_path, asset_path) != 0 - || sqlite3_column_int64(statement, 2) != (sqlite3_int64)size_bytes - || sqlite3_column_int(statement, 3) != LARDON3D_DB_IMAGE_ASSET_READY) { - result = LARDON3D_PROJECT_DB_CORRUPT; - } else asset_id = sqlite3_column_int64(statement, 0); - (void)sqlite3_finalize(statement); statement = NULL; - } - if (result == LARDON3D_PROJECT_DB_OK) result = prepare(database, - "INSERT INTO images(scanset_id,asset_id,original_name,source_path,producer_task_id,imported_at) VALUES(?1,?2,?3,?4,?5,?6) ON CONFLICT(scanset_id,asset_id) DO NOTHING", &statement); - int inserted = 0; - if (result == LARDON3D_PROJECT_DB_OK) { - (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)scanset_id); - (void)sqlite3_bind_int64(statement, 2, asset_id); - (void)sqlite3_bind_text(statement, 3, original_name, -1, SQLITE_TRANSIENT); - (void)sqlite3_bind_text(statement, 4, source_path, -1, SQLITE_TRANSIENT); - if (producer_task_id) (void)sqlite3_bind_int64(statement, 5, (sqlite3_int64)producer_task_id); - else (void)sqlite3_bind_null(statement, 5); - (void)sqlite3_bind_int64(statement, 6, imported_at); - result = step_done(database, statement, "insert logical image"); statement = NULL; - inserted = sqlite3_changes(database->connection); - } - sqlite3_int64 image_id = 0; - if (result == LARDON3D_PROJECT_DB_OK) result = prepare(database, - "SELECT image_id FROM images WHERE scanset_id=?1 AND asset_id=?2", &statement); - if (result == LARDON3D_PROJECT_DB_OK) { - (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)scanset_id); - (void)sqlite3_bind_int64(statement, 2, asset_id); - int code = sqlite3_step(statement); - if (code != SQLITE_ROW || (image_id = sqlite3_column_int64(statement, 0)) <= 0) - result = LARDON3D_PROJECT_DB_CORRUPT; - (void)sqlite3_finalize(statement); statement = NULL; - } - if (result == LARDON3D_PROJECT_DB_OK) result = execute(database, "COMMIT", "commit image register"); - if (result != LARDON3D_PROJECT_DB_OK) (void)execute(database, "ROLLBACK", "rollback image register"); - (void)pthread_mutex_unlock(&database->mutex); - if (result != LARDON3D_PROJECT_DB_OK) return result; - *status = inserted == 1 ? LARDON3D_PROJECT_DB_IMAGE_REGISTERED - : LARDON3D_PROJECT_DB_IMAGE_ALREADY_PRESENT; - return lardon3d_project_db_load_image(database, (uint64_t)image_id, image, - &(Lardon3DProjectDbImageAsset){0}); -} - -Lardon3DProjectDbResult -lardon3d_project_db_load_image(Lardon3DProjectDb *database, uint64_t image_id, - Lardon3DProjectDbImage *image, Lardon3DProjectDbImageAsset *asset) -{ - if (!database || !valid_catalog_id(image_id) || !image || !asset) - return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; - memset(image, 0, sizeof(*image)); memset(asset, 0, sizeof(*asset)); - (void)pthread_mutex_lock(&database->mutex); - char sql[768]; (void)snprintf(sql, sizeof(sql), "%s WHERE i.image_id=?1", image_select); - sqlite3_stmt *statement = NULL; - Lardon3DProjectDbResult result = prepare(database, sql, &statement); - if (result == LARDON3D_PROJECT_DB_OK) { - (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)image_id); - int code = sqlite3_step(statement); - if (code == SQLITE_DONE) result = LARDON3D_PROJECT_DB_NOT_FOUND; - else if (code != SQLITE_ROW || !read_image(statement, image, asset)) - result = LARDON3D_PROJECT_DB_CORRUPT; - (void)sqlite3_finalize(statement); - } - (void)pthread_mutex_unlock(&database->mutex); - return result; -} - -Lardon3DProjectDbResult -lardon3d_project_db_list_images(Lardon3DProjectDb *database, - uint64_t scanset_id, uint64_t after_image_id, Lardon3DProjectDbImage *images, - Lardon3DProjectDbImageAsset *assets, size_t capacity, size_t *count) -{ - if (count) *count = 0; - if (!database || !valid_catalog_id(scanset_id) || after_image_id > INT64_MAX - || !images || !assets || !count || capacity == 0 - || capacity > LARDON3D_PROJECT_DB_CATALOG_PAGE_MAX) - return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; - Lardon3DProjectDbScanSet scanset; - Lardon3DProjectDbResult exists = lardon3d_project_db_load_scanset( - database, scanset_id, &scanset); - if (exists != LARDON3D_PROJECT_DB_OK) return exists; - (void)pthread_mutex_lock(&database->mutex); - char sql[896]; (void)snprintf(sql, sizeof(sql), - "%s WHERE i.scanset_id=?1 AND i.image_id>?2 ORDER BY i.image_id LIMIT ?3", image_select); - sqlite3_stmt *statement = NULL; - Lardon3DProjectDbResult result = prepare(database, sql, &statement); - if (result == LARDON3D_PROJECT_DB_OK) { - (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)scanset_id); - (void)sqlite3_bind_int64(statement, 2, (sqlite3_int64)after_image_id); - (void)sqlite3_bind_int64(statement, 3, (sqlite3_int64)capacity); - int code = SQLITE_DONE; - while (*count < capacity && (code = sqlite3_step(statement)) == SQLITE_ROW) { - memset(&images[*count], 0, sizeof(images[*count])); - memset(&assets[*count], 0, sizeof(assets[*count])); - if (!read_image(statement, &images[*count], &assets[*count])) { - result = LARDON3D_PROJECT_DB_CORRUPT; break; - } - ++*count; - } - if (result == LARDON3D_PROJECT_DB_OK && *count < capacity && code != SQLITE_DONE) - result = sqlite_result(database, code, "list images"); - (void)sqlite3_finalize(statement); - } - (void)pthread_mutex_unlock(&database->mutex); - return result; -} - -Lardon3DProjectDbResult -lardon3d_project_db_count_images(Lardon3DProjectDb *database, - uint64_t scanset_id, uint64_t *count) -{ - if (count) *count = 0; - if (!database || !valid_catalog_id(scanset_id) || !count) - return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; - Lardon3DProjectDbScanSet scanset; - Lardon3DProjectDbResult exists = lardon3d_project_db_load_scanset( - database, scanset_id, &scanset); - if (exists != LARDON3D_PROJECT_DB_OK) return exists; - (void)pthread_mutex_lock(&database->mutex); - sqlite3_stmt *statement = NULL; - Lardon3DProjectDbResult result = prepare(database, - "SELECT count(*) FROM images WHERE scanset_id=?1", &statement); - if (result == LARDON3D_PROJECT_DB_OK) { - (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)scanset_id); - int code = sqlite3_step(statement); sqlite3_int64 value = sqlite3_column_int64(statement, 0); - if (code != SQLITE_ROW || value < 0) result = LARDON3D_PROJECT_DB_CORRUPT; - else *count = (uint64_t)value; - (void)sqlite3_finalize(statement); - } - (void)pthread_mutex_unlock(&database->mutex); - return result; -} - -Lardon3DProjectDbResult -lardon3d_project_db_create_artifact(Lardon3DProjectDb *database, const Lardon3DProjectDbArtifact *artifact) -{ - if (!database || !artifact || !bounded_text(artifact->artifact_id, sizeof(artifact->artifact_id), false) - || !bounded_text(artifact->kind, sizeof(artifact->kind), false) || !bounded_text(artifact->path, sizeof(artifact->path), false) - || artifact->state != LARDON3D_DB_ARTIFACT_STAGED || artifact->size_bytes != 0 || artifact->created_at < 0 - || artifact->updated_at < artifact->created_at || (artifact->has_producer_task && !valid_task_id(artifact->producer_task_id))) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; - (void)pthread_mutex_lock(&database->mutex); sqlite3_stmt *statement = NULL; - Lardon3DProjectDbResult result = prepare(database, "INSERT INTO artifacts(artifact_id,kind,path,state,size_bytes,producer_task_id,created_at,updated_at) VALUES(?1,?2,?3,0,0,?4,?5,?6)", &statement); - if (result == LARDON3D_PROJECT_DB_OK) { (void)sqlite3_bind_text(statement, 1, artifact->artifact_id, -1, SQLITE_TRANSIENT); (void)sqlite3_bind_text(statement, 2, artifact->kind, -1, SQLITE_TRANSIENT); - (void)sqlite3_bind_text(statement, 3, artifact->path, -1, SQLITE_TRANSIENT); if (artifact->has_producer_task) (void)sqlite3_bind_int64(statement, 4, (sqlite3_int64)artifact->producer_task_id); else (void)sqlite3_bind_null(statement, 4); - (void)sqlite3_bind_int64(statement, 5, artifact->created_at); (void)sqlite3_bind_int64(statement, 6, artifact->updated_at); result = step_done(database, statement, "create artifact"); } - (void)pthread_mutex_unlock(&database->mutex); return result; -} - -Lardon3DProjectDbResult -lardon3d_project_db_mark_artifact_ready(Lardon3DProjectDb *database, const char *artifact_id, int64_t updated_at) -{ - if (!database || !bounded_text(artifact_id, LARDON3D_PROJECT_DB_ID_CAPACITY, false) || updated_at < 0) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; - char path[LARDON3D_PROJECT_DB_PATH_CAPACITY]; Lardon3DProjectDbArtifact artifact; - Lardon3DProjectDbResult result = lardon3d_project_db_load_artifact(database, artifact_id, &artifact); - if (result != LARDON3D_PROJECT_DB_OK) return result; (void)snprintf(path, sizeof(path), "%s", artifact.path); - struct stat information; if (stat(path, &information) != 0 || !S_ISREG(information.st_mode) || information.st_size < 0) return LARDON3D_PROJECT_DB_IO_ERROR; - (void)pthread_mutex_lock(&database->mutex); sqlite3_stmt *statement = NULL; - result = prepare(database, "UPDATE artifacts SET state=1,size_bytes=?1,updated_at=?2 WHERE artifact_id=?3 AND path=?4", &statement); - if (result == LARDON3D_PROJECT_DB_OK) { (void)sqlite3_bind_int64(statement, 1, information.st_size); (void)sqlite3_bind_int64(statement, 2, updated_at); - (void)sqlite3_bind_text(statement, 3, artifact_id, -1, SQLITE_TRANSIENT); (void)sqlite3_bind_text(statement, 4, path, -1, SQLITE_TRANSIENT); result = step_done(database, statement, "mark artifact ready"); - if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(database->connection) != 1) result = LARDON3D_PROJECT_DB_NOT_FOUND; } - (void)pthread_mutex_unlock(&database->mutex); return result; -} - -Lardon3DProjectDbResult -lardon3d_project_db_load_artifact(Lardon3DProjectDb *database, const char *artifact_id, Lardon3DProjectDbArtifact *artifact) -{ - if (!database || !artifact || !bounded_text(artifact_id, LARDON3D_PROJECT_DB_ID_CAPACITY, false)) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; - (void)pthread_mutex_lock(&database->mutex); sqlite3_stmt *statement = NULL; - Lardon3DProjectDbResult result = prepare(database, "SELECT artifact_id,kind,path,state,size_bytes,producer_task_id,created_at,updated_at FROM artifacts WHERE artifact_id=?1", &statement); - if (result == LARDON3D_PROJECT_DB_OK) { (void)sqlite3_bind_text(statement, 1, artifact_id, -1, SQLITE_TRANSIENT); int code = sqlite3_step(statement); - if (code == SQLITE_DONE) result = LARDON3D_PROJECT_DB_NOT_FOUND; - else if (code != SQLITE_ROW || !copy_column(statement, 0, artifact->artifact_id, sizeof(artifact->artifact_id)) || !copy_column(statement, 1, artifact->kind, sizeof(artifact->kind)) || !copy_column(statement, 2, artifact->path, sizeof(artifact->path))) result = LARDON3D_PROJECT_DB_CORRUPT; - else { sqlite3_int64 size = sqlite3_column_int64(statement, 4); sqlite3_int64 producer = sqlite3_column_int64(statement, 5); - artifact->state = (Lardon3DProjectDbArtifactState)sqlite3_column_int(statement, 3); artifact->size_bytes = size >= 0 ? (uint64_t)size : 0; - artifact->has_producer_task = sqlite3_column_type(statement, 5) != SQLITE_NULL; artifact->producer_task_id = artifact->has_producer_task && producer > 0 ? (uint64_t)producer : 0; - artifact->created_at = sqlite3_column_int64(statement, 6); artifact->updated_at = sqlite3_column_int64(statement, 7); } - (void)sqlite3_finalize(statement); } - if (result == LARDON3D_PROJECT_DB_OK && (artifact->state < LARDON3D_DB_ARTIFACT_STAGED - || artifact->state > LARDON3D_DB_ARTIFACT_READY || (artifact->has_producer_task && artifact->producer_task_id == 0))) result = LARDON3D_PROJECT_DB_CORRUPT; - (void)pthread_mutex_unlock(&database->mutex); return result; -} - -#ifdef LARDON3D_PROJECT_DB_TESTING -Lardon3DProjectDbResult -lardon3d_project_db_test_orphan_checkpoint(Lardon3DProjectDb *database) -{ - if (!database) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; - (void)pthread_mutex_lock(&database->mutex); - sqlite3_stmt *statement = NULL; - Lardon3DProjectDbResult result = prepare(database, - "INSERT INTO checkpoints(task_id,path,format_version,durability,updated_at) VALUES(9223372036854775807,'orphan',1,0,0)", + (void)sqlite3_finalize(statement); + statement = NULL; + } + if (result == LARDON3D_PROJECT_DB_OK) { + result = prepare( + database, + "INSERT INTO " + "images(scanset_id,asset_id,original_name,source_path,producer_task_id,imported_at) " + "VALUES(?1,?2,?3,?4,?5,?6) ON CONFLICT(scanset_id,asset_id) DO NOTHING", &statement); - if (result == LARDON3D_PROJECT_DB_OK) result = step_done(database, statement, "orphan checkpoint"); - (void)pthread_mutex_unlock(&database->mutex); + } + int inserted = 0; + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)scanset_id); + (void)sqlite3_bind_int64(statement, 2, asset_id); + (void)sqlite3_bind_text(statement, 3, original_name, -1, SQLITE_TRANSIENT); + (void)sqlite3_bind_text(statement, 4, source_path, -1, SQLITE_TRANSIENT); + if (producer_task_id) { + (void)sqlite3_bind_int64(statement, 5, (sqlite3_int64)producer_task_id); + } else { + (void)sqlite3_bind_null(statement, 5); + } + (void)sqlite3_bind_int64(statement, 6, imported_at); + result = step_done(database, statement, "insert logical image"); + statement = NULL; + inserted = sqlite3_changes(database->connection); + } + sqlite3_int64 image_id = 0; + if (result == LARDON3D_PROJECT_DB_OK) { + result = prepare(database, "SELECT image_id FROM images WHERE scanset_id=?1 AND asset_id=?2", + &statement); + } + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)scanset_id); + (void)sqlite3_bind_int64(statement, 2, asset_id); + int code = sqlite3_step(statement); + if (code != SQLITE_ROW || (image_id = sqlite3_column_int64(statement, 0)) <= 0) { + result = LARDON3D_PROJECT_DB_CORRUPT; + } + (void)sqlite3_finalize(statement); + statement = NULL; + } + if (result == LARDON3D_PROJECT_DB_OK) { + result = execute(database, "COMMIT", "commit image register"); + } + if (result != LARDON3D_PROJECT_DB_OK) { + (void)execute(database, "ROLLBACK", "rollback image register"); + } + (void)pthread_mutex_unlock(&database->mutex); + if (result != LARDON3D_PROJECT_DB_OK) { return result; + } + *status = inserted == 1 ? LARDON3D_PROJECT_DB_IMAGE_REGISTERED + : LARDON3D_PROJECT_DB_IMAGE_ALREADY_PRESENT; + return lardon3d_project_db_load_image(database, (uint64_t)image_id, image, + &(Lardon3DProjectDbImageAsset){0}); +} + +Lardon3DProjectDbResult lardon3d_project_db_load_image(Lardon3DProjectDb *database, + uint64_t image_id, + Lardon3DProjectDbImage *image, + Lardon3DProjectDbImageAsset *asset) { + if (!database || !valid_catalog_id(image_id) || !image || !asset) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + memset(image, 0, sizeof(*image)); + memset(asset, 0, sizeof(*asset)); + (void)pthread_mutex_lock(&database->mutex); + char sql[768]; + (void)snprintf(sql, sizeof(sql), "%s WHERE i.image_id=?1", image_select); + sqlite3_stmt *statement = NULL; + Lardon3DProjectDbResult result = prepare(database, sql, &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)image_id); + int code = sqlite3_step(statement); + if (code == SQLITE_DONE) { + result = LARDON3D_PROJECT_DB_NOT_FOUND; + } else if (code != SQLITE_ROW || !read_image(statement, image, asset)) { + result = LARDON3D_PROJECT_DB_CORRUPT; + } + (void)sqlite3_finalize(statement); + } + (void)pthread_mutex_unlock(&database->mutex); + return result; +} + +Lardon3DProjectDbResult lardon3d_project_db_list_images(Lardon3DProjectDb *database, + uint64_t scanset_id, + uint64_t after_image_id, + Lardon3DProjectDbImage *images, + Lardon3DProjectDbImageAsset *assets, + size_t capacity, size_t *count) { + if (count) { + *count = 0; + } + if (!database || !valid_catalog_id(scanset_id) || after_image_id > INT64_MAX || !images || + !assets || !count || capacity == 0 || capacity > LARDON3D_PROJECT_DB_CATALOG_PAGE_MAX) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + Lardon3DProjectDbScanSet scanset; + Lardon3DProjectDbResult exists = lardon3d_project_db_load_scanset(database, scanset_id, &scanset); + if (exists != LARDON3D_PROJECT_DB_OK) { + return exists; + } + (void)pthread_mutex_lock(&database->mutex); + char sql[896]; + (void)snprintf(sql, sizeof(sql), + "%s WHERE i.scanset_id=?1 AND i.image_id>?2 ORDER BY i.image_id LIMIT ?3", + image_select); + sqlite3_stmt *statement = NULL; + Lardon3DProjectDbResult result = prepare(database, sql, &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)scanset_id); + (void)sqlite3_bind_int64(statement, 2, (sqlite3_int64)after_image_id); + (void)sqlite3_bind_int64(statement, 3, (sqlite3_int64)capacity); + int code = SQLITE_DONE; + while (*count < capacity && (code = sqlite3_step(statement)) == SQLITE_ROW) { + memset(&images[*count], 0, sizeof(images[*count])); + memset(&assets[*count], 0, sizeof(assets[*count])); + if (!read_image(statement, &images[*count], &assets[*count])) { + result = LARDON3D_PROJECT_DB_CORRUPT; + break; + } + ++*count; + } + if (result == LARDON3D_PROJECT_DB_OK && *count < capacity && code != SQLITE_DONE) { + result = sqlite_result(database, code, "list images"); + } + (void)sqlite3_finalize(statement); + } + (void)pthread_mutex_unlock(&database->mutex); + return result; +} + +Lardon3DProjectDbResult lardon3d_project_db_count_images(Lardon3DProjectDb *database, + uint64_t scanset_id, uint64_t *count) { + if (count) { + *count = 0; + } + if (!database || !valid_catalog_id(scanset_id) || !count) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + Lardon3DProjectDbScanSet scanset; + Lardon3DProjectDbResult exists = lardon3d_project_db_load_scanset(database, scanset_id, &scanset); + if (exists != LARDON3D_PROJECT_DB_OK) { + return exists; + } + (void)pthread_mutex_lock(&database->mutex); + sqlite3_stmt *statement = NULL; + Lardon3DProjectDbResult result = + prepare(database, "SELECT count(*) FROM images WHERE scanset_id=?1", &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)scanset_id); + int code = sqlite3_step(statement); + sqlite3_int64 value = sqlite3_column_int64(statement, 0); + if (code != SQLITE_ROW || value < 0) { + result = LARDON3D_PROJECT_DB_CORRUPT; + } else { + *count = (uint64_t)value; + } + (void)sqlite3_finalize(statement); + } + (void)pthread_mutex_unlock(&database->mutex); + return result; } Lardon3DProjectDbResult -lardon3d_project_db_test_delete_catalog_identity(Lardon3DProjectDb *database, - uint64_t scanset_id, uint64_t image_id, uint64_t asset_id) -{ - if (!database || !valid_catalog_id(scanset_id) - || !valid_catalog_id(image_id) || !valid_catalog_id(asset_id)) - return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; - (void)pthread_mutex_lock(&database->mutex); - Lardon3DProjectDbResult result = execute(database, "BEGIN IMMEDIATE", - "begin catalog identity deletion test"); - const char *sql[] = { - "DELETE FROM images WHERE image_id=?1", - "DELETE FROM image_assets WHERE asset_id=?1", - "DELETE FROM scansets WHERE scanset_id=?1", - }; - const uint64_t ids[] = {image_id, asset_id, scanset_id}; - for (size_t index = 0; index < 3 && result == LARDON3D_PROJECT_DB_OK; ++index) { - sqlite3_stmt *statement = NULL; - result = prepare(database, sql[index], &statement); - if (result == LARDON3D_PROJECT_DB_OK) { - (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)ids[index]); - result = step_done(database, statement, "delete catalog identity test"); - if (result == LARDON3D_PROJECT_DB_OK - && sqlite3_changes(database->connection) != 1) - result = LARDON3D_PROJECT_DB_NOT_FOUND; - } +lardon3d_project_db_create_artifact(Lardon3DProjectDb *database, + const Lardon3DProjectDbArtifact *artifact) { + if (!database || !artifact || + !bounded_text(artifact->artifact_id, sizeof(artifact->artifact_id), false) || + !bounded_text(artifact->kind, sizeof(artifact->kind), false) || + !bounded_text(artifact->path, sizeof(artifact->path), false) || + artifact->state != LARDON3D_DB_ARTIFACT_STAGED || artifact->size_bytes != 0 || + artifact->created_at < 0 || artifact->updated_at < artifact->created_at || + (artifact->has_producer_task && !valid_task_id(artifact->producer_task_id))) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + (void)pthread_mutex_lock(&database->mutex); + sqlite3_stmt *statement = NULL; + Lardon3DProjectDbResult result = + prepare(database, + "INSERT INTO " + "artifacts(artifact_id,kind,path,state,size_bytes,producer_task_id,created_at," + "updated_at) VALUES(?1,?2,?3,0,0,?4,?5,?6)", + &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_text(statement, 1, artifact->artifact_id, -1, SQLITE_TRANSIENT); + (void)sqlite3_bind_text(statement, 2, artifact->kind, -1, SQLITE_TRANSIENT); + (void)sqlite3_bind_text(statement, 3, artifact->path, -1, SQLITE_TRANSIENT); + if (artifact->has_producer_task) { + (void)sqlite3_bind_int64(statement, 4, (sqlite3_int64)artifact->producer_task_id); + } else { + (void)sqlite3_bind_null(statement, 4); } - if (result == LARDON3D_PROJECT_DB_OK) - result = execute(database, "COMMIT", "commit catalog identity deletion test"); - if (result != LARDON3D_PROJECT_DB_OK) - (void)execute(database, "ROLLBACK", "rollback catalog identity deletion test"); - (void)pthread_mutex_unlock(&database->mutex); + (void)sqlite3_bind_int64(statement, 5, artifact->created_at); + (void)sqlite3_bind_int64(statement, 6, artifact->updated_at); + result = step_done(database, statement, "create artifact"); + } + (void)pthread_mutex_unlock(&database->mutex); + return result; +} + +Lardon3DProjectDbResult lardon3d_project_db_mark_artifact_ready(Lardon3DProjectDb *database, + const char *artifact_id, + int64_t updated_at) { + if (!database || !bounded_text(artifact_id, LARDON3D_PROJECT_DB_ID_CAPACITY, false) || + updated_at < 0) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + char path[LARDON3D_PROJECT_DB_PATH_CAPACITY]; + Lardon3DProjectDbArtifact artifact; + Lardon3DProjectDbResult result = + lardon3d_project_db_load_artifact(database, artifact_id, &artifact); + if (result != LARDON3D_PROJECT_DB_OK) { return result; + } + (void)snprintf(path, sizeof(path), "%s", artifact.path); + struct stat information; + if (stat(path, &information) != 0 || !S_ISREG(information.st_mode) || information.st_size < 0) { + return LARDON3D_PROJECT_DB_IO_ERROR; + } + (void)pthread_mutex_lock(&database->mutex); + sqlite3_stmt *statement = NULL; + result = prepare( + database, + "UPDATE artifacts SET state=1,size_bytes=?1,updated_at=?2 WHERE artifact_id=?3 AND path=?4", + &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_int64(statement, 1, information.st_size); + (void)sqlite3_bind_int64(statement, 2, updated_at); + (void)sqlite3_bind_text(statement, 3, artifact_id, -1, SQLITE_TRANSIENT); + (void)sqlite3_bind_text(statement, 4, path, -1, SQLITE_TRANSIENT); + result = step_done(database, statement, "mark artifact ready"); + if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(database->connection) != 1) { + result = LARDON3D_PROJECT_DB_NOT_FOUND; + } + } + (void)pthread_mutex_unlock(&database->mutex); + return result; +} + +Lardon3DProjectDbResult lardon3d_project_db_load_artifact(Lardon3DProjectDb *database, + const char *artifact_id, + Lardon3DProjectDbArtifact *artifact) { + if (!database || !artifact || + !bounded_text(artifact_id, LARDON3D_PROJECT_DB_ID_CAPACITY, false)) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + (void)pthread_mutex_lock(&database->mutex); + sqlite3_stmt *statement = NULL; + Lardon3DProjectDbResult result = prepare( + database, + "SELECT artifact_id,kind,path,state,size_bytes,producer_task_id,created_at,updated_at FROM " + "artifacts WHERE artifact_id=?1", + &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_text(statement, 1, artifact_id, -1, SQLITE_TRANSIENT); + int code = sqlite3_step(statement); + if (code == SQLITE_DONE) { + result = LARDON3D_PROJECT_DB_NOT_FOUND; + } else if (code != SQLITE_ROW || + !copy_column(statement, 0, artifact->artifact_id, sizeof(artifact->artifact_id)) || + !copy_column(statement, 1, artifact->kind, sizeof(artifact->kind)) || + !copy_column(statement, 2, artifact->path, sizeof(artifact->path))) { + result = LARDON3D_PROJECT_DB_CORRUPT; + } else { + sqlite3_int64 size = sqlite3_column_int64(statement, 4); + sqlite3_int64 producer = sqlite3_column_int64(statement, 5); + artifact->state = (Lardon3DProjectDbArtifactState)sqlite3_column_int(statement, 3); + artifact->size_bytes = size >= 0 ? (uint64_t)size : 0; + artifact->has_producer_task = sqlite3_column_type(statement, 5) != SQLITE_NULL; + artifact->producer_task_id = + artifact->has_producer_task && producer > 0 ? (uint64_t)producer : 0; + artifact->created_at = sqlite3_column_int64(statement, 6); + artifact->updated_at = sqlite3_column_int64(statement, 7); + } + (void)sqlite3_finalize(statement); + } + if (result == LARDON3D_PROJECT_DB_OK && + (artifact->state < LARDON3D_DB_ARTIFACT_STAGED || + artifact->state > LARDON3D_DB_ARTIFACT_READY || + (artifact->has_producer_task && artifact->producer_task_id == 0))) { + result = LARDON3D_PROJECT_DB_CORRUPT; + } + (void)pthread_mutex_unlock(&database->mutex); + return result; +} + +static bool canonical_feature_path(const unsigned char hash[32], const char *path) { + static const char digits[] = "0123456789abcdef"; + char hex[65]; + for (size_t i = 0; i < 32; ++i) { + hex[2 * i] = digits[hash[i] >> 4]; + hex[2 * i + 1] = digits[hash[i] & 15]; + } + hex[64] = '\0'; + char expected[LARDON3D_PROJECT_DB_PATH_CAPACITY]; + int n = snprintf(expected, sizeof(expected), "assets/features/%c%c/%s", hex[0], hex[1], hex); + return n > 0 && (size_t)n < sizeof(expected) && strcmp(expected, path) == 0; +} + +static bool read_feature_set(sqlite3_stmt *s, Lardon3DProjectDbFeatureSet *set) { + memset(set, 0, sizeof(*set)); + sqlite3_int64 id = sqlite3_column_int64(s, 0), image = sqlite3_column_int64(s, 1), + asset = sqlite3_column_int64(s, 2); + sqlite3_int64 version = sqlite3_column_int64(s, 4), count = sqlite3_column_int64(s, 7), + type = sqlite3_column_int64(s, 8), dim = sqlite3_column_int64(s, 9); + const void *fp = sqlite3_column_blob(s, 5), *source = sqlite3_column_blob(s, 6), + *hash = sqlite3_column_blob(s, 12); + sqlite3_int64 size = sqlite3_column_int64(s, 14), durability = sqlite3_column_int64(s, 15), + producer = sqlite3_column_int64(s, 10); + if (id <= 0 || image <= 0 || asset <= 0 || version <= 0 || version > UINT32_MAX || count < 0 || + count > 8192 || type < 1 || type > 2 || dim <= 0 || dim > 4096 || + sqlite3_column_bytes(s, 5) != 32 || sqlite3_column_bytes(s, 6) != 32 || + sqlite3_column_bytes(s, 12) != 32 || !fp || !source || !hash || size < 0 || durability < 0 || + durability > 1 || !copy_column(s, 3, set->extractor_kind, sizeof(set->extractor_kind)) || + !lardon3d_task_kind_is_valid(set->extractor_kind) || + !copy_column(s, 13, set->asset.path, sizeof(set->asset.path))) { + return false; + } + set->feature_set_id = (uint64_t)id; + set->image_id = (uint64_t)image; + set->feature_asset_id = (uint64_t)asset; + set->extractor_version = (uint32_t)version; + memcpy(set->parameter_fingerprint, fp, 32); + memcpy(set->source_image_sha256, source, 32); + set->feature_count = (uint32_t)count; + set->descriptor_type = (uint32_t)type; + set->descriptor_dimension = (uint32_t)dim; + set->has_producer_task = sqlite3_column_type(s, 10) != SQLITE_NULL; + if (set->has_producer_task && producer <= 0) { + return false; + } + set->producer_task_id = set->has_producer_task ? (uint64_t)producer : 0; + set->created_at = sqlite3_column_int64(s, 11); + set->asset.feature_asset_id = (uint64_t)asset; + memcpy(set->asset.sha256, hash, 32); + set->asset.size_bytes = (uint64_t)size; + set->asset.durability = (Lardon3DProjectDbFeatureDurability)durability; + set->asset.created_at = sqlite3_column_int64(s, 16); + return set->created_at >= 0 && set->asset.created_at >= 0 && + canonical_feature_path(set->asset.sha256, set->asset.path); +} + +static const char feature_select[] = + "SELECT " + "f.feature_set_id,f.image_id,f.feature_asset_id,f.extractor_kind,f.extractor_version,f." + "parameter_fingerprint,f.source_image_sha256,f.feature_count,f.descriptor_type,f.descriptor_" + "dimension,f.producer_task_id,f.created_at,a.sha256,a.path,a.size_bytes,a.durability,a.created_" + "at FROM feature_sets f JOIN feature_assets a ON a.feature_asset_id=f.feature_asset_id "; + +static Lardon3DProjectDbResult verify_feature_source_locked(Lardon3DProjectDb *db, + uint64_t image_id, + const unsigned char source[32]) { + sqlite3_stmt *statement = NULL; + Lardon3DProjectDbResult result = prepare(db, + "SELECT a.sha256 FROM images i JOIN image_assets a ON " + "a.asset_id=i.asset_id WHERE i.image_id=?1", + &statement); + if (result != LARDON3D_PROJECT_DB_OK) { + return result; + } + sqlite3_bind_int64(statement, 1, (sqlite3_int64)image_id); + int code = sqlite3_step(statement); + if (code != SQLITE_ROW || sqlite3_column_bytes(statement, 0) != 32 || + memcmp(sqlite3_column_blob(statement, 0), source, 32) != 0) { + result = code == SQLITE_DONE ? LARDON3D_PROJECT_DB_NOT_FOUND : LARDON3D_PROJECT_DB_CONSTRAINT; + } + sqlite3_finalize(statement); + return result; +} + +static Lardon3DProjectDbResult find_or_insert_feature_asset_locked( + Lardon3DProjectDb *db, const unsigned char hash[32], const char *path, uint64_t size, + Lardon3DProjectDbFeatureDurability durability, int64_t created, sqlite3_int64 *asset_id) { + sqlite3_stmt *statement = NULL; + Lardon3DProjectDbResult result = + prepare(db, + "INSERT INTO feature_assets(sha256,path,size_bytes,durability,created_at) " + "VALUES(?1,?2,?3,?4,?5) ON CONFLICT(sha256) DO NOTHING", + &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + sqlite3_bind_blob(statement, 1, hash, 32, SQLITE_TRANSIENT); + sqlite3_bind_text(statement, 2, path, -1, SQLITE_TRANSIENT); + sqlite3_bind_int64(statement, 3, (sqlite3_int64)size); + sqlite3_bind_int(statement, 4, (int)durability); + sqlite3_bind_int64(statement, 5, created); + result = step_done(db, statement, "insert feature asset"); + statement = NULL; + } + if (result == LARDON3D_PROJECT_DB_OK) { + result = prepare(db, + "SELECT feature_asset_id,path,size_bytes,durability " + "FROM feature_assets WHERE sha256=?1", + &statement); + } + if (result == LARDON3D_PROJECT_DB_OK) { + sqlite3_bind_blob(statement, 1, hash, 32, SQLITE_TRANSIENT); + int code = sqlite3_step(statement); + char stored[4096]; + sqlite3_int64 stored_durability = sqlite3_column_int64(statement, 3); + if (code != SQLITE_ROW || !copy_column(statement, 1, stored, sizeof(stored)) || + strcmp(stored, path) != 0 || sqlite3_column_int64(statement, 2) != (sqlite3_int64)size || + stored_durability < 0 || stored_durability > 1) { + result = LARDON3D_PROJECT_DB_CORRUPT; + } else { + *asset_id = sqlite3_column_int64(statement, 0); + } + sqlite3_finalize(statement); + statement = NULL; + } + if (result == LARDON3D_PROJECT_DB_OK && durability == LARDON3D_DB_FEATURE_ASSET_DURABLE) { + result = prepare(db, + "UPDATE feature_assets SET durability=?1 " + "WHERE feature_asset_id=?2 AND durability=?3", + &statement); + } + if (result == LARDON3D_PROJECT_DB_OK && durability == LARDON3D_DB_FEATURE_ASSET_DURABLE) { + sqlite3_bind_int(statement, 1, LARDON3D_DB_FEATURE_ASSET_DURABLE); + sqlite3_bind_int64(statement, 2, *asset_id); + sqlite3_bind_int(statement, 3, LARDON3D_DB_FEATURE_ASSET_PUBLISHED_NOT_DURABLE); + result = step_done(db, statement, "promote feature asset durability"); + } + return result; +} + +static Lardon3DProjectDbResult +insert_feature_set_locked(Lardon3DProjectDb *db, uint64_t image_id, sqlite3_int64 asset_id, + const char *kind, uint32_t version, const unsigned char fp[32], + const unsigned char source[32], uint32_t count, uint32_t type, + uint32_t dim, uint64_t producer, int64_t created) { + sqlite3_stmt *statement = NULL; + Lardon3DProjectDbResult result = + prepare(db, + "INSERT INTO feature_sets(image_id,feature_asset_id,extractor_kind,extractor_version," + "parameter_fingerprint,source_image_sha256,feature_count,descriptor_type," + "descriptor_dimension,producer_task_id,created_at) " + "VALUES(?1,?2,?3,?4,?5,?6,?7,?8,?9,?10,?11) ON CONFLICT(image_id,extractor_kind," + "extractor_version,parameter_fingerprint) DO NOTHING", + &statement); + if (result != LARDON3D_PROJECT_DB_OK) { + return result; + } + sqlite3_bind_int64(statement, 1, (sqlite3_int64)image_id); + sqlite3_bind_int64(statement, 2, asset_id); + sqlite3_bind_text(statement, 3, kind, -1, SQLITE_TRANSIENT); + sqlite3_bind_int64(statement, 4, version); + sqlite3_bind_blob(statement, 5, fp, 32, SQLITE_TRANSIENT); + sqlite3_bind_blob(statement, 6, source, 32, SQLITE_TRANSIENT); + sqlite3_bind_int64(statement, 7, count); + sqlite3_bind_int64(statement, 8, type); + sqlite3_bind_int64(statement, 9, dim); + if (producer) { + sqlite3_bind_int64(statement, 10, (sqlite3_int64)producer); + } else { + sqlite3_bind_null(statement, 10); + } + sqlite3_bind_int64(statement, 11, created); + return step_done(db, statement, "insert feature set"); +} + +Lardon3DProjectDbResult lardon3d_project_db_register_feature_set( + Lardon3DProjectDb *db, uint64_t image_id, const char *kind, uint32_t version, + const unsigned char fp[32], const unsigned char source[32], uint32_t count, uint32_t type, + uint32_t dim, const unsigned char hash[32], const char *path, uint64_t size, + Lardon3DProjectDbFeatureDurability durability, uint64_t producer, int64_t created, + Lardon3DProjectDbFeatureSet *out) { + if (!db || !valid_catalog_id(image_id) || !kind || !lardon3d_task_kind_is_valid(kind) || + version == 0 || !fp || !source || count > 8192 || type < 1 || type > 2 || dim == 0 || + dim > 4096 || !hash || !canonical_feature_path(hash, path) || size > INT64_MAX || + durability < 0 || durability > 1 || (producer && !valid_task_id(producer)) || created < 0 || + !out) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } +#ifdef LARDON3D_PROJECT_DB_TESTING + const char *failure = getenv("LARDON3D_TEST_PROJECT_DB_FAIL_FEATURE_REGISTER"); + if (failure && strcmp(failure, "1") == 0) { + return LARDON3D_PROJECT_DB_BUSY; + } +#endif + (void)pthread_mutex_lock(&db->mutex); + Lardon3DProjectDbResult result = execute(db, "BEGIN IMMEDIATE", "begin feature register"); + if (result == LARDON3D_PROJECT_DB_OK) { + result = verify_feature_source_locked(db, image_id, source); + } + sqlite3_int64 asset_id = 0; + if (result == LARDON3D_PROJECT_DB_OK) { + result = + find_or_insert_feature_asset_locked(db, hash, path, size, durability, created, &asset_id); + } + if (result == LARDON3D_PROJECT_DB_OK) { + result = insert_feature_set_locked(db, image_id, asset_id, kind, version, fp, source, count, + type, dim, producer, created); + } + if (result == LARDON3D_PROJECT_DB_OK) { + result = execute(db, "COMMIT", "commit feature register"); + } + if (result != LARDON3D_PROJECT_DB_OK) { + (void)execute(db, "ROLLBACK", "rollback feature register"); + } + (void)pthread_mutex_unlock(&db->mutex); + return result == LARDON3D_PROJECT_DB_OK + ? lardon3d_project_db_find_feature_set(db, image_id, kind, version, fp, out) + : result; +} + +Lardon3DProjectDbResult lardon3d_project_db_find_feature_set(Lardon3DProjectDb *db, + uint64_t image_id, const char *kind, + uint32_t version, + const unsigned char fp[32], + Lardon3DProjectDbFeatureSet *out) { + if (!db || !valid_catalog_id(image_id) || !lardon3d_task_kind_is_valid(kind) || version == 0 || + !fp || !out) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + (void)pthread_mutex_lock(&db->mutex); + char sql[1024]; + snprintf(sql, sizeof(sql), + "%s WHERE f.image_id=?1 AND f.extractor_kind=?2 AND f.extractor_version=?3 AND " + "f.parameter_fingerprint=?4", + feature_select); + sqlite3_stmt *s = NULL; + Lardon3DProjectDbResult result = prepare(db, sql, &s); + if (result == LARDON3D_PROJECT_DB_OK) { + sqlite3_bind_int64(s, 1, (sqlite3_int64)image_id); + sqlite3_bind_text(s, 2, kind, -1, SQLITE_TRANSIENT); + sqlite3_bind_int64(s, 3, version); + sqlite3_bind_blob(s, 4, fp, 32, SQLITE_TRANSIENT); + int code = sqlite3_step(s); + if (code == SQLITE_DONE) { + result = LARDON3D_PROJECT_DB_NOT_FOUND; + } else if (code != SQLITE_ROW || !read_feature_set(s, out)) { + result = LARDON3D_PROJECT_DB_CORRUPT; + } + sqlite3_finalize(s); + } + (void)pthread_mutex_unlock(&db->mutex); + return result; +} + +Lardon3DProjectDbResult lardon3d_project_db_load_feature_set(Lardon3DProjectDb *db, uint64_t id, + Lardon3DProjectDbFeatureSet *out) { + if (!db || !valid_catalog_id(id) || !out) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + (void)pthread_mutex_lock(&db->mutex); + char sql[1024]; + snprintf(sql, sizeof(sql), "%s WHERE f.feature_set_id=?1", feature_select); + sqlite3_stmt *s = NULL; + Lardon3DProjectDbResult result = prepare(db, sql, &s); + if (result == LARDON3D_PROJECT_DB_OK) { + sqlite3_bind_int64(s, 1, (sqlite3_int64)id); + int code = sqlite3_step(s); + if (code == SQLITE_DONE) { + result = LARDON3D_PROJECT_DB_NOT_FOUND; + } else if (code != SQLITE_ROW || !read_feature_set(s, out)) { + result = LARDON3D_PROJECT_DB_CORRUPT; + } + sqlite3_finalize(s); + } + (void)pthread_mutex_unlock(&db->mutex); + return result; +} + +Lardon3DProjectDbResult lardon3d_project_db_list_feature_sets(Lardon3DProjectDb *db, uint64_t after, + Lardon3DProjectDbFeatureSet *sets, + size_t capacity, size_t *count) { + if (count) { + *count = 0; + } + if (!db || after > INT64_MAX || !sets || !count || capacity == 0 || capacity > 256) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + (void)pthread_mutex_lock(&db->mutex); + char sql[1024]; + snprintf(sql, sizeof(sql), "%s WHERE f.feature_set_id>?1 ORDER BY f.feature_set_id LIMIT ?2", + feature_select); + sqlite3_stmt *s = NULL; + Lardon3DProjectDbResult result = prepare(db, sql, &s); + if (result == LARDON3D_PROJECT_DB_OK) { + sqlite3_bind_int64(s, 1, (sqlite3_int64)after); + sqlite3_bind_int64(s, 2, (sqlite3_int64)capacity); + int code = SQLITE_DONE; + while (*count < capacity && (code = sqlite3_step(s)) == SQLITE_ROW) { + if (!read_feature_set(s, &sets[*count])) { + result = LARDON3D_PROJECT_DB_CORRUPT; + break; + } + ++*count; + } + if (result == LARDON3D_PROJECT_DB_OK && *count < capacity && code != SQLITE_DONE) { + result = sqlite_result(db, code, "list feature sets"); + } + sqlite3_finalize(s); + } + (void)pthread_mutex_unlock(&db->mutex); + return result; +} + +Lardon3DProjectDbResult +lardon3d_project_db_load_feature_extract_task(Lardon3DProjectDb *db, uint64_t task_id, + Lardon3DProjectDbFeatureExtractTask *p) { + if (!db || !valid_task_id(task_id) || !p) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + memset(p, 0, sizeof(*p)); + (void)pthread_mutex_lock(&db->mutex); + sqlite3_stmt *s = NULL; + Lardon3DProjectDbResult result = + prepare(db, + "SELECT " + "image_id,extractor_kind,extractor_version,max_features,pyramid_levels,fast_" + "threshold,parameter_fingerprint FROM feature_extract_tasks WHERE task_id=?1", + &s); + if (result == LARDON3D_PROJECT_DB_OK) { + sqlite3_bind_int64(s, 1, (sqlite3_int64)task_id); + int code = sqlite3_step(s); + sqlite3_int64 image = sqlite3_column_int64(s, 0), version = sqlite3_column_int64(s, 2), + max = sqlite3_column_int64(s, 3), levels = sqlite3_column_int64(s, 4), + threshold = sqlite3_column_int64(s, 5); + const void *fp = sqlite3_column_blob(s, 6); + if (code == SQLITE_DONE) { + result = LARDON3D_PROJECT_DB_NOT_FOUND; + } else if (code != SQLITE_ROW || image <= 0 || version <= 0 || version > UINT32_MAX || + max <= 0 || max > 8192 || levels <= 0 || levels > 16 || threshold <= 0 || + threshold > 255 || + !copy_column(s, 1, p->extractor_kind, sizeof(p->extractor_kind)) || + !lardon3d_task_kind_is_valid(p->extractor_kind) || + sqlite3_column_bytes(s, 6) != 32 || !fp) { + result = LARDON3D_PROJECT_DB_CORRUPT; + } else { + p->task_id = task_id; + p->image_id = (uint64_t)image; + p->extractor_version = (uint32_t)version; + p->max_features = (uint32_t)max; + p->pyramid_levels = (uint32_t)levels; + p->fast_threshold = (uint32_t)threshold; + memcpy(p->parameter_fingerprint, fp, 32); + } + sqlite3_finalize(s); + } + (void)pthread_mutex_unlock(&db->mutex); + return result; +} + +#ifdef LARDON3D_PROJECT_DB_TESTING +Lardon3DProjectDbResult lardon3d_project_db_test_delete_feature_identity( + Lardon3DProjectDb *database, uint64_t feature_set_id, uint64_t feature_asset_id) { + if (!database || !valid_catalog_id(feature_set_id) || !valid_catalog_id(feature_asset_id)) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + (void)pthread_mutex_lock(&database->mutex); + Lardon3DProjectDbResult result = + execute(database, "BEGIN IMMEDIATE", "begin delete feature identity"); + sqlite3_stmt *s = NULL; + const char *sql[2] = {"DELETE FROM feature_sets WHERE feature_set_id=?1", + "DELETE FROM feature_assets WHERE feature_asset_id=?1"}; + uint64_t ids[2] = {feature_set_id, feature_asset_id}; + for (size_t i = 0; i < 2 && result == LARDON3D_PROJECT_DB_OK; ++i) { + result = prepare(database, sql[i], &s); + if (result == LARDON3D_PROJECT_DB_OK) { + sqlite3_bind_int64(s, 1, (sqlite3_int64)ids[i]); + result = step_done(database, s, "delete feature identity"); + s = NULL; + if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(database->connection) != 1) { + result = LARDON3D_PROJECT_DB_NOT_FOUND; + } + } + } + if (result == LARDON3D_PROJECT_DB_OK) { + result = execute(database, "COMMIT", "commit delete feature identity"); + } + if (result != LARDON3D_PROJECT_DB_OK) { + (void)execute(database, "ROLLBACK", "rollback delete feature identity"); + } + (void)pthread_mutex_unlock(&database->mutex); + return result; +} + +Lardon3DProjectDbResult lardon3d_project_db_test_orphan_checkpoint(Lardon3DProjectDb *database) { + if (!database) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + (void)pthread_mutex_lock(&database->mutex); + sqlite3_stmt *statement = NULL; + Lardon3DProjectDbResult result = + prepare(database, + "INSERT INTO checkpoints(task_id,path,format_version,durability,updated_at) " + "VALUES(9223372036854775807,'orphan',1,0,0)", + &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + result = step_done(database, statement, "orphan checkpoint"); + } + (void)pthread_mutex_unlock(&database->mutex); + return result; +} + +Lardon3DProjectDbResult +lardon3d_project_db_test_delete_catalog_identity(Lardon3DProjectDb *database, uint64_t scanset_id, + uint64_t image_id, uint64_t asset_id) { + if (!database || !valid_catalog_id(scanset_id) || !valid_catalog_id(image_id) || + !valid_catalog_id(asset_id)) { + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + } + (void)pthread_mutex_lock(&database->mutex); + Lardon3DProjectDbResult result = + execute(database, "BEGIN IMMEDIATE", "begin catalog identity deletion test"); + const char *sql[] = { + "DELETE FROM images WHERE image_id=?1", + "DELETE FROM image_assets WHERE asset_id=?1", + "DELETE FROM scansets WHERE scanset_id=?1", + }; + const uint64_t ids[] = {image_id, asset_id, scanset_id}; + for (size_t index = 0; index < 3 && result == LARDON3D_PROJECT_DB_OK; ++index) { + sqlite3_stmt *statement = NULL; + result = prepare(database, sql[index], &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + (void)sqlite3_bind_int64(statement, 1, (sqlite3_int64)ids[index]); + result = step_done(database, statement, "delete catalog identity test"); + if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(database->connection) != 1) { + result = LARDON3D_PROJECT_DB_NOT_FOUND; + } + } + } + if (result == LARDON3D_PROJECT_DB_OK) { + result = execute(database, "COMMIT", "commit catalog identity deletion test"); + } + if (result != LARDON3D_PROJECT_DB_OK) { + (void)execute(database, "ROLLBACK", "rollback catalog identity deletion test"); + } + (void)pthread_mutex_unlock(&database->mutex); + return result; } #endif diff --git a/src/task_kinds.c b/src/task_kinds.c index 63e8d5a..d08e2a8 100644 --- a/src/task_kinds.c +++ b/src/task_kinds.c @@ -1,17 +1,22 @@ +#include #include #include -const Lardon3DTaskKindRegistry * -lardon3d_task_kind_registry_production(void) -{ - static const Lardon3DTaskKindDescriptor descriptors[] = {{ - .kind = LARDON3D_IMAGE_IMPORT_TASK_KIND, - .kind_version = LARDON3D_IMAGE_IMPORT_TASK_KIND_VERSION, - .reconstruct = lardon3d_image_import_reconstruct, - }}; - static const Lardon3DTaskKindRegistry registry = { - .descriptors = descriptors, - .count = sizeof(descriptors) / sizeof(descriptors[0]), - }; - return ®istry; +const Lardon3DTaskKindRegistry *lardon3d_task_kind_registry_production(void) { + static const Lardon3DTaskKindDescriptor descriptors[] = { + { + .kind = LARDON3D_IMAGE_IMPORT_TASK_KIND, + .kind_version = LARDON3D_IMAGE_IMPORT_TASK_KIND_VERSION, + .reconstruct = lardon3d_image_import_reconstruct, + }, + { + .kind = LARDON3D_FEATURE_EXTRACT_TASK_KIND, + .kind_version = LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION, + .reconstruct = lardon3d_feature_extract_reconstruct, + }}; + static const Lardon3DTaskKindRegistry registry = { + .descriptors = descriptors, + .count = sizeof(descriptors) / sizeof(descriptors[0]), + }; + return ®istry; } diff --git a/tests/test_feature_store.c b/tests/test_feature_store.c new file mode 100644 index 0000000..e5a28c7 --- /dev/null +++ b/tests/test_feature_store.c @@ -0,0 +1,432 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +Lardon3DProjectDbResult lardon3d_project_db_test_delete_feature_identity(Lardon3DProjectDb *, + uint64_t, uint64_t); + +typedef struct { + Lardon3DAppState *state; + uint64_t image_id; + const Lardon3DFeatureExtractorParameters *parameters; + const Lardon3DExtractedFeatures *features; + Lardon3DFeatureStoreResult result; + Lardon3DProjectDbFeatureSet feature_set; +} PublishThreadContext; + +static void *publish_thread(void *userdata) { + PublishThreadContext *context = userdata; + context->result = + lardon3d_feature_store_publish(context->state, context->image_id, 0, context->parameters, + context->features, &context->feature_set); + return NULL; +} + +#define CHECK(x) \ + do { \ + if (!(x)) { \ + fprintf(stderr, "Échec ligne %d: %s\n", __LINE__, #x); \ + return false; \ + } \ + } while (0) + +static bool join_path(char out[PATH_MAX], const char *a, const char *b) { + int n = snprintf(out, PATH_MAX, "%s/%s", a, b); + return n > 0 && (size_t)n < PATH_MAX; +} +static bool remove_tree(const char *path) { + struct stat st; + if (lstat(path, &st) != 0) { + return errno == ENOENT; + } + if (!S_ISDIR(st.st_mode)) { + return unlink(path) == 0; + } + DIR *d = opendir(path); + if (!d) { + return false; + } + bool ok = true; + for (struct dirent *e = readdir(d); e; e = readdir(d)) { + if (!strcmp(e->d_name, ".") || !strcmp(e->d_name, "..")) { + continue; + } + char child[PATH_MAX]; + if (!join_path(child, path, e->d_name) || !remove_tree(child)) { + ok = false; + } + } + if (closedir(d) != 0 || rmdir(path) != 0) { + ok = false; + } + return ok; +} +static size_t count_files(const char *path) { + DIR *d = opendir(path); + if (!d) { + return 0; + } + size_t count = 0; + for (struct dirent *e = readdir(d); e; e = readdir(d)) { + if (!strcmp(e->d_name, ".") || !strcmp(e->d_name, "..")) { + continue; + } + char child[PATH_MAX]; + struct stat s; + if (join_path(child, path, e->d_name) && lstat(child, &s) == 0) { + if (S_ISDIR(s.st_mode)) { + count += count_files(child); + } else if (S_ISREG(s.st_mode)) { + ++count; + } + } + } + closedir(d); + return count; +} + +static bool write_pgm(const char *path, bool uniform) { + FILE *f = fopen(path, "wb"); + if (!f) { + return false; + } + if (fprintf(f, "P5\n128 128\n255\n") <= 0) { + fclose(f); + return false; + } + for (unsigned int y = 0; y < 128; ++y) { + for (unsigned int x = 0; x < 128; ++x) { + unsigned char v = uniform ? 128U : (unsigned char)((((x / 8U) ^ (y / 8U)) & 1U) ? 240U : 15U); + if (fwrite(&v, 1, 1, f) != 1) { + fclose(f); + return false; + } + } + } + return fclose(f) == 0; +} + +static uint64_t read_u64_le(const unsigned char *bytes) { + uint64_t value = 0; + for (unsigned int index = 0; index < 8; ++index) { + value |= (uint64_t)bytes[index] << (8U * index); + } + return value; +} + +static bool restore_file(const char *path, const unsigned char *bytes, size_t size) { + int descriptor = open(path, O_WRONLY | O_TRUNC | O_CLOEXEC); + if (descriptor < 0) { + return false; + } + size_t written = 0; + while (written < size) { + ssize_t amount = write(descriptor, bytes + written, size - written); + if (amount <= 0) { + close(descriptor); + return false; + } + written += (size_t)amount; + } + return close(descriptor) == 0; +} + +static bool expect_reader_result(const char *root, const Lardon3DProjectDbFeatureSet *feature_set, + Lardon3DFeatureStoreResult expected) { + Lardon3DFeatureReader *reader = NULL; + Lardon3DFeatureFileMetadata metadata; + Lardon3DFeatureStoreResult result = + lardon3d_feature_reader_open(root, feature_set, &reader, &metadata); + lardon3d_feature_reader_close(reader); + return result == expected; +} + +static bool run_test(void) { + char root[] = "/tmp/lardon3d-feature-store-XXXXXX"; + CHECK(mkdtemp(root)); + char dbpath[PATH_MAX], structured[PATH_MAX], uniform[PATH_MAX]; + CHECK(join_path(dbpath, root, "project.db") && join_path(structured, root, "structured.pgm") && + join_path(uniform, root, "uniform.pgm")); + CHECK(write_pgm(structured, false) && write_pgm(uniform, true)); + Lardon3DProjectDb *db = NULL; + char error[256]; + CHECK(lardon3d_project_db_open(dbpath, &db, error) == LARDON3D_PROJECT_DB_OK); + Lardon3DAppState state; + lardon3d_app_state_init(&state); + state.project_loaded = true; + state.project_db = db; + CHECK(snprintf(state.project_path, sizeof(state.project_path), "%s", root) > 0); + Lardon3DProjectDbScanSet a, b; + CHECK(lardon3d_image_catalog_create_scanset(&state, "A", &a) && + lardon3d_image_catalog_create_scanset(&state, "B", &b)); + Lardon3DProjectDbImage ia, ib, iu; + Lardon3DProjectDbImageAsset aa, ab, au; + CHECK(lardon3d_image_catalog_import_file(&state, a.scanset_id, structured, 0, &ia, &aa) == + LARDON3D_IMAGE_CATALOG_IMPORTED); + CHECK(lardon3d_image_catalog_import_file(&state, b.scanset_id, structured, 0, &ib, &ab) == + LARDON3D_IMAGE_CATALOG_IMPORTED && + aa.asset_id == ab.asset_id); + CHECK(lardon3d_image_catalog_import_file(&state, a.scanset_id, uniform, 0, &iu, &au) == + LARDON3D_IMAGE_CATALOG_IMPORTED); + Lardon3DFeatureExtractorParameters p = { + .max_features = 512, .pyramid_levels = 4, .fast_threshold = 10}; + CHECK(lardon3d_feature_extractor_parameters_valid(&p)); + CHECK(!lardon3d_feature_extractor_parameters_valid( + &(Lardon3DFeatureExtractorParameters){0, 4, 10})); + CHECK(!lardon3d_feature_extractor_parameters_valid( + &(Lardon3DFeatureExtractorParameters){8193, 4, 10})); + CHECK(!lardon3d_feature_extractor_parameters_valid( + &(Lardon3DFeatureExtractorParameters){512, 0, 10})); + CHECK(!lardon3d_feature_extractor_parameters_valid( + &(Lardon3DFeatureExtractorParameters){512, 17, 10})); + CHECK(!lardon3d_feature_extractor_parameters_valid( + &(Lardon3DFeatureExtractorParameters){512, 4, 0})); + CHECK(!lardon3d_feature_extractor_parameters_valid( + &(Lardon3DFeatureExtractorParameters){512, 4, 256})); + unsigned char fp[32], fp2[32]; + lardon3d_feature_extractor_parameter_fingerprint(&p, fp); + lardon3d_feature_extractor_parameter_fingerprint(&p, fp2); + CHECK(memcmp(fp, fp2, 32) == 0); + Lardon3DFeatureExtractorParameters p2 = p; + p2.max_features = 256; + lardon3d_feature_extractor_parameter_fingerprint(&p2, fp2); + CHECK(memcmp(fp, fp2, 32) != 0); + Lardon3DExtractedFeatures f; + CHECK(lardon3d_feature_extract_orb(structured, &p, &f) == LARDON3D_FEATURE_EXTRACT_OK && + f.feature_count > 0 && f.feature_count <= 512); + Lardon3DProjectDbFeatureSet sa, same, sb; + CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &p, &f, &sa) == + LARDON3D_FEATURE_STORE_OK); + CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &p, &f, &same) == + LARDON3D_FEATURE_STORE_ALREADY_PRESENT && + same.feature_set_id == sa.feature_set_id); + CHECK(lardon3d_feature_store_publish(&state, ib.image_id, 0, &p, &f, &sb) == + LARDON3D_FEATURE_STORE_OK && + sb.feature_set_id != sa.feature_set_id && sb.feature_asset_id == sa.feature_asset_id); + Lardon3DFeatureReader *reader = NULL; + Lardon3DFeatureFileMetadata metadata; + CHECK(lardon3d_feature_reader_open(root, &sa, &reader, &metadata) == LARDON3D_FEATURE_STORE_OK && + metadata.feature_count == f.feature_count); + size_t chunk = metadata.feature_count < 8 ? metadata.feature_count : 8; + Lardon3DFeatureKeypoint points[8]; + unsigned char descriptors[8 * 32]; + CHECK(lardon3d_feature_reader_keypoints(reader, 0, points, chunk) == LARDON3D_FEATURE_STORE_OK); + CHECK(lardon3d_feature_reader_descriptors(reader, 0, descriptors, chunk, sizeof(descriptors)) == + LARDON3D_FEATURE_STORE_OK); + for (size_t i = 0; i < chunk; ++i) { + CHECK(isfinite(points[i].x) && memcmp(descriptors + i * 32, f.descriptors + i * 32, 32) == 0); + } + CHECK(lardon3d_feature_reader_keypoints(reader, metadata.feature_count, points, 1) == + LARDON3D_FEATURE_STORE_INVALID_ARGUMENT); + CHECK(lardon3d_feature_reader_descriptors(reader, UINT32_MAX, descriptors, 1, + sizeof(descriptors)) == + LARDON3D_FEATURE_STORE_INVALID_ARGUMENT); + CHECK(lardon3d_feature_reader_keypoints(reader, 0, points, 257) == + LARDON3D_FEATURE_STORE_INVALID_ARGUMENT); + CHECK(lardon3d_feature_reader_descriptors(reader, 0, descriptors, 257, sizeof(descriptors)) == + LARDON3D_FEATURE_STORE_INVALID_ARGUMENT); + lardon3d_feature_reader_close(reader); + + const char *invalid_paths[] = { + "/assets/features/00/hash", + "../assets/features/00/hash", + "assets/features/AA/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "assets/features/aa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "assets/features/0/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "assets/features/aa/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + }; + for (size_t index = 0; index < sizeof(invalid_paths) / sizeof(invalid_paths[0]); ++index) { + Lardon3DProjectDbFeatureSet invalid_path_set = sa; + CHECK(snprintf(invalid_path_set.asset.path, sizeof(invalid_path_set.asset.path), "%s", + invalid_paths[index]) > 0); + CHECK(lardon3d_feature_reader_open(root, &invalid_path_set, &reader, &metadata) == + LARDON3D_FEATURE_STORE_INVALID_ARGUMENT); + } + + Lardon3DExtractedFeatures invalid_features = f; + Lardon3DFeatureKeypoint invalid_point = f.keypoints[0]; + invalid_point.x = NAN; + invalid_features.keypoints = &invalid_point; + invalid_features.feature_count = 1; + CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &p2, &invalid_features, &same) == + LARDON3D_FEATURE_STORE_INVALID_ARGUMENT); + + Lardon3DFeatureExtractorParameters concurrent_parameters = {508, 4, 10}; + PublishThreadContext concurrent[2] = { + {.state = &state, + .image_id = ia.image_id, + .parameters = &concurrent_parameters, + .features = &f}, + {.state = &state, + .image_id = ia.image_id, + .parameters = &concurrent_parameters, + .features = &f}, + }; + pthread_t publish_threads[2]; + CHECK(pthread_create(&publish_threads[0], NULL, publish_thread, &concurrent[0]) == 0 && + pthread_create(&publish_threads[1], NULL, publish_thread, &concurrent[1]) == 0); + CHECK(pthread_join(publish_threads[0], NULL) == 0 && pthread_join(publish_threads[1], NULL) == 0); + CHECK((concurrent[0].result == LARDON3D_FEATURE_STORE_OK || + concurrent[0].result == LARDON3D_FEATURE_STORE_ALREADY_PRESENT) && + (concurrent[1].result == LARDON3D_FEATURE_STORE_OK || + concurrent[1].result == LARDON3D_FEATURE_STORE_ALREADY_PRESENT) && + concurrent[0].feature_set.feature_set_id == concurrent[1].feature_set.feature_set_id); + Lardon3DExtractedFeatures uf; + CHECK(lardon3d_feature_extract_orb(uniform, &p, &uf) == LARDON3D_FEATURE_EXTRACT_OK && + uf.feature_count == 0); + Lardon3DProjectDbFeatureSet us; + CHECK(lardon3d_feature_store_publish(&state, iu.image_id, 0, &p, &uf, &us) == + LARDON3D_FEATURE_STORE_OK); + CHECK(lardon3d_feature_reader_open(root, &us, &reader, &metadata) == LARDON3D_FEATURE_STORE_OK && + metadata.feature_count == 0); + lardon3d_feature_reader_close(reader); + Lardon3DFeatureExtractorParameters large_parameters = {8192, 4, 10}; + Lardon3DExtractedFeatures large = { + .image_width = 128, .image_height = 128, .feature_count = 8192}; + large.keypoints = calloc(large.feature_count, sizeof(*large.keypoints)); + large.descriptors = calloc(large.feature_count, 32); + CHECK(large.keypoints && large.descriptors); + for (uint32_t i = 0; i < large.feature_count; ++i) { + large.keypoints[i] = + (Lardon3DFeatureKeypoint){(float)(i % 128), (float)((i / 128) % 64), 1, 0, 1, 0}; + memset(large.descriptors + (size_t)i * 32, (int)(i & 255), 32); + } + Lardon3DProjectDbFeatureSet large_set; + CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &large_parameters, &large, + &large_set) == LARDON3D_FEATURE_STORE_OK); + CHECK(lardon3d_feature_reader_open(root, &large_set, &reader, &metadata) == + LARDON3D_FEATURE_STORE_OK && + metadata.feature_count == 8192); + Lardon3DFeatureKeypoint last; + unsigned char last_descriptor[32]; + CHECK(lardon3d_feature_reader_keypoints(reader, 8191, &last, 1) == LARDON3D_FEATURE_STORE_OK && + lardon3d_feature_reader_descriptors(reader, 8191, last_descriptor, 1, + sizeof(last_descriptor)) == LARDON3D_FEATURE_STORE_OK && + last_descriptor[0] == 255); + lardon3d_feature_reader_close(reader); + lardon3d_extracted_features_destroy(&large); + char features_dir[PATH_MAX]; + CHECK(join_path(features_dir, root, "assets/features")); + size_t before = count_files(features_dir); + Lardon3DFeatureExtractorParameters failed_parameters = {511, 4, 10}; + CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_FEATURE_REGISTER", "1", 1) == 0); + Lardon3DProjectDbFeatureSet failed_set; + CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &failed_parameters, &f, + &failed_set) == LARDON3D_FEATURE_STORE_DB_BUSY); + CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_FEATURE_REGISTER") == 0 && + count_files(features_dir) == before + 1); + CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &failed_parameters, &f, + &failed_set) == LARDON3D_FEATURE_STORE_OK); + Lardon3DFeatureExtractorParameters nondurable_parameters = {510, 4, 10}; + CHECK(setenv("LARDON3D_TEST_FEATURE_FAIL_DIRECTORY_SYNC", "1", 1) == 0); + Lardon3DProjectDbFeatureSet nondurable; + CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &nondurable_parameters, &f, + &nondurable) == + LARDON3D_FEATURE_STORE_PUBLISHED_NOT_DURABLE && + nondurable.asset.durability == LARDON3D_DB_FEATURE_ASSET_PUBLISHED_NOT_DURABLE); + CHECK(unsetenv("LARDON3D_TEST_FEATURE_FAIL_DIRECTORY_SYNC") == 0); + Lardon3DProjectDbFeatureSet promoted; + CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &nondurable_parameters, &f, + &promoted) == LARDON3D_FEATURE_STORE_ALREADY_PRESENT && + promoted.feature_set_id == nondurable.feature_set_id && + promoted.asset.durability == LARDON3D_DB_FEATURE_ASSET_DURABLE); + uint64_t deleted_set = nondurable.feature_set_id, deleted_asset = nondurable.feature_asset_id; + CHECK(lardon3d_project_db_test_delete_feature_identity(db, deleted_set, deleted_asset) == + LARDON3D_PROJECT_DB_OK); + Lardon3DFeatureExtractorParameters replacement_parameters = {509, 4, 10}; + Lardon3DProjectDbFeatureSet replacement; + CHECK(lardon3d_feature_store_publish(&state, ia.image_id, 0, &replacement_parameters, &f, + &replacement) == LARDON3D_FEATURE_STORE_OK && + replacement.feature_set_id > deleted_set && replacement.feature_asset_id > deleted_asset); + Lardon3DProjectDbFeatureSet page[8]; + size_t count = 0; + CHECK(lardon3d_project_db_list_feature_sets(db, 0, page, 1, &count) == LARDON3D_PROJECT_DB_OK && + count == 1); + CHECK(lardon3d_project_db_list_feature_sets(db, page[0].feature_set_id, page, 8, &count) == + LARDON3D_PROJECT_DB_OK && + count == 6); + for (uint32_t index = 0; index < 2000; ++index) { + unsigned char metadata_fingerprint[32] = {0}; + metadata_fingerprint[0] = (unsigned char)index; + metadata_fingerprint[1] = (unsigned char)(index >> 8); + metadata_fingerprint[2] = 0xa5; + Lardon3DProjectDbFeatureSet metadata_set; + CHECK(lardon3d_project_db_register_feature_set( + db, ia.image_id, "metadata.fixture", 1, metadata_fingerprint, aa.sha256, + sa.feature_count, sa.descriptor_type, sa.descriptor_dimension, sa.asset.sha256, + sa.asset.path, sa.asset.size_bytes, sa.asset.durability, 0, index, + &metadata_set) == LARDON3D_PROJECT_DB_OK); + } + Lardon3DProjectDbFeatureSet *metadata_page = calloc(256, sizeof(*metadata_page)); + CHECK(metadata_page); + const size_t page_sizes[] = {1, 8, 64, 256}; + for (size_t index = 0; index < sizeof(page_sizes) / sizeof(page_sizes[0]); ++index) { + CHECK(lardon3d_project_db_list_feature_sets(db, 0, metadata_page, page_sizes[index], &count) == + LARDON3D_PROJECT_DB_OK && + count == page_sizes[index]); + } + CHECK(lardon3d_project_db_list_feature_sets(db, 0, metadata_page, 257, &count) == + LARDON3D_PROJECT_DB_INVALID_ARGUMENT); + free(metadata_page); + char asset_path[PATH_MAX]; + CHECK(join_path(asset_path, root, sa.asset.path)); + int fd = open(asset_path, O_RDONLY | O_CLOEXEC); + CHECK(fd >= 0); + struct stat asset_info; + CHECK(fstat(fd, &asset_info) == 0 && asset_info.st_size >= 160); + size_t original_size = (size_t)asset_info.st_size; + unsigned char *original = malloc(original_size); + CHECK(original && read(fd, original, original_size) == (ssize_t)original_size && close(fd) == 0); + + uint64_t descriptor_offset = read_u64_le(original + 48); + const off_t truncated_sizes[] = { + 100, + (off_t)descriptor_offset - 1, + (off_t)descriptor_offset, + (off_t)original_size - 1, + }; + for (size_t index = 0; index < sizeof(truncated_sizes) / sizeof(truncated_sizes[0]); ++index) { + CHECK(truncate(asset_path, truncated_sizes[index]) == 0 && + expect_reader_result(root, &sa, LARDON3D_FEATURE_STORE_CORRUPT) && + restore_file(asset_path, original, original_size)); + } + + const size_t corrupt_offsets[] = {12, 16, 20, 24, 28, 40, 48, 56, 148, 144, 128}; + for (size_t index = 0; index < sizeof(corrupt_offsets) / sizeof(corrupt_offsets[0]); ++index) { + unsigned char mutated = (unsigned char)(original[corrupt_offsets[index]] ^ 0x5aU); + fd = open(asset_path, O_WRONLY | O_CLOEXEC); + CHECK(fd >= 0 && pwrite(fd, &mutated, 1, (off_t)corrupt_offsets[index]) == 1 && + close(fd) == 0 && expect_reader_result(root, &sa, LARDON3D_FEATURE_STORE_CORRUPT) && + restore_file(asset_path, original, original_size)); + } + + unsigned char future_version[4] = {2, 0, 0, 0}; + fd = open(asset_path, O_WRONLY | O_CLOEXEC); + CHECK(fd >= 0 && pwrite(fd, future_version, sizeof(future_version), 8) == 4 && close(fd) == 0 && + expect_reader_result(root, &sa, LARDON3D_FEATURE_STORE_UNSUPPORTED_VERSION) && + restore_file(asset_path, original, original_size)); + + fd = open(asset_path, O_WRONLY | O_TRUNC | O_CLOEXEC); + CHECK(fd >= 0 && write(fd, "bad", 3) == 3 && close(fd) == 0 && + expect_reader_result(root, &sa, LARDON3D_FEATURE_STORE_CORRUPT)); + free(original); + lardon3d_extracted_features_destroy(&f); + lardon3d_extracted_features_destroy(&uf); + lardon3d_project_db_close(db); + CHECK(remove_tree(root)); + return true; +} + +int main(void) { return run_test() ? 0 : 1; } diff --git a/tests/test_feature_task.c b/tests/test_feature_task.c new file mode 100644 index 0000000..1050b6f --- /dev/null +++ b/tests/test_feature_task.c @@ -0,0 +1,188 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +#define CHECK(x) \ + do { \ + if (!(x)) { \ + fprintf(stderr, "Échec ligne %d: %s\n", __LINE__, #x); \ + return false; \ + } \ + } while (0) +static bool join_path(char out[PATH_MAX], const char *a, const char *b) { + int n = snprintf(out, PATH_MAX, "%s/%s", a, b); + return n > 0 && (size_t)n < PATH_MAX; +} +static bool remove_tree(const char *p) { + struct stat s; + if (lstat(p, &s) != 0) { + return errno == ENOENT; + } + if (!S_ISDIR(s.st_mode)) { + return unlink(p) == 0; + } + DIR *d = opendir(p); + if (!d) { + return false; + } + bool ok = true; + for (struct dirent *e = readdir(d); e; e = readdir(d)) { + if (!strcmp(e->d_name, ".") || !strcmp(e->d_name, "..")) { + continue; + } + char c[PATH_MAX]; + if (!join_path(c, p, e->d_name) || !remove_tree(c)) { + ok = false; + } + } + if (closedir(d) || rmdir(p)) { + ok = false; + } + return ok; +} +static bool write_pgm(const char *p) { + FILE *f = fopen(p, "wb"); + if (!f) { + return false; + } + fprintf(f, "P5\n192 192\n255\n"); + for (unsigned y = 0; y < 192; y++) { + for (unsigned x = 0; x < 192; x++) { + unsigned char v = (unsigned char)((((x / 6) ^ (y / 6)) & 1) ? 245 : 10); + if (fwrite(&v, 1, 1, f) != 1) { + fclose(f); + return false; + } + } + } + return fclose(f) == 0; +} +static bool runtime(Lardon3DAppState *s) { + s->hardware_profile = (Lardon3DHardwareProfile){.logical_cpu_count = 64, + .page_size_bytes = 4096, + .memory_total_bytes = UINT64_MAX, + .cpu_architecture = "test"}; + Lardon3DResourcePolicy p = { + .maximum_cpu_load_ratio = 1, .maximum_io_pressure_avg10 = 100, .io_slot_capacity = 2}; + s->resource_governor = lardon3d_resource_governor_create(&s->hardware_profile, &p); + s->task_queue = s->resource_governor ? lardon3d_task_queue_create(s->resource_governor, 4) : NULL; + return s->task_queue != NULL; +} +static bool wait_state(Lardon3DTaskQueue *q, uint64_t id, Lardon3DTaskState wanted, + Lardon3DTaskSnapshot *out) { + for (size_t i = 0; i < 2000000; i++) { + if (lardon3d_task_queue_get(q, id, out) && out->state == wanted) { + return true; + } + sched_yield(); + } + return false; +} + +static bool run_test(void) { + char root[] = "/tmp/lardon3d-feature-task-XXXXXX"; + CHECK(mkdtemp(root) && setenv("LARDON3D_PROJECTS_ROOT", root, 1) == 0); + char source[PATH_MAX]; + CHECK(join_path(source, root, "source.pgm") && write_pgm(source)); + Lardon3DAppState state; + lardon3d_app_state_init(&state); + CHECK(runtime(&state) && lardon3d_project_create(&state, "Features")); + Lardon3DProjectDbScanSet scanset; + CHECK(lardon3d_image_catalog_create_scanset(&state, "A", &scanset)); + Lardon3DProjectDbImage image; + Lardon3DProjectDbImageAsset asset; + CHECK(lardon3d_image_catalog_import_file(&state, scanset.scanset_id, source, 0, &image, &asset) == + LARDON3D_IMAGE_CATALOG_IMPORTED); + Lardon3DFeatureExtractorParameters parameters = {512, 4, 10}; + CHECK(setenv("LARDON3D_TEST_FEATURE_PAUSE_BEFORE_PUBLISH", "1", 1) == 0 && + setenv("LARDON3D_TEST_FEATURE_SKIP_FINISHED_CHECKPOINT", "1", 1) == 0); + uint64_t task_id = 0; + CHECK(lardon3d_project_enqueue_feature_extract(&state, image.image_id, ¶meters, &task_id)); + Lardon3DTaskSnapshot snapshot; + CHECK(wait_state(state.task_queue, task_id, TASK_PAUSED, &snapshot)); + lardon3d_task_queue_destroy(state.task_queue); + state.task_queue = NULL; + lardon3d_project_close(&state); + lardon3d_resource_governor_destroy(state.resource_governor); + state.resource_governor = NULL; + CHECK(unsetenv("LARDON3D_TEST_FEATURE_PAUSE_BEFORE_PUBLISH") == 0 && + unsetenv("LARDON3D_TEST_FEATURE_SKIP_FINISHED_CHECKPOINT") == 0); + lardon3d_app_state_init(&state); + CHECK(runtime(&state) && lardon3d_project_open(&state, "Features")); + Lardon3DProjectRecoverySummary summary; + CHECK(lardon3d_project_last_recovery_summary(&state, &summary) && summary.resumed == 1); + CHECK(wait_state(state.task_queue, task_id, TASK_COMPLETED, &snapshot) && + snapshot.progress == 100); + unsigned char fp[32]; + lardon3d_feature_extractor_parameter_fingerprint(¶meters, fp); + Lardon3DProjectDbFeatureSet set; + CHECK(lardon3d_project_db_find_feature_set(state.project_db, image.image_id, + LARDON3D_FEATURE_EXTRACTOR_KIND, 1, fp, + &set) == LARDON3D_PROJECT_DB_OK); + lardon3d_task_queue_destroy(state.task_queue); + state.task_queue = NULL; + lardon3d_project_close(&state); + lardon3d_resource_governor_destroy(state.resource_governor); + state.resource_governor = NULL; + lardon3d_app_state_init(&state); + CHECK(runtime(&state) && lardon3d_project_open(&state, "Features")); + CHECK(lardon3d_project_last_recovery_summary(&state, &summary) && summary.resumed == 0); + CHECK(lardon3d_project_db_load_feature_set(state.project_db, set.feature_set_id, &set) == + LARDON3D_PROJECT_DB_OK); + CHECK(unlink(source) == 0); + Lardon3DFeatureReader *reader = NULL; + Lardon3DFeatureFileMetadata metadata; + CHECK(lardon3d_feature_reader_open(state.project_path, &set, &reader, &metadata) == + LARDON3D_FEATURE_STORE_OK); + lardon3d_feature_reader_close(reader); + + CHECK(write_pgm(source)); + Lardon3DProjectDbImage missing_image; + Lardon3DProjectDbImageAsset missing_asset; + CHECK(lardon3d_image_catalog_import_file(&state, scanset.scanset_id, source, 0, &missing_image, + &missing_asset) == + LARDON3D_IMAGE_CATALOG_ALREADY_PRESENT); + char managed_path[PATH_MAX]; + CHECK(join_path(managed_path, state.project_path, missing_asset.path)); + CHECK(unlink(managed_path) == 0); + uint64_t missing_task_id = 0; + Lardon3DFeatureExtractorParameters missing_parameters = {511, 4, 10}; + CHECK(lardon3d_project_enqueue_feature_extract(&state, missing_image.image_id, + &missing_parameters, &missing_task_id)); + CHECK(wait_state(state.task_queue, missing_task_id, TASK_FAILED, &snapshot)); + + char invalid_source[PATH_MAX]; + CHECK(join_path(invalid_source, root, "invalid.pgm")); + FILE *invalid = fopen(invalid_source, "wb"); + CHECK(invalid && fwrite("not an image", 1, 12, invalid) == 12 && fclose(invalid) == 0); + Lardon3DProjectDbImage invalid_image; + Lardon3DProjectDbImageAsset invalid_asset; + CHECK(lardon3d_image_catalog_import_file(&state, scanset.scanset_id, invalid_source, 0, + &invalid_image, + &invalid_asset) == LARDON3D_IMAGE_CATALOG_IMPORTED); + uint64_t invalid_task_id = 0; + Lardon3DFeatureExtractorParameters invalid_parameters = {510, 4, 10}; + CHECK(lardon3d_project_enqueue_feature_extract(&state, invalid_image.image_id, + &invalid_parameters, &invalid_task_id)); + CHECK(wait_state(state.task_queue, invalid_task_id, TASK_FAILED, &snapshot)); + lardon3d_task_queue_destroy(state.task_queue); + state.task_queue = NULL; + lardon3d_project_close(&state); + lardon3d_resource_governor_destroy(state.resource_governor); + CHECK(remove_tree(root)); + return true; +} +int main(void) { return run_test() ? 0 : 1; } diff --git a/tests/test_project_db.c b/tests/test_project_db.c index 6e4e121..fd1aa16 100644 --- a/tests/test_project_db.c +++ b/tests/test_project_db.c @@ -11,432 +11,561 @@ #include -#define CHECK(condition) do { if (!(condition)) { \ - (void)fprintf(stderr, "Échec ligne %d : %s\n", __LINE__, #condition); return false; \ -} } while (0) +#define CHECK(condition) \ + do { \ + if (!(condition)) { \ + (void)fprintf(stderr, "Échec ligne %d : %s\n", __LINE__, #condition); \ + return false; \ + } \ + } while (0) Lardon3DProjectDbResult lardon3d_project_db_test_orphan_checkpoint(Lardon3DProjectDb *database); -Lardon3DProjectDbResult lardon3d_project_db_test_delete_catalog_identity( - Lardon3DProjectDb *database, uint64_t scanset_id, uint64_t image_id, - uint64_t asset_id); +Lardon3DProjectDbResult +lardon3d_project_db_test_delete_catalog_identity(Lardon3DProjectDb *database, uint64_t scanset_id, + uint64_t image_id, uint64_t asset_id); -static void -asset_path_for_hash(const unsigned char hash[LARDON3D_PROJECT_DB_SHA256_SIZE], - char path[LARDON3D_PROJECT_DB_PATH_CAPACITY]) -{ - static const char digits[] = "0123456789abcdef"; - char hex[65]; - for (size_t index = 0; index < sizeof(hex) / 2; ++index) { - hex[index * 2] = digits[hash[index] >> 4]; - hex[index * 2 + 1] = digits[hash[index] & 15U]; - } - hex[64] = '\0'; - (void)snprintf(path, LARDON3D_PROJECT_DB_PATH_CAPACITY, - "assets/images/%c%c/%s", hex[0], hex[1], hex); +static void asset_path_for_hash(const unsigned char hash[LARDON3D_PROJECT_DB_SHA256_SIZE], + char path[LARDON3D_PROJECT_DB_PATH_CAPACITY]) { + static const char digits[] = "0123456789abcdef"; + char hex[65]; + for (size_t index = 0; index < sizeof(hex) / 2; ++index) { + hex[index * 2] = digits[hash[index] >> 4]; + hex[index * 2 + 1] = digits[hash[index] & 15U]; + } + hex[64] = '\0'; + (void)snprintf(path, LARDON3D_PROJECT_DB_PATH_CAPACITY, "assets/images/%c%c/%s", hex[0], hex[1], + hex); } -static Lardon3DTaskDurableSnapshot -task_snapshot(uint64_t id, Lardon3DTaskState saved) -{ - Lardon3DTaskDurableSnapshot snapshot = { - .id = id, - .estimate = {.minimum_batch_size = 1, .maximum_batch_size = 1, .desired_cpu_threads = 1}, - .progress = saved == TASK_COMPLETED ? 100 : 25, - .saved_state = saved, - .recovery_state = saved == TASK_RUNNING || saved == TASK_PAUSED ? TASK_PENDING : saved, - .started_at = {.tv_sec = 10, .tv_nsec = 20}, - .finished_at = {.tv_sec = 30, .tv_nsec = 40}, - .sequence_count = 2, - }; - (void)snprintf(snapshot.name, sizeof(snapshot.name), "Tâche %llu", (unsigned long long)id); - return snapshot; +static Lardon3DTaskDurableSnapshot task_snapshot(uint64_t id, Lardon3DTaskState saved) { + Lardon3DTaskDurableSnapshot snapshot = { + .id = id, + .estimate = {.minimum_batch_size = 1, .maximum_batch_size = 1, .desired_cpu_threads = 1}, + .progress = saved == TASK_COMPLETED ? 100 : 25, + .saved_state = saved, + .recovery_state = saved == TASK_RUNNING || saved == TASK_PAUSED ? TASK_PENDING : saved, + .started_at = {.tv_sec = 10, .tv_nsec = 20}, + .finished_at = {.tv_sec = 30, .tv_nsec = 40}, + .sequence_count = 2, + }; + (void)snprintf(snapshot.name, sizeof(snapshot.name), "Tâche %llu", (unsigned long long)id); + return snapshot; } -typedef struct { Lardon3DProjectDb *database; bool success; } ThreadContext; -typedef struct { Lardon3DProjectDb *database; uint64_t id; bool success; } IdThreadContext; +typedef struct { + Lardon3DProjectDb *database; + bool success; +} ThreadContext; +typedef struct { + Lardon3DProjectDb *database; + uint64_t id; + bool success; +} IdThreadContext; -static void * -read_thread(void *userdata) -{ - ThreadContext *context = userdata; - context->success = true; - for (size_t index = 0; index < 100; ++index) { - Lardon3DProjectDbTask task; - if (lardon3d_project_db_load_task(context->database, 1, &task) != LARDON3D_PROJECT_DB_OK) { - context->success = false; - break; - } - } - return NULL; -} - -static void * -allocate_id_thread(void *userdata) -{ - IdThreadContext *context = userdata; - context->success = lardon3d_project_db_allocate_task_id( - context->database, &context->id) == LARDON3D_PROJECT_DB_OK; - return NULL; -} - -static bool -create_future_database(const char *path) -{ - sqlite3 *connection = NULL; - if (sqlite3_open(path, &connection) != SQLITE_OK) return false; - bool ok = sqlite3_exec(connection, "CREATE TABLE metadata(key TEXT PRIMARY KEY,value INTEGER NOT NULL);" - "INSERT INTO metadata VALUES('schema_version',5);", NULL, NULL, NULL) == SQLITE_OK; - return sqlite3_close(connection) == SQLITE_OK && ok; -} - -static bool -create_v1_database(const char *path) -{ - static const char sql[] = - "PRAGMA foreign_keys=ON;" - "CREATE TABLE metadata(key TEXT PRIMARY KEY,value INTEGER NOT NULL);" - "INSERT INTO metadata VALUES('schema_version',1);" - "CREATE TABLE project(singleton INTEGER PRIMARY KEY CHECK(singleton=1),stable_id TEXT NOT NULL UNIQUE,name TEXT NOT NULL,created_at INTEGER NOT NULL,updated_at INTEGER NOT NULL);" - "CREATE TABLE tasks(task_id INTEGER PRIMARY KEY CHECK(task_id>0),name TEXT NOT NULL,saved_state INTEGER NOT NULL CHECK(saved_state BETWEEN 0 AND 5),recovery_state INTEGER NOT NULL CHECK(recovery_state BETWEEN 0 AND 5),progress INTEGER NOT NULL CHECK(progress BETWEEN 0 AND 100),sequence_count INTEGER NOT NULL CHECK(sequence_count>=0),started_sec INTEGER NOT NULL,started_nsec INTEGER NOT NULL CHECK(started_nsec BETWEEN 0 AND 999999999),finished_sec INTEGER NOT NULL,finished_nsec INTEGER NOT NULL CHECK(finished_nsec BETWEEN 0 AND 999999999),updated_at INTEGER NOT NULL);" - "CREATE INDEX tasks_recovery_state_idx ON tasks(recovery_state,task_id);" - "CREATE TABLE checkpoints(task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE CASCADE,path TEXT NOT NULL,format_version INTEGER NOT NULL CHECK(format_version>0),durability INTEGER NOT NULL CHECK(durability BETWEEN 0 AND 1),updated_at INTEGER NOT NULL);" - "CREATE TABLE artifacts(artifact_id TEXT PRIMARY KEY,kind TEXT NOT NULL,path TEXT NOT NULL,state INTEGER NOT NULL CHECK(state BETWEEN 0 AND 1),size_bytes INTEGER NOT NULL CHECK(size_bytes>=0),producer_task_id INTEGER REFERENCES tasks(task_id),created_at INTEGER NOT NULL,updated_at INTEGER NOT NULL);" - "CREATE INDEX artifacts_state_idx ON artifacts(state,artifact_id);" - "CREATE INDEX artifacts_producer_idx ON artifacts(producer_task_id);" - "INSERT INTO project VALUES(1,'legacy-project','Legacy',1,1);" - "INSERT INTO tasks VALUES(9,'Legacy task',1,0,12,3,1,0,0,0,2);" - "INSERT INTO checkpoints VALUES(9,'legacy.chk',1,0,2);" - "INSERT INTO artifacts VALUES('legacy-artifact','legacy','legacy.bin',0,0,9,2,2);"; - sqlite3 *connection = NULL; - if (sqlite3_open(path, &connection) != SQLITE_OK) return false; - bool ok = sqlite3_exec(connection, sql, NULL, NULL, NULL) == SQLITE_OK; - return sqlite3_close(connection) == SQLITE_OK && ok; -} - -static bool -create_v2_database(const char *path) -{ - if (!create_v1_database(path)) return false; - sqlite3 *connection = NULL; - if (sqlite3_open(path, &connection) != SQLITE_OK) return false; - bool ok = sqlite3_exec(connection, - "ALTER TABLE tasks ADD COLUMN task_kind TEXT;" - "ALTER TABLE tasks ADD COLUMN task_kind_version INTEGER CHECK(task_kind_version IS NULL OR task_kind_version>0);" - "UPDATE tasks SET task_kind='test.work',task_kind_version=1 WHERE task_id=9;" - "UPDATE metadata SET value=2 WHERE key='schema_version'", - NULL, NULL, NULL) == SQLITE_OK; - return sqlite3_close(connection) == SQLITE_OK && ok; -} - -static bool -create_v3_database(const char *path) -{ - if (!create_v2_database(path)) return false; - sqlite3 *connection = NULL; - if (sqlite3_open(path, &connection) != SQLITE_OK) return false; - bool ok = sqlite3_exec(connection, - "CREATE TABLE image_import_tasks(task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE CASCADE,source_path TEXT NOT NULL);" - "INSERT INTO image_import_tasks VALUES(9,'/legacy/source');" - "INSERT INTO metadata(key,value) VALUES('next_task_id',10);" - "UPDATE metadata SET value=3 WHERE key='schema_version'", - NULL, NULL, NULL) == SQLITE_OK; - return sqlite3_close(connection) == SQLITE_OK && ok; -} - -static bool -query_integer(const char *path, const char *sql, sqlite3_int64 expected) -{ - sqlite3 *connection = NULL; - sqlite3_stmt *statement = NULL; - if (sqlite3_open_v2(path, &connection, SQLITE_OPEN_READONLY, NULL) != SQLITE_OK - || sqlite3_prepare_v2(connection, sql, -1, &statement, NULL) != SQLITE_OK) { - if (statement) (void)sqlite3_finalize(statement); - if (connection) (void)sqlite3_close(connection); - return false; - } - bool matches = sqlite3_step(statement) == SQLITE_ROW - && sqlite3_column_int64(statement, 0) == expected - && sqlite3_step(statement) == SQLITE_DONE; - return sqlite3_finalize(statement) == SQLITE_OK - && sqlite3_close(connection) == SQLITE_OK && matches; -} - -static bool -run_test(void) -{ - char directory[] = "/tmp/lardon3d-project-db-XXXXXX"; - CHECK(mkdtemp(directory)); - char database_path[512], artifact_path[512], future_path[512], corrupt_path[512]; - char legacy_path[512], failed_migration_path[512], v2_path[512]; - char failed_v3_migration_path[512], v3_path[512], failed_v4_path[512]; - CHECK(snprintf(database_path, sizeof(database_path), "%s/project.db", directory) > 0); - CHECK(snprintf(artifact_path, sizeof(artifact_path), "%s/artifact.bin", directory) > 0); - CHECK(snprintf(future_path, sizeof(future_path), "%s/future.db", directory) > 0); - CHECK(snprintf(corrupt_path, sizeof(corrupt_path), "%s/corrupt.db", directory) > 0); - CHECK(snprintf(legacy_path, sizeof(legacy_path), "%s/legacy.db", directory) > 0); - CHECK(snprintf(failed_migration_path, sizeof(failed_migration_path), "%s/failed-migration.db", directory) > 0); - CHECK(snprintf(v2_path, sizeof(v2_path), "%s/v2.db", directory) > 0); - CHECK(snprintf(failed_v3_migration_path, sizeof(failed_v3_migration_path), - "%s/failed-v3-migration.db", directory) > 0); - CHECK(snprintf(v3_path, sizeof(v3_path), "%s/v3.db", directory) > 0); - CHECK(snprintf(failed_v4_path, sizeof(failed_v4_path), - "%s/failed-v4-migration.db", directory) > 0); - - char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]; - Lardon3DProjectDb *database = NULL; - CHECK(lardon3d_project_db_open(database_path, &database, error) == LARDON3D_PROJECT_DB_OK); - CHECK(database && lardon3d_project_db_schema_version(database) == 4); - bool legacy_pending = true; - CHECK(lardon3d_project_db_legacy_catalog_pending(database, &legacy_pending) - == LARDON3D_PROJECT_DB_OK && !legacy_pending); - - Lardon3DProjectDbScanSet deleted_scanset; - CHECK(lardon3d_project_db_create_scanset(database, "Deleted identity", - &deleted_scanset) == LARDON3D_PROJECT_DB_OK); - unsigned char first_hash[LARDON3D_PROJECT_DB_SHA256_SIZE] = {1}; - char first_asset_path[LARDON3D_PROJECT_DB_PATH_CAPACITY]; - asset_path_for_hash(first_hash, first_asset_path); - Lardon3DProjectDbImageRegisterStatus identity_status; - Lardon3DProjectDbImage deleted_image; - CHECK(lardon3d_project_db_register_image(database, - deleted_scanset.scanset_id, first_hash, first_asset_path, 1, - "deleted.jpg", "/source/deleted.jpg", 0, 1, &identity_status, - &deleted_image) == LARDON3D_PROJECT_DB_OK); - CHECK(lardon3d_project_db_test_delete_catalog_identity(database, - deleted_scanset.scanset_id, deleted_image.image_id, - deleted_image.asset_id) == LARDON3D_PROJECT_DB_OK); - Lardon3DProjectDbScanSet replacement_scanset; - CHECK(lardon3d_project_db_create_scanset(database, "Replacement identity", - &replacement_scanset) == LARDON3D_PROJECT_DB_OK - && replacement_scanset.scanset_id > deleted_scanset.scanset_id); - unsigned char second_hash[LARDON3D_PROJECT_DB_SHA256_SIZE] = {2}; - char second_asset_path[LARDON3D_PROJECT_DB_PATH_CAPACITY]; - asset_path_for_hash(second_hash, second_asset_path); - Lardon3DProjectDbImage replacement_image; - CHECK(lardon3d_project_db_register_image(database, - replacement_scanset.scanset_id, second_hash, second_asset_path, 1, - "replacement.jpg", "/source/replacement.jpg", 0, 2, - &identity_status, &replacement_image) == LARDON3D_PROJECT_DB_OK - && replacement_image.image_id > deleted_image.image_id - && replacement_image.asset_id > deleted_image.asset_id); - - Lardon3DProjectDbProject project = {.created_at = 100, .updated_at = 100}; - (void)snprintf(project.stable_id, sizeof(project.stable_id), "project-0001"); - (void)snprintf(project.name, sizeof(project.name), "Projet test"); - CHECK(lardon3d_project_db_set_project(database, &project) == LARDON3D_PROJECT_DB_OK); - Lardon3DProjectDbProject loaded_project; - CHECK(lardon3d_project_db_get_project(database, &loaded_project) == LARDON3D_PROJECT_DB_OK); - CHECK(strcmp(loaded_project.stable_id, project.stable_id) == 0); - project.updated_at = 101; (void)snprintf(project.name, sizeof(project.name), "Projet renommé"); - CHECK(lardon3d_project_db_set_project(database, &project) == LARDON3D_PROJECT_DB_OK); - Lardon3DProjectDbProject conflict = project; (void)snprintf(conflict.stable_id, sizeof(conflict.stable_id), "other"); - CHECK(lardon3d_project_db_set_project(database, &conflict) == LARDON3D_PROJECT_DB_CONSTRAINT); - char copied_error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]; - CHECK(lardon3d_project_db_last_error(database, copied_error)); - CHECK(copied_error[0] != '\0'); - project = loaded_project; memset(project.name, 'x', sizeof(project.name)); - CHECK(lardon3d_project_db_set_project(database, &project) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT); - - Lardon3DTaskDurableSnapshot running = task_snapshot(1, TASK_RUNNING); - Lardon3DProjectDbCheckpoint checkpoint = {.format_version = 1, .durability = LARDON3D_DB_CHECKPOINT_DURABLE, .updated_at = 200}; - (void)snprintf(checkpoint.path, sizeof(checkpoint.path), "%s/checkpoints/task-1.chk", directory); - CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 200) == LARDON3D_PROJECT_DB_OK); - running.progress = 30; running.sequence_count = 3; - CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 201) == LARDON3D_PROJECT_DB_OK); - Lardon3DProjectDbScanSet import_scanset; - CHECK(lardon3d_project_db_create_scanset(database, "Legacy import", - &import_scanset) == LARDON3D_PROJECT_DB_OK); +static void *read_thread(void *userdata) { + ThreadContext *context = userdata; + context->success = true; + for (size_t index = 0; index < 100; ++index) { Lardon3DProjectDbTask task; - CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK); - CHECK(task.recovery_state == TASK_PENDING && task.progress == 30 && task.sequence_count == 3); - CHECK(task.has_task_kind && strcmp(task.task_kind, "test.work") == 0 - && task.task_kind_version == 1); - CHECK(task.has_checkpoint && strcmp(task.checkpoint.path, checkpoint.path) == 0); - checkpoint.durability = LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE; - CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 201) == LARDON3D_PROJECT_DB_OK); - CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK); - CHECK(task.checkpoint.durability == LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE); - CHECK(lardon3d_project_db_record_task(database, &running, "test.other", 1, - &checkpoint, 202) == LARDON3D_PROJECT_DB_CONSTRAINT); - CHECK(lardon3d_project_db_record_task(database, &running, "Test.invalid", 1, - &checkpoint, 202) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT); - CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 0, - &checkpoint, 202) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT); - CHECK(lardon3d_project_db_record_image_import_task(database, &running, - "import.images", 1, &checkpoint, "/tmp/source-a", - import_scanset.scanset_id, 202) - == LARDON3D_PROJECT_DB_CONSTRAINT); - - Lardon3DTaskDurableSnapshot completed = task_snapshot(2, TASK_COMPLETED); - CHECK(lardon3d_project_db_record_task(database, &completed, "test.work", 1, NULL, 202) == LARDON3D_PROJECT_DB_OK); - Lardon3DTaskDurableSnapshot no_checkpoint = task_snapshot(4, TASK_PENDING); - CHECK(lardon3d_project_db_record_task(database, &no_checkpoint, NULL, 0, NULL, 202) == LARDON3D_PROJECT_DB_OK); - Lardon3DProjectDbTask page[1]; size_t count = 0; - CHECK(lardon3d_project_db_list_recoverable(database, 0, page, 1, &count) == LARDON3D_PROJECT_DB_OK); - CHECK(count == 1 && page[0].task_id == 1); - CHECK(lardon3d_project_db_list_recoverable(database, 1, page, 1, &count) == LARDON3D_PROJECT_DB_OK && count == 0); - CHECK(lardon3d_project_db_list_recoverable(database, 0, page, LARDON3D_PROJECT_DB_RECOVERY_PAGE_MAX + 1, &count) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT); - - Lardon3DTaskDurableSnapshot image_import = task_snapshot(5, TASK_PENDING); - CHECK(lardon3d_project_db_record_image_import_task(database, &image_import, - "import.images", 1, &checkpoint, "/tmp/source-a", - import_scanset.scanset_id, 202) - == LARDON3D_PROJECT_DB_OK); - Lardon3DProjectDbImageImport import_parameters; - CHECK(lardon3d_project_db_load_image_import(database, 5, - &import_parameters) == LARDON3D_PROJECT_DB_OK); - CHECK(import_parameters.task_id == 5 - && strcmp(import_parameters.source_path, "/tmp/source-a") == 0 - && import_parameters.scanset_id == import_scanset.scanset_id); - CHECK(lardon3d_project_db_record_image_import_task(database, &image_import, - "import.images", 1, &checkpoint, "/tmp/source-b", - import_scanset.scanset_id, 203) - == LARDON3D_PROJECT_DB_CONSTRAINT); - - Lardon3DTaskDurableSnapshot rollback_task = task_snapshot(3, TASK_PENDING); - CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_CHECKPOINT", "1", 1) == 0); - CHECK(lardon3d_project_db_record_task(database, &rollback_task, "test.work", 1, &checkpoint, 203) == LARDON3D_PROJECT_DB_IO_ERROR); - CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_CHECKPOINT") == 0); - CHECK(lardon3d_project_db_load_task(database, 3, &task) == LARDON3D_PROJECT_DB_NOT_FOUND); - CHECK(lardon3d_project_db_test_orphan_checkpoint(database) == LARDON3D_PROJECT_DB_CONSTRAINT); - - Lardon3DProjectDbArtifact artifact = {.state = LARDON3D_DB_ARTIFACT_STAGED, .has_producer_task = true, .producer_task_id = 1, .created_at = 300, .updated_at = 300}; - (void)snprintf(artifact.artifact_id, sizeof(artifact.artifact_id), "artifact-1"); - (void)snprintf(artifact.kind, sizeof(artifact.kind), "generic-test"); - (void)snprintf(artifact.path, sizeof(artifact.path), "%s", artifact_path); - CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_OK); - Lardon3DProjectDbArtifact loaded_artifact; - CHECK(lardon3d_project_db_load_artifact(database, artifact.artifact_id, &loaded_artifact) == LARDON3D_PROJECT_DB_OK); - CHECK(loaded_artifact.state == LARDON3D_DB_ARTIFACT_STAGED); - CHECK(lardon3d_project_db_mark_artifact_ready(database, artifact.artifact_id, 301) == LARDON3D_PROJECT_DB_IO_ERROR); - int descriptor = open(artifact_path, O_WRONLY | O_CREAT | O_TRUNC, 0600); CHECK(descriptor >= 0); - const char payload[] = "validated"; CHECK(write(descriptor, payload, sizeof(payload)) == (ssize_t)sizeof(payload)); CHECK(close(descriptor) == 0); - CHECK(lardon3d_project_db_mark_artifact_ready(database, artifact.artifact_id, 301) == LARDON3D_PROJECT_DB_OK); - CHECK(lardon3d_project_db_load_artifact(database, artifact.artifact_id, &loaded_artifact) == LARDON3D_PROJECT_DB_OK); - CHECK(loaded_artifact.state == LARDON3D_DB_ARTIFACT_READY && loaded_artifact.size_bytes == sizeof(payload)); - CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_CONSTRAINT); - artifact.producer_task_id = 999; - (void)snprintf(artifact.artifact_id, sizeof(artifact.artifact_id), "orphan-artifact"); - CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_CONSTRAINT); - artifact = loaded_artifact; artifact.state = LARDON3D_DB_ARTIFACT_STAGED; artifact.size_bytes = 0; - memset(artifact.path, 'x', sizeof(artifact.path)); - CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT); - - ThreadContext contexts[2] = {{.database = database}, {.database = database}}; pthread_t threads[2]; - CHECK(pthread_create(&threads[0], NULL, read_thread, &contexts[0]) == 0); - CHECK(pthread_create(&threads[1], NULL, read_thread, &contexts[1]) == 0); - CHECK(pthread_join(threads[0], NULL) == 0 && pthread_join(threads[1], NULL) == 0); - CHECK(contexts[0].success && contexts[1].success); - IdThreadContext id_contexts[2] = {{.database = database}, {.database = database}}; - CHECK(pthread_create(&threads[0], NULL, allocate_id_thread, - &id_contexts[0]) == 0); - CHECK(pthread_create(&threads[1], NULL, allocate_id_thread, - &id_contexts[1]) == 0); - CHECK(pthread_join(threads[0], NULL) == 0 - && pthread_join(threads[1], NULL) == 0); - CHECK(id_contexts[0].success && id_contexts[1].success - && id_contexts[0].id != id_contexts[1].id - && id_contexts[0].id > 5 && id_contexts[1].id > 5); - - char too_long[LARDON3D_PROJECT_DB_PATH_CAPACITY + 1]; memset(too_long, 'x', sizeof(too_long)); too_long[sizeof(too_long) - 1] = '\0'; - CHECK(lardon3d_project_db_open(too_long, &database, error) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT); - lardon3d_project_db_close(contexts[0].database); database = NULL; - CHECK(query_integer(database_path, "SELECT value FROM metadata WHERE key='schema_version'", 4)); - CHECK(query_integer(database_path, "SELECT count(*) FROM tasks WHERE task_id=1", 1)); - CHECK(lardon3d_project_db_open(database_path, &database, error) == LARDON3D_PROJECT_DB_OK); - CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK); - CHECK(lardon3d_project_db_load_artifact(database, "artifact-1", &loaded_artifact) == LARDON3D_PROJECT_DB_OK); - lardon3d_project_db_close(database); - - CHECK(create_future_database(future_path)); - CHECK(lardon3d_project_db_open(future_path, &database, error) == LARDON3D_PROJECT_DB_UNSUPPORTED_SCHEMA); - descriptor = open(corrupt_path, O_WRONLY | O_CREAT | O_TRUNC, 0600); CHECK(descriptor >= 0); - CHECK(write(descriptor, "not sqlite", 10) == 10 && close(descriptor) == 0); - CHECK(lardon3d_project_db_open(corrupt_path, &database, error) == LARDON3D_PROJECT_DB_CORRUPT); - - CHECK(create_v1_database(legacy_path)); - CHECK(lardon3d_project_db_open(legacy_path, &database, error) == LARDON3D_PROJECT_DB_OK); - CHECK(lardon3d_project_db_schema_version(database) == 4); - CHECK(lardon3d_project_db_get_project(database, &loaded_project) == LARDON3D_PROJECT_DB_OK - && strcmp(loaded_project.stable_id, "legacy-project") == 0); - CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK); - CHECK(!task.has_task_kind && task.has_checkpoint - && strcmp(task.checkpoint.path, "legacy.chk") == 0); - CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact", - &loaded_artifact) == LARDON3D_PROJECT_DB_OK); - lardon3d_project_db_close(database); database = NULL; - CHECK(query_integer(legacy_path, "SELECT value FROM metadata WHERE key='schema_version'", 4)); - - CHECK(create_v1_database(failed_migration_path)); - CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V2", "1", 1) == 0); - CHECK(lardon3d_project_db_open(failed_migration_path, &database, error) - == LARDON3D_PROJECT_DB_IO_ERROR); - CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V2") == 0); - CHECK(query_integer(failed_migration_path, - "SELECT value FROM metadata WHERE key='schema_version'", 1)); - CHECK(lardon3d_project_db_open(failed_migration_path, &database, error) - == LARDON3D_PROJECT_DB_OK); - CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK - && !task.has_task_kind); - lardon3d_project_db_close(database); database = NULL; - - CHECK(create_v2_database(v2_path)); - CHECK(lardon3d_project_db_open(v2_path, &database, error) - == LARDON3D_PROJECT_DB_OK); - CHECK(lardon3d_project_db_load_task(database, 9, &task) - == LARDON3D_PROJECT_DB_OK && task.has_task_kind - && strcmp(task.task_kind, "test.work") == 0); - CHECK(task.has_checkpoint - && strcmp(task.checkpoint.path, "legacy.chk") == 0); - CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact", - &loaded_artifact) == LARDON3D_PROJECT_DB_OK); - lardon3d_project_db_close(database); database = NULL; - CHECK(query_integer(v2_path, - "SELECT value FROM metadata WHERE key='schema_version'", 4)); - - CHECK(create_v2_database(failed_v3_migration_path)); - CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V3", "1", 1) == 0); - CHECK(lardon3d_project_db_open(failed_v3_migration_path, &database, error) - == LARDON3D_PROJECT_DB_IO_ERROR); - CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V3") == 0); - CHECK(query_integer(failed_v3_migration_path, - "SELECT value FROM metadata WHERE key='schema_version'", 2)); - CHECK(query_integer(failed_v3_migration_path, - "SELECT count(*) FROM sqlite_master WHERE type='table' AND name='image_import_tasks'", 0)); - - CHECK(create_v3_database(v3_path)); - CHECK(lardon3d_project_db_open(v3_path, &database, error) - == LARDON3D_PROJECT_DB_OK); - CHECK(lardon3d_project_db_load_image_import(database, 9, - &import_parameters) == LARDON3D_PROJECT_DB_OK); - CHECK(strcmp(import_parameters.source_path, "/legacy/source") == 0 - && import_parameters.scanset_id > 0); - CHECK(lardon3d_project_db_legacy_catalog_pending(database, &legacy_pending) - == LARDON3D_PROJECT_DB_OK && legacy_pending); - CHECK(lardon3d_project_db_load_task(database, 9, &task) - == LARDON3D_PROJECT_DB_OK && task.has_task_kind); - CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact", - &loaded_artifact) == LARDON3D_PROJECT_DB_OK); - lardon3d_project_db_close(database); database = NULL; - CHECK(query_integer(v3_path, - "SELECT value FROM metadata WHERE key='schema_version'", 4)); - - CHECK(create_v3_database(failed_v4_path)); - CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V4", "1", 1) == 0); - CHECK(lardon3d_project_db_open(failed_v4_path, &database, error) - == LARDON3D_PROJECT_DB_IO_ERROR); - CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V4") == 0); - CHECK(query_integer(failed_v4_path, - "SELECT value FROM metadata WHERE key='schema_version'", 3)); - CHECK(query_integer(failed_v4_path, - "SELECT count(*) FROM sqlite_master WHERE type='table' AND name='scansets'", 0)); - - CHECK(unlink(artifact_path) == 0); CHECK(unlink(database_path) == 0); CHECK(unlink(future_path) == 0); CHECK(unlink(corrupt_path) == 0); - CHECK(unlink(legacy_path) == 0); CHECK(unlink(failed_migration_path) == 0); - CHECK(unlink(v2_path) == 0); CHECK(unlink(failed_v3_migration_path) == 0); - CHECK(unlink(v3_path) == 0); CHECK(unlink(failed_v4_path) == 0); - CHECK(rmdir(directory) == 0); - return true; + if (lardon3d_project_db_load_task(context->database, 1, &task) != LARDON3D_PROJECT_DB_OK) { + context->success = false; + break; + } + } + return NULL; } -int main(void) { return run_test() ? EXIT_SUCCESS : EXIT_FAILURE; } +static void *allocate_id_thread(void *userdata) { + IdThreadContext *context = userdata; + context->success = lardon3d_project_db_allocate_task_id(context->database, &context->id) == + LARDON3D_PROJECT_DB_OK; + return NULL; +} + +static bool create_future_database(const char *path) { + sqlite3 *connection = NULL; + if (sqlite3_open(path, &connection) != SQLITE_OK) { + return false; + } + bool ok = sqlite3_exec(connection, + "CREATE TABLE metadata(key TEXT PRIMARY KEY,value INTEGER NOT NULL);" + "INSERT INTO metadata VALUES('schema_version',6);", + NULL, NULL, NULL) == SQLITE_OK; + return sqlite3_close(connection) == SQLITE_OK && ok; +} + +static bool create_v1_database(const char *path) { + static const char sql[] = + "PRAGMA foreign_keys=ON;" + "CREATE TABLE metadata(key TEXT PRIMARY KEY,value INTEGER NOT NULL);" + "INSERT INTO metadata VALUES('schema_version',1);" + "CREATE TABLE project(singleton INTEGER PRIMARY KEY CHECK(singleton=1),stable_id TEXT NOT " + "NULL UNIQUE,name TEXT NOT NULL,created_at INTEGER NOT NULL,updated_at INTEGER NOT NULL);" + "CREATE TABLE tasks(task_id INTEGER PRIMARY KEY CHECK(task_id>0),name TEXT NOT " + "NULL,saved_state INTEGER NOT NULL CHECK(saved_state BETWEEN 0 AND 5),recovery_state " + "INTEGER NOT NULL CHECK(recovery_state BETWEEN 0 AND 5),progress INTEGER NOT NULL " + "CHECK(progress BETWEEN 0 AND 100),sequence_count INTEGER NOT NULL " + "CHECK(sequence_count>=0),started_sec INTEGER NOT NULL,started_nsec INTEGER NOT NULL " + "CHECK(started_nsec BETWEEN 0 AND 999999999),finished_sec INTEGER NOT NULL,finished_nsec " + "INTEGER NOT NULL CHECK(finished_nsec BETWEEN 0 AND 999999999),updated_at INTEGER NOT " + "NULL);" + "CREATE INDEX tasks_recovery_state_idx ON tasks(recovery_state,task_id);" + "CREATE TABLE checkpoints(task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE " + "CASCADE,path TEXT NOT NULL,format_version INTEGER NOT NULL " + "CHECK(format_version>0),durability INTEGER NOT NULL CHECK(durability BETWEEN 0 AND " + "1),updated_at INTEGER NOT NULL);" + "CREATE TABLE artifacts(artifact_id TEXT PRIMARY KEY,kind TEXT NOT NULL,path TEXT NOT " + "NULL,state INTEGER NOT NULL CHECK(state BETWEEN 0 AND 1),size_bytes INTEGER NOT NULL " + "CHECK(size_bytes>=0),producer_task_id INTEGER REFERENCES tasks(task_id),created_at " + "INTEGER NOT NULL,updated_at INTEGER NOT NULL);" + "CREATE INDEX artifacts_state_idx ON artifacts(state,artifact_id);" + "CREATE INDEX artifacts_producer_idx ON artifacts(producer_task_id);" + "INSERT INTO project VALUES(1,'legacy-project','Legacy',1,1);" + "INSERT INTO tasks VALUES(9,'Legacy task',1,0,12,3,1,0,0,0,2);" + "INSERT INTO checkpoints VALUES(9,'legacy.chk',1,0,2);" + "INSERT INTO artifacts VALUES('legacy-artifact','legacy','legacy.bin',0,0,9,2,2);"; + sqlite3 *connection = NULL; + if (sqlite3_open(path, &connection) != SQLITE_OK) { + return false; + } + bool ok = sqlite3_exec(connection, sql, NULL, NULL, NULL) == SQLITE_OK; + return sqlite3_close(connection) == SQLITE_OK && ok; +} + +static bool create_v2_database(const char *path) { + if (!create_v1_database(path)) { + return false; + } + sqlite3 *connection = NULL; + if (sqlite3_open(path, &connection) != SQLITE_OK) { + return false; + } + bool ok = + sqlite3_exec(connection, + "ALTER TABLE tasks ADD COLUMN task_kind TEXT;" + "ALTER TABLE tasks ADD COLUMN task_kind_version INTEGER " + "CHECK(task_kind_version IS NULL OR task_kind_version>0);" + "UPDATE tasks SET task_kind='test.work',task_kind_version=1 WHERE task_id=9;" + "UPDATE metadata SET value=2 WHERE key='schema_version'", + NULL, NULL, NULL) == SQLITE_OK; + return sqlite3_close(connection) == SQLITE_OK && ok; +} + +static bool create_v3_database(const char *path) { + if (!create_v2_database(path)) { + return false; + } + sqlite3 *connection = NULL; + if (sqlite3_open(path, &connection) != SQLITE_OK) { + return false; + } + bool ok = sqlite3_exec(connection, + "CREATE TABLE image_import_tasks(task_id INTEGER PRIMARY KEY REFERENCES " + "tasks(task_id) ON DELETE CASCADE,source_path TEXT NOT NULL);" + "INSERT INTO image_import_tasks VALUES(9,'/legacy/source');" + "INSERT INTO metadata(key,value) VALUES('next_task_id',10);" + "UPDATE metadata SET value=3 WHERE key='schema_version'", + NULL, NULL, NULL) == SQLITE_OK; + return sqlite3_close(connection) == SQLITE_OK && ok; +} + +static bool create_v4_database(const char *path) { + if (!create_v3_database(path)) { + return false; + } + sqlite3 *connection = NULL; + if (sqlite3_open(path, &connection) != SQLITE_OK) { + return false; + } + const char *sql = + "CREATE TABLE scansets(scanset_id INTEGER PRIMARY KEY AUTOINCREMENT " + "CHECK(scanset_id>0),name TEXT NOT NULL,created_at INTEGER NOT NULL,updated_at INTEGER NOT " + "NULL);" + "CREATE TABLE image_assets(asset_id INTEGER PRIMARY KEY AUTOINCREMENT " + "CHECK(asset_id>0),sha256 BLOB NOT NULL UNIQUE,path TEXT NOT NULL UNIQUE,size_bytes " + "INTEGER NOT NULL,state INTEGER NOT NULL,created_at INTEGER NOT NULL);" + "CREATE TABLE images(image_id INTEGER PRIMARY KEY AUTOINCREMENT " + "CHECK(image_id>0),scanset_id INTEGER NOT NULL REFERENCES scansets(scanset_id),asset_id " + "INTEGER NOT NULL REFERENCES image_assets(asset_id),original_name TEXT NOT " + "NULL,source_path TEXT NOT NULL,producer_task_id INTEGER REFERENCES " + "tasks(task_id),imported_at INTEGER NOT NULL,UNIQUE(scanset_id,asset_id));" + "CREATE INDEX images_scanset_idx ON images(scanset_id,image_id);" + "CREATE INDEX images_producer_idx ON images(producer_task_id,image_id);" + "ALTER TABLE image_import_tasks ADD COLUMN scanset_id INTEGER REFERENCES " + "scansets(scanset_id);" + "INSERT INTO scansets(name,created_at,updated_at) VALUES('Legacy',0,0);" + "UPDATE image_import_tasks SET scanset_id=1;" + "INSERT INTO metadata VALUES('legacy_image_catalog_pending',1);" + "UPDATE metadata SET value=4 WHERE key='schema_version';"; + bool ok = sqlite3_exec(connection, sql, NULL, NULL, NULL) == SQLITE_OK; + return sqlite3_close(connection) == SQLITE_OK && ok; +} + +static bool query_integer(const char *path, const char *sql, sqlite3_int64 expected) { + sqlite3 *connection = NULL; + sqlite3_stmt *statement = NULL; + if (sqlite3_open_v2(path, &connection, SQLITE_OPEN_READONLY, NULL) != SQLITE_OK || + sqlite3_prepare_v2(connection, sql, -1, &statement, NULL) != SQLITE_OK) { + if (statement) { + (void)sqlite3_finalize(statement); + } + if (connection) { + (void)sqlite3_close(connection); + } + return false; + } + bool matches = sqlite3_step(statement) == SQLITE_ROW && + sqlite3_column_int64(statement, 0) == expected && + sqlite3_step(statement) == SQLITE_DONE; + return sqlite3_finalize(statement) == SQLITE_OK && sqlite3_close(connection) == SQLITE_OK && + matches; +} + +static bool run_test(void) { + char directory[] = "/tmp/lardon3d-project-db-XXXXXX"; + CHECK(mkdtemp(directory)); + char database_path[512], artifact_path[512], future_path[512], corrupt_path[512]; + char legacy_path[512], failed_migration_path[512], v2_path[512]; + char failed_v3_migration_path[512], v3_path[512], failed_v4_path[512]; + char v4_path[512], failed_v5_path[512]; + CHECK(snprintf(database_path, sizeof(database_path), "%s/project.db", directory) > 0); + CHECK(snprintf(artifact_path, sizeof(artifact_path), "%s/artifact.bin", directory) > 0); + CHECK(snprintf(future_path, sizeof(future_path), "%s/future.db", directory) > 0); + CHECK(snprintf(corrupt_path, sizeof(corrupt_path), "%s/corrupt.db", directory) > 0); + CHECK(snprintf(legacy_path, sizeof(legacy_path), "%s/legacy.db", directory) > 0); + CHECK(snprintf(failed_migration_path, sizeof(failed_migration_path), "%s/failed-migration.db", + directory) > 0); + CHECK(snprintf(v2_path, sizeof(v2_path), "%s/v2.db", directory) > 0); + CHECK(snprintf(failed_v3_migration_path, sizeof(failed_v3_migration_path), + "%s/failed-v3-migration.db", directory) > 0); + CHECK(snprintf(v3_path, sizeof(v3_path), "%s/v3.db", directory) > 0); + CHECK(snprintf(failed_v4_path, sizeof(failed_v4_path), "%s/failed-v4-migration.db", directory) > + 0); + CHECK(snprintf(v4_path, sizeof(v4_path), "%s/v4.db", directory) > 0); + CHECK(snprintf(failed_v5_path, sizeof(failed_v5_path), "%s/failed-v5-migration.db", directory) > + 0); + + char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]; + Lardon3DProjectDb *database = NULL; + CHECK(lardon3d_project_db_open(database_path, &database, error) == LARDON3D_PROJECT_DB_OK); + CHECK(database && lardon3d_project_db_schema_version(database) == 5); + bool legacy_pending = true; + CHECK(lardon3d_project_db_legacy_catalog_pending(database, &legacy_pending) == + LARDON3D_PROJECT_DB_OK && + !legacy_pending); + + Lardon3DProjectDbScanSet deleted_scanset; + CHECK(lardon3d_project_db_create_scanset(database, "Deleted identity", &deleted_scanset) == + LARDON3D_PROJECT_DB_OK); + unsigned char first_hash[LARDON3D_PROJECT_DB_SHA256_SIZE] = {1}; + char first_asset_path[LARDON3D_PROJECT_DB_PATH_CAPACITY]; + asset_path_for_hash(first_hash, first_asset_path); + Lardon3DProjectDbImageRegisterStatus identity_status; + Lardon3DProjectDbImage deleted_image; + CHECK(lardon3d_project_db_register_image(database, deleted_scanset.scanset_id, first_hash, + first_asset_path, 1, "deleted.jpg", + "/source/deleted.jpg", 0, 1, &identity_status, + &deleted_image) == LARDON3D_PROJECT_DB_OK); + CHECK(lardon3d_project_db_test_delete_catalog_identity( + database, deleted_scanset.scanset_id, deleted_image.image_id, deleted_image.asset_id) == + LARDON3D_PROJECT_DB_OK); + Lardon3DProjectDbScanSet replacement_scanset; + CHECK(lardon3d_project_db_create_scanset(database, "Replacement identity", + &replacement_scanset) == LARDON3D_PROJECT_DB_OK && + replacement_scanset.scanset_id > deleted_scanset.scanset_id); + unsigned char second_hash[LARDON3D_PROJECT_DB_SHA256_SIZE] = {2}; + char second_asset_path[LARDON3D_PROJECT_DB_PATH_CAPACITY]; + asset_path_for_hash(second_hash, second_asset_path); + Lardon3DProjectDbImage replacement_image; + CHECK(lardon3d_project_db_register_image(database, replacement_scanset.scanset_id, second_hash, + second_asset_path, 1, "replacement.jpg", + "/source/replacement.jpg", 0, 2, &identity_status, + &replacement_image) == LARDON3D_PROJECT_DB_OK && + replacement_image.image_id > deleted_image.image_id && + replacement_image.asset_id > deleted_image.asset_id); + + Lardon3DProjectDbProject project = {.created_at = 100, .updated_at = 100}; + (void)snprintf(project.stable_id, sizeof(project.stable_id), "project-0001"); + (void)snprintf(project.name, sizeof(project.name), "Projet test"); + CHECK(lardon3d_project_db_set_project(database, &project) == LARDON3D_PROJECT_DB_OK); + Lardon3DProjectDbProject loaded_project; + CHECK(lardon3d_project_db_get_project(database, &loaded_project) == LARDON3D_PROJECT_DB_OK); + CHECK(strcmp(loaded_project.stable_id, project.stable_id) == 0); + project.updated_at = 101; + (void)snprintf(project.name, sizeof(project.name), "Projet renommé"); + CHECK(lardon3d_project_db_set_project(database, &project) == LARDON3D_PROJECT_DB_OK); + Lardon3DProjectDbProject conflict = project; + (void)snprintf(conflict.stable_id, sizeof(conflict.stable_id), "other"); + CHECK(lardon3d_project_db_set_project(database, &conflict) == LARDON3D_PROJECT_DB_CONSTRAINT); + char copied_error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]; + CHECK(lardon3d_project_db_last_error(database, copied_error)); + CHECK(copied_error[0] != '\0'); + project = loaded_project; + memset(project.name, 'x', sizeof(project.name)); + CHECK(lardon3d_project_db_set_project(database, &project) == + LARDON3D_PROJECT_DB_INVALID_ARGUMENT); + + Lardon3DTaskDurableSnapshot running = task_snapshot(1, TASK_RUNNING); + Lardon3DProjectDbCheckpoint checkpoint = { + .format_version = 1, .durability = LARDON3D_DB_CHECKPOINT_DURABLE, .updated_at = 200}; + (void)snprintf(checkpoint.path, sizeof(checkpoint.path), "%s/checkpoints/task-1.chk", directory); + CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 200) == + LARDON3D_PROJECT_DB_OK); + running.progress = 30; + running.sequence_count = 3; + CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 201) == + LARDON3D_PROJECT_DB_OK); + Lardon3DProjectDbScanSet import_scanset; + CHECK(lardon3d_project_db_create_scanset(database, "Legacy import", &import_scanset) == + LARDON3D_PROJECT_DB_OK); + Lardon3DProjectDbTask task; + CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK); + CHECK(task.recovery_state == TASK_PENDING && task.progress == 30 && task.sequence_count == 3); + CHECK(task.has_task_kind && strcmp(task.task_kind, "test.work") == 0 && + task.task_kind_version == 1); + CHECK(task.has_checkpoint && strcmp(task.checkpoint.path, checkpoint.path) == 0); + checkpoint.durability = LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE; + CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 1, &checkpoint, 201) == + LARDON3D_PROJECT_DB_OK); + CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK); + CHECK(task.checkpoint.durability == LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE); + CHECK(lardon3d_project_db_record_task(database, &running, "test.other", 1, &checkpoint, 202) == + LARDON3D_PROJECT_DB_CONSTRAINT); + CHECK(lardon3d_project_db_record_task(database, &running, "Test.invalid", 1, &checkpoint, 202) == + LARDON3D_PROJECT_DB_INVALID_ARGUMENT); + CHECK(lardon3d_project_db_record_task(database, &running, "test.work", 0, &checkpoint, 202) == + LARDON3D_PROJECT_DB_INVALID_ARGUMENT); + CHECK(lardon3d_project_db_record_image_import_task( + database, &running, "import.images", 1, &checkpoint, "/tmp/source-a", + import_scanset.scanset_id, 202) == LARDON3D_PROJECT_DB_CONSTRAINT); + + Lardon3DTaskDurableSnapshot completed = task_snapshot(2, TASK_COMPLETED); + CHECK(lardon3d_project_db_record_task(database, &completed, "test.work", 1, NULL, 202) == + LARDON3D_PROJECT_DB_OK); + Lardon3DTaskDurableSnapshot no_checkpoint = task_snapshot(4, TASK_PENDING); + CHECK(lardon3d_project_db_record_task(database, &no_checkpoint, NULL, 0, NULL, 202) == + LARDON3D_PROJECT_DB_OK); + Lardon3DProjectDbTask page[1]; + size_t count = 0; + CHECK(lardon3d_project_db_list_recoverable(database, 0, page, 1, &count) == + LARDON3D_PROJECT_DB_OK); + CHECK(count == 1 && page[0].task_id == 1); + CHECK(lardon3d_project_db_list_recoverable(database, 1, page, 1, &count) == + LARDON3D_PROJECT_DB_OK && + count == 0); + CHECK(lardon3d_project_db_list_recoverable(database, 0, page, + LARDON3D_PROJECT_DB_RECOVERY_PAGE_MAX + 1, + &count) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT); + + Lardon3DTaskDurableSnapshot image_import = task_snapshot(5, TASK_PENDING); + CHECK(lardon3d_project_db_record_image_import_task( + database, &image_import, "import.images", 1, &checkpoint, "/tmp/source-a", + import_scanset.scanset_id, 202) == LARDON3D_PROJECT_DB_OK); + Lardon3DProjectDbImageImport import_parameters; + CHECK(lardon3d_project_db_load_image_import(database, 5, &import_parameters) == + LARDON3D_PROJECT_DB_OK); + CHECK(import_parameters.task_id == 5 && + strcmp(import_parameters.source_path, "/tmp/source-a") == 0 && + import_parameters.scanset_id == import_scanset.scanset_id); + CHECK(lardon3d_project_db_record_image_import_task( + database, &image_import, "import.images", 1, &checkpoint, "/tmp/source-b", + import_scanset.scanset_id, 203) == LARDON3D_PROJECT_DB_CONSTRAINT); + + Lardon3DTaskDurableSnapshot rollback_task = task_snapshot(3, TASK_PENDING); + CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_CHECKPOINT", "1", 1) == 0); + CHECK(lardon3d_project_db_record_task(database, &rollback_task, "test.work", 1, &checkpoint, + 203) == LARDON3D_PROJECT_DB_IO_ERROR); + CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_CHECKPOINT") == 0); + CHECK(lardon3d_project_db_load_task(database, 3, &task) == LARDON3D_PROJECT_DB_NOT_FOUND); + CHECK(lardon3d_project_db_test_orphan_checkpoint(database) == LARDON3D_PROJECT_DB_CONSTRAINT); + + Lardon3DProjectDbArtifact artifact = {.state = LARDON3D_DB_ARTIFACT_STAGED, + .has_producer_task = true, + .producer_task_id = 1, + .created_at = 300, + .updated_at = 300}; + (void)snprintf(artifact.artifact_id, sizeof(artifact.artifact_id), "artifact-1"); + (void)snprintf(artifact.kind, sizeof(artifact.kind), "generic-test"); + (void)snprintf(artifact.path, sizeof(artifact.path), "%s", artifact_path); + CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_OK); + Lardon3DProjectDbArtifact loaded_artifact; + CHECK(lardon3d_project_db_load_artifact(database, artifact.artifact_id, &loaded_artifact) == + LARDON3D_PROJECT_DB_OK); + CHECK(loaded_artifact.state == LARDON3D_DB_ARTIFACT_STAGED); + CHECK(lardon3d_project_db_mark_artifact_ready(database, artifact.artifact_id, 301) == + LARDON3D_PROJECT_DB_IO_ERROR); + int descriptor = open(artifact_path, O_WRONLY | O_CREAT | O_TRUNC, 0600); + CHECK(descriptor >= 0); + const char payload[] = "validated"; + CHECK(write(descriptor, payload, sizeof(payload)) == (ssize_t)sizeof(payload)); + CHECK(close(descriptor) == 0); + CHECK(lardon3d_project_db_mark_artifact_ready(database, artifact.artifact_id, 301) == + LARDON3D_PROJECT_DB_OK); + CHECK(lardon3d_project_db_load_artifact(database, artifact.artifact_id, &loaded_artifact) == + LARDON3D_PROJECT_DB_OK); + CHECK(loaded_artifact.state == LARDON3D_DB_ARTIFACT_READY && + loaded_artifact.size_bytes == sizeof(payload)); + CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_CONSTRAINT); + artifact.producer_task_id = 999; + (void)snprintf(artifact.artifact_id, sizeof(artifact.artifact_id), "orphan-artifact"); + CHECK(lardon3d_project_db_create_artifact(database, &artifact) == LARDON3D_PROJECT_DB_CONSTRAINT); + artifact = loaded_artifact; + artifact.state = LARDON3D_DB_ARTIFACT_STAGED; + artifact.size_bytes = 0; + memset(artifact.path, 'x', sizeof(artifact.path)); + CHECK(lardon3d_project_db_create_artifact(database, &artifact) == + LARDON3D_PROJECT_DB_INVALID_ARGUMENT); + + ThreadContext contexts[2] = {{.database = database}, {.database = database}}; + pthread_t threads[2]; + CHECK(pthread_create(&threads[0], NULL, read_thread, &contexts[0]) == 0); + CHECK(pthread_create(&threads[1], NULL, read_thread, &contexts[1]) == 0); + CHECK(pthread_join(threads[0], NULL) == 0 && pthread_join(threads[1], NULL) == 0); + CHECK(contexts[0].success && contexts[1].success); + IdThreadContext id_contexts[2] = {{.database = database}, {.database = database}}; + CHECK(pthread_create(&threads[0], NULL, allocate_id_thread, &id_contexts[0]) == 0); + CHECK(pthread_create(&threads[1], NULL, allocate_id_thread, &id_contexts[1]) == 0); + CHECK(pthread_join(threads[0], NULL) == 0 && pthread_join(threads[1], NULL) == 0); + CHECK(id_contexts[0].success && id_contexts[1].success && + id_contexts[0].id != id_contexts[1].id && id_contexts[0].id > 5 && id_contexts[1].id > 5); + + char too_long[LARDON3D_PROJECT_DB_PATH_CAPACITY + 1]; + memset(too_long, 'x', sizeof(too_long)); + too_long[sizeof(too_long) - 1] = '\0'; + CHECK(lardon3d_project_db_open(too_long, &database, error) == + LARDON3D_PROJECT_DB_INVALID_ARGUMENT); + lardon3d_project_db_close(contexts[0].database); + database = NULL; + CHECK(query_integer(database_path, "SELECT value FROM metadata WHERE key='schema_version'", 5)); + CHECK(query_integer(database_path, "SELECT count(*) FROM tasks WHERE task_id=1", 1)); + CHECK(lardon3d_project_db_open(database_path, &database, error) == LARDON3D_PROJECT_DB_OK); + CHECK(lardon3d_project_db_load_task(database, 1, &task) == LARDON3D_PROJECT_DB_OK); + CHECK(lardon3d_project_db_load_artifact(database, "artifact-1", &loaded_artifact) == + LARDON3D_PROJECT_DB_OK); + lardon3d_project_db_close(database); + + CHECK(create_future_database(future_path)); + CHECK(lardon3d_project_db_open(future_path, &database, error) == + LARDON3D_PROJECT_DB_UNSUPPORTED_SCHEMA); + descriptor = open(corrupt_path, O_WRONLY | O_CREAT | O_TRUNC, 0600); + CHECK(descriptor >= 0); + CHECK(write(descriptor, "not sqlite", 10) == 10 && close(descriptor) == 0); + CHECK(lardon3d_project_db_open(corrupt_path, &database, error) == LARDON3D_PROJECT_DB_CORRUPT); + + CHECK(create_v1_database(legacy_path)); + CHECK(lardon3d_project_db_open(legacy_path, &database, error) == LARDON3D_PROJECT_DB_OK); + CHECK(lardon3d_project_db_schema_version(database) == 5); + CHECK(lardon3d_project_db_get_project(database, &loaded_project) == LARDON3D_PROJECT_DB_OK && + strcmp(loaded_project.stable_id, "legacy-project") == 0); + CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK); + CHECK(!task.has_task_kind && task.has_checkpoint && + strcmp(task.checkpoint.path, "legacy.chk") == 0); + CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact", &loaded_artifact) == + LARDON3D_PROJECT_DB_OK); + lardon3d_project_db_close(database); + database = NULL; + CHECK(query_integer(legacy_path, "SELECT value FROM metadata WHERE key='schema_version'", 5)); + + CHECK(create_v1_database(failed_migration_path)); + CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V2", "1", 1) == 0); + CHECK(lardon3d_project_db_open(failed_migration_path, &database, error) == + LARDON3D_PROJECT_DB_IO_ERROR); + CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V2") == 0); + CHECK(query_integer(failed_migration_path, + "SELECT value FROM metadata WHERE key='schema_version'", 1)); + CHECK(lardon3d_project_db_open(failed_migration_path, &database, error) == + LARDON3D_PROJECT_DB_OK); + CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK && + !task.has_task_kind); + lardon3d_project_db_close(database); + database = NULL; + + CHECK(create_v2_database(v2_path)); + CHECK(lardon3d_project_db_open(v2_path, &database, error) == LARDON3D_PROJECT_DB_OK); + CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK && + task.has_task_kind && strcmp(task.task_kind, "test.work") == 0); + CHECK(task.has_checkpoint && strcmp(task.checkpoint.path, "legacy.chk") == 0); + CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact", &loaded_artifact) == + LARDON3D_PROJECT_DB_OK); + lardon3d_project_db_close(database); + database = NULL; + CHECK(query_integer(v2_path, "SELECT value FROM metadata WHERE key='schema_version'", 5)); + + CHECK(create_v2_database(failed_v3_migration_path)); + CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V3", "1", 1) == 0); + CHECK(lardon3d_project_db_open(failed_v3_migration_path, &database, error) == + LARDON3D_PROJECT_DB_IO_ERROR); + CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V3") == 0); + CHECK(query_integer(failed_v3_migration_path, + "SELECT value FROM metadata WHERE key='schema_version'", 2)); + CHECK(query_integer( + failed_v3_migration_path, + "SELECT count(*) FROM sqlite_master WHERE type='table' AND name='image_import_tasks'", 0)); + + CHECK(create_v3_database(v3_path)); + CHECK(lardon3d_project_db_open(v3_path, &database, error) == LARDON3D_PROJECT_DB_OK); + CHECK(lardon3d_project_db_load_image_import(database, 9, &import_parameters) == + LARDON3D_PROJECT_DB_OK); + CHECK(strcmp(import_parameters.source_path, "/legacy/source") == 0 && + import_parameters.scanset_id > 0); + CHECK(lardon3d_project_db_legacy_catalog_pending(database, &legacy_pending) == + LARDON3D_PROJECT_DB_OK && + legacy_pending); + CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK && + task.has_task_kind); + CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact", &loaded_artifact) == + LARDON3D_PROJECT_DB_OK); + lardon3d_project_db_close(database); + database = NULL; + CHECK(query_integer(v3_path, "SELECT value FROM metadata WHERE key='schema_version'", 5)); + + CHECK(create_v3_database(failed_v4_path)); + CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V4", "1", 1) == 0); + CHECK(lardon3d_project_db_open(failed_v4_path, &database, error) == LARDON3D_PROJECT_DB_IO_ERROR); + CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V4") == 0); + CHECK(query_integer(failed_v4_path, "SELECT value FROM metadata WHERE key='schema_version'", 3)); + CHECK(query_integer(failed_v4_path, + "SELECT count(*) FROM sqlite_master WHERE type='table' AND name='scansets'", + 0)); + + CHECK(create_v4_database(v4_path)); + Lardon3DProjectDbResult v4_result = lardon3d_project_db_open(v4_path, &database, error); + if (v4_result != LARDON3D_PROJECT_DB_OK) { + fprintf(stderr, "Migration v4 (%d): %s\n", (int)v4_result, error); + } + CHECK(v4_result == LARDON3D_PROJECT_DB_OK); + CHECK(lardon3d_project_db_schema_version(database) == 5); + CHECK(lardon3d_project_db_load_task(database, 9, &task) == LARDON3D_PROJECT_DB_OK); + CHECK(lardon3d_project_db_load_artifact(database, "legacy-artifact", &loaded_artifact) == + LARDON3D_PROJECT_DB_OK); + lardon3d_project_db_close(database); + database = NULL; + + CHECK(create_v4_database(failed_v5_path)); + CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V5", "1", 1) == 0); + CHECK(lardon3d_project_db_open(failed_v5_path, &database, error) == LARDON3D_PROJECT_DB_IO_ERROR); + CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V5") == 0); + CHECK(query_integer(failed_v5_path, "SELECT value FROM metadata WHERE key='schema_version'", 4)); + CHECK(query_integer( + failed_v5_path, + "SELECT count(*) FROM sqlite_master WHERE type='table' AND name='feature_sets'", 0)); + + CHECK(unlink(artifact_path) == 0); + CHECK(unlink(database_path) == 0); + CHECK(unlink(future_path) == 0); + CHECK(unlink(corrupt_path) == 0); + CHECK(unlink(legacy_path) == 0); + CHECK(unlink(failed_migration_path) == 0); + CHECK(unlink(v2_path) == 0); + CHECK(unlink(failed_v3_migration_path) == 0); + CHECK(unlink(v3_path) == 0); + CHECK(unlink(failed_v4_path) == 0); + CHECK(unlink(v4_path) == 0); + CHECK(unlink(failed_v5_path) == 0); + CHECK(rmdir(directory) == 0); + return true; +} + +int main(void) { + return run_test() ? EXIT_SUCCESS : EXIT_FAILURE; +}