Skip to content

Latest commit

 

History

History
37 lines (27 loc) · 5.73 KB

TODO.md

File metadata and controls

37 lines (27 loc) · 5.73 KB

TODO

Known gaps, not scheduled yet. Remove an item once it is done.

EcoTaxa interactions: robustness

Review of 23 Oct 2026, after the Explore branch. The Explore reads are protected (stats kept in sample_ecotaxa_stats and refreshed at most once a day, at most 2 EcoTaxa calls in flight, batches of 50, partial answers, see src/domain/use-cases/explore/explore-ecotaxa.ts). The points below concern every EcoTaxa interaction and existed before.

  1. Generic account: single point of failure, not monitored. Every import, read, export and deletion uses the generic EcoTaxa account of the instance. Its expiry check is commented out (getEcotaxaGenericAccountForInstance, src/domain/repositories/ecotaxa_account-repository.ts), nothing renews its token and nothing warns when it dies: imports then fail with an EcoTaxa error, the Explore EcoTaxa figures stay empty and the deletions skip EcoTaxa silently. To do: a periodic check (api/users/me per instance) with its status in the admin pages and an alert, and a written renewal procedure.

  2. No reconciliation between the two applications.

    • Deleting an EcoPart project ignores an EcoTaxa failure (delete-project.ts): the objects stay orphaned in EcoTaxa. Since 29 Oct 2026 the deletion runs as a task and the failure is written in its log and result (WARNING); a sample deletion keeps the sample instead. Nothing retries the EcoTaxa side afterwards.
    • Objects deleted in EcoTaxa leave the sample marked as imported in EcoPart.
    • An EcoTaxa import interrupted by an EcoPart restart lets the EcoTaxa job finish without the sample being marked imported; relaunching uploads the same files again.
    • To do: have the daily Explore sweep also detect samples that no longer exist in EcoTaxa, and record the EcoTaxa job id of an import so that a recovered task can check its outcome.
  3. Import files never cleaned up on the EcoTaxa side. Each import uploads the whole archive (TSV and images) into the generic account's file space (from_EcoPart/<project_id>, importEcoTaxaSamplesInEcoTaxa) and never removes it, which fills the EcoTaxa disk over time. To do: delete the uploaded files once the import job has ended.

  4. No per-client limit on the open Explore routes. Every /explore route is served to anonymous users. On /explore/ecotaxa and /explore/images the shared throttle protects EcoTaxa, but a client looping on them keeps its two slots busy and slows the EcoTaxa part of Explore for everyone; /explore/graphs reads the particle and CTD files of up to 100 samples per request, a cache being the only protection. To do: a rate limit per address or session on these routes (see the rate limits in src/main.ts).

  5. Depth offset changes are not propagated to EcoTaxa. The original spec asks that editing the project depth offset update the object depths of the linked EcoTaxa project. Not done and not decided: EcoPart adds the offset when it reads EcoTaxa depths (Explore images) and leaves the EcoTaxa objects as imported.

Structural, to keep in mind (not fixable in EcoPart alone):

  • The generic account is manager of every linked EcoTaxa project: a compromised EcoPart server would have that right everywhere. EcoTaxa has no read-only service role for this case.
  • Legacy samples are matched to EcoTaxa samples by name (orig_id, migration of EcoTaxa projects and id resolution): a name that differs between the two applications breaks the link.

Explore

  • EcoTaxa concentrations (/explore/graphs/ecotaxa): the descent filter and the first and last image of the particle settings are not applied to the objects yet, every object with a depth is counted while the water volume only counts the images kept (as legacy EcoPart did). The front card says so. Found on 19 Nov 2026, to decide before coding:
    • UVP6: exact. The EcoTaxa orig_id of an object is <image id>_<n>, the image id of the pivot (checked on HS1447C1: the 858 objects find their image in particules.csv at the same depth, the filter drops 470 of 62,434 images and 11 objects). The pass of BinnedSampleBuilder that gives the volumes can tell which images are kept; the objects would then be read with obj.orig_id.
    • UVP5: no link to the image in EcoTaxa (orig_id is the vignette number, objtime the start of the profile for every object). The vignettes are numbered in acquisition order and their depth never decreased on the 4 local omer2 samples, whose datfile goes up and down: Zooprocess seems to cut the descent only. Proposal: the same running-maximum rule on the vignettes in their order. To check on real UVP5 projects of the production EcoTaxa first.
  • EcoTaxa concentrations of time series: not drawn (the card says "coming soon"). Legacy EcoPart computed a 1 h histogram from objdate + objtime (GenerateTaxonomyHistogram) but never drew it.
  • Access request endpoint: the front opens a mailto: to the managers for now.
  • "Export selection" button of the exportable tab: not wired to the export routes yet.
  • Renaming / grouping of EcoTaxa categories (modal of the mockups).
  • Samples imported before the ISO conversion keep their UVP date formats (20120520080213, 20230406-144332-820). Explore reads them (EXPLORE_SAMPLING_TIME in sqlite-sample-data-source.ts), but the oldest-sample date of the privacy delays ignores them (OLDEST_SAMPLE_UTC_DATE_TIME_SQL), so a project with only such samples stays private. A migration converting them to ISO would align both.

Front

  • Project deletion overlay (front ProjectDetailsPage, since front v0.0.47): the deletion now runs as a DELETE_PROJECT task that the page follows until it ends (29 Oct 2026). Check whether the overlay still earns its place, or whether a redirect to the task page (global /tasks/:id) with a snackbar would do, since a long deletion keeps the user on a veiled page.