[Issue]: Versioning conformance tests

[DragaDoncila]

Describe the issue

@clbarnes and other folks, I wanted to open this issue to discuss versioning for the conformance tests and testing framework.

However we choose to package/distribute the tests for downstream consumption, we'll want to make sure that:

• the conformance package version and spec version can be kept in sync
• the conformance package version and spec version can be changed independently

In GEFF we use a hybrid versioning scheme consisting of 5 numbers. For the conformance testing package, this could look like vX.Y.Z.A.B where vA.B would be the highest supported version of the spec. The numbers X.Y.Z could be bumped according to semver, effver, or any other suitable versioning scheme.

Let's say we cut a new version of this conformance package (say "v0.1.0") for spec release v0.6. Under the hybrid versioning scheme, this could be v0.1.0.0.6. If we realized there was a bug in the package, we could release v0.1.1.0.6. If version 0.7 of the spec is released, we'd probably be adding new features to the tests, so we could release v0.2.0.0.7... etc.

We'll have to think about what that looks like with v0.4 and v0.5 re. "the highest supported version".

[clbarnes]

Would you want to have a single conformance test package which has all the OME-Zarr versions' test cases in? So

conformance/
├── script.py
└── cases/
    ├── v0.4/
    │   ├── attributes/
    │   │   ├── valid/
    │   │   │   ├── case1.json
    │   │   │   └── case2.json
    │   │   └── invalid/
    │   │       ├── case3.json
    │   │       └── case4.json
    │   └── zarr/
    │       ├── valid/
    │       │   ├── case5.ome.zarr/
    │       │   └── case6.ome.zarr/
    │       └── invalid/
    │           ├── case7.ome.zarr/
    │           └── case8.ome.zarr/
    ├── v0.5/
    │   └── ...
    └── v0.6/
        └── ...

as opposed to implementors pulling in different versions of the conformance tests:

conformance/
├── script.py
└── cases/
    ├── attributes/
    │   ├── valid/
    │   │   ├── case1.json
    │   │   └── case2.json
    │   └── invalid/
    │       ├── case3.json
    │       └── case4.json
    └── zarr/
        ├── valid/
        │   ├── case5.ome.zarr/
        │   └── case6.ome.zarr/
        └── invalid/
            ├── case7.ome.zarr/
            └── case8.ome.zarr/

I think option 1 could be a better fit if the conformance tests are to be packaged outside of ngff-spec, and option 2 is definitely a better fit if they are to be kept in this repo.

[DragaDoncila]

In my head you should be able to toggle which versions of the spec you're testing against without having to change versions of your package.

I do see the value of strictly mapping conformance tests to spec versions using specific commits, I'm just not sure it's as practical or flexible.

I also think the conformance test cases + infrastructure might grow in scope/scale, and having it in this repo could become unwieldy?

Wdyt?

[clbarnes]

That all makes sense to me, and makes it easier to fix past versions' tests if they're found to be broken, and/or update the testing infrastructure if we decide we prefer different patterns.