Consider tools for multi-configuration testing

[nettle]

Describe the task

We need to support multi-configuration testing

• Bazel versions 6, 7 and 8
• CodeChecker versions 6.25, 6.26, 6.27
• probably different Python versions
• might be different Clang and Clang-Tidy versions
• It would be also good to install and use dev tools like pylint, pycodestyle, buildifier

Scope

1. Define requirements for multi-configuration testing
2. Consider different tools
3. Discuss pros/cons, benefit and limitations
4. Suggests ways forward: tools, approaches
5. Agree on tools, approaches

Definition of Done

Discussed and agreed on tools and approaches for multi-configuration testing

[nettle]

Tools to consider for multi-configuration testing

Tool	Pros & Cons
Mise	✅ It does work, see #132 ✅ Language agnostic ✅ Supports multiple backends/repos (including github, conda, pip) ✅ Supports multiple versions in one configuration ✅ Easy to use ✅ Relatively fast ⛔ Often timeouts ⛔ Unstable, many bugs
UV	Not evaluated... ⛔ Initially it is for Python
Conda	⛔ Quite heavy
Micromamba	✅ It does work, see #167 ✅ Language agnostic ✅ Relatively easy to use ✅ Relatively stable ✅ Relatively fast in CI ⛔ Mess with environments (possible?) ⛔ Slow environments
Nix	Not evaluated... ✅ Language agnostic ⛔ Requires root?

[nettle]

Hi @furtib, @Szelethus,
I've tried to play with Mise (#132) and MicroMamba (#167). Both work. I can do demo.

• ✅ Mise is more flexible and easier to use, but buggy.
• ✅ MicroMamba seems more stable and faster in CI, but not that flexible as Mise.
• ⛔ Classical Conda is too heavy for us I think
• ❔UV, Nix, somthing else - not tried

Please share your thoughts

[furtib]

I have played around with Nix a bit. (I couldn't get it to run tests successfully) It seems to be really robust/stable.
(https://github.com/furtib/codechecker_bazel/tree/nix-testing run with nix flake check -L --impure --option sandbox false)
BUT nix by default manages everything in a sandbox with no network connection, so we either disable the sandbox (everything after -L), or prefetch bazel dependencies(?). Also, Nix doesn't seem to have a newer codechecker than 6.24 or a Bazel older than 7. So in summary: Its more trouble than it's worth.

I have only taken a look at Mise after it was working, so I didn't see much of its "buggy" side. I don't have any ill feelings towards it.

I have tried MicroMamba; if I understand right, it does not provide that test-running capability as Mise. If I understand correctly, we will need to create a separate yaml file for each configuration and have a script loop over them. From a CI standpoint, I would love the faster startup. I think we may have already run out of free minutes for this month. I have since found a tool named nox, which can use the mamba backend to run the tests similarly to Mise.

I don't know much about conda, but there seems to be a plugin (Tox, well it's not really a plugin, but it has its own conda integration) to run tests in a list of environments. At first glance, it seems pretty promising too. Conda seems like an unviable option due to license problems.
Micromamba seems to fall under the conda umbrella. Are we sure its license does allow us to use it?

[nettle]

I have played around with Nix a bit. (I couldn't get it to run tests successfully) It seems to be really robust/stable. (https://github.com/furtib/codechecker_bazel/tree/nix-testing run with nix flake check -L --impure --option sandbox false) BUT nix by default manages everything in a sandbox with no network connection, so we either disable the sandbox (everything after -L), or prefetch bazel dependencies(?). Also, Nix doesn't seem to have a newer codechecker than 6.24 or a Bazel older than 7. So in summary: Its more trouble than it's worth.

Good to know. Thanks for sharing @furtib!
I think we can skip Nix so far.

I have only taken a look at Mise after it was working, so I didn't see much of its "buggy" side. I don't have any ill feelings towards it.

Well, I see some problems with Mise. First of all it requires PAT (Personal Access Token) when working with GitHub,
which is actually GitHub problem, not Mise, but MicroMamba fetches packages from https://prefix.dev and does not need PAT.
Then, Mise is failing with timeout often.
Does not work well with task environments etc.

I have tried MicroMamba; if I understand right, it does not provide that test-running capability as Mise. If I understand correctly, we will need to create a separate yaml file for each configuration and have a script loop over them. From a CI standpoint, I would love the faster startup. I think we may have already run out of free minutes for this month.

Right, MicroMamaba is not that flexible, but might be better alternative for CI.
And that's what I'm looking for too...

I have since found a tool named nox, which can use the mamba backend to run the tests similarly to Mise.

Thanks interesting! I will check: https://github.com/wntrblm/nox
Thanks for sharing!

I don't know much about conda, but there seems to be a plugin (Tox, well it's not really a plugin, but it has its own conda integration) to run tests in a list of environments. At first glance, it seems pretty promising too. Conda seems like an unviable option due to license problems. Micromamba seems to fall under the conda umbrella. Are we sure its license does allow us to use it?

Yes, Micromamba is the lightest tool from Conda family (Conda, Miniconda, Mamba, Micromamba).
Conda was originally developer for Python, but now supports more than Python.
As to license I see no problem it is BSD-3-Clause (see https://github.com/mamba-org/mamba).
Besides we just use this tool for dev environment.

[nettle]

Meanwhile I'm thinking about something like:

├── .ci
│   ├── mise
│   ├── micromamba
│   ├── jenkins
│   └── something-else
├── .github
│   └── workflows
├── src
└── test

What do you think?

[furtib]

I like the directory structure; it makes things much cleaner.
I wish .github could be placed under .ci too.

I actually tried Nox myself, just opened a PR with it (#185, this really is just a show it can do it PR, I haven't thought about details that much). I found it very customizable and easy to work with.

Under license troubles, I meant the Anaconda TOS. I would link it here, but I can't even open their website, hence my lack of knowledge about the issue at hand.
My understanding is that as long as we use the conda-forge repository, we are in the clear.