I know of the existence of aunit, ahven, gnattest, but my experience is anecdotal and outdated. Before doing my own research, maybe some of you already have the answer ready.
I’m looking for something that generates the boilerplate (harness project? and test stubs) and is able to find new subprograms in subsequent generation runs without botching existing tests.
Ada for the tooling is not mandatory, as long as the generated test stubs are in Ada. Not sure if such a beast exists though…
Trendy Test doesn’t use any generated code, but instead hijacks the exception system to handle registration and running, relying on GNAT unused procedure warnings to ensure that all tests get added to the test pools.
The most likely way to get auto-registration to work as-is would be to write a test framework that follows what Criterion does. It uses macros to write test functions to a special separate linker section and then read that out of the ELF or COFF at test startup time to effectively give it the ability to dynamically determine the list of all tests to run. This relies on having appropriate binary analysis to look at either of these file types at startup.
Thinking about this some more, you wouldn’t even have to parse the elf/coff, if you made a custom pragma for unit test registration (similar to #[test] for Rust, iirc, C# and F# have something similar), the implementation should ignore it, so you could write a script to extract of these, and dump them into an entry procedure (with some wrapper for catching exceptions and reporting). I had thought of pitching an RFC a while ago for a Unit_Test attribute, but thought it might be too specialized.
pragma Unit_Test (Test_Function);
-- Maybe it should be an function instead, idk.
procedure Test_Function is begin ... end;
Frankly, before I reach the point of registering custom tests, I’d like to have automatically registered tests for every visible subprogram that XFAIL until I fill them.
Just curious: in what domains are people actually using extensive unit tests (as opposed to functional tests or integration tests that test whole packages or projects) ? I have started a few times with the intention of having unit tests for a library or something, but it generally becomes quite visible that this is a major test that’s not worth the trouble in general. Also it somewhat deter from refactoring the code and split it into more subprograms.
For me it depends on what is being developed. I work on a lot of custom message protocols, so when we have a brand new one, I do heavy unit testing on them to ensure there are no bugs in the implementation (and to some extent catch any conflicts in the design). If it’s a more vetted system, I’ll rely only on functional testing.
On my hobby projects: The NES emulator that I am putzing around with has a ton of unit tests for the CPU. I wanted to make darn sure my instruction execution was as sound as possible as I figured it was the hardest place to debug later on once I integrated the other parts. Now that I am past that part I’ll probably rely more on functional testing.
I find a good balance works well for me. I don’t know of any good tools for Mosteo though. I’ve always rolled my own, which is time consuming.
EDIT: I guess if someone was really feeling froggy they could make an application based on libadalang that just parsed files and looked for library level function / procedure declarations and generate the stubs. Might be a fun project for someone.
I don’t know about libadalang, but I could probably do it in Raffle. I was planning on doing this to be able to automatically make Lua bindings for packages, and also to make better searchable docs based on spec files, like “I have a Foo, what subprograms return a Foo and which take a Foo as a parameter?”
Oh, I do unit tests differently where I focus tests on cases, not necessarily specific subprograms. I see what you mean now.
My experience has also been that full end-to-end tests are much more powerful and long-lived. BBT has been good for this. Unit tests are great for “did I write this thing correctly?” but I also very liberally use Pre/Pre and constraints and run with all checks on.
In my case is small focused libraries with relatively few public subprograms. I was thinking that it would be better to start with these tests rather than with an ad-hoc, haphazard collection of tests (which is what I usually end doing).
But it may well turn out that what I want to try is not really practical.
My three small open source app (acc, smk and bbt) are CLI oriented, and easy to end-to-end test.
I have very very rare needs for unit testing. Last example is a function to compute a relative shortest path from, lets say, /home/lionel/bin/x to /home/lionel/y/z : taking into account the Windows drive, Windows vs Unix separators, etc.
This easily lead to many test cases.
So there is one unit test, not even for a package, just for this procedure.
I noticed that in CLI, there is often commands or options that do not execute the full process the app is intended to run, and activate only a small part of the code. For example dry-run options, or command to list input files recursively found by the app.
Those command or options sometimes offers a way to have a stable end-to-end test on a limited scope, that would have been a integration or a unit test otherwise.
Going further, I don’t exclude if the need arises to cheat, and have a secret option exposing some stable internal data only for test purposes.
Using those internal data in (false) end-to-end tests, depending on the command line interface and not on the package/procedure profiles, seems to me less prone to change when the code evolves, and do not deter refactoring.
All in all, I have 98% end-to-end tests, 2% unit testing, no integration tests.
My small CLI utilities seems to me an easy case for testing, things would probably be not so simple on larger app, server, app with GUI, app with high coverage requirement, etc.
We have an asserts package (a generic of course), based on GNATCOLL.Asserts. Much nicer to use than a straight pragma Assert, because in case of errors you see both sides of the “=” (or any other operator). Also has support for retrying (for those times when you are dealing with asynchronous things), comparing arrays,… I am sure a lot of people have similar packages, or maybe use directly GNATCOLL.Asserts… @Fabien.C maybe Cesar could also gather ideas and later improve that package to make writing tests even nicer.
I just use AUnit and just write my own. Yes, AUnit needs a lot of boilerplate but it’s mostly copy paste and I wrote my own extensions to make it more bareable.
That would indeed be nice. There is a reason all other unit test frameworks I know of do it that way.
It do. It’s called test driven development where you write the test first and the the method. But more importantly: It’s not either or. You can do both.
Which is what I suggest for Android. With the unit test designed to run without the actual device. Since testing on device is significantly slower.
I too have the contracts active when running unit tests and it did indeed find problems I would not have found otherwise.
I hope that won’t break my AUnit tests I already have in my Alire crates and already run by the Alire build server.
I posted a first draft of what I had in mind for a built-in testing feature, on the Alire github repository. This first draft is intended to collect ideas and feedback, in order to build something that would be a good fit for actual crates in the wild so feel free to comment and add ideas!
Following on my RFC, I implemented a demo of what I had in mind for a default test runner in alire.
The point is still to have a minimal runner in alire, with opinionated-ish defaults (but easy to replace with a different runner whenever those choices don’t align with a project’s specific needs), and I’m still very much looking for feedback
In this draft PR there’s just the runner, with no manifest integration yet. That will come soon, I hope
so feel free to comment and add ideas!