[HOWTO] Easily add unit tests to an Alire crate

Hi!

I recently found myself finding out how to do this once again, so I decided to write it down and share it here :slight_smile:

*Replace <main_crate> below with the name of your main crate.

Set up testing in a sub-crate

Create binary sub-crate tests

# Change to main crate directory
$ alr init --bin tests

# Enter name "Unit tests for <main_crate>"
# Enter other data as desired

Add dependency to main crate (in sub-crate)

# Change to `tests` sub-crate directory
$ alr with <main_crate> --use=..

Add action hook (in main crate)

# Change to main crate directory
# Apply the following patch to alire.toml:

+
+[[actions]]
+type = "test"
+command = ["alr", "run"]
+directory = "tests"

Try it out!

# Change to main crate directory
$ alr test

Since the sub-crate tests depends on the main crate, this command triggers recompilation of both and executes the test runner, which should pass/fail based on defined test cases.

Option A: Define your own testing framework

You have full freedom in defining your testing framework, just put your code under tests/src and define the test runner in tests/src/tests.adb.

However, it may be more convenient to build on what is already available (see other options below).

Option B: Use AUnit to manually define your test cases/suites/etc.

Add AUnit dependency

# Change to `tests` sub-crate directory
$ alr with aunit

Define test runner

Save the following as tests/src/tests.adb:

with AUnit.Reporter.Text;
with AUnit.Run;
with AUnit;

with My_Suite; -- This is one of our test suites

procedure Tests is
   use AUnit; -- Needed for exit status comparison

   Failed_Tests : exception;

   function Run is new AUnit.Run.Test_Runner_With_Status (My_Suite.Suite);
   Reporter : AUnit.Reporter.Text.Text_Reporter;
begin
   if Run (Reporter) /= Success then
      raise Failed_Tests; -- So that `alr test` fails
   end if;
end Tests;

Define test suite My_Suite

Save the following as tests/src/my_suite.ads:

with AUnit.Test_Suites;
package My_Suite is
   function Suite return AUnit.Test_Suites.Access_Test_Suite;
end My_Suite;

Save the following as tests/src/my_suite.ads:

with Test_Case_1; -- This is one of our test cases

with AUnit.Test_Caller;

package body My_Suite is

   package Caller_1 is new AUnit.Test_Caller (Test_Case_1.Test);

   function Suite return AUnit.Test_Suites.Access_Test_Suite is
      Ret : constant AUnit.Test_Suites.Access_Test_Suite :=
         AUnit.Test_Suites.New_Suite;
   begin
      Ret.Add_Test
        (Caller_1.Create
           (Name => "Test My_Function",
            Test => Test_Case_1.Test_My_Function'Access));
      return Ret;
   end Suite;

end My_Suite;

Define test case Test_Case_1 (simple fixture)

Save the following as tests/src/test_case_1.ads:

with AUnit.Test_Fixtures;

package Test_Case_1 is

   type Test is new AUnit.Test_Fixtures.Test_Fixture with null record;

   procedure Test_My_Function (T : in out Test);

end Test_Case_1;

Save the following as tests/src/test_case_1.adb:

with AUnit.Assertions; use AUnit.Assertions;

--  with My_Package; -- This is one of our packages

package body Test_Case_1 is

   procedure Test_My_Function (T : in out Test) is
   begin
      --  Do something with My_Package.My_Function
      Assert (True, "Failed assertion");
   end Test_My_Func;

end Test_Case_1;

Try it out!

# Change to main crate directory
$ alr test

# Read the nice log output
$ less alire/alr_test_local.log

Go further

Check out the AUnit Cookbook.

Some of the things you can do are:

  • Define Set_Up and Tear_Down procedures for each test case.
  • Define more complex test cases using AUnit.Test_Cases.Test_Case instead of AUnit.Test_Fixtures.Test_Fixture.
  • Define more test cases by simply defining packages Test_Case_2, etc. and adding them to the test suite.
  • Define more test suites by using different sets of test cases, or composing previously defined test suites.
  • Filter which tests to run using the package AUnit.Test_Filters.

Option C: Use gnattest to automatically generate test case skeletons

gnattest helps automate test generation, saving time and keeping tests in sync with the source code.

We are going to piggyback gnattest on our alire sub-crate tests, so we can still use alr test.

Install gnattest

In case you don’t have it installed yet, you can do:

$ alr install gnattest

Add AUnit dependency

# Change to `tests` sub-crate directory
$ alr with aunit

Run gnattest

To create/update the test framework, do the following:

# Change to main crate directory
gnattest -v -P<main_crate>.gpr \
    --subdirs=../tests/gnattest/src \
    --harness-dir=../../tests/gnattest/harness \
    --include-subp-name \
    --exit-status=on

This results in:

  • Test harness code under tests/gnattest/harness
  • Test skeleton code under tests/gnattest/src
  • Subprogram names in log
  • Failed alr test when any test fails/crashes (desirable)

Modify sub-crate

# Change to `tests` sub-crate directory
# Apply the following patch to alire.toml:

-executables = ["tests"]
+executables = ["test_runner"]
+project-files = ["gnattest/harness/test_driver.gpr"]

Update test framework before every test run (optional)

# Change to `tests` sub-crate directory
# Apply the following patch to alire.toml:

+
+[[actions]]
+type = "pre-build"
+command = ["gnattest", "-v", "-P<main_crate>.gpr",
+    "--subdirs=../tests/gnattest/src",
+    "--harness-dir=../../tests/gnattest/harness",
+    "--include-subp-name",
+    "--exit-status=on"]
+directory = ".."

Try it out!

# Change to main crate directory
$ alr test

# Read the nice log output
$ less alire/alr_test_local.log

Put test framework under version control

Put the following files under version control (as per gnattest docs):

  • tests/gnattest/harness/gnattest_common.gpr
  • tests/gnattest/harness/coverage_settings.mk
  • tests/gnattest/src/*test_data.adb
  • tests/gnattest/src/*tests.adb

Go further

Read the GNATtest User’s Guide.

Option D: Use both gnattest and your own AUnit tests

To use both kinds of tests, automated and handcrafted, simply pass the flag --additional-tests=tests/tests.gpr to gnattest, and place your tests under tests/src (as in option B).

Any test case derived from AUnit.Test_Fixtures.Test_Fixture will be automatically included.

Add additional tests to pre-build action

# Change to `tests` sub-crate directory
# Apply the following patch to alire.toml:

     "--harness-dir=../../tests/gnattest/harness",
+    "--additional-tests=tests/tests.gpr",

Notes

  • By default, Alire will compile your main crate with the release profile when doing alr test, as it is a dependency of the tests sub-crate (and dependencies are compiled by default with the release profile). This is convenient for unit tests, as they will run on the production code. However, it does take additional time, so alr test may be a bit slow.

Having a template create the files would be very useful here.

Having an alire command to generate/update it all would even be better.

That’s what I meant. There is already a templater inside alire apparently.

This works well for native applications, but requires a little adaption for embedded:

Since the embedded application uses a different runtime and target, you cannot alr with the embedded application in the unit test project. Instead you need to explicitly include the files that are platform independent and mock packages that cannot be compiled with a native compiler.

Of course you could also split the application into a platform independent library and a minimal application and then only unit test the library. But that is only worth it for rather large applications.

Very nice! Thanks for this. I’m going to have to give this a try, as I wasn’t aware of sub-crates, which seems like the way I’d want to set it up (but didn’t realize I could).

There seems to be some ongoing tests-related work in the Alire repo, which I was unaware of until now. Perhaps when that is finished there will be an official easy way to do this :slight_smile:

That does sound more complex indeed. I wonder if gnattest could pick up that use case and automate generation of the requisite mock packages. Otherwise, perhaps some automation could be achieved with the help of libadalang and some custom pragmas/comments.

See here

Funny, I was working on very similar document. Which is now out:

There is a bit of overlap but not that much.

Indeed, I implemented a basic test setup in Alire that basically does the subcrate thing and has a default built-in test runner. It generates the required files automatically with alr init.

It basically takes (for now) all .adb files in the subcrate’s src/ dir (and subdirs), and runs them as procedures. Supporting code can go in the common/ directory of the subcrate, and we’re also exploring having an explicit way of declaring what is a test and what isn’t in the .adb files.

This test runner is a very basic implementation, and for more advanced uses (embedded, etc) it can be replaced in the alire.toml with

[test]
command = ["a", "custom", "command"]

For now it’s only available in Alire nightly builds, but it will be in Alire 3.0 (whenever that comes out).

Hi @Seirios and @krischik,

I loved both of your Guides/Tutorials! I was thinking that maybe they could be added to the main Ada-Lang website so that anybody that may be interested can find them easily? I think it would be better to have the content there than in a forum post (though @krischik’s website is also good, but dispersed). What do you think?

Best regards,
Fer

I was thinking the same thing. Forum posts tend to disappear rather quickly, so a more permanent home would be useful.

I have been using Hugo (Markdown + Git) for my Ada-related sites for a while now. It gives me several advantages:

  • Markdown is familiar to most programmers.
  • Editing is almost as easy as a wiki, but the source is safely stored in Git.
  • Offline compilation means I never have to worry about spam or vandalism.
  • The result is static HTML that can be hosted almost anywhere.
  • Adding new authors is straightforward via Git.

Seirios’ post is already in Markdown, so it would literally take only a few minutes to turn it into a proper page. I’m happy to host it (or any other Ada tutorial) on one of my sites:

I’m also perfectly open to additional authors and contributors on any of these sites — the more good material we have, the better for the whole Ada community.

The only reason I haven’t pushed my own tutorials onto the main Ada-Lang website yet is that I prefer the “publish fast, even if not perfect” wiki-style approach. Ada-Lang is very professional and polished, which is great, but it might not be the best fit for my rapid-iteration workflow. Hugo gives me the freedom of a wiki without the risk of vandalism or sudden deletions.

What do you think? Would you like me to prepare a nicely formatted version of Seirios’ tutorial for one of the sites above?

Martin