[HOWTO] Easily add unit tests to an Alire crate

General
,
1

Hi!

I recently found myself finding out how to do this once again, so I decided to write it down and share it here :slight_smile:

*Replace <main_crate> below with the name of your main crate.

Set up testing in a sub-crate

Create binary sub-crate tests

# Change to main crate directory
$ alr init --bin tests

# Enter name "Unit tests for <main_crate>"
# Enter other data as desired

Add dependency to main crate (in sub-crate)

# Change to `tests` sub-crate directory
$ alr with <main_crate> --use=..

Add action hook (in main crate)

# Change to main crate directory
# Apply the following patch to alire.toml:

+
+[[actions]]
+type = "test"
+command = ["alr", "run"]
+directory = "tests"

Try it out!

# Change to main crate directory
$ alr test

Since the sub-crate tests depends on the main crate, this command triggers recompilation of both and executes the test runner, which should pass/fail based on defined test cases.

Option A: Define your own testing framework

You have full freedom in defining your testing framework, but it may be more convenient to build on what is already available (see other options below).

Option B: Use AUnit to manually define your test cases/suites/etc.

Add AUnit dependency

# Change to `tests` sub-crate directory
$ alr with aunit

Define test runner

Save the following as tests/src/tests.adb:

with AUnit.Reporter.Text;
with AUnit.Run;
with AUnit;

with My_Suite; -- This is one of our test suites

procedure Tests is
   use AUnit;

   Failed_Tests : exception;

   function Run is new AUnit.Run.Test_Runner_With_Status (My_Suite.Suite);
   Reporter : AUnit.Reporter.Text.Text_Reporter;
begin
   if Run (Reporter) /= Success then
      raise Failed_Tests;
   end if;
end Tests;

Define test suite My_Suite

Save the following as tests/src/my_suite.ads:

with AUnit.Test_Suites;
package My_Suite is
   function Suite return AUnit.Test_Suites.Access_Test_Suite;
end My_Suite;

Save the following as tests/src/my_suite.ads:

with Test_Case_1; -- This is one of our test cases

with AUnit.Test_Caller;

package body My_Suite is

   package Caller_1 is new AUnit.Test_Caller (Test_Case_1.Test);

   function Suite return AUnit.Test_Suites.Access_Test_Suite is
      Ret : constant AUnit.Test_Suites.Access_Test_Suite :=
         AUnit.Test_Suites.New_Suite;
   begin
      Ret.Add_Test
        (Caller_1.Create
           (Name => "Test My_Function",
            Test => Test_Case_1.Test_My_Function'Access));
      return Ret;
   end Suite;

end My_Suite;

Define test case Test_Case_1 (simple fixture)

Save the following as tests/src/test_case_1.ads:

with AUnit.Test_Fixtures;

package Test_Case_1 is

   type Test is new AUnit.Test_Fixtures.Test_Fixture with null record;

   procedure Test_My_Function (T : in out Test);

end Test_Case_1;

Save the following as tests/src/test_case_1.adb:

with AUnit.Assertions; use AUnit.Assertions;

package body Test_Case_1 is

   procedure Test_My_Function (T : in out Test) is
   begin
      Assert (True, "Failed assertion");
   end Test_My_Func;

end Test_Case_1;

Try it out!

# Change to main crate directory
$ alr test

# Read the nice log output
$ less alire/alr_test_local.log

Go further

Check out the AUnit Cookbook.

Some of the things you can do are:

  • Define Set_Up and Tear_Down procedures for each test case.
  • Define more complex test cases using AUnit.Test_Cases.Test_Case instead of AUnit.Test_Fixtures.Test_Fixture.
  • Define more test cases by simply defining packages Test_Case_2, etc. and adding them to the test suite.
  • Define more test suites by using different sets of test cases, or composing previously defined test suites.
  • Filter which tests to run using the package AUnit.Test_Filters.

Option C: Use gnattest to automatically generate test case skeletons

gnattest helps automate test generation, saving time and keeping tests in sync with the source code.

We are going to piggyback gnattest on our alire sub-crate tests, so we can still use alr test.

Install gnattest

In case you don’t have it installed yet, you can do:

$ alr install gnattest

Add AUnit dependency

# Change to `tests` sub-crate directory
$ alr with aunit

Run gnattest

To create/update the test framework, do the following:

# Change to main crate directory
gnattest -v -P<main_crate>.gpr \
    --subdirs=../tests/gnattest/src \
    --harness-dir=../../tests/gnattest/harness \
    --include-subp-name \
    --exit-status=on

This results in:

  • Test harness code under tests/gnattest/harness
  • Test skeleton code under tests/gnattest/src
  • Subprogram names in log
  • Failed alr test when any test fails/crashes

Modify sub-crate

# Change to `tests` sub-crate directory
# Apply the following patch to alire.toml:

-executables = ["tests"]
+executables = ["test_runner"]
+project-files = ["gnattest/harness/test_driver.gpr"]

Update test framework before every test run (optional)

# Change to `tests` sub-crate directory
# Apply the following patch to alire.toml:

+
+[[actions]]
+type = "pre-build"
+command = ["gnattest", "-v", "-P<main_crate>.gpr",
+    "--subdirs=../tests/gnattest/src",
+    "--harness-dir=../../tests/gnattest/harness",
+    "--include-subp-name",
+    "--exit-status=on"]
+directory = ".."

Try it out!

# Change to main crate directory
$ alr test

# Read the nice log output
$ less alire/alr_test_local.log

Put test framework under version control

Put the following files under version control:

  • tests/gnattest/harness/gnattest_common.gpr
  • tests/gnattest/harness/coverage_settings.mk
  • tests/gnattest/src/*test_data.adb
  • tests/gnattest/src/*tests.adb

Go further

Read the GNATtest User’s Guide.

Option D: Use both gnattest and your own AUnit tests

To use both kinds of tests, automated and handcrafted, simply pass the flag --additional-tests=tests/tests.gpr to gnattest, and place your tests under tests/src (as in option B).

Any test case derived from AUnit.Test_Fixtures.Test_Fixture will be automatically included.

Add additional tests to pre-build action

# Change to `tests` sub-crate directory
# Apply the following patch to alire.toml:

     "--harness-dir=../../tests/gnattest/harness",
+    "--additional-tests=tests/tests.gpr",

Notes

  • By default, Alire will compile your main crate with the release profile when doing alr test, as it is a dependency of the tests sub-crate (and dependencies are compiled by default with the release profile). This is convenient for unit tests, as they will run on the production code. However, it does take additional time, so alr test may be a bit slow.
2

Having a template create the files would be very useful here.