[ANN] bbt 0.0.5

Releases
, ,
1

I am pleased to make this first announce of bbt.

bbt is a simple tool to “black box” check the behavior of an executable (hence the name, bbt stands for Black Box Tester) in command line mode.

btt special feature is that it aims at directly using your behavior description as the test script.
The description is in Markdown, structured with the classical Gherkin Given / When / Then.

Here is an example of test scenario (but I could say of behavior description) : bbt/docs/tests/examples/hello_word.md at 11dffae8e35f2ad7245c036b2063563e9e376aeb · LionelDraghi/bbt · GitHub

This documentation file is the only input for btt.
No preprocessing, no scripting, no manual adjustment, just “run the doc”.

Version 0.0.5 is available in Alire :

alr get bbt
cd bbt
alr run

I run the tests only on Linux yet, but it is supposed to be platform independant.

Any feedback is welcomed!

Lionel

7 Likes
2

That looks like a really nice and useful tool. I definitely need to try that out. BTW, your Archicheck tool also looks like a very handy tool as well.

3 Likes
3

I’ve been trying this out a bit on Windows, and it seems like a great fit to test something that I’m working on.

Is there a way to test a written file against another file? I see output against a file, but not a created file against a pre-existing file. I want to specify a very large output file outside of the Markdown and have it diff’d against a file created when a program is run.

I’d trying to do something like:

- When I run `foo`
- Then `output_file.html` is equal to `expected/reference.html`
4

Hi Paul, sounds like a reasonable expectation, I’m on it!

1 Like
5

But first of all, I’m happy to have a feedback on windows, thank you.

(especially if it works!)

1 Like
6

Done!
The code was ready for it, I just had to add a single line of code (in the grammar definition). :grinning:
And the test, obviously.

Due to the fact that there is no syntax difference between strings and file name, the keyword file must be used :

  • Then file1 is string → check that file1 contains exactly string
    versus
  • Then file1 is file file2 → check that file1 and file2 are equal

(Cf. the first Warning in bbt/docs/UG.md at e9ba55452863ce4e04e1583906b06a39d1acd6d6 · LionelDraghi/bbt · GitHub on this topic)

1 Like
7

Been using BBT for a project for a few weeks, one thing that’d be really great, would be providing a directory for all of the temporary files when I’m not letting bbt do cleanup. This would allow me to add that directory to .gitignore but still inspect program runs manually when debugging.

8

Happy to read that!

Side note : consequence of you using it seriously on Windows, I decided to be more proactive on it, and to run the Makefile on Windows.
First result was kind of discouraging, with half of the tests failing!
But most of the failures where due to :

  • one function, File_Utilities.Short_Path, that was not able to process correctly DOS formatted PATH;
  • the “.exe” at the end of the exec name;
  • app I was using on Linux for the examples, or in the make file (rpl, gcc, mlc, etc.)

I found most of those apps on windows, modify the Makefile with some lines specifics to Windows, and modify Short_Path (and more than double the unit test volume).

Now, 62 out of 65 bbt tests are OK on Windows.Two of the still failing tests contains PATH in their expected results, and I don’t know for now how to specify with bbt that the expected result is /my/path on Linux and c:\my\path on Windows. And the last failing is a test on escaping character in the command line specific to sh/bash.

Nice experience for me, I was not used to such a multi-plateform context.

Anyhow, I pushed the result on github today with the v0.0.6 tag. (and it’s on the Alire pipe)

1 Like
9

Isn’t the --exec_dir option doing the job ?

If you run

bbt -ed .bbt scenario.md

you could put .bbt in your .gitignore?

10

When I do this it changes into that directory before running any of my commands so none of my sample files I use as inputs can be found.

I really need to be able to run in one directory, and have test output files get put into a different directory.

11

I added you as a collaborator to my repo so you can see what I’m doing.

12

Yes, it’s the expected behavior : bbt/docs/features/B020_Exec_dir.md at 05a0abb3d4dc162ae3b72e8e0953c2d604319835 · LionelDraghi/bbt · GitHub

And it’s not what you’re asking for.

To understand, when you says “output files”, it’s not THE output file that is provided on the command line with “-o”?

13

Ok, thanks, I have a look.

14

After private discussion with Paul : --tmp_dir added in the latest on github

1 Like
15

Many thanks to @LionelDraghi for the quick updates! bbt has been super useful on my project.

2 Likes