bbt is the most effective solution I’m aware of for documenting and testing CLI software.
As a reminder, the principle is to describe your examples / scenarios in near-natural language by using the usual Gherkin framework (Given/When/Then) and bbt syntax for the steps.
And then you just run your scenarios with (for example) bbt my_features.md, meaning that the same file is both your documentation and the test script, and you get both for the price of one.
Version 0.3.0 brings some more user assistance, including:
A bbt explain command that clearly shows what bbt understands from the submitted file and what will be executed / tested;
An integrated tutorial accessible with bbt help tutorial;
Generation of an example with bbt help example;
A complete grammar with an example for each recognized step syntax via bbt help grammar.
There are also some new syntax features, such as “when I run x or y”:
# Scenario: testing command line help
- When I run `awk` or `awk --help`
- Then the output contains `Usage: awk etc.`
Or the ability to execute a command in preconditions or postconditions, as in:
# Example: repairing an xml file
- Given that `xmllint my.xml` fails
- When I run `repair --auto my.xml`
- Then I successfully run `xmllint my.xml`
I started using it to validate macOS software (antimalware daemon) in my day job. The biggest problems I’ve found is that I need to write several shell scripts that take care of things bbt cannot do natively because it doesn’t launch programs as a shell.
So far, I’m very happy about it, even though I have to rely on those scripts to do the heavy lifting.
## Feature : Command line
bbt is using quotes for joining arguments as the shell, and the quotes are
not passed to the executed program.
### Scenario : command with quoted arguments
- When I run `sh -c "if true; then echo True; fi"`
- Then I get no error
- And Output is `True`
$ bbt test.md
# Document: [test.md](test.md)
## Feature: Command line
### Scenario: [command with quoted arguments](test.md):
- OK : When I run `sh -c "if $(true); then echo True; fi"`
- OK : Then I get no error
- OK : And Output is `True`
- [X] scenario [command with quoted arguments](test.md) pass
## Summary : **Success**, 1 scenarios OK
## Scenario: trying to run the same script created with the executable attribute should succeed
- Given the new executable file `cmd2.sh` containing
```
#!/bin/sh
echo "bbt rules!"
```
- When I run `./cmd2.sh`
- Then I get `bbt rules!`
That said, this is not more portable than what was suggested by Manuel.
Side note: I realize now that attributes are not documented !
For the record, there is two attributes for now:
executable so that the created file has +x attributes
unordered so that result is OK even if lines in the output are not in the expected order
- Then output contains file `Cities` (unordered)
I was envisioning to extend this list of atributes, for example to ignore case or blanks on file compare. It’s already possible thanks to command line options, but that goes against the spirit of the tools : the single source of truth is the doc, so the expected behavior must be in the doc, not on the command line. If you have good names to suggest…
Does it implement the full gherkin/cucumber feature set? I got stuck ages ago when doing the ray tracer challenge book because a certain feature wasn’t implemented in GNATBDD. I’d like to go back to redo it and complete it.
Pending action on my side, describing the differences with Gherkin!
Mostly:
Some Gherkin features are not implemented in bbt:
table input (Scenario Outlines / Examples).
the new “Rule” keyword (I don’t know what it is for)
Some differences :
the file structure of gherkin files is simple: one feature per file, optionally a background, and some scenarios. bbt is more flexible, zero, one or more features, one optional background at document level and one per feature;
in Gherkin, tags are words starting with “@” put just below the scenario title. With bbt there is no special marker, whatever string may be a tag. You can use @windows if you want.
(in bbt tests, to specify different file path format output, I use Windows_Only and Unix_Only as tag)
Some features are specific to bbt:
for example the ability to specify a common output on several commands, within a single scenario:
when I run add "1+2" or add "2+1" or add "0+3"
then i get 3
But the most important difference is in the use:
gherkin is a doc format, it has no defined semantic within scenarios: to run stuff, you have to write code.
bbt is dedicated to cli black box testing, you do not have to write code, but you cannot use it for unit or integration or GUI testing.
Yes I can and I’m using (abusing) it for integration testing. But I have to use several shell scripts to do part of the hard work. I don’t particularly like that workflow, but I do like having an md file that acts as documentation for a specific feature and tests for it to validate it.
I suppose canvas and canvas_to_ppm are functions from your code?
This illustrate what GNATBDD, Cucumber, behave, etc. are good at. bbt wasn’t design for this.
Here you could twist things to use it, because the output PPM P3 is a text format, but it requires to also have the ability to pass the canvas to your exe on the command line, or in a text file. It would then looks like:
# Scenario: Constructing the PPM header
- given the file `canvas.txt`
```
height=5
width=3
```
- when I run `myraytracer canvas.txt`
- then the output contains
```
P3
5 3
255
```
(Note also that bbt has no “start with” semantics for now.)
