About QUTest™

QUTest™ (pronounced "cutest") is a unit testing harness (a.k.a. unit testing framework), which is specifically designed for deeply embedded systems, but also supports unit testing of embedded code on host computers ("dual targeting"). QUTest™ is the fundamental tooling for Test-Driven Development (TDD) of QP/C/C++ applications, which is a highly recommended best-practice.

Note: Even though QUTest™ has been primarily designed for testing of event-driven systems, it can also be used to test any embedded C or C++ code. To demonstrate this capability, the QUTest™ comes with a few examples from the conventional Unity testing framework, which among others demonstrate such techniques as using test doubles and mocks with QUTest™.

How it works?

In a nutshell, working with QUTest™ is similar to "debugging with printf" (or sprintf or similar), where you instrument the code to output information about its execution. You then run the code with a controlled set of inputs, and examine the produced output from the printfs to determine whether the code under test operates correctly. The main differences from using printfs are: (1) that the much more efficient QP/Spy output mechanism is used instead and (2) that both generating the inputs and the checking of the test outputs are automated.

The process of testing embedded code with QUTest™ involves the following components:

The Target that runs an instrumented test fixture code, whose job is to exercise your CUT (Code Under Test). Please note that a test fixture only exercises the CUT and reports the results to the host using QP/Spy, but the test fixture does not check if the CUT operates "correctly".

NOTE: Several examples of test fixtures are explained in the QUTest Tutorial. The details of test fixtures are described in QUTest Fixture Reference.
The QSPY "back-end" application that receives the tracing data from the Target and also opens a communication channel for the QUTest "front-end".
The QUTest "front-end" that executes test scripts, which drive the tests and check the QSPY output against the expectations ("test-assertions").

NOTE: Several examples of test scripts are explained in the QUTest Tutorial. The details of test scripts are described in QUTest Script Reference.

Communication between Target, QSPY, and QUTest

Remarks: The separation between CUT execution and checking the test results has many benefits. One of the most important ones is that CUT execution (test fixture) and checking the results (test script) can be done in different programming languages. To this end QUTest™ provides support for writing the test scripts in Python.

The general QUTest™ structure just described corresponds to running tests on an embedded Target. But QUTest™ can also execute tests on the host computer. In that case (shown in the figure below), the test fixture is a host executable that communicates with the QSPY host application via a TCP/IP socket (QSPY started with the -t command-line option). In this case all QUTest™ components execute on the host computer.

QUTest with Host Executable

Remarks: To work effectively with QUTest™, you need to understand how much output to expect from any given input. Additionally, you need to understand how QUTest ensures that for every input to the test fixture, the generated QSPY output matches all test expectations and that there are no extra expectations or missing expectations after every command from the test script. These subjects are explained in the section Run-to-Completion Processing.

What's Special about QUTest™?

Unlike other existing unit testing harnesses for embedded systems (e.g., Unity or CppUTest) QUTest™ is not based on xUnit that was originally designed to run tests on host computers. Instead, QUTest™ is geared towards unit testing of deeply embedded systems. Here is a list of QUTest™ unique features, specifically designed for this purpose:

QUTest™ separates the execution of the CUT (Code Under Test) from checking of the "test assertions". The embedded target is concerned only with running a test fixture that exercises the CUT and produces QP/Spy™ trace, but it does not check the "test assertions". Checking the "test assertions" against the expectations is performed on the host computer by means of test scripts.
The QUTest™ approach is more intuitive for embedded developers, because it is conceptually like automated "debugging by printf" that most embedded developers use extensively. As it turns out, this approach also simplifies the development of all sorts of test doubles, including mocks, without breaking encapsulation of the CUT.
QUTest™ is a unique test harness on the embedded market that supports scripting. QUTest test scripts run on the Host, which skips compilation and uploading the code to the Target and thus shortens the TDD micro-cycle.

NOTE: QUTest™ supports test scripts written in Python (3.3+).

QUTest™ supports resetting the Target for each individual test, if needed. This goes far beyond providing test setup() and teardown() functions that other test fixtures offer (and of course QUTest supports as well). Clean reset of the Target avoids erroneous tests that implicitly rely on side effects from previously executed code. This is particularly important for embedded systems and for state machines, so that each test can start from a known reset condition.
QUTest™ supports testing Design by Contract (assertions in C or C++, not to be confused with "test assertions") in the CUT. This is a carefully designed, unique feature of QUTest not available in other test harnesses. A successful test of DbC might actually mean breaking an assertion in the Target code.
QUTest™ test fixtures that run on the Target do not require dynamic memory allocation (malloc()/free() in C or new/delete in C++). This means that you don't need to commit any of your precious embedded RAM to the heap (you can set the heap size to zero) and you don't need to link the heap management code. Avoiding dynamic memory allocation is one of the best practices of real-time embedded programming, which you don't need to compromise to run QUTest.
QUTest™ test fixtures that run on the Target do not require non-local jumps (setjmp()()/longjmp() in C or throw/catch in C++), which are needed by other test harnesses to discontinue failing tests. QUTest™ test fixtures do not need to discontinue failing tests, because they don't check "testing assertions", so a test fixture does not "know" if it is failing or passing. Should a test fixture crash on the Target, it simply waits for the target reset commanded by a test script.
QUTest™ test fixtures can be based on the actual application code. For example you can reuse the same main() function in a test fixture and in your final application. This means that you can either grow your test fixture into a final application through TDD, or you can more easily add unit tests to an existing application.

Note: Even though QUTest™ is particularly suitable for running tests on deeply embedded targets, it also fully supports running the same tests on your host computer (Windows, Linux, and MacOS are supported). In fact, running the tests as much as possible on the host and thus avoiding the target-hardware bottleneck is the highly recommended best-practice of embedded TDD. QUTest™ supports fully-automated unit testing, both on the embedded target and on the host computer.

Installation & Use

The qutest.py script can be used standalone, without any installation in your Python system (see Running QUTest™ below).

Note: The qutest.py script is included in the QTools™ collection. Also, the QTools™ collection for Windows already includes Python (3.8), so you don't need to install anything extra.

Alternatively, you can use *your own Python** installation, into which you can install the latest QUTest™ with pip from the PyPi index by executing the following command:

pip install qutest

Running QUTest™

If you are using QUTest™ as a standalone Python script, you invoke it as follows:

python3 <path-to-qutest-script>/qutest.py [-x] [test-scripts] [host_exe] [qspy_host[:udp_port]] [qspy_tcp_port]

Alternatively, if you've installed QView™ with pip, you invoke it as follows:

qutest [-x] [test-scripts] [host_exe] [qspy_host[:udp_port]] [qspy_tcp_port]

Command-line Options

-x - optional flag that causes qutest to exit on first test failure.
test_scripts - optional specification of the Python test scripts to run. If not specified, qutest will try to run all *.py files in the current directory as test scripts
host_exe | "" | DEBUG - optional specification of the test-fixture compiled for the host (host executable) for testing on the host computer. The placeholder value "" (empty string) can be used for running test fixtures on an embedded Target. The special value DEBUG means that qutest will run in the "debug mode", in which it will NOT launch the host executables and it will wait for the Target reset and other responses from the Target. If host_exe is not specified, an embedded target is assumed (which is loaded with the test fixture already).
qspy_host[:udp_port] - optional host-name/IP-address:port for the host running the QSPY host utility. If not specified, the default is 'localhost:7701'.
tcp_port- optional the QSpy TCP port number for connecting host executables. If not specified, the default is 6601.

Note: The command-line options for qutest.py are positional, meaning that an option must be at the specific position in the command-line to be parsed correctly. For example, if you wish to run QUTest on an embedded target at the specified [qspy_host] host-name, you still need to provide a placeholder for the host executable [host_exe] ("") option, to get to the right position for the [qspy_host] option.

Examples

Windows Hosts

python3 %QTOOLS%\qutest\qutest.py

runs all the test scripts (*.py) in the current directory.

python3 %QTOOLS%\qutest\qutest.py *.py

runs the test scripts (*.py) in the current directory.

python3 %QTOOLS%\qutest\qutest.py *.py build\dpp.exe

runs the test scripts (*.py) in the current directory and uses the host executable: build\dpp.exe

python3 %QTOOLS%\qutest\qutest.py *.py "" 192.168.1.100:7705

runs the test scripts (*.py) in the current directory, without a host executable: (""), and connects to QSPY at 192.168.1.100:7705

qutest *.py build\dpp.exe 192.168.1.100:7705

runs "qutest" (installed with pip) to execute the test scripts (*.py) in the current directory, uses the host executable: build\dpp.exe, and connects to QSPY at 192.168.1.100:7705

qutest *.py build\dpp.exe localhost:7701 6605

runs "qutest" (installed with pip) to execute the test scripts (*.py) in the current directory, uses the host executable: build\dpp.exe, and connects to QSPY at localhost:7701, using the local UDP port 6605

Linux/MacOS Hosts

python3 $(QTOOLS)/qutest/qutest.py

runs all the test scripts (*.py) in the current directory.

python3 $(QTOOLS)/qutest/utest.py *.py

runs the test scripts (*.py) in the current directory.

python3 $(QTOOLS)/qutest/qutest.py *.py build/dpp

runs the test scripts (*.py) in the current directory and uses the host executable: build/dpp

python3 $(QTOOLS)/qutest/utest.py *.py "" 192.168.1.100:7705

runs the test scripts (*.py) in the current directory, without a host executable: (""), and connects to QSPY at 192.168.1.100:7705

qutest *.py build/dpp 192.168.1.100:7705

runs "qutest" (installed with pip) to execute the test scripts (*.py) in the current directory, uses the host executable: build/dpp, and connects to QSPY at 192.168.1.100:7705

qutest *.py build/dpp localhost:7701 6605

runs "qutest" (installed with pip) to execute the test scripts (*.py) in the current directory, uses the host executable: build/dpp, and connects to QSPY at localhost:7701, using the local UDP port 6605

Next: QUTest™ Tutorial