Remarks: This simple example runs QUTest tests on the host (Windows, Linux, or MacOS) for both Unity and QUTest. The QUTest version also runs on embedded boards (TivaC LaunchPad from Texas Instruments and EFM32 Pearl-Gecko board from Silicon Labs). The instructions for building and running the code on the embedded boards are located at the end of this lesson.

Code Under Test (CUT)

Note: Because it uses pure C code, the Basic Unity example is only available in the QP/C framework (and is not available in the QP/C++ framework).

The CUT in this example is the file ProductionCode.c located in the directory C:\qp\qpc\examples\qutest\unity_basic\src. The CUT contains just two functions with some fairly obvious errors. The purpose of the example is to find these errors by unit-testing this "production code":

  
 #include "ProductionCode.h"
  
 int Counter = 0;
 int NumbersToFind[9] = { 0, 34, 55, 66, 32, 11, 1, 77, 888 }; /* some obnoxious array to search that is 1-based indexing instead of 0. */
  
 /* This function is supposed to search through NumbersToFind and find a particular number.
  * If it finds it, the index is returned.  Otherwise 0 is returned which sorta makes sense since
  * NumbersToFind is indexed from 1.  Unfortunately it's broken
  * (and should therefore be caught by our tests) */
 int FindFunction_WhichIsBroken(int NumberToFind)
 {
     int i = 0;
     while (i < 8) /* Notice I should have been in braces */
         i++;
         if (NumbersToFind[i] == NumberToFind) /* Yikes!  I'm getting run after the loop finishes instead of during it! */
             return i;
     return 0;
 }
  
 int FunctionWhichReturnsLocalVariable(void)
 {
     return Counter;
 }

Running the Test with Unity

The complete code for the basic Unity example is provided in the QP/C framework, directory C:\qp\qpc\examples\qutest\unity_basic\test_unity. To run the basic Unity test (on Windows), open a command-prompt and type:

cd C:\qp\qpc\examples\qutest\unity_basic\test_unity
make

Note: The provided Makefile will also work on Linux and MacOS. The only difference from Winows is that you open a terminal window and change directory to ~/qp/qpc/examples/qutest/unity_basic/test_unity.

This will build the test fixture as a host executable and then it will run it. The screen shot below shows the output produced from the make command.

Unity example_1 test build and run with Unity

Remarks: As you can see, two out of five Unity tests fail. This is intentional, because the CUT is faulty.

Running the Test with QUTest

The complete code for the basic Unity example is provided in the QP/C framework, directory C:\qp\qpc\examples\qutest\unity_basic\test. To run the basic test (on Windows), open a command prompt and type:

qspy

This will start the QSPY host utility with the TCP/IP connection to the Target.

Next, open a second command prompt window and type:

cd C:\qp\qpc\examples\qutest\unity_basic\test
make

Note: The provided Makefile will also work on Linux and MacOS. The only difference from Windows is that you open a terminal window and change directory to ~/qp/qpc/examples/qutest/unity_basic/test.

This will build the test fixture as a host executable and then it will run the test script (in Python). The screen shot below shows the output produced in these two command-prompt windows.

Unity example_1 test build and run with QUTest (left) and QSPY output (right).

Remarks: As you can see, two out of five QUTest tests fail. These are the same tests (adapted for QUTest) that failed during testing with Unity above.

Test Fixture

The job of a test fixture is to exercise the CUT (ProductionCode.c in this case) and report the results back to the QSPY host application. Note that a test fixture in QUTest™ is not supposed to perform any checks whether the CUT operates "correctly". Instead, your test fixture should only provide facilities to thoroughly exercise the CUT remotely from the test script(s). A properly written test fixture can be typically used for many tests (implemented in multiple test scripts).

Remarks: Coming up with a "good" test fixture requires some practice, but when you study the examples in this Tutorial, you will see some instances of flexible test fixtures that allow you to to run a wide variety of tests on them.

The following listing shows the complete QUTest test fixture for the basic tests (file test_ProductionCode.c in the directory C:\qp\qpc\examples\qutest\unity_basic\test). The explanation section following the listing clarifies the interesting lines of code (lines starting with [xx] labels).

 [1] #include "qpc.h"  /* QUTest interface */
 [2] #include "ProductionCode.h" /* CUT interface */
 
 [3] Q_DEFINE_THIS_FILE
 
     /*--------------------------------------------------------------------------*/
     // sometimes you may want to get at local data in a module.
     // for example: If you plan to pass by reference, this could be useful
     // however, it should often be avoided
 [4] extern int Counter;
 
     /*--------------------------------------------------------------------------*/
 [5] int main(int argc, char *argv[]) {
 
 [6]     QF_init();  /* initialize the framework */
 
         /* initialize the QS software tracing */
 [7]     Q_ALLEGE(QS_INIT(argc > 1 ? argv[1] : (void *)0));
 
         /* dictionaries... */
 [8]     QS_FUN_DICTIONARY(&FindFunction_WhichIsBroken);
 [9]     QS_FUN_DICTIONARY(&FunctionWhichReturnsLocalVariable);
[10]     QS_OBJ_DICTIONARY(&Counter);
 
         /* filter setup... */
[11]     QS_FILTER_ON(QS_ALL_RECORDS);
 
[12]     return QF_run(); /* run the tests */
     }
 
     /*--------------------------------------------------------------------------*/
[13] void QS_onTestSetup(void) {
     }
     /*..........................................................................*/
[14] void QS_onTestTeardown(void) {
     }
 
     /*..........................................................................*/
[15] void QS_onCommand(uint8_t cmdId,
                       uint32_t param1, uint32_t param2, uint32_t param3)
     {
         switch (cmdId) {
             case 0: { /* call the CUT: FindFunction_WhichIsBroken */
[16]             int ret = FindFunction_WhichIsBroken((int)param1);
[17]             QS_BEGIN(QS_USER + cmdId, (void *)0) /* user-specific record */
[18]                 QS_FUN(&FindFunction_WhichIsBroken); /* function called */
[19]                 QS_I32(0, (int32_t)ret); /* returned value */
[20]                 QS_I16(0, (int16_t)param1); /* parameter */
[21]             QS_END()
                 break;
             }
             case 1: { /* call the CUT: FunctionWhichReturnsLocalVariable */
[22]             int ret = FunctionWhichReturnsLocalVariable();
[23]             QS_BEGIN(QS_USER + cmdId, (void *)0) /* user-specific record */
[24]                 QS_FUN(&FunctionWhichReturnsLocalVariable); /* function called */
[25]                 QS_U32_HEX(0, (uint32_t)ret); /* returned value */
[26]             QS_END()
                 break;
             }
             default:
                 break;
         }
 
         /* unused parametrers... */
         //(void)param1;
         (void)param2;
         (void)param3;
     }
     /*..........................................................................*/
     /* host callback function to "massage" the event, if necessary */
[27] void QS_onTestEvt(QEvt *e) {
         (void)e;
     #ifdef Q_HOST  /* is this test compiled for a desktop Host computer? */
     #else /* this test is compiled for an embedded Target system */
     #endif
     }
     /*..........................................................................*/
[28] void QS_onTestPost(void const *sender, QActive *recipient,
                        QEvt const *e, bool status)
     {
         (void)sender;
         (void)recipient;
         (void)e;
         (void)status;
     }

1: The "qpc.h" header file contains the QP/C framework API, which includes the QUTest interface. Typically, you need to include this header file in QUTest test doubles.

NOTE: for test fixtures based on the QP/C++ framework, you need to include the "qpcpp.h" header file.
2: You also need to include the interface to the CUT, which is ProductionCode.h in this case.
3: The macro Q_DEFINE_THIS_FILE is needed for DbC assertions (they have nothing to do with test assertions). Later in this file, a DbC assertion is used to guard against failure in the initialization of the QS target-resident component (see step [7]).
4: The variable Counter is used to control the return value from the CUT (see ProductionCode.c line 4).

NOTE: this approach breaks encapsulation of the CUT, but it is copied here from the original Unity test.
5: A QUTest test fixture code needs the main() function. This main() function can be in a separate file, but in this simple example it is placed in test_ProductionCode.c. Either way, the main() function has the usual structure of a QP/C application (and in fact in the more advanced tests it can be the same function as used by the actual QP/C application). But here, it contains the bare minimum function calls, as described below.
6: The main() function must start with calling QF_init() to initialize the QP framework.
7: Next, you need to initialize the QS target-resident component (QS_INIT()). This macro is wrapped with the Q_ALLEGE() assertion, which will fire if the QS initialization fails. (In which case continuationon of the test makes no sense).
8-10: Next, you produce QS dictionaries for all functions you wish to test as well as objects you might need to inspect.

NOTE: you need to do this, so that the test scripts can refer to the functions and objects by the same symbolic names as the CUT/test-fixture.
11: The QS_FILTER_ON() macro sets the global filter in the Target. Here all output is enabled (QS_ALL_RECORDS).
12: Finally, at the end of main() you need to call QF_run() to run the tests.
13: The callback function QS_onTestSetup() allows you to include code that will be run at the beginning of each test. Here this simple CUT does not need any setup, but you still need to provide (an empty) implementation to satisfy the linker.
14: The callback function QS_onTestTeardown() allows you to include code that will be run at the end of each test. Here this simple CUT does not need any teardown, but you still need to provide (an empty) implementation to satisfy the linker.
15: The callback function QS_onCommand() allows you to remotely execute commands inside the Target. Here is where you execute the CUT and report results back to QSPY.
16: The command with cmdId==0 will be used to call the FindFunction_WhichIsBroken() CUT.

NOTE: You can use other cmdIds to call other pieces of CUT or to provide different variants of calling the same CUT, as you see fit. Much of the art of writing test fixtures lies in constructing flexible remote commands that exercise your CUT.
17: The QS_BEGIN() macro starts the application-specific trace record that will report results of calling the FindFunction_WhichIsBroken() CUT to the test script.
18: The QS_FUN() macro sends the address of the function to the test script. This address will be converted to the name of the function, because the dictionary for this function has been generated in setp 8.
19: The QS_I32() macro sends a 32-bit signed integer (int32_t) to the test script. Here you output the return value from FindFunction_WhichIsBroken().
20: The QS_I16() macro sends a 16-bit signed integer (int16_t) to the test script. Here you output the argument passed to FindFunction_WhichIsBroken().
21: The QS_END() macro ends the application-specific trace record.
22: The command with cmdId==1 will be used to call the FunctionWhichReturnsLocalVariable() CUT.
23-26: Again, the application-specific trace record gets generated that reports the function address and the return value from this function call.
27: The QS_onTestEvt() callback function is not used in this test, but needs to be provided to satisfy the linker.
28: The QS_onTestPost() callback function is not used in this test, but needs to be provided to satisfy the linker.

Test Script

A test script contains a group of related tests (a "test group"). The basic job of these tests is to send inputs to the test fixture running in the Target and to compare the produced QSPY textual output with the expectations of the test. The following listing shows the test script (Python) for the Unity basic tests (file test_ProductionCode.py). The explanation section following the listing clarifies the interesting lines of code (lines starting with [xx] labels).

Remarks: Even though a test script uses Python under the hood, you don"t need to be a Python expert to write effective test scripts. The actual set of commands that you use and need to know about forms a small @ref qutest_script "Domain Specific Language (DSL) for unit testing", which just happens to be implemented with Python as a command interpreter.

Note: The following test script performs the same tests as the Unity test fixture TestProductionCode.c in the directory qpc\examples\qutest\unity_basic\test_unity.

 [1] # QUTEST script corresponding to the test_ProductionCode.c test fixture.
     # This test fixture corresponds to ../test_unity/TestProductionCode.c fixture.
     # see https://www.state-machine.com/qtools/qutest.html
 
     # preamble...
 [2] def on_setup():
         # This is run before EACH TEST
 [3]     current_obj(OBJ_AP, "Counter")
 [4]     poke(0, 4, pack("<L", 0x5A5A))
 
     # tests...
 [5] test("FindFunction_WhichIsBroken() Should Return Zero If Item Is Not In List, Which Works Even In Our Broken Code")
     # All of these should pass
 [6] command(0, 78)
 [7] expect("0000000001 USER+000 FindFunction_WhichIsBroken 0 78")
 [8] expect("0000000002 Trg-Done QS_RX_COMMAND")
     command(0, 1)
     expect("0000000003 USER+000 FindFunction_WhichIsBroken 0 1")
     expect("0000000004 Trg-Done QS_RX_COMMAND")
     command(0, 33)
 [9] expect("@timestamp USER+000 FindFunction_WhichIsBroken 0 33")
     expect("@timestamp Trg-Done QS_RX_COMMAND")
     command(0, 999)
     expect("@timestamp USER+000 FindFunction_WhichIsBroken 0 999")
     expect("@timestamp Trg-Done QS_RX_COMMAND")
[10] command(0, (-1) & 0xFFFFFFFF)
     expect("@timestamp USER+000 FindFunction_WhichIsBroken 0 -1")
     expect("@timestamp Trg-Done QS_RX_COMMAND")
 
[11] test("FindFunction_WhichIsBroken() Should Return The Index For Items In List, Which Will Fail Because Our Function Under Test Is Broken")
[12] command(0, 34)
[13] expect("@timestamp USER+000 FindFunction_WhichIsBroken 1 34")
     # Notice the rest of these didn"t get a chance to run because the line above failed.
     # Unit tests abort each test on the first sign of trouble.
     # Then NEXT test runs as normal.
     expect("@timestamp Trg-Done QS_RX_COMMAND")
     command(0, 8888)
     expect("@timestamp USER+000 FindFunction_WhichIsBroken 8 8888")
     expect("@timestamp Trg-Done QS_RX_COMMAND")
 
[14] test("FunctionWhichReturnsLocalVariable() Should Return The Current Counter Value")
[15] command(1)
     expect("@timestamp USER+001 FunctionWhichReturnsLocalVariable 0x5A5A")
     expect("@timestamp Trg-Done QS_RX_COMMAND")
     # This should be true because we can still change our answer
[16] poke(0, 4, pack("<L", 0x1234))
     command(1)
     expect("@timestamp USER+001 FunctionWhichReturnsLocalVariable 0x1234")
     expect("@timestamp Trg-Done QS_RX_COMMAND")
 
[17] test("FunctionWhichReturnsLocalVariable() Should Return The Current Counter Value Again", NORESET)
     # This should be true again because setup was rerun before this test (and after we changed it to 0x1234)
     command(1)
     expect("@timestamp USER+001 FunctionWhichReturnsLocalVariable 0x5A5A")
     expect("@timestamp Trg-Done QS_RX_COMMAND")
 
     test("FunctionWhichReturnsLocalVariable() Should Return Current Counter, But Fails Because This Test Is Actually Flawed", NORESET)
     # Sometimes you get the test wrong.  When that happens, you get a failure too... and a quick look should tell
     # you what actually happened...which in this case was a failure to setup the initial condition.
     command(1)
     expect("@timestamp USER+001 FunctionWhichReturnsLocalVariable 0x1234")
     expect("@timestamp Trg-Done QS_RX_COMMAND")

1: Lines starting with a pound sign ('#') or empty lines are comments which are ignored by QUTest.

"preamble" defines the startup code common to all tests in the group:

2: The function on_setup() is executed at the beginning of each test in the group (see test()), including both tests that reset and do not reset the Target.
3: The current_obj() command sets the "current object" of the application-specific kind (OBJ_AP) in the Target. Subsequent commands (such as poke() in the next step) will act on this "current object".
4: The poke() command pokes the specified "Application Current Object" starting with the specified offset from the beginning of the object in memory (here 0) with the data elements of size 4 (the second argument) with the data provided in the third argument pack() "pack("<L", 0x5A5A)" into the previously established "Application Current Object".

Test: "FindFunction_WhichIsBroken() Should Return Zero..." checks that the CUT returns 0 when a given number is not found:

5: The test() command starts a test and gives it a name (in double quotes). The name of test will be displayed as the test is executed and should be a quick reminder about the objective of this test. This test command also resets the Target, which brings the Target into a well-defined initial state and produces the QS dictionary records (see test-fixture[12])

NOTE: The ability to perform the full Target reset is a unique feature of QUTest. Other unit testing frameworks, including Unity and CppUTest, don't reset the Target. They merely call the test setup()/tearDown() functions at the beginning and end of the test, respectively. QUTest also calls onReset()/onSetup()/onTeardown(), but obviously the full Target reset is a much better guarantee that the Target starts in exactly the same state.
6: The command() command causes the invocation of the QS_onCommand() callback inside the Target. The argument 0 is the cmdId parameter (see test-fixture[16])

NOTE: The first parameter of command() "command" here is just a number (0), but it is also possible to use a symbolic name for the first parameter. This symbolic name will be looked up in the user dictionary (QS_USR_DICTIONARY())
7: The expect() command represents an expectation of this test (a.k.a. test assertion). This is the expected output generated by the command(0) command from the previous step. You need to consult the test fixture to determine what you should expect in this case.

NOTE: the expected string starts with a number 0000000001, which is the Target Time-Stamp. In QUTest, the "timestamp" simply counts all the QS trace records produced, so that you know that no entries have been lost. In the later tests you will see how you can count the steps automatically with the @timestamp placeholder.
8: The test finishes with the expectation for the Trg-Done QS_RX_COMMAND trace record, which means that all output generated by command() has been generated.
9: This expect() command illustrate the use of the @timestamp placeholder to account for the test steps automatically.
10: This command() illustrates how to pass negative numbers as arguments. As you can see, you need to binary-and the number with the all-bits-on bitmask 0xFFFFFFFF.

Test: "FindFunction_WhichIsBroken() Should Return The Index..." checks that the CUT fails to return the expected index, because of the internal bug:

11: The test() command starts a next test.
12: This command(0, 34) should cause the function FindFunction_WhichIsBroken() to return index 1, because the number 34 is actually in the list.
13: This expect() command codifies the expected result of the function call.

NOTE: Due to the bug in the CUT, however, the expectation will fail, in which case the rest of the test is skipped until the next test() command.

Test: "FunctionWhichReturnsLocalVariable() Should Return The Current Counter Value":

14: The test() command starts a next test.
15: This command(1) runs the second function FunctionWhichReturnsLocalVariable() in the CUT.
16: This poke() command changes the value of the Counter variable, because this is the "current AP object" established in the on_setup() callback in step [3].

Attention: The QSPY host application limits the QS dictionaries to the first 63 characters. This means that longer names of functions or objects will be truncated to this limit. Please keep this limit in mind when choosing various names in your application.

Test: "FunctionWhichReturnsLocalVariable() Should Return The Current Counter..." checks whether FunctionWhichReturnsLocalVariable() CUT returns the value poked into the Counter variable:

17: The test() command starts a next test.

NOTE: Unlike all previous tests so far, this test does not reset the Target (argument NORESET)

Remarks: As an exercise you should modify the file ProductionCode.c to fix the bug and make the tests pass.

Running the Test on Embedded Targets

As mentioned at the initial description of this example, the directory C:\qp\qpc\examples\qutest\unity_basic\test contains makefiles to build the code and run the tests on the embedded boards (TivaC LaunchPad from Texas Instruments and EFM32 Pearl-Gecko board from Silicon Labs). Both these boards open a virtual COM port on the machine they are attached to via the USB cable. This virtual COM port provides an ideal connection for the QS communication with the QSPY host utility.

Targets for running QUTests. From the left: host computer, TivaC LaunchPad and EFM32 Pearl-Gecko

For example, to test the EFM32 Pearl-Gecko board (ARM Cortex-M4), open a command prompt (on Windows) and type:

qspy -c COM6

This will start the QSPY host utility with com-port connection to the embedded board. Of course, you need to adjust the serial port number to the actual number of the virtual COM port on your machine.

Next, open a second command prompt window and type:

cd C:\qp\qpc\examples\qutest\unity_basic\test
make -f make_efm32

The rest of the test is then exactly the same as with the host executable described above, except that the test fixture runs on the embedded board.

To run the tests on the TivaC LaunchPad (TM4C123 board), you use the make_tm4c123 in the last step.

Next: Unity Mock Example