tmk_core/tool/mbed/mbed-sdk/docs/TESTING.md

   1 # Mbed SDK automated test suite
   2 ## Introduction
   3
   4 Test suit allows users to run locally on their machines Mbed SDK’s tests included in Mbed SDK repository. It also allows users to create their own tests and for example add new tests to test set as they progress with their project. If test is generic enough it could be included into official Mbed SDK test pool just do it via normal pull request!
   5
   6 Each test is supervised by python script called “host test” which will at least Test suite is using build script API to compile and build test source together with required by test libraries like CMSIS, Mbed, Ethernet, USB etc.
   7
   8 ## What is host test?
   9 Test suite supports test supervisor concept. This concept is realized by separate Python script called ```host test```. Host tests can be found in ```mbed/workspace_tools/host_tests/``` directory. Note: In newer mbed versions (mbed OS) host tests will be separate library.
  10
  11 Host test script is executed in parallel with test runner to monitor test execution. Basic host test just monitors device's default serial port for test results returned by test runner. Simple tests will print test result on serial port. In other cases host tests can for example judge by test results returned by test runner is test passed or failed. It all depends on test itself.
  12
  13 In some cases host test can be TCP server echoing packets from test runner and judging packet loss. In other cases it can just check if values returned from accelerometer are actually valid (sane).
  14
  15 ## Test suite core: singletest.py script
  16
  17 ```singletest.py``` script located in ```mbed/workspace_tools/``` is a test suite script which allows users to compile, build tests and test runners (also supports CppUTest unit test library). Script also is responsible for test execution on devices selected by configuration files.
  18
  19 ### Parameters of singletest.py
  20
  21 Test execution script ```singletest.py``` is a fairly powerful tool to run tests for mbed SDK platform. It is flexible and allows users to configure test execution process and define which mbed platforms will be tested.
  22
  23 By specifying external configuration files (in JSON format) you can gain flexibility and prepare many different test scenarios. Just pass configuration file names to your script and run it.
  24
  25 #### MUTs Specification
  26 You can easily configure your MUTs (Mbed Under Test) by creating configuration file with MUTs description.
  27 Note: This configuration file must be in [JSON format](http://www.w3schools.com/json/).
  28 Note: Unfortunately JSON format is not allowing you to have comments inside JSON code.
  29
  30 Let’s see some example and let's try to configure small "test farm" with three devices connected to your host computer. In this example no peripherals (like SD card or EEPROM) are connected to our Mbed boards. We will use three platforms in this example:
  31 * [NXP LPC1768](https://mbed.org/platforms/mbed-LPC1768) board.
  32 * \[Freescale KL25Z](https://mbed.org/platforms/KL25Z) board and
  33 * [STMicro Nucleo F103RB](https://mbed.org/platforms/ST-Nucleo-F103RB) board.
  34 After connecting boards to our host machine (PC) we can check which serial ports and disks they occupy. For our example let's assume that:
  35 * ```LPC1768``` serial port is on ```COM4``` and disk drive is ```J:```.
  36 * ```KL25Z``` serial port is on ```COM39``` and disk drive is ```E:```.
  37 * ```NUCLEO_F103RB``` serial port is on ```COM11``` and disk drive is ```I:```.
  38 If you are working under Linux your port and disk could look like /dev/ttyACM5 and /media/usb5.
  39
  40 This information is needed to create ```muts_all.json``` configuration file. You can create in in ```mbed/workspace_tools/``` directory:
  41 ```
  42 $ touch muts_all.json
  43 ```
  44
  45 Its name will be passed to ```singletest.py``` script after ```-M``` (MUTs specification) switch. Let’s see how this file's content would look like in our example below:
  46 ```json
  47 {
  48     "1" : {"mcu": "LPC1768",
  49         "port":"COM4",
  50         "disk":"J:\\",
  51         "peripherals": []
  52     },
  53
  54     "2" : {"mcu": "KL25Z",
  55         "port":"COM39",
  56         "disk":"E:\\",
  57         "peripherals": []
  58     },
  59
  60     "3" : {"mcu": "NUCLEO_F103RB",
  61         "port":"COM11",
  62         "disk":"I:\\",
  63         "peripherals": []
  64     }
  65 }
  66 ```
  67
  68 Note: We will leave field ```peripherals``` empty for the sake of this example. We will explain it later. All you need to do now is to properly fill fields ```mcu```, ```port``` and ```disk```.
  69
  70 Note: Please make sure files muts_all.json and test_spec.json are in workspace_tools/ directory. We will assume in this example they are.
  71 Where to find ```mcu``` names? You can use option ```-S``` of ```build.py``` script (in ```mbed/workspace_tools/``` directory) to check all supported off-line MCUs names.
  72
  73 Note: If you update mbed device firmware or even disconnect / reconnect mbed device you may find that serial port / disk configuration changed. You need to update configuration file accordingly or you will face connection problems and obviously tests will not run.
  74
  75 #### Peripherals testing
  76 When using MUTs configuration file (switch ```-M```) you can define in MUTs JSON file peripherals connected to your device:
  77 ```json
  78 {
  79     "1" : {"mcu" : "KL25Z",
  80            "port" : "COM39",
  81            "disk" : "E:\\",
  82            "peripherals" : ["SD", "24LC256"]}
  83 }
  84 ```
  85 You can force test suite to run only common tests (switch ```-C```) or only peripheral tests (switch ```-P```).
  86 ```
  87 $ python singletest.py -i test_spec.json -M muts_all.json -C
  88 ```
  89 will not include tests for SD card and EEPROM 24LC256.
  90 ```
  91 $ python  singletest.py -i test_spec.json -M muts_all.json -P
  92 ```
  93 will only run tests bind to SD card and EEPROM 24LC256.
  94
  95 Note: option ```-P``` is useful for example in cases when you have same platform and different shields you want to test. No need to test common part all the time (timers, RTC, RTOS etc.). You can force to test peripherals only on some devices and for example only common tests on other devices.
  96
  97 #### Additional MUTs configuration file settings
  98 You can add extra information to each MUT configuration. In particular you can specify which flashing (binary copy method) should be used, how to reset target and for example set reset timeout (used to delay test execution just after reset).
  99
 100 muts_all.json:
 101 ```json
 102 {
 103     "1" : {"mcu" : "LPC1768",
 104            "port" : "COM77",
 105            "disk" : "G:\\",
 106            "peripherals" : ["TMP102", "digital_loop", "port_loop", "analog_loop", "SD"]},
 107
 108     "2" : {"mcu" : "KL25Z",
 109            "port" : "COM89",
 110            "disk" : "F:\\",
 111            "peripherals" : ["SD", "24LC256", "KL25Z"],
 112            "copy_method" : "copy",
 113            "reset_type" : "default",
 114            "reset_tout" : "2"},
 115
 116     "3" : {"mcu" : "LPC11U24",
 117            "port" : "COM76",
 118            "disk" : "E:\\",
 119            "peripherals" : []}
 120 }
 121 ```
 122 Please note that for MUT no. 2 few extra parameters were defined: ```copy_method```, ```reset_type``` and ```reset_tout```. Using this extra options you can tell test suite more about MUT you are using. This will allow you to be more flexible in terms of how you configure and use your MUTs.
 123
 124 * ```copy_method``` - STRING - tells test suite which binary copy method should be used.
 125 You may notice that ```singletest.py``` command line help contains description about:
 126  * Option ```-c``` (in MUTs file called ```copy_method```) with available copy methods supported by test suite plugin system.
 127  * Option ```-r``` (in MUTs file called reset_type) with available reset methods supported by test suite plugin system.
 128 * ```reset_type``` - STRING - some boards may require special reset handling, for example vendor specific command must be executed to reset device.
 129 * ```reset_tout``` - INTEGER - extra timeout just after device is reseted. May be used to wait for few seconds so device may finish booting, flashing data internally etc.
 130
 131 Part of help listing for singletest.py:
 132 ```
 133   -c COPY_METHOD, --copy-method=COPY_METHOD
 134                         Select binary copy (flash) method. Default is Python's
 135                         shutil.copy() method. Plugin support: copy, cp,
 136                         default, eACommander, eACommander-usb, xcopy
 137   -r MUT_RESET_TYPE, --reset-type=MUT_RESET_TYPE
 138                         Extra reset method used to reset MUT by host test
 139                         script. Plugin support: default, eACommander,
 140                         eACommander-usb
 141 ```
 142
 143 ----
 144
 145 Now we've already defined how our devices are connected to our host PC. We can continue and define which of this MUTs will be tested and which compilers we will use to compile and build Mbed SDK and tests. To do so we need to create test specification file (let's call it ```test_spec.json```) and put inside our configuration file information about which MUTs we actually want to test. We will pass this file's name to ```singletest.py``` script using ```-i``` switch.
 146
 147 Below we can see how sample ```test_spec.json``` file content could look like. (I've included all possible toolchains, we will change it in a moment):
 148 ```json
 149 {
 150     "targets": {
 151         "LPC1768" : ["ARM", "uARM", "GCC_ARM", "GCC_CS", "GCC_CR", "IAR"],
 152         "KL25Z" : ["ARM", "GCC_ARM"],
 153         "NUCLEO_F103RB" : ["ARM", "uARM"]
 154      }
 155 }
 156 ```
 157 Above example configuration will force tests for LPC1768, KL25Z, NUCLEO_F103RB platforms and:
 158
 159 * Compilers: ```ARM```, ```uARM```, ```GCC_ARM```, ```GCC_CS```, ```GCC_CR``` and ```IAR``` will be used to compile tests for NXP's ```LPC1768```.
 160 * Compilers: ```ARM``` and ```GCC_ARM``` will be used for Freescales' ```KL25Z``` platform.
 161 * Compilers: ```ARM``` and ```uARM``` will be used for STMicro's ```NUCLEO_F103RB``` platform.
 162
 163 For our example purposes let's assume we only have Keil ARM compiler, so let's change configuration in ```test_spec.json``` file and reduce number of compiler to those we actually have:
 164 ```json
 165 {
 166     "targets": {
 167         "LPC1768" : ["ARM", "uARM"],
 168         "KL25Z" : ["ARM"],
 169         "NUCLEO_F103RB" : ["ARM", "uARM"]
 170      }
 171 }
 172 ```
 173 #### Run your tests
 174
 175 After you configure all your MUTs and compilers you are ready to run tests. Make sure your devices are connected and your configuration files reflect your current configuration (serial ports, devices). Go to workspace_tools directory in your mbed location.
 176 ```
 177 $ cd workspace_tools/
 178 ```
 179 and execute test suite script.
 180 ```
 181 $ python singletest.py -i test_spec.json -M muts_all.json
 182 ```
 183 To check your configuration before test execution please use ```--config``` switch:
 184 ```
 185 $ python singletest.py -i test_spec.json -M muts_all.json --config
 186 MUTs configuration in m.json:
 187 +-------+-------------+---------------+------+-------+
 188 | index | peripherals | mcu           | disk | port  |
 189 +-------+-------------+---------------+------+-------+
 190 | 1     |             | LPC1768       | J:\  | COM4  |
 191 | 3     |             | NUCLEO_F103RB | I:\  | COM11 |
 192 | 2     |             | KL25Z         | E:\  | COM39 |
 193 +-------+-------------+---------------+------+-------+
 194
 195 Test specification in t.json:
 196 +---------------+-----+------+
 197 | mcu           | ARM | uARM |
 198 +---------------+-----+------+
 199 | NUCLEO_F103RB | Yes | Yes  |
 200 | KL25Z         | Yes | -    |
 201 | LPC1768       | Yes | Yes  |
 202 +---------------+-----+------+
 203 ```
 204 It should help you localize basic problems with configuration files and toolchain configuration.
 205 Note: Configurations with issues will be marked with ```*``` sign.
 206
 207 Having multiple configuration files allows you to manage your test scenarios in more flexible manner. You can:
 208
 209 * Set up all platforms and toolchains used during testing.
 210 * Define (using script's ```-n``` switch) which tests you want to run during testing.
 211 * Just run regression (all tests). Regression is default setting for test script.
 212
 213 You can also force ```singletest.py``` script to:
 214 * Run only peripherals' tests (switch ```-P```) or
 215 * Just skip peripherals' tests (switch ```-C```).
 216 * Build mbed SDK, libraries and corresponding tests with multiple cores, just use ```-j X``` option where ```X``` is number of cores you want to use for compilation.
 217 ```
 218 $ python singletest.py -i test_spec.json -M muts_all.json -j 8
 219 ```
 220 * Print test cases console output using ```-V``` option.
 221 * Only build mbed SDK, tests and dependant libraries with switch ```-O```:
 222 ```
 223 $ python singletest.py -i test_spec.json -M muts_all.json -j 8 -O
 224 ```
 225 * Execute each test case multiple times with ```--global-loops X``` option, where ```X``` number of repeats. Additionally use option ```-W``` to continue repeating test cases execution only if they continue to fail.
 226 ```
 227 $ python singletest.py -i test_spec.json -M muts_all.json --global-loops 3 -W
 228 ```
 229 * Option ```--loops``` can be used to overwrite global loop count and redefine loop count for particular tests. Define test loops as ```TEST_ID=X``` where ```X``` is integer and separate loops count definitions by comma if necessary. E.g. ```TEST_1=5,TEST_2=20,TEST_3=2```.
 230 ```
 231 $ python singletest.py -i test_spec.json -M muts_all.json RTOS_1=10,RTOS_2=5
 232 ```
 233 This will execute test ```RTOS_1``` ten (10) times and test ```RTOS_2``` five (5) times.
 234 * Force non default copy method. Note that mbed platforms can be flashed with just binary drag&drop. We simply copy file onto mbed's disk and interface chip flashes target MCU with given binary. Force non standard (Python specific) copy method by using option ```-c COPY_METHOD``` where ```COPY_METHOD``` can be shell, command line copy command like: ```cp```, ```copy````, ```xcopy``` etc. Make sure those commands are available from command line!
 235 ```
 236 $ python singletest.py -i test_spec.json -M muts_all.json -c cp
 237 ```
 238 * Run only selected tests. You can select which tests should be executed when you run test suite. Use ```-n``` switch to define tests by their ids you want to execute. Use comma to separate test ids:
 239 ```
 240 $ python singletest.py -i test_spec.json -M muts_all.json -n RTOS_1,RTOS_2,RTOS_3,MBED_10,MBED_16,MBED_11
 241 ```
 242 * Set common output binary name for all tests. In some cases you would like to have the same name for all tests. You can use switch ```--firmware-name``` to specify (without extension) build script output binary name.
 243 In below example we would like to have all test binaries called ```firmware.bin`` (or with other extension like .elf, .hex depending on target accepted format).
 244 ```
 245 $ python singletest.py -i test_spec.json -M muts_all.json --firmware-name firmware
 246 ```
 247 * Where to find test list? Tests are defined in file ```tests.py``` in ```mbed/workspace_tools/``` directory. ```singletest.py``` uses test metadata in ```tests.py``` to resolve libraries dependencies and build tests for proper platforms and peripherals. Option ```-R``` can be used to get test names and direct path and test configuration.
 248 ```
 249 $ python singletest.py -R
 250 +-------------+-----------+---------------------------------------+--------------+-------------------+----------+--------------------------------------------------------+
 251 | id          | automated | description                           | peripherals  | host_test         | duration | source_dir                                             |
 252 +-------------+-----------+---------------------------------------+--------------+-------------------+----------+--------------------------------------------------------+
 253 | MBED_1      | False     | I2C SRF08                             | SRF08        | host_test         |       10 | C:\Work\mbed\libraries\tests\mbed\i2c_SRF08            |
 254 | MBED_10     | True      | Hello World                           | -            | host_test         |       10 | C:\Work\mbed\libraries\tests\mbed\hello                |
 255 | MBED_11     | True      | Ticker Int                            | -            | host_test         |       20 | C:\Work\mbed\libraries\tests\mbed\ticker               |
 256 | MBED_12     | True      | C++                                   | -            | host_test         |       10 | C:\Work\mbed\libraries\tests\mbed\cpp                  |
 257 | MBED_13     | False     | Heap & Stack                          | -            | host_test         |       10 | C:\Work\mbed\libraries\tests\mbed\heap_and_stack       |
 258 | MBED_14     | False     | Serial Interrupt                      | -            | host_test         |       10 | C:\Work\mbed\libraries\tests\mbed\serial_interrupt     |
 259 | MBED_15     | False     | RPC                                   | -            | host_test         |       10 | C:\Work\mbed\libraries\tests\mbed\rpc                  |
 260 | MBED_16     | True      | RTC                                   | -            | host_test         |       15 | C:\Work\mbed\libraries\tests\mbed\rtc                  |
 261 | MBED_17     | False     | Serial Interrupt 2                    | -            | host_test         |       10 | C:\Work\mbed\libraries\tests\mbed\serial_interrupt_2   |
 262 | MBED_18     | False     | Local FS Directory                    | -            | host_test         |       10 | C:\Work\mbed\libraries\tests\mbed\dir                  |
 263 ...
 264 ```
 265 Note: you can filter tests by ```id``` column, just use ```-f``` option and give test name or regular expression:
 266 ```
 267 $ python singletest.py -R -f RTOS
 268 +--------------+-----------+-------------------------+-------------+-----------+----------+---------------------------------------------------+
 269 | id           | automated | description             | peripherals | host_test | duration | source_dir                                        |
 270 +--------------+-----------+-------------------------+-------------+-----------+----------+---------------------------------------------------+
 271 | CMSIS_RTOS_1 | False     | Basic                   | -           | host_test |       10 | C:\Work\mbed\libraries\tests\rtos\cmsis\basic     |
 272 | CMSIS_RTOS_2 | False     | Mutex                   | -           | host_test |       20 | C:\Work\mbed\libraries\tests\rtos\cmsis\mutex     |
 273 | CMSIS_RTOS_3 | False     | Semaphore               | -           | host_test |       20 | C:\Work\mbed\libraries\tests\rtos\cmsis\semaphore |
 274 | CMSIS_RTOS_4 | False     | Signals                 | -           | host_test |       10 | C:\Work\mbed\libraries\tests\rtos\cmsis\signals   |
 275 | CMSIS_RTOS_5 | False     | Queue                   | -           | host_test |       20 | C:\Work\mbed\libraries\tests\rtos\cmsis\queue     |
 276 | CMSIS_RTOS_6 | False     | Mail                    | -           | host_test |       20 | C:\Work\mbed\libraries\tests\rtos\cmsis\mail      |
 277 | CMSIS_RTOS_7 | False     | Timer                   | -           | host_test |       10 | C:\Work\mbed\libraries\tests\rtos\cmsis\timer     |
 278 | CMSIS_RTOS_8 | False     | ISR                     | -           | host_test |       10 | C:\Work\mbed\libraries\tests\rtos\cmsis\isr       |
 279 | RTOS_1       | True      | Basic thread            | -           | host_test |       15 | C:\Work\mbed\libraries\tests\rtos\mbed\basic      |
 280 | RTOS_2       | True      | Mutex resource lock     | -           | host_test |       20 | C:\Work\mbed\libraries\tests\rtos\mbed\mutex      |
 281 | RTOS_3       | True      | Semaphore resource lock | -           | host_test |       20 | C:\Work\mbed\libraries\tests\rtos\mbed\semaphore  |
 282 | RTOS_4       | True      | Signals messaging       | -           | host_test |       10 | C:\Work\mbed\libraries\tests\rtos\mbed\signals    |
 283 | RTOS_5       | True      | Queue messaging         | -           | host_test |       10 | C:\Work\mbed\libraries\tests\rtos\mbed\queue      |
 284 | RTOS_6       | True      | Mail messaging          | -           | host_test |       10 | C:\Work\mbed\libraries\tests\rtos\mbed\mail       |
 285 | RTOS_7       | True      | Timer                   | -           | host_test |       15 | C:\Work\mbed\libraries\tests\rtos\mbed\timer      |
 286 | RTOS_8       | True      | ISR (Queue)             | -           | host_test |       10 | C:\Work\mbed\libraries\tests\rtos\mbed\isr        |
 287 | RTOS_9       | True      | SD File write-read      | SD          | host_test |       10 | C:\Work\mbed\libraries\tests\rtos\mbed\file       |
 288 +--------------+-----------+-------------------------+-------------+-----------+----------+---------------------------------------------------+
 289 ```
 290
 291 * Shuffle your tests. We strongly encourage you to shuffle your test order each time you execute test suite script.
 292 Rationale: It is probable that tests executed in one particular order will pass and in other will fail. To shuffle your tests’ order please use option ```–u``` (or ```--shuffle```):
 293 ```
 294 $ python singletest.py -i test_spec.json -M muts_all.json --shuffle
 295 ```
 296 Above command with force test script to randomly generate shuffle seed and shuffle test order execution. Note: Shuffle seed is float in ```[0.0, 1.0)```.
 297
 298 You can always recreate particular test order by forcing shuffle (```-u``` or ```--shuffle```} switch) and passing shuffle seed back to test suite using ```--shuffle-seed``` switch:
 299 ```
 300 $ python singletest.py -i test_spec.json -M muts_all.json --shuffle --shuffle-seed 0.4041028336
 301 ```
 302 Note: You can also supply your own randomly generated shuffle seed to drive particular test execution order scenarios. Just make sure shuffle seed is float in ```[0.0, 1.0)```.
 303 You can find test shuffle seed in test summary:
 304 ```
 305 ...
 306 | OK     | LPC1768 | ARM       | MBED_A9   | Serial Echo at 115200       |        2.84        |       10      |  1/1  |
 307 +--------+---------+-----------+-----------+-----------------------------+--------------------+---------------+-------+
 308 Result: 1 FAIL / 22 OK
 309 Shuffle Seed: 0.4041028336
 310
 311 Completed in 234.85 sec
 312 ```
 313
 314 ### Exmple of device configuration (one device connected to host computer)
 315
 316 This example will show you how to configure single device, run general tests or only peripheral tests. We will also show some real test result examples.
 317
 318 1. We will test only one board STMIcro Nucleo ```F334R8``` board connected to our PC (port ```COM46``` and disk is ```E:```).
 319 2. We will also connect EEPROM ```24LC256``` to SDA, SCL pins of our Nucleo board and define 24LC256 peripheral to make sure our test suite will run all available tests for ```24LC256```.
 320
 321 Let's configure our one MUT and set uARM as the only compiler we will use to compiler Mbed SDK and tests.
 322 We also need to create two configuration files ```muts_all.json``` and ```test_spec.json``` to pass our small testbed configuration to test script.
 323
 324 muts_all.json:
 325 ```json
 326 {
 327     "1" : {
 328             "mcu": "NUCLEO_F334R8",
 329             "port":"COM46",
 330             "disk":"E:\\",
 331             "peripherals": ["24LC256"]
 332     }
 333 }
 334 ```
 335 Note: By defining ```"peripherals": ["24LC256"]``` we are passing to test suite information that this particular board has EEPROM 24LC256 connected to our board.
 336
 337 test_spec.json:
 338 ```json
 339 {
 340     "targets": {
 341         "NUCLEO_F334R8" : ["uARM"]
 342      }
 343 }
 344 ```
 345 Note:
 346 * Please make sure device is connected before we will start running tests.
 347 * Please make sure files ```muts_all.json``` and ```test_spec.json``` are in ```mbed/workspace_tools/``` directory.
 348 Now you can call test suite and execute tests:
 349 ```
 350 $ python singletest.py -i test_spec.json -M muts_all.json
 351 ...
 352 Test summary:
 353 +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
 354 | Result | Target        | Toolchain | Test ID   | Test Description                | Elapsed Time (sec) | Timeout (sec) |
 355 +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
 356 | OK     | NUCLEO_F334R8 | uARM      | MBED_A25  | I2C EEPROM line read/write test |       12.41        |       15      |
 357 | OK     | NUCLEO_F334R8 | uARM      | MBED_A1   | Basic                           |        3.42        |       10      |
 358 | OK     | NUCLEO_F334R8 | uARM      | EXAMPLE_1 | /dev/null                       |        3.42        |       10      |
 359 | OK     | NUCLEO_F334R8 | uARM      | MBED_24   | Timeout Int us                  |       11.47        |       15      |
 360 | OK     | NUCLEO_F334R8 | uARM      | MBED_25   | Time us                         |       11.43        |       15      |
 361 | OK     | NUCLEO_F334R8 | uARM      | MBED_26   | Integer constant division       |        3.37        |       10      |
 362 | OK     | NUCLEO_F334R8 | uARM      | MBED_23   | Ticker Int us                   |       12.43        |       15      |
 363 | OK     | NUCLEO_F334R8 | uARM      | MBED_A19  | I2C EEPROM read/write test      |       11.42        |       15      |
 364 | OK     | NUCLEO_F334R8 | uARM      | MBED_11   | Ticker Int                      |       12.43        |       20      |
 365 | OK     | NUCLEO_F334R8 | uARM      | MBED_10   | Hello World                     |        2.42        |       10      |
 366 | OK     | NUCLEO_F334R8 | uARM      | MBED_12   | C++                             |        3.42        |       10      |
 367 | OK     | NUCLEO_F334R8 | uARM      | MBED_16   | RTC                             |        4.76        |       15      |
 368 | UNDEF  | NUCLEO_F334R8 | uARM      | MBED_2    | stdio                           |       20.42        |       20      |
 369 | UNDEF  | NUCLEO_F334R8 | uARM      | MBED_A9   | Serial Echo at 115200           |       10.37        |       10      |
 370 +--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
 371 Result: 2 UNDEF / 12 OK
 372
 373 Completed in 160 sec
 374 ```
 375
 376 If we want to get additional test summary with results in separate columns please use option ```-t```.
 377 ```
 378 $ python singletest.py -i test_spec.json -M muts_all.json -t
 379 ...
 380 Test summary:
 381 +---------------+-----------+---------------------------------+-------+
 382 | Target        | Test ID   | Test Description                | uARM  |
 383 +---------------+-----------+---------------------------------+-------+
 384 | NUCLEO_F334R8 | EXAMPLE_1 | /dev/null                       | OK    |
 385 | NUCLEO_F334R8 | MBED_10   | Hello World                     | OK    |
 386 | NUCLEO_F334R8 | MBED_11   | Ticker Int                      | OK    |
 387 | NUCLEO_F334R8 | MBED_12   | C++                             | OK    |
 388 | NUCLEO_F334R8 | MBED_16   | RTC                             | OK    |
 389 | NUCLEO_F334R8 | MBED_2    | stdio                           | UNDEF |
 390 | NUCLEO_F334R8 | MBED_23   | Ticker Int us                   | OK    |
 391 | NUCLEO_F334R8 | MBED_24   | Timeout Int us                  | OK    |
 392 | NUCLEO_F334R8 | MBED_25   | Time us                         | OK    |
 393 | NUCLEO_F334R8 | MBED_26   | Integer constant division       | OK    |
 394 | NUCLEO_F334R8 | MBED_A1   | Basic                           | OK    |
 395 | NUCLEO_F334R8 | MBED_A19  | I2C EEPROM read/write test      | OK    |
 396 | NUCLEO_F334R8 | MBED_A25  | I2C EEPROM line read/write test | OK    |
 397 | NUCLEO_F334R8 | MBED_A9   | Serial Echo at 115200           | UNDEF |
 398 +---------------+-----------+---------------------------------+-------+
 399 ```
 400 ----
 401 Please do not forget you can combine few options together to get result you want. For example you want to repeat few tests multiple number of times, shuffle test ids execution order and select only tests which are critical for you at this point. You can do it using switch -n, --global-loops with --loops and --shuffle:
 402
 403 Execute above command to:
 404
 405 * Run only tests: ```RTOS_1```, ```RTOS_2```, ```RTOS_3```, ```MBED_10```, ```MBED_16```, ```MBED_11```.
 406 * Shuffle test execution order. Note tests in loops will not be shuffled.
 407 * Set global loop count to 3 - each test will repeated 3 times.
 408 * Overwrite global loop count (set above to 3) and:
 409  * Force to loop test RTOS_1 to execute 3 times.
 410  * Force to loop test RTOS_2 to execute 4 times.
 411  * Force to loop test RTOS_3 to execute 5 times.
 412  * Force to loop test MBED_11 to execute 5 times.
 413
 414 ```
 415 $ python singletest.py -i test_spec.json -M muts_all.json -n RTOS_1,RTOS_2,RTOS_3,MBED_10,MBED_16,MBED_11 --shuffle --global-loops 3 --loops RTOS_1=3,RTOS_2=4,RTOS_3=5,MBED_11=5
 416 ```
 417
 418 # CppUTest unit test library support
 419 ## CppUTest in Mbed SDK testing introduction
 420 [CppUTest](http://cpputest.github.io/) is a C / C++ based unit xUnit test framework for unit testing and for test-driving your code. It is written in C++ but is used in C and C++ projects and frequently used in embedded systems but it works for any C / C++ project.
 421
 422 Mbed SDK test suite supports writing tests using CppUTest. All you need to do it to provide CppUTest sources and includes with Mbed SDK port. This is already done for you so all you need to do it to get proper sources in your project directory.
 423 CppUTest’s core design principles are:
 424 * Simple in design and simple in use.
 425 * Portable to old and new platforms.
 426 * Build with Test-driven Development in mind.
 427
 428 ## From where you can get more help about CppUTest library and unit testing
 429 •     You can read [CppUTest manual](http://cpputest.github.io/manual.html)
 430 * [CppUTest forum](https://groups.google.com/forum/?fromgroups#!forum/cpputest)
 431 * [CppUTest on GitHub](https://github.com/cpputest/cpputest)
 432 * Finally, if you think unit testing is new concept for you, you can have a grasp of it on Wikipedia pages about [unit testing](http://en.wikipedia.org/wiki/Unit_testing) and continue from there.
 433
 434 ## How to add CppUTest to your current Mbed SDK installation
 435
 436 ### Do I need CppUTest port for Mbed SDK?
 437 Yes, you do. If you want to use CppUTest with Mbed SDK you need to have CppUTest version with ARMCC compiler (only ARM flavor for now) port and Mbed SDK console port (if you want to have output on serial port). All is already prepared by Mbed engineers and you can get it for example here: http://mbed.org/users/rgrover1/code/CppUTest/
 438
 439 ### Prerequisites
 440 * Installed [git client](http://git-scm.com/downloads/).
 441 * Installed [Mercurial client](http://mercurial.selenic.com/).
 442
 443 ### How / where to install
 444 We want to create directory structure similar to one below:
 445 ```
 446 \your_project_directory
 447 │
 448 ├───cpputest
 449 │   ├───include
 450 │   └───src
 451 └───mbed
 452     ├───libraries
 453     ├───travis
 454     └───workspace_tools
 455 ```
 456
 457 Please go to directory with your project. For example it could be c:\Projects\Project.
 458 ```
 459 $ cd c:\Projects\Project
 460 ```
 461 If your project directory already has your mbed SDK repository included just execute below command (Mercurial console client). It should download CppUTest with Mbed SDK port.
 462 ```
 463 $ hg clone https://mbed.org/users/rgrover1/code/cpputest/
 464 ```
 465
 466 You should see something like this after you execute Mercurial clone command:
 467 ```
 468 $ hg clone https://mbed.org/users/rgrover1/code/cpputest/
 469 destination directory: cpputest
 470 requesting all changes
 471 adding changesets
 472 adding manifests
 473 adding file changes
 474 added 3 changesets with 69 changes to 42 files
 475 updating to branch default
 476 41 files updated, 0 files merged, 0 files removed, 0 files unresolved
 477 ```
 478
 479 Confirm your project structure. It should look more or less like this:
 480 ```
 481 $ ls
 482 cpputest  mbed
 483 ```
 484 From now on CppUTest is in correct path. Each time you want to compile unit tests for CppUTest build script will always look for CppUTest library in the same directory where mbed library is.
 485
 486 ## New off-line mbed SDK project with CppUTest support
 487
 488 If you are creating new mbed SDK project and you want to use CppUTest with it you need to download both mbed SDK and CppUTest with mbed port to the same directory. You can do it like this:
 489 ```
 490 $ cd c:\Projects\Project
 491 $ git clone https://github.com/mbedmicro/mbed.git
 492 $ hg clone https://mbed.org/users/rgrover1/code/cpputest/
 493 ```
 494
 495 After above three steps you should have proper directory structure. All you need to do now is to configure your ```private_settings.py``` in ```mbed/workspace_tools/``` directory. Please refer to mbed SDK build script documentation for details.
 496
 497 ## CppUTest with mbed port
 498 To make sure you actualy have CppUTest library with mbed SDK port you can go to CppUTest ```armcc``` platform directory:
 499 ```
 500 $ cd c:/Projects/Project/cpputest/src/Platforms/armcc/
 501 ```
 502 And open file ```UtestPlatform.cpp```.
 503
 504 You should find part of code responsible for porting console on default serial port of the mbed device:
 505 ```c++
 506 #include "Serial.h"
 507 using namespace mbed;
 508
 509 int PlatformSpecificPutchar(int c)
 510 {
 511     /* Please modify this block for test results to be reported as
 512      * console output. The following is a sample implementation using a
 513      * Serial object connected to the console. */
 514 #define NEED_TEST_REPORT_AS_CONSOLE_OUTPUT 1
 515 #if NEED_TEST_REPORT_AS_CONSOLE_OUTPUT
 516     extern Serial console;
 517
 518     #define NEED_LINE_FEED_IN_ADDITION_TO_NEWLINE 1
 519     #if NEED_LINE_FEED_IN_ADDITION_TO_NEWLINE
 520     /* CppUTest emits \n line terminators in its reports; some terminals
 521      * need the linefeed (\r) in addition. */
 522     if (c == '\n') {
 523         console.putc('\r');
 524     }
 525     #endif /* #if NEED_LINE_FEED_IN_ADDITION_TO_NEWLINE */
 526
 527     return (console.putc(c));
 528 #else /* NEED_TEST_REPORT_AS_CONSOLE_OUTPUT */
 529     return (0);
 530 #endif /* NEED_TEST_REPORT_AS_CONSOLE_OUTPUT */
 531 }
 532 ```
 533
 534 You can find cpputest UT test runner main function in mbed sources: ```c:/Projects/Project/mbed/libraries/tests/utest/testrunner/testrunner.cpp```. Test runner code (in ```testrunner.cpp```) only defined console object and executes all unit tests:
 535 ```c++
 536 #include "CommandLineTestRunner.h"
 537 #include <stdio.h>
 538 #include "mbed.h"
 539 #include "testrunner.h"
 540 #include "test_env.h"
 541
 542 /**
 543 Object 'mbed_cpputest_console' is used to show prints on console.
 544 It is declared in \cpputest\src\Platforms\armcc\UtestPlatform.cpp
 545 */
 546 Serial mbed_cpputest_console(STDIO_UART_TX, STDIO_UART_RX);
 547
 548 int main(int ac, char** av) {
 549     MBED_HOSTTEST_TIMEOUT(20);
 550     MBED_HOSTTEST_SELECT(default_auto);
 551     MBED_HOSTTEST_DESCRIPTION(Unit test);
 552     MBED_HOSTTEST_START("UT");
 553
 554     unsigned failureCount = 0;
 555     {
 556         // Some compilers may not pass ac, av so we need to supply them ourselves
 557         int ac = 2;
 558         char* av[] = {__FILE__, "-v"};
 559         failureCount = CommandLineTestRunner::RunAllTests(ac, av);
 560     }
 561
 562     MBED_HOSTTEST_RESULT(failureCount == 0);
 563     return failureCount;
 564 }
 565 ```
 566
 567 ## Unit test location
 568 Unit tests source code is located in below directory: ```c:/Projects/Project/mbed/libraries/tests/utest/```
 569
 570 Each sub directory except testrunner contains compilable unit test source files with test groups and test cases. You can see utest structure below. Please note this is just example and in the future this directory will contain many sub directories with unit tests.
 571 ```
 572 $ c:\Projects\Project\mbed\libraries\tests\utest> tree
 573 utest
 574 ├───basic
 575 ├───semihost_fs
 576 └───testrunner
 577 ```
 578
 579 ## Define unit tests in mbed SDK test suite structure
 580 All tests defined in test suite are described in ```mbed/workspace_tools/tests.py``` file. This file stores data structure ```TESTS``` which is a list of simple structures describing each test. Below you can find example of ```TESTS``` structure which is configuring one of the unit tests.
 581 ```
 582 .
 583 .
 584 .
 585     {
 586         "id": "UT_2", "description": "Semihost file system",
 587         "source_dir": join(TEST_DIR, "utest", "file"),
 588         "dependencies": [MBED_LIBRARIES, TEST_MBED_LIB, CPPUTEST_LIBRARY],
 589         "automated": False,
 590         "mcu": ["LPC1768", "LPC2368", "LPC11U24"]
 591     },
 592 .
 593 .
 594 .
 595 ```
 596 Note: In dependency section we've added library ```CPPUTEST_LIBRARY``` which is pointing build script to CppUTest library with mbed port. This is a must for unit tests to be compiled with CppUTest library.
 597
 598 ### Tests are now divided into two types:
 599 #### 'Hello world' tests
 600 First type of test cases we call 'hello world' tests. They do not dependent on CppUTest library and are monolithic programs usually composed of one main function. Yo can find this tests in below example directories:
 601
 602 * ```mbed/libraries/tests/mbed/```
 603 * ```mbed/libraries/tests/net/```
 604 * ```mbed/libraries/tests/rtos/```
 605 * ```mbed/libraries/tests/usb/```
 606
 607 Usually ‘hello world’ test cases are using ```test_env.cpp``` and ```test_env.h``` files which implement simple test framework used to communicate with host test and help test framework instrument your tests.
 608
 609 Below you can see listing of ```test_env.h``` file which contains simple macro definitions used to communicate (via serial port printouts) between test case (on hardware) and host test script (on host computer).
 610 Each use case should print on console basic information like:
 611 * Default test case timeout.
 612 * Which host test should be used to supervise test case execution.
 613 * Test description and test case ID (short identifier).
 614
 615 ```c++
 616 .
 617 .
 618 .
 619 // Test result related notification functions
 620 void notify_start();
 621 void notify_completion(bool success);
 622 bool notify_completion_str(bool success, char* buffer);
 623 void notify_performance_coefficient(const char* measurement_name, const int value);
 624 void notify_performance_coefficient(const char* measurement_name, const unsigned int value);
 625 void notify_performance_coefficient(const char* measurement_name, const double value);
 626
 627 // Host test auto-detection API
 628 void notify_host_test_name(const char *host_test);
 629 void notify_timeout(int timeout);
 630 void notify_test_id(const char *test_id);
 631 void notify_test_description(const char *description);
 632
 633 // Host test auto-detection API
 634 #define MBED_HOSTTEST_START(TESTID)      notify_test_id(TESTID); notify_start()
 635 #define MBED_HOSTTEST_SELECT(NAME)       notify_host_test_name(#NAME)
 636 #define MBED_HOSTTEST_TIMEOUT(SECONDS)   notify_timeout(SECONDS)
 637 #define MBED_HOSTTEST_DESCRIPTION(DESC)  notify_test_description(#DESC)
 638 #define MBED_HOSTTEST_RESULT(RESULT)     notify_completion(RESULT)
 639
 640 /**
 641     Test auto-detection preamble example:
 642     main() {
 643         MBED_HOSTTEST_TIMEOUT(10);
 644         MBED_HOSTTEST_SELECT( host_test );
 645         MBED_HOSTTEST_DESCRIPTION(Hello World);
 646         MBED_HOSTTEST_START("MBED_10");
 647         // Proper 'host_test.py' should take over supervising of this test
 648
 649         // Test code
 650         bool result = ...;
 651
 652         MBED_HOSTTEST_RESULT(result);
 653     }
 654 */
 655 .
 656 .
 657 .
 658 ```
 659
 660 Example of 'hello world' test:
 661 ```c++
 662 #include "mbed.h"
 663 #include "test_env.h"
 664
 665 #define CUSTOM_TIME  1256729737
 666
 667 int main() {
 668     MBED_HOSTTEST_TIMEOUT(20);
 669     MBED_HOSTTEST_SELECT(rtc_auto);
 670     MBED_HOSTTEST_DESCRIPTION(RTC);
 671     MBED_HOSTTEST_START("MBED_16");
 672
 673     char buffer[32] = {0};
 674     set_time(CUSTOM_TIME);  // Set RTC time to Wed, 28 Oct 2009 11:35:37
 675     while(1) {
 676         time_t seconds = time(NULL);
 677         strftime(buffer, 32, "%Y-%m-%d %H:%M:%S %p", localtime(&seconds));
 678         printf("MBED: [%ld] [%s]\r\n", seconds, buffer);
 679         wait(1);
 680     }
 681 }
 682 ```
 683
 684 #### 'Unit test' test cases
 685 Second group of tests are unit tests. They are using CppUTest library and require you to write ```TEST_GROUP```s and ```TEST```s in your test files. Test suite will add test runner sources to your test automatically so you can concentrate on writing tests.
 686
 687 Example of unit test:
 688 ```c++
 689 #include "TestHarness.h"
 690 #include <utility>
 691 #include "mbed.h"
 692
 693 TEST_GROUP(BusOut_mask)
 694 {
 695 };
 696
 697 TEST(BusOut_mask, led_1_2_3)
 698 {
 699     BusOut bus_data(LED1, LED2, LED3);
 700     CHECK_EQUAL(0x07, bus_data.mask());
 701 }
 702
 703 TEST(BusOut_mask, led_nc_nc_nc_nc)
 704 {
 705     BusOut bus_data(NC, NC, NC, NC);
 706     CHECK_EQUAL(0x00, bus_data.mask());
 707 }
 708
 709 TEST(BusOut_mask, led_1_2_3_nc_nc)
 710 {
 711     BusOut bus_data(LED1, LED2, LED3, NC, NC);
 712     CHECK_EQUAL(0x07, bus_data.mask());
 713 }
 714
 715 TEST(BusOut_mask, led_1_nc_2_nc_nc_3)
 716 {
 717     BusOut bus_data(LED1, NC, LED2, NC, NC, LED3);
 718     CHECK_EQUAL(0x25, bus_data.mask());
 719 }
 720
 721 ///////////////////////////////////////////////////////////////////////////////
 722
 723 #ifdef MBED_OPERATORS
 724 TEST_GROUP(BusOut_digitalout_write)
 725 {
 726 };
 727
 728 TEST(BusOut_digitalout_write, led_nc)
 729 {
 730     BusOut bus_data(NC);
 731     CHECK_EQUAL(false, bus_data[0].is_connected())
 732 }
 733
 734 TEST(BusOut_digitalout_write, led_1_2_3)
 735 {
 736     BusOut bus_data(LED1, LED2, LED3);
 737     bus_data[0].write(1);
 738     bus_data[1].write(1);
 739     bus_data[2].write(1);
 740     CHECK(bus_data[0].read());
 741     CHECK(bus_data[1].read());
 742     CHECK(bus_data[2].read());
 743 }
 744
 745 TEST(BusOut_digitalout_write, led_1_2_3_nc_nc)
 746 {
 747     BusOut bus_data(LED1, LED2, LED3, NC, NC);
 748     bus_data[0].write(0);
 749     bus_data[1].write(0);
 750     bus_data[2].write(0);
 751     CHECK(bus_data[0].read() == 0);
 752     CHECK(bus_data[1].read() == 0);
 753     CHECK(bus_data[2].read() == 0);
 754 }
 755
 756 TEST(BusOut_digitalout_write, led_1_nc_2_nc_nc_3)
 757 {
 758     BusOut bus_data(LED1, NC, LED2, NC, NC, LED3);
 759     bus_data[0].write(1);
 760     bus_data[2].write(0);
 761     bus_data[5].write(0);
 762     CHECK(bus_data[0].read());
 763     CHECK(bus_data[2].read() == 0);
 764     CHECK(bus_data[5].read() == 0);
 765 }
 766 #endif
 767 ```
 768
 769 ## Example
 770 In below example we will run two example unit tests that are now available. tests ```UT_1``` and ```UT_2``` are unit tests used for now only to check if mbed SDK works with CppUTest library and if tests are being executed. In future number of unit tests will increase, nothing is also should stop you from writing and executing your own unit tests!
 771
 772 ### Example configuration
 773 By default unit tests ```UT_1``` and ```UT_2``` are not automated - simply test suite will ignore them. Also we do not want to create dependency to CppUTest library each time someone executes automation.
 774
 775 Note: To compile ```UT_1``` and ```UT_2``` tests CppUTest library described above installation is needed and not all users wish to add UT libs to their project. Also new to mbed users may find it difficult. This is why unit testing is an extra feature active only after you deliberately install and enable needed components.
 776
 777 Bellow snippet shows how to modify 'automated' flag so test suite will consider unit tests ```UT_1``` and ```UT_2``` as part of "automated test portfolio". Just change flag 'automated' from ```False``` to ```True```.
 778
 779 ```tests.py``` listing related to ```UT_1``` and ```UT_2```:
 780 ```python
 781 .
 782 .
 783 .
 784     # CPPUTEST Library provides Unit testing Framework
 785     #
 786     # To write TESTs and TEST_GROUPs please add CPPUTEST_LIBRARY to 'dependencies'
 787     #
 788     # This will also include:
 789     # 1. test runner - main function with call to CommandLineTestRunner::RunAllTests(ac, av)
 790     # 2. Serial console object to print test result on serial port console
 791     #
 792
 793     # Unit testing with cpputest library
 794     {
 795         "id": "UT_1", "description": "Basic",
 796         "source_dir": join(TEST_DIR, "utest", "basic"),
 797         "dependencies": [MBED_LIBRARIES, TEST_MBED_LIB, CPPUTEST_LIBRARY],
 798         "automated": True,
 799     },
 800     {
 801         "id": "UT_2", "description": "Semihost file system",
 802         "source_dir": join(TEST_DIR, "utest", "semihost_fs"),
 803         "dependencies": [MBED_LIBRARIES, TEST_MBED_LIB, CPPUTEST_LIBRARY],
 804         "automated": True,
 805         "mcu": ["LPC1768", "LPC2368", "LPC11U24"]
 806     },
 807 .
 808 .
 809 .
 810 ```
 811
 812 ### Execute tests
 813 In my test I will use common [LPC1768](http://developer.mbed.org/platforms/mbed-LPC1768/) mbed-enabled board because unit test ```UT_2``` is checking semi-host functionality which is available on this board and handful of others.
 814
 815 Configure your ```test_spec.json``` and ```muts_all.json``` files (refer to test suite build script and automation description) and set mbed disk and serial port.
 816
 817 ```
 818 $ singletest.py -i test_spec.json -M muts_all.json -n UT_1,UT_2 -V
 819 Building library CMSIS (LPC1768, ARM)
 820 Building library MBED (LPC1768, ARM)
 821 Building library CPPUTEST (LPC1768, ARM)
 822 Building project BASIC (LPC1768, ARM)
 823 Executing 'python host_test.py -p COM77 -d E:\ -t 10'
 824 Test::Output::Start
 825 Host test instrumentation on port: "COM77" and disk: "E:\"
 826 TEST(FirstTestGroup, FirstTest) - 0 ms
 827
 828 OK (1 tests, 1 ran, 3 checks, 0 ignored, 0 filtered out, 3 ms)
 829
 830 {{success}}
 831 {{end}}
 832 Test::Output::Finish
 833 TargetTest::LPC1768::ARM::UT_1::Basic [OK] in 2.43 of 10 sec
 834 Building library CPPUTEST (LPC1768, ARM)
 835 Building project SEMIHOST_FS (LPC1768, ARM)
 836 Executing 'python host_test.py -p COM77 -d E:\ -t 10'
 837 Test::Output::Start
 838 Host test instrumentation on port: "COM77" and disk: "E:\"
 839 TEST(FirstTestGroup, FirstTest) - 9 ms
 840
 841 OK (1 tests, 1 ran, 10 checks, 0 ignored, 0 filtered out, 10 ms)
 842
 843 {{success}}
 844 {{end}}
 845 Test::Output::Finish
 846 TargetTest::LPC1768::ARM::UT_2::Semihost file system [OK] in 2.43 of 10 sec
 847 Test summary:
 848 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
 849 | Result | Target  | Toolchain | Test ID | Test Description     | Elapsed Time (sec) | Timeout (sec) | Loops |
 850 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
 851 | OK     | LPC1768 | ARM       | UT_1    | Basic                |        2.43        |       10      |  1/1  |
 852 | OK     | LPC1768 | ARM       | UT_2    | Semihost file system |        2.43        |       10      |  1/1  |
 853 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
 854 Result: 2 OK
 855
 856 Completed in 12.02 sec
 857 ```
 858
 859 You can compile unit tests using various number of supported compilers, below just few examples with working solutions:
 860 ```
 861 Test summary:
 862 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
 863 | Result | Target  | Toolchain | Test ID | Test Description     | Elapsed Time (sec) | Timeout (sec) | Loops |
 864 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
 865 | OK     | LPC1768 | ARM       | UT_1    | Basic                |        2.43        |       10      |  1/1  |
 866 | OK     | LPC1768 | ARM       | UT_2    | Semihost file system |        2.43        |       10      |  1/1  |
 867 | OK     | LPC1768 | uARM      | UT_1    | Basic                |        2.43        |       10      |  1/1  |
 868 | OK     | LPC1768 | uARM      | UT_2    | Semihost file system |        2.43        |       10      |  1/1  |
 869 | OK     | LPC1768 | GCC_ARM   | UT_1    | Basic                |        2.43        |       10      |  1/1  |
 870 | OK     | LPC1768 | GCC_ARM   | UT_2    | Semihost file system |        2.43        |       10      |  1/1  |
 871 | OK     | LPC1768 | GCC_CR    | UT_1    | Basic                |        3.44        |       10      |  1/1  |
 872 | OK     | LPC1768 | GCC_CR    | UT_2    | Semihost file system |        3.43        |       10      |  1/1  |
 873 +--------+---------+-----------+---------+----------------------+--------------------+---------------+-------+
 874 Result: 8 OK
 875
 876 Completed in 55.85 sec
 877 ```