OpenUxAS/tests/cpp/README.md

Table of Contents

1. Background
2. Integration of GCOV in OpenUxAS
   1. Compilation Options
   2. Modifications to UxAS executable
   3. Usage during testsuite run
3. How to write tests using the pylmcp module
   1. Setup UxAS
   2. Launch server
   3. Execute the test

Background

Gcov is a test coverage program. Use it in concert with GCC to analyze your programs to help create more efficient, faster running code and to discover untested parts of your program. You can use gcov as a profiling tool to help discover where your optimization efforts will best affect your code. You can also use gcov along with the other profiling tool, gprof, to assess which parts of your code use the greatest amount of computing time. (from gcov documentation at link).

The main advantage of gcov is that it is freely available as part of the GCC toolchain. The tool can be used for statement coverage at source level and decision coverage at object level. An important note is that this tool cannot be used in the context of DO178 certification because of some limitations.

As gcov uses instrumentation to gather coverage information the program must be compiled with GCC using some special options. Also, gcov requires a file system to dump coverage information (so it won’t be suitable for purely embedded development).

Integration of GCOV in OpenUxAS

Compilation Options

The following compilation options were used: -fprofile-arcs -ftestcoverage.

Modifications to UxAS executable

The UxAS executable is meant to run indefinitely. Gcov by default dumps all the coverage information at program termination. To ensure that the UxAS program dumps the coverage information during testing, we had to add some code that ensures that when receiving a SIGINT signal we do manually the dumps. The resulting patch was:

diff --git a/src/UxAS_Main.cpp b/src/UxAS_Main.cpp
index 9e3fe289..0b4951d9 100644
--- a/src/UxAS_Main.cpp
+++ b/src/UxAS_Main.cpp
@@ -37,6 +37,10 @@
#include <thread>
#include <locale>

+#ifdef GCOV_MODE
+#include <csignal>
+#endif
+
#define ARG_CFG_PATH "-cfgPath"
#define ARG_VERSION "-version"
#define ARG_RUN_UNTIL "-runUntil"
@@ -47,6 +51,15 @@

#define BEFORE_LOG_MANAGER_INITIALIZATION_LOG_MESSAGE(message) std::cout << message << std::endl; std::cout.flush();

+#ifdef GCOV_MODE
+extern "C" void __gcov_flush();
+
+void signalHandler( int signum ) {
+    __gcov_flush();
+    std::exit(signum);
+}
+#endif
+
int
main(int argc, char** argv)
{
@@ -67,6 +80,10 @@ main(int argc, char** argv)
    // declare relative paths of configuration files with default values
    // example arguments: -cfgBasePath ./cfg/cfgbase2.xml
    //
+
+    #ifdef GCOV_MODE
+    signal (SIGINT, signalHandler);
+    #endif
    std::string cfgPath{"cfg.xml"};
       uint32_t runUntil_sec = 0;

In order not to include the code when not performing coverage activities, the additional code is controlled by the C macro GCOV_MODE (code is enabled by adding -DGCOV_MODE=1 during compilation).

Usage during testsuite run

Gcov produces various files both at compilation time and execution time:

1. *.gcda: this is the coverage information produce during the execution
2. *.gcno: contains information to reconstruct the basic block graphs and assign source line numbers to blocks. This file is produced during compilation and placed with the object files
3. *.gcov: file produced by gcov once the gcda and gcno files are available. These files are basically annotated source files

During test-suite runs, we use the options: GCOV_PREFIX and GCOV_PREFIX_STRIP in order to produce the gcda files inside the directory specific to the testsuite. We ensure that at the beginning of each testsuite run, the gcda files are reset.

Once all tests have been executed, we gather gcno files along with the produced gcda files and call the gcov tool to generate *.gcov files (one for each source). Some of the produced files are then ignored as not part of the project (system includes, …, …)

For more details see the function dump_gcov_summary in run-tests. The function is quite generic. The only part that might need adjustment is the invocation of gcov itself. The directory from where it should be invoked might depend on the build itself as some path information (to locate sources) stored in the gcno files might be relative paths.

How to write tests using the pylmcp module

In the testsuite, each test case will define a specific scenario. In the following paragraphs, we will explain how the test arv/correct_automation_request/test.py has been created. It can be modified to create other tests based on the same pattern.

Setup UxAS

The first thing to do is to configure the bridge.

bridge_cfg = UxASConfig()
bridge_cfg += AutomationRequestValidator()

We initialize the bridge_cfg variable to the base Object that contains the configuration, UxASConfig(). Then, the desired service is added to the configuration. Here, it is the AutomationRequestValidator. The services are defined in pylmcp/uxas.py. It is possible to add more service just by creating a new class in this package. The script will create the corresponding config.xml passed to UxAS.

Launch server

After adding the configuration to bridge_cfg, the server can be launched along with UxAS

with Server(bridge_cfg=bridge_cfg) as server:
	try:

After this line, the python script UxAS is running and it is possible to send messages on the bridge with the python code.

Execute the test

The first thing done is to send configuration messages to the ARV service so it can respond to the Automation Request.

for obj in (Object(class_name='AirVehicleConfiguration', ID=400,
		   randomize=True),
	    Object(class_name='AirVehicleConfiguration', ID=500,
		   randomize=True),
	    Object(class_name='AirVehicleState', ID=400,
		   randomize=True),
	    Object(class_name='AirVehicleState', ID=500,
		   randomize=True),
	    Object(class_name='KeepInZone', ZoneID=1,
		   randomize=True),
	    Object(class_name='KeepOutZone', ZoneID=2,
		   randomize=True),
	    Object(class_name='OperatingRegion', ID=3,
		   KeepInAreas=[1], KeepOutAreas=[2]),
	    Object(class_name='cmasi.LineSearchTask', TaskID=1000,
		   randomize=True),
	    Object(class_name='TaskInitialized', TaskID=1000,
		   randomize=True)):
    server.send_msg(obj)
    time.sleep(0.1)

We send nine messages on the bridge. Each message corresponds to an instance of an Object. Let’s focus on the first one, and decompose it:

Object(class_name='AirVehicleConfiguration', ID=400,
randomize=True)

The class name corresponds to the message name in the mdms files. When it is the only message with this name, it is possible to only put the name, and not the full path. However, when message names can clash, it is needed to put the path that disambiguates it, e.g. for LineSearchTask, we use class_name='cmasi.LineSearchTask'. Then, it is possible to set the attributes that need to be re-used later, or that we want to keep fixed. Here, it is done with the ID attribute that will be used in the Automation Request (in the requested EntityList). Finally, all attributes that are not set can be randomized. This is enabled by adding randomize=True to the object initialization.

For each created Object, we send it on the bridge using the server.send_msg(Object) method, and wait for 0.1 second to give some time to UxAS to process the message.

After the setup of UxAS internal data (Vehicle states, configurations, regions, …) has been done, we send the Automation Request:

obj = Object(class_name='cmasi.AutomationRequest',
	     TaskList=[1000], EntityList=[400, 500],
	     OperatingRegion=3, randomize=True)
server.send_msg(obj)

As said before, we can reuse the fixed attributes in the message to create a correct Automation Request to send to the ARV service. Since it is well formed, the service is supposed to send a corresponding Unique Automation Request. We can catch it with the server.wait_formsg method:

msg = server.wait_for_msg(
		descriptor='uxas.messages.task.UniqueAutomationRequest',
		timeout=10.0)

If the server doesn’t see an UniqueAutomationRequest message within 10 seconds, it will return an error and the test will end. When it is sent, the message is stored in the msg variable.

It enables the user two things:

• First, it is possible to assert properties on the received message. In the test, we assert the fact that the field OriginalRequest of the UniqueAutomationRequest is equal to the sent AutomationRequest:

  assert(msg.obj['OriginalRequest'] == obj)
  

• Second, it can be used to store data to further send coherent messages. In the test, the ARV service generates a unique ID corresponding to the Automation Request. If we want to check the nominal behavior of the ARV, it is necessary for it to receive a UniqueAutomationResponse with a ResponseID that is equal to the RequestID of the UniqueAutomationRequest within 5 seconds.

  unique_id = msg.obj.data["RequestID"]
  obj = Object(
  	       class_name='uxas.messages.task.UniqueAutomationResponse',
  	       ResponseID=unique_id, randomize=True)
  server.send_msg(obj)
  

  By storing the received RequestID, we are able to create a correct UniqueAutomationResponse.

The only action left is waiting for the corresponding AutomationResponse, and test that it is equal to the OriginalResponse in the sent UniqueAutomationResponse.

msg = server.wait_for_msg(descriptor="afrl.cmasi.AutomationResponse",
 timeout=10.0)
assert (msg.descriptor == "afrl.cmasi.AutomationResponse")
assert (msg.obj == obj['OriginalResponse'])

After executing the test, gcov and lcov can generate a coverage report that will point to the unread lines. It is possible to create as many tests as needed to cover the entire file.