Assert statements are a great tool for programming defensively. This is especially true in embedded systems where we don't typically have a lot of user interface to help our users figure out an error. Often it's better to crash or reset the application programmatically than risk executing the code in and undefined state.
But how do you write unit tests for code that can assert? This can be tricky with unit test tools, especially in C. It's also difficult if we're using a host-based testing strategy and running our tests on host PC.
As part of our unit tests we want to both allow asserts to happen and to verify whether or not an assert occurred in some particular condition.
The first thing to consider is what an assert really is -- it's an alternate return path from a function. When an assert failure occurs we're essentially aborting the entire program. When we're testing a single function in a unit test though, an assertion failure just looks like a different way for the function to return.
For example, consider a function that we might want to test, that looks like this:
void foo(mystruct_t* mystruct){
// Assert when a null pointer is detected.
ASSERT(NULL != mystruct);
// Do something with the struct.
mystruct->a = 1;
mystruct->b = 2;
}This trivial function manipulates a custom structure passed in through a pointer. The code here does a defensive check on the pointer passed in, and asserts if it's a null pointer -- so we don't try to modify any memory we shouldn't be touching (and avoid getting oursleves into more trouble).
Getting it to build and run
The first problem is just to get our unit tests to run (with all our assert-filled code) on our host PC.
Consider a custom assert implementation that might look something like this:
#define ASSERT(condition) if (!(condition)) reset()In this system reset() is called in the case of an assertion failure. Here reset() is some hardware specific function to do a software reset of the target. When running host based tests however, this function likely won't exist. If you write a test that calls foo() and an assert does occur, you'll get an error when it's time for reset to be called.
So the first thing we need to do is redefine our ASSERT implementation when we're running tests. Conveniently, when running unit tests with Ceedling the macro TEST is defined. This makes it relatively simple to create an alternate definition of ASSERT:
#if TEST
// Our fake ASSERT implementation.
#define ASSERT(condition)
#else
// The original ASSERT implementation.
#define ASSERT(condition) if (!(condition)) reset()
#endifIn this naïve example, we essentially remove the assertion checks by defining ASSERT to an empty macro. This allows us to write a test that uses a valid pointer like this:
void test_foo_sets_values_on_valid_struct(void)
{
mystruct_t ms = { .a = 5, .b = 5 };
foo(&ms);
TEST_ASSERT_EQUAL(1, ms.a);
TEST_ASSERT_EQUAL(2, ms.b);
}Aborting on an assert failure
So far our tests work when we don't have an assertion failure. However if we write a test that passes a null pointer, we're going to have a problem. Say the test looks like this:
void test_foo_does_nothing_on_a_null_struct_pointer(void)
{
foo(NULL);
}This isn't a great example (because it doesn't really test anything) but if we run it we'll see an error something like this:
ERROR: Test executable "test_code_under_test.out" failed.
> Produced no final test result counts in $stdout:
1 [main] test_code_under_test.out 11788 cygwin_exception::open_stackdumpfile: Dumping stack trace to test_code_under_test.out.stackdump
> And exited with status: [0] (count of failed tests).
> This is often a symptom of a bad memory access in source or test code.The problem is that because we've removed the null assertion check, foo continues to run to completion even when we pass it a null pointer, and it attempts to modify the memory at the null address.
To fix this we need foo to stop executing and return when the assertion check fails. Now we could put a return statement in our fake ASSERT macro to stop execution... but that is not going to work universally for all functions because different functions are going to have different return types. You'll end up with nagging warnings when running your tests.
A cleaner solution is to use an "exception" which implements a kind of "jump" out of a function. If you're using Ceedling, there is the handy CException library right at your finger tips (since it's used in Ceedling's implementation). To use it in your tests, just set the project, use_exceptions setting in the project.yml configuration to TRUE:
:project:
:use_exceptions: TRUE # <-- Set this TRUE (it's FALSE by default).
:use_test_preprocessor: TRUE
:use_auxiliary_dependencies: TRUE
:build_root: buildThen you use CException by including CException.h. For our purposes we want to throw an exception when an assertion fails, which we can do in our fake ASSERT macro like this:
#if TEST
#include "CException.h"
#define ASSERT(condition) if (!(condition)) Throw(0)
#else
#define ASSERT(condition) if (!(condition)) reset()
#endifHere we call Throw(0) to throw the exception (0 is just an integer to identify what we're throwing, but we're not going really going to use this).
Now if we run our test on the null pointer, we get a result like this:
[test_code_under_test.c]
Test: test_foo_does_nothing_on_a_null_struct_pointer
At line (54): "Expected 0x5A5A5A5A Was 0x00000000. Unhandled Exception!"Here we threw an exception, but we didn't handle it. We'll need to put in a CException Try block to handle the exception. And -- while we're at it -- what we really want is to hide the exception nonsense under some macros.
Expecting assertion failures
When you're writing tests, you're going to want to test that assertions occur under certain conditions. For our null pointer check, a useful test might look like this:
void test_foo_asserts_on_null_struct(void)
{
mystruct_t * null_struct = NULL;
TEST_ASSERT_FAIL_ASSERT(foo(null_struct));
}In this test we pass a null pointer to foo and we want to test that an assert failure does occur. Here we've introduced a new macro TEST_ASSERT_FAIL_ASSERT(). This macro will cause our test to fail if an assertion failure does not occur. This is where we're going to put the Try block:
#define TEST_ASSERT_FAIL_ASSERT(_code_under_test) \
{ \
CEXCEPTION_T e; \
Try { \
_code_under_test; \
TEST_FAIL_MESSAGE("Code under test did not assert"); \
} Catch(e) {} \
}Here we're just calling the code we pass to the function inside the Try block and failing the test if we don't get an exception.
Similarly, we can create a macro to expect that an assert will not occur like this:
#define TEST_ASSERT_PASS_ASSERT(_code_under_test) \
{ \
CEXCEPTION_T e; \
Try { \
_code_under_test; \
} Catch(e) { \
TEST_FAIL_MESSAGE("Code under test failed an assertion"); \
} \
}And write tests that ensure that we do not get an assert under specific conditions:
void test_foo_does_not_assert_on_vaild_struct(void)
{
mystruct_t ms;
TEST_ASSERT_PASS_ASSERT(foo(&ms));
}A good place to put both of these test macros would be their own test support file in the test/support folder, because this folder path is included by default. A filename like assert_test_helpers.h might be a good choice. Also, make sure to include CException.h wherever you put these macros.
Summary
Exceptions are just alternate return paths from functions, but in embedded environments our assert handling is often implemented in hardware specific ways. This makes the assertion behavior difficult to test.
To get around this we can create special assert implementations (with an exception library) that get used only when running host-based unit tests. In addition, we can create helper macros that allow us to control whether or not we expect an assert to occur in a test.
Finally, this allows us to write useful unit tests for conditions where asserts do occur and do not occur, all while allowing other traditional (not worried about asserts) unit tests for other behavior.