CMock vs FFF - A Comparison of C Mocking Frameworks

Did you know you have options when it comes to creating mocks for your C-language unit tests?

I've been spending a lot of time working with CMock -- since it's used by Ceedling -- but I've just been checking out FFF (the "fake function framework"). It's well done and I think it deserves a closer look.

To compare CMock and FFF, I created an example software module (the event_processor) to build some unit tests for. The event_processor interacts with a display module, which is the interface that is mocked during the tests.

Note that a mocking framework is nothing without a unit test framework (i.e. some "test" or "assert" functions to use in our tests).

In the CMock tests, I use Unity as the unit test framework. To preserve consistency in the FFF tests, I've created a simple unit test framework of my own, with a TEST_ASSERT_EQUAL() macro identical to the one provided by Unity. It's loosely based on MinUnit.

Setup

CMock

I did not find it possible to configure CMock to use it on it's own. What little documentation there is was no help. So instead for these examples I used CMock in a Ceedling project. Ceedling just makes it a whole lot easier to autogenerate the mocks needed for each test.

FFF

FFF is very simple to set up. It comes as a single header file (fff.h) which you can drop right into your source folder. Then you just #include "fff.h" and you're ready to start using it in you tests.

One note: I did need to #include <string.h> ahead of the include for fff.h to get it to build. Looks like FFF is missing an include.

You create a fake function by using either the FAKE_VOID_FUNC() or FAKE_VALUE_FUNC() macro. These macros take the function name, arguments and return value (if applicable) as arguments.

There are also two other important FFF macros to be aware of. RESET_FAKE() is used clear the internal stats for each fake function. FFF_RESET_HISTORY() resets the call history for all fake function calls. Typically, both of these would be called before each test. These calls are left out of the examples below for clarity, but are done in the "set up" function of the test framework... before each test is executed.

Now lets look at a few different examples of how to use each framework.

Test that a single function was called

CMock

void test_whenTheDeviceIsReset_thenTheStatusLedIsTurnedOff(void)
{
  // Then
  display_turnOffStatusLed_Expect();

  // When
  event_deviceReset();
}

FFF

FAKE_VOID_FUNC(display_turnOffStatusLed);

void test_whenTheDeviceIsReset_thenTheStatusLedIsTurnedOff(void)
{
  // When
  event_deviceReset();

  // Then
  TEST_ASSERT_EQUAL(1, display_turnOffStatusLed_fake.call_count);
}

A lot of times, all I want to do is verify that a particular function was called.

With CMock, I set up the "call chain" before calling the function under test. CMock itself verifies that everything listed in the call chain is called correctly. Note that this usually means reversing the order of the "then" and the "when" in my test.

With FFF, each "fake function" behaves more like a traditional mock object. I create the mock, call the function under test, and then I can inspect the mock after-the-fact. I use the unit test framework to make sure it was called the correct number of times. This lets me write the test in the logical given-when-then order that things actually occur.

Another issue here is when an expectation fails with CMock, the line number reported is the line number of the start of the test function... which is not very much help. When you use the unit framework to inspect the mock (as with FFF), any failures yields the real line number at which the failure occurred.

Test that a single function was NOT called, while other functions may be called

CMock

void test_whenTheDeviceIsReset_thenTheStatusLedIsNotTurnedOn(void) 
{
  // Then
  // NOTE: We have to ignore a function we know could be called.
  display_turnOffStatusLed_Ignore();

  // When
  event_deviceReset();
}

FFF

FAKE_VOID_FUNC(display_turnOnStatusLed);

void test_whenTheDeviceIsReset_thenTheStatusLedIsNotTurnedOn()
{
  // When
  event_deviceReset();

  // Then
  TEST_ASSERT_EQUAL(0, display_turnOnStatusLed_fake.call_count);
}

CMock requires that every expected function call appears in the call chain. Even if we don't care when or how often a particular function is called, we need to "ignore" it explicitly. If we're not careful, this can lead to long chains of expectations.

With FFF, calls are ignored by default. As long as we've created a fake function, we don't need to tell FFF what we're expecting. We can control what to test for ourselves after running the function under test.

I think this is the single biggest benefit of using FFF. By only looking at the interactions that we actually care about, it becomes possible to write simpler (and less fragile) tests.

Test that a single function was called with the correct argument

CMock

void test_whenTheVolumeKnobIsMaxed_thenVolumeDisplayIsSetTo11(void)
{
  // Then
  display_setVolume_Expect(11);

  // When
  event_volumeKnobMaxed();
}

FFF

FAKE_VOID_FUNC(display_setVolume, int);

void test_whenTheVolumeKnobIsMaxed_thenVolumeDisplayIsSetTo11(void)
{
  // When
  event_volumeKnobMaxed();

  // Then
  TEST_ASSERT_EQUAL(1, display_setVolume_fake.call_count);
  TEST_ASSERT_EQUAL(11, display_setVolume_fake.arg0_val);
}

The FFF version includes one extra line because the function call count is tested independently from the expected value.

Test a sequence of calls

CMock

void test_whenTheModeSelectButtonIsPressed_thenTheDisplayModeIsCycled(void)
{ 
  // Then 
  display_setModeToMinimum_Expect();
  display_setModeToMaximum_Expect();
  display_setModeToAverage_Expect();

  // When
  event_modeSelectButtonPressed();
  event_modeSelectButtonPressed();
  event_modeSelectButtonPressed();
}

FFF

FAKE_VOID_FUNC(display_setModeToMinimum); FAKE_VOID_FUNC(display_setModeToMaximum); FAKE_VOID_FUNC(display_setModeToAverage);

void test_whenTheModeSelectButtonIsPressed_thenTheDisplayModeIsCycled(void) 
{
  //When
  event_modeSelectButtonPressed();
  event_modeSelectButtonPressed(); 
  event_modeSelectButtonPressed();

  // Then
  TEST_ASSERT_EQUAL(fff.call_history[0],
          (void *)display_setModeToMinimum);
  TEST_ASSERT_EQUAL(fff.call_history[1],
          (void *)display_setModeToMaximum);
  TEST_ASSERT_EQUAL(fff.call_history[2],
          (void *)display_setModeToAverage);
}

The CMock syntax here is a bit nicer, but I'd argue that we don't want to make tests like this (with long call chains) easier to write. This is the slippery slope to brittle and un-maintainable tests.

Mock a return value from a function

CMock

void test_givenTheDisplayHasAnError_whenTheDeviceIsPoweredOn_thenTheDisplayIsPoweredDown(void)
{
  // Given
  display_isError_ExpectAndReturn(true);

  // Then
  display_powerDown_Expect();

  // When
  event_devicePoweredOn();
}

FFF

FAKE_VALUE_FUNC(bool, display_isError);
FAKE_VOID_FUNC(display_powerDown);

void test_givenTheDisplayHasAnError_whenTheDeviceIsPoweredOn_thenTheDisplayIsPoweredDown(void)
{
  // Given
  display_isError_fake.return_val = true;

  // When
  event_devicePoweredOn();

  // Then
  TEST_ASSERT_EQUAL(1, display_powerDown_fake.call_count);
}

Both tests are pretty simple, although the FFF test can be written in the correct "given-when-then" order.

Mock a function with a value returned by reference

CMock

Note: This CMock test doesn't work out-of-the-box. The "returnthruptr" and "ignorearg" plug-ins need to be enabled in the Ceedling project.yaml file._

void test_givenTheUserHasTypedSleep_whenItIsTimeToCheckTheKeyboard_theDisplayIsPoweredDown(void)
{
  // Given
  char entry[] = "sleep";
  display_getKeyboardEntry_Expect(0,0);
  display_getKeyboardEntry_ReturnArrayThruPtr_entry(entry, strlen(entry));
  display_getKeyboardEntry_IgnoreArg_entry();
  display_getKeyboardEntry_IgnoreArg_length();

  // Then
  display_powerDown_Expect();

  // When
  event_keyboardCheckTimerExpired();
}

FFF

FAKE_VOID_FUNC(display_getKeyboardEntry, char *, int);
FAKE_VOID_FUNC(display_powerDown);

void test_givenTheUserHasTypedSleep_whenItIsTimeToCheckTheKeyboard_theDisplayIsPoweredDown(void)
{
  // Given
  char mockedEntry[] = "sleep";
  void return_mock_value(char * entry, int length)
  { 
      if (length > strlen(mockedEntry))
      { 
          strncpy(entry, mockedEntry, length); 
      }
  } 
  display_getKeyboardEntry_fake.custom_fake = return_mock_value;

  // When
  event_keyboardCheckTimerExpired();

  // Then
  TEST_ASSERT_EQUAL(1, display_powerDown_fake.call_count);
}

CMock starts to get a little hairy in this test. Writing this test requires a lot of knowledge of the CMock API. With FFF, you do a little more work in the test, but it's just C so you already know how to do it.

When writing the CMock version, I had to refer back to the documentation a few times, but with FFF, I got it working on the first try.

Mock a function with a function pointer parameter

CMock

void test_givenNewDataIsAvailable_whenTheDisplayHasUpdated_thenTheEventIsComplete(void)
{
  // A mock function for capturing the callback handler function pointer.
  void(*registeredCallback)(void) = 0;
  void mock_display_updateData(int data, void(*callback)(void), int numCalls)
  {
      //Save the callback function.
      registeredCallback = callback;
  }

  // Given
  display_updateData_StubWithCallback(mock_display_updateData);
  event_newDataAvailable(10);

  // When
  if (registeredCallback != 0)
  {
      registeredCallback();
  }

  // Then
  TEST_ASSERT_EQUAL(true, eventProcessor_isLastEventComplete());
}

FFF

typedef void (*displayCompleteCallback) (void);
FAKE_VOID_FUNC(display_updateData, int, displayCompleteCallback);

void test_givenNewDataIsAvailable_whenTheDisplayHasUpdated_thenTheEventIsComplete(void)
{
  // A mock function for capturing the callback handler function pointer.
  void(*registeredCallback)(void) = 0;
  void mock_display_updateData(int data, void(*callback)(void))
  {
      //Save the callback function.
      registeredCallback = callback;
  }
  display_updateData_fake.custom_fake = mock_display_updateData;

  // Given
  event_newDataAvailable(10);

  // When
  if (registeredCallback != 0)
  {
      registeredCallback();
  }

  // Then
  TEST_ASSERT_EQUAL(true, eventProcessor_isLastEventComplete());
}

In this case, both CMock and FFF work similarly. You register your own function to be called when the mock is called, and set up your function to capture the provided function pointer. Then you can manually do whatever you need to with it.

Summary

CMock and FFF take very different approaches.

With CMock, you run it ahead of time on a header files to generate a whole bunch of mock functions for you. There is a lot more "pre-processing machinery" that runs ahead of time.

And with CMock, you need to explicitly declare all expected function calls ahead of time. This can lead to tight coupling of tests and code.

FFF is much lighter weight. You just include the single header file, and you're ready to go.

It works more like a traditional "mock object" you'd see in an object-oriented mocking framework. There is no need to set up all expectations ahead of time -- you just inspect the mock when you're done... and only for the things that you care about.

CMock is certainly easy to get started with via Ceedling, which provides automatic discovery, generation and running of tests and mocks.

FFF has me excited to work with it further though. In particular, being able to write tests in a more logical given-when-then order, and the ability to ignore mocked calls by default looks to be powerful.

With FFF there is a bit more manual work to set up each test, but I think I may be able to write some better quality tests.

Get the test project source on GitHub.

Mocking Embedded Hardware Interfaces with Ceedling and CMock

Get the source code for this example on GitHub.

How can you unit test your embedded software? What about your hardware dependencies?

The secret is mocking.

We can mock the interfaces to our hardware so that we don't need the actual hardware to test. This allows us to run our tests more quickly and before the hardware might even be available.

The Plan

If we're developing the software for an embedded microcontroller, we're probably going to be using the microcontroller-provided hardware modules for things like SPI, I2C, timers, etc.

Application software interfaces to microcontroller hardware.

Application software interfaces to microcontroller hardware.

For each of these hardware interfaces, we want to have a corresponding software module containing the microcontroller hardware dependencies (i.e. hardware register accesses).

Each hardware interface has a corresponding software module.

Each hardware interface has a corresponding software module.

We can then mock each of these hardware interfaces, eliminating our hardware dependencies but still allowing us to unit test our application. Instead of compiling these tests for the embedded microcontroller, we compile them for and run them on our host PC.

Mocks for software modules allow us to isolate the application software from the hardware.

Mocks for software modules allow us to isolate the application software from the hardware.

To help you create your mocks you want to use a mocking framework. The mocking framework included with Ceedling is CMock. It allows you to create mocks of individual software modules from their header files. Ceedling improves the experience by automatically using CMock to generating the mocks that we need.

A Test Driven Example

Note that this example assumes that we already have an existing Ceedling project. See my other article for help creating one.

Imagine that we want to talk to an external I2C temperature sensor.

Microcontroller connected to external temperature sensor via I2C.

Microcontroller connected to external temperature sensor via I2C.

Create the Temperature Sensor Module

Let's create a module that will be our temperature sensor driver.

$ rake module:create[tempSensor]
Generating 'tempSensor'...
mkdir -p ./test/.
mkdir -p ./src/.
File ./test/./test_tempSensor.c created
File ./src/./tempSensor.c created
File ./src/./tempSensor.h created

Write Our First Test

What is the first thing I want to be able to do with this sensor? I'd like to be able to read the current temperature value.

Cool. So I take a look at the datasheet for my fictional temperature sensor and I can see that it has a bunch of 16-bit registers -- each with 8-bit addresses -- one of which is the temperature register.

The scaling of the values is such that a register value of 0 is -100.0°C and a register value of 0x3FF is +104.6°C. This makes each bit equivalent to 0.2°C.

The senor temperature register values and their corresponding temperatures.

The senor temperature register values and their corresponding temperatures.

Now lets add our first test to test_tempSensor.c. I want to know that when I read a temperature register value of 0x3FF that the temperature calcualted is 104.6.

void test_whenTempRegisterReadsMaxValue_thenTheTempIsTheMaxValue(void)
{
    uint8_t tempRegisterAddress = 0x03;
    float expectedTemperature = 104.6f;
    float tolerance = 0.1f;

    //When
    i2c_readRegister_ExpectAndReturn(tempRegisterAddress, 0x3ff);

    //Then
    float actualTemperature = tempSensor_getTemperature();
    TEST_ASSERT_FLOAT_WITHIN(tolerance, expectedTemperature, 
        actualTemperature);
}

First we set up some variables to hold our expected values. Then in the "when" clause, we need to simulate (or mock) the I2C module returning a value of 0x3ff on a read of the temperature address.

For the moment, we pretend that there is another i2c module (it doesn't actually exist yet) which handles the I2C communication with the temperature sensor. This is where our hardware dependent code will eventually go.

So, the i2c_readReadgister_ExpectAndReturn function is actually a mock function used to simulate a call to a function called i2c_readRegister in the i2c module. We'll come back to this in a moment.

The "then" clause is where we test that the tempSensor module actually returns the correct temperature when we call tempSensor_getTemperature. This function doesn't exist yet either.

Create the Function Under Test

Lets create the tempSensor_getTemperature function with a dummy implementation:

tempSensor.h:

# ifndef tempSensor_H
# define tempSensor_H

float tempSensor_getTemperature(void);

# endif // tempSensor_H

tempSensor.c:

# include "tempSensor.h"

float tempSensor_getTemperature(void)
{ 
    return 0.0f;
}

Mock the I2C Interface

If we try and run the test now, the compiler will complain that it doesn't know about the i2c_readReadgister_ExpectAndReturn mock function. This is because the i2c_readRegister function doesn't exist and we haven't yet told Ceedling to mock it.

We don't actually need to implement this function however. It's enough to declare the function prototype in a header file and tell Ceedling to mock it with CMock.

Create the header file, i2c.h:

# ifndef i2c_H
# define i2c_H

# include <stdint.h>

uint16_t i2c_readRegister(uint8_t registerAddress);

# endif // i2c_H

The way we tell Ceedling to mock this module is to add this line to test_tempSensor.c:

# include "mock_i2c.h"

This tells Ceedling: You know the i2c.h header you see over there? Well... use CMock to generate the implementation and compile it in for us, okay?

When CMock gets a hold of the header file it looks at all the functions defined there and generates several mock functions for each... including the i2c_readRegister_ExpectAndReturn function we used in the test. This mock function appends an additional argument to the original i2c_readRegister function, which is the value we want the function to return to the calling function.

For more details on all the mock functions available with CMock, see the CMock documentation.

Implement the Function Under Test

Now we can implement the logic for our tempSensor_getTemperature function. Our new tempSensor.c is:

# include "tempSensor.h"
# include "i2c.h"
# include <stdint.h>

float tempSensor_getTemperature(void)
{
    uint16_t rawValue = i2c_readRegister(0x03);

    return -100.0f + (0.2f * (float)rawValue);
}

If we run our test, it should pass now.

Adding Another Test

We'll next want to add more tests for other possible return values from i2c_readRegister. This is easily done by changing the return value provided to the mock function.

For example, to test that the minimum temperature value is read correctly:

void test_whenTempRegisterReadsMinValue_thenTheTempIsTheMinValue(void)
{
    uint8_t tempRegisterAddress = 0x03;
    float expectedTemperature = -100.0f;
    float tolerance = 0.1f;

    //When
    i2c_readRegister_ExpectAndReturn(tempRegisterAddress, 0x0);

    //Then
    float actualTemperature = tempSensor_getTemperature();
    TEST_ASSERT_FLOAT_WITHIN(tolerance, expectedTemperature,
        actualTemperature);
}

Now we have a driver for an external hardware device that we can test without any of the hardware. We can continue to develop the driver -- adding more tests and features -- by building and testing on our host PC.

By putting all of the microcontroller-dependent I2C operations into their own module, we easily mocked them with Ceedling and CMock. In fact, we didn't even have to implement this module yet -- we just had to define its interface in the header file.

Using our mocks, we created unit tests that verify the behavior of our temperature sensor driver. As the rest of our application is developed, we can easily run these unit tests at any time to make sure the driver will still work correctly.

Get the source code for this example on GitHub.

Using Rake to Build a Simple C Application

In this example we'll set up a simple build system for a C application to be complied with gcc. Using a similar method, we could use a cross-compliler (gcc or otherwise) to compile for whatever embedded target we would like.

Read More