Over the years I've worked in and with many different firmware organizations. In my experience most people are really trying to do their best but I see many struggling with the same, common problems of organizational maturity.
When an organization is mature it makes disciplined use of best practices and tools to regularly and continuously deliver high quality firmware.
When an organization is immature, things are done in an ad hoc (often reactionary) way. Successful results are not easily repeatable and it is often the responsibility of just one or two individuals to try to hold it all together.
Here are some questions to ask to assess a firmware organization's maturity:
- Do you use source control?
- Do you use feature branches?
- Can you set up a new developer in just a few hours?
- Can you build in one step?
- Can you deploy a release without a developer?
- Do you have requirements?
- Is your architecture documented?
- Does everyone know the schedule?
- Do you do independent testing?
- Do you have automated unit tests?
- Do you have automated hardware-in-loop tests?
- Do you run your automated tests with each deployment?
You get one point for each question you can answer with yes. When you're at a 12 you're set up to efficiently deliver quality code. The lower your score, the closer you are to chaos.
This is inspired by the Joel test, but it's a modern version specifically for firmware development.
Why not do these things? The most common objection is that "there isn't time." The irony though is that these best practices are an investment. Choosing not to do them is a short-sighted approach — in the long term they will save you tons of time.
1. Do you use source control?
I really hope I don't have to tell you to use source control by now. And, you should probably be using Git. It's the modern, de-facto standard which the best developers want to use (they'll probably complain if you don't). And when you use a service like Github, Bitbucket or GitLab, you get really great support for branching, code reviews, merging and build server integration.
2. Do you use feature branches?
Source control is great, but it can be a nightmare if you can't use it effectively to isolate and track changes. The simplest way to do this is with feature branches. Create new branches for each feature. This keeps all the changes for a particular feature in one place where they can be reviewed before merging to the main development branch.
3. Can you set up a new developer in just a few hours?
If you can set up a developer quickly it's likely that you have a well-documented, controlled, and easily-reproducible build environment. If it takes longer than this you probably have a development environment that is understood by only a few people (or no one!) that is difficult to set up. In this case I'd bet that you're also not using the same environment to develop and build all of your releases.
4. Can you build in one step?
Your IDE will usually let you press a single button to compile, but that's not usually enough here. A complete build typically involves additional steps like building other libraries, running code generation tools, or running post-build scripts (for things like changing formats or combining binaries). Another common situation involves producing different binaries from the same code — either through #defines or other source file changes.
You won't need to handle each of these on every compile during development, but in order to produce a full, clean build — for test or production — you want to be able build everything from its source. That way you know exactly what is going into each build, each time.
Any manual steps are prone to error or likely to be forgotten. Don't make people have to remember how to do it. Give them a script that makes it easy by building everything in one step.
5. Can you deploy a release without a developer?
If you rely on your development team to produce all of the release builds it's going to slow them down and make it harder to get releases out regularly. Even worse is relying on a single person who knows how to do a build. This is a huge risk that will become incredibly painful should they ever decide to leave (or aren't available temporariliy).
Alternatively, if your build system automatically generates builds on every source code push you never have to wait for the development team to "do a build." You always have the latest build that can be pulled, tested or deployed... by anyone.
6. Do you have requirements?
If you don't know what you need to build then writing code is likely to be a waste of time. Sure you can experiment and prototype here-and-there, but in order to build the right thing someone has to figure out what that is. And write it down. You can call it something else — e.g. a specification, feature list, user stories, etc. — but these are requirements.
7. Is your architecture documented?
After building the darn thing, finding your way around the code is the next hardest problem when coming up-to-speed on a project. Documenting the architecture helps your team do this.
When everyone understands the architecture, they are more likely to make changes that work with it rather than against it. These changes are also faster to implement and less error prone.
8. Does everyone know the schedule?
Each stakeholder should know and understand the schedule. What needs to be delivered when, and how their piece fits into the whole. This is about the team having a "shared vision" that they are working toward. This helps everyone make the right micro-decisions about how and what to work on as they go along.
9. Do you do independent testing?
Developer testing is good — you need your developers to test their code as they write it. But time and time again "super-confident-developer-finished" code ends up with issues when tested independently. This doesn't necessarily have to be done be separate test group, it could be done by another developer on the team. Just changing the perspective a bit helps a lot with finding problems, especially unintended regression issues.
10. Do you have automated unit tests?
This is a particular favorite of mine. When you have automated unit tests, developers can quickly and easily detect things that they've broken accidentally. The tests are documentation (as executable code) for what the code should do. Testing at the unit level also detects bugs that will be difficult or impossible to find at the system level.
11. Do you have automated hardware-in-loop tests?
In a hardware-in-loop (HIL) test you have extra hardware connected around your device that allows you to test the entire system including the hardware/software interfaces. This sort of testing can save a ton of time typically spent in manual testing for each release, letting you deploy releases faster and with more confidence.
12. Do you run your automated tests with each deployment?
Once you have automated tests in place, the last step is to make sure that they all run (and pass!) each time a new release build is deployed. This can be done manually, but the best way to do this is with a build server. With a build server you can run all the tests on every push to source control. This makes detecting problems automatic and prevents the release of bugs to the field.