Firmware sits at the edge between software and hardware. Often there are very little guard rails/guides in terms of how to go about writing good firmware all while different firmware operating on different devices face very different challenges and constraints. This is the understanding I have gained in over 3 years of building firmwre for 100kW+ conversion systems

All systems are different

When I started embedded electronics I had the thought that there would be some sort of 'universal' design principle, that when followed would produce clean code that one could understand effortlessly. One would read the code and simply get what is going on, it was just necessary to write it in the right way. While there definitely is a distinction between bad code and good code, there is no way to write code in such a way that it can be read and understood 'effortlessly'.

  • Real systems will have weird edge cases that require some confusing logic to handle well.
  • Real systems will have some sort of delivery date (even though in my experience these delivery dates are more flexible than it may seem).
  • Real systems will have memory and timing constraints that force a certain desing even though it may not be very legible.

This is all to say that real systems are actually pretty complicated. This in turn means, that the code for these systems cannot magically be simpler than that. This is in fact one of my key insights, compex systems are complex and that means that it will take some effort to understand why these systems are designed and act the way they do. Trying to 'abstract' certain thigns will in fact only make the system more complicated as now one needs to understand the system (which you will always need to) just to then also need to understand the abstraction to actually be able to figure out what the code does.

There are a few exceptions to this (like external memory should preferrably have code that looks like a file acess or simple read(address, size, &buffer) and write calls) but over all there is very little actual value in abstracting specific peripherals beyond writing convenience functions that translate access from weird sounding register acronyms into functions with more descriptive names.

Keep it simple stupid

There is no 'universal abstraction', the application can only be understood when it is 'embedded' in it's environment. So keep the code minimal, keep it simple stupid.

There is a related saying that comes from the 'zmq' guide that I have also found to be very very true: premature optimization is the root of all evil. This statement can be interpreted in different ways. One of the interpretations implies that code was optimized for performance. Performance optimizing, while technically challenging and interesting, causes more code to be written and that code to become less readable. This is unavoidable as performance optimization comes with increased complexity as some of the abstractions that are commonly used and that exist to make things more understandable for humans are discarded in order to allow for code that exploits certain traits of the hardware and program context to arange data and operations in such a way as to perform the same task in less time. Abandoning abstractions needs to be done carefully. Preferrably the part of the code that is optimized is encapsulated and documented so that the motivation and approach of the optimization is clear before starting to read the code.

When to write peripheral Drivers and when not to

An MCU would not be an MCU if it was not stuffed full of specialty peripherals. Most of the silicon area of modern MCUs or DSPs as they are sometimes called is devoted to these peripherals. For my work I tend to care about PWM generating peripherals (so timers mostly) and ADCs, together with some fancy interconnects and interrupts. Then there are more common things like USART, I2C, SPI and more and more common USB. Peripherals seem to fall into a few categories, Communication, Memory and what I'll call 'Special'. Drivers tend to make sense for comms and memory because drivers provide an abstraction over 'details'. In the case of communications interfaces, the 'details' are the setup and management of that particular peripheral, while the rest of the application only really cares about the 'pipe' (USART or USB-CDC) or 'transmission unit' (CAN, I2C, SPI) that has been received or needs to be sent. The API is relatively simple. As I use CAN a lot, I'll use this as an example. I care if I have received a CAN frame or I have a CAN frame to be sent. That means there needs to be a 'read/get_transmission_unit' function and a 'write/send_transmission_unit' function. The interface essentially is the transmission unit. The driver and all the things that interact with that 'bus' share a common struct that holds the transmission unit in the firmware. The peripheral drivers job is straight forward, and the details live in the 'init' function of the driver and it's tx/rx functions. I can port most of the 'logic' that actually matters to the application to another chip without really needing to care. Yes a new driver is needed but it slots in to the rest of the application without difficulty, using the same transmission unit struct as the driver for the previous chip. I2C may be a bit tricky, as it's read/write in a single operation, but an I2CTransaction struct can be written that encodes this properly. With such a struct in hand the same application logic (think readout procedure for a I2C connected Sensor) works on every chip that has a driver supporting the I2CTransaction struct.

The other end of the spectrum are the 'specialty' peripherals. These peripherals tend to be, well, special. You can write code for that peripheral, but it will only be useful on devices that ship this particular piece of hardware. On top of that, peripherals are often designed such that they support various applications through different configuration options. The job of the person writing the firmware is to figure out how the application that the MCU is used in maps to the particular peripheral of the chosen device. There is not much to 'abstract' here. The firmware is married to the quirks of the peripheral. There are ways for an application to state an 'intent' but this is pretty particular to the application. So while it may increase readability of the code, it will not really increase portability.

In this case a 'driver' as such also does not seem to make much sense. Sure playing around with the device to get to understand how the peripheral works is a good idea, and a debugger with a nice interface will definitively be a good idea here, but at the end of the day the best one can do is to understand the peripheral hardware itself and then write an init function for your application, maybe a few functions that quickly update parameters that need updating on the fly (like duty cycle in case of a PWM for a half bridge) and you are effectively done. Trying to write a driver has in my experience only led to awkward looking code that does stuff like keeping a copy of the hardware registers lying around for no good reason (the CPU can just read the real hardware registers after all). Treat the hardware registers as what they are, an 'object'. You can wrap this in a struct and that may be fine, but at the end you are just writing bits into registers. I have not seen a good reason to treat this like anything else.

There are some things like displays that are actual hybrids. The abstraction here is clear, a framebuffer. But the hardware wiring now includes DMA + SPI peripherals that need to be combined in very specific ways. So at the end you are kind of back to a 'specialty' hardware setup that happens to have a nice API (the framebuffer) on top. The app writes to the framebuffer through a singelton pointer and thats the end of it (sometimes there is fancy hardware to do color space transforms as well, but this is really just to make the software's life simpler and to free up CPU cycles for more interesting stuff).

The verification trap

When firmware developers write firmware for systems - or parts thereof - they sometimes try to address some axiety they might have. A lot of emphasis and code gets written to 'verify' something that is essentially guaranteed by the hardware itself, or really hard to actually verify.

There is a variant of this error checking behaviour that is particular to embedded systems: 'checking' peripheral configuration. In Desktop environments where many programs need to cooperate resource management dynamic allocation is key. Thus the code can not simply assume that certain resources are available. For a bare metal embedded project, life is a little simpler as there only is one thing that runs and it has access to all the resources, which avoids needing any coordination overhead. Eithe a peripheral is on the chip or it is not and that's that. In contrast to traditional software, the environment is known and the state of the system tends to be very consistent. It can therfore be assumed that a peripheral is available and operational at boot.

Assume it works, until it does not

Once a valid configuration procedure has been developed, it is ok to simply replay this configuration and assume it works (until proven otherwise). If it is needed to check the functionality of a hardware peripheral before operation, develop a separate self test routine. Possibly select a MCU that incorporates self test facilities in the peripheral to facilitate self test firmware. Without these hardware capabilities it is essentially impossible to make a statement about the funcionality of a peripheral so there is no need to write code that performs 'pseudo checks' only to soothe the nerves of the developer.

We implicitly assume that every memory read that is performed and does not need to be checked. If we actually need to know if the memory is good, special CRC memory needs to be used and checked. If the results of a computation really need to be trusted, triplicated compute facilities are used to allow for one CPU to fail without causing an issue. This obviosly needs specialty hardware. and is essentially impossible to do without having hardware support for it, so there is no point in trying without the hardware in the loop. Functional safety standards essentially say the same thing: either there is specialty hardware or it is not functionally safe. Hardware can fail, this is an inherant risk that we need to live with. If it is important, verify the functioning via independent hardware first. Firmware can not magically make something safe if it is not designed to be this to begin with.

This is not to say that things don't fail, they do. In most cases however, especially during development, it is ok to be optimistic about the functioning of the device. There are of course exceptions, but in most cases this will be clear from the start. People tend to overemphasise issues that have not acutally been shown to exist and subsequently spend time and resources (and lines of code) on trying to mitigate things that have or actually may never end up happening.

There are of course exceptions to this but often the hardware needs to provide adequate means for software to control something. If stuff on the hardware fails, there is often not much a control system software can do. And if there is, it has been explicitly designed into the system from the start (think redundancy or 'fail safe' hardware etc.)

The cost of this focus on possible errors is real. Lots of work is put into code that will rarely if ever get executed. If a branch is added to code, it is preferrably also tested. This means even more code is needed. All this is work that is not about actually writing the code that attempts to handle a fault, but instead having to maintain that code long term. This leads to the age old wisdom that the best code is no code at all. A component that does not exist cannot fail. This is true for mechanical engineering and this is true for software.

Machines will fail. That is ok. Once it does, and it proves important enough, then you can go and think about ways of mitigating the failure.

How to actually handle failures

Failures are of few types. As embedded firmware goes, it is always part of a whole system and this system will have failure modes designed in. These failures are mitigatable, think overheating of some component. The system is designed to expect this failure and can mitigate it to some extent. In case of the overheating example it stops providing power until the temperature falls below the lock out threshold. Then a 'clear fault' command, allows the system to restart operations. Then there are failures that the system can detect but not mitigate, think the config eeprom can no longer be read. This renders the system permanently inoperable, a case for RMA/Waranty/customer Support, nothing that firmware can solve. And at last there are failures that are not detectable/catastrophic (a short circuit that fries your mosfets through the body diode). You can turn off the mosfets, they are still going to cook. The device was simply operated outside of its specified operating range. There is nothing that firmware can do about that. The issue here is that it becomes really difficult to detect that the mosfets are shot. Special sampling times would be needed to validate that inductor current follows expected paths, and this gets much more complicated for more tricky failure modes. Here trust needs to be placed into the user that they don't simply reconnect the semi exploded device and hope it still works.

Communicating failures is important. Especially when it comes to what caused the failure. This again ends up being more of a documentation challenge than a firmware challenge. When something goes wrong, the firmware detects it and stops operations. It is up to the human to decide what to do now. That decision is only as good as the information that is available to them.

So for something that you design a fault code/flag for, explain what exactly causes the flag and what to do when it is raised. The job of the firmware is to do what it is told by the user, not try and protect them from stupid behaviour (there are a few exceptions to that but over all a device should do as instructed).

My first instinct normally is to try and work around the issue, To try and compensate for it. This is normally a dead end. In many applications there is very little that firmware can actually do by itself. If something fails that is in some way core to the function of the device, it simply needs to stop operating. The challenge here is figuring out how to stop operating before it is to late. In power converters, failure means to stop switching immediately and to hope that the Mosfets dont die of overvoltage.

Thigs tend to fail either very slowly, like fateaguing metal, or very quickly. The former is difficult to diagnose, because there is very little measurable degradation in performance and measurements tend to be noisy. The latter is difficult to diagnose because you need a very fine grained log of what happened in what order, as the same end resuly may have many causes and these causes can be interlinked. Even if a log exists, it is often hard to tell what exactly failed. So lab testing is often needed alongside good logging to figure out what happened.

The ability to have a machine react 'intelligently' to it's environment is intreaguing, but I in the end it is better not to make the appliance too smart. Let the human be smart. Most of the time it is the best strategy simply to tell the human what has gone wrong in as much detail as possible so that they can then make the call on what to do with this information. It is very difficult to take all the various environmental conditions in to account. In a power converter it is essentially impossible to know how it is wired and thus if things is wired correctly. So don't try and be smart. Just fail (prefferably before there is any damage or harm) and try and tell the humans what happened. Focus on the diagnostics and not the mitigation. If there is a way to mitigate things this will emerge over time and once these strategies have been verified, they can be integrated into the firmware as a possible automatic mitigation. Dont forget to have an option to enable or disable the behaviour to allow for backwards compatibility.

Larger systems have mitigation strategies built in

In larger systems the whole system may still continue to operate if a part of it fails. As a designer you need to understand how you can provide functionality even with a failed component, and what can and cannot be accomplished in this state. This is again a system design problem more than a 'firmware' problem and really the firmware executes the things that have been designed into the device intentionally. Redundancy is a good way to keep operating even if a component failed. Eliminating unnecessary components is another good strategy for improving reliability.

It is also important to keep in mind that the device is there to help humans. Again cars are a very good example. The car may not know what the world looks like so it should not make decisions for the driver that the driver did not ask for. I think that devices should do what they are told even if that means degredation or destruction of the device. If you are in a snowstorm and need to get home or you freeze you don't care that the battery of the EV might not survive the ride home, as long as you do.

The state of modern Tech

A device that does what it is told without being fussy about it is sadly somethign that is being designed out in search for profit margins. This still does not make it good design. The world in the end is really unpredictable and humans in particular are pretty creative. So a device should provide capabilities, it should be engineered to be safe and if possible, it should be designed to be rugged and handle as much abuse as possible. There are lots of real world tensions to this. Cost, integration, time to market etc. These are real challenges to get right. Firmware and software sadly are used more and more to serve the interests of the company making the devices rather than the person using those devices. So if you are designing something, be mindful of what firmware can and cannot do. Think in terms of the whole, not only the software and accept that things normally only operate allong very narrow and well understood paths and are practically useless outside of that path (our coffee machine is very needy, and will only make coffee if everything is just right). then try and be as helpful in finding what went wrong when things do.

Happy Hacking