Old tricks for new dogs

How debug logging helped get an embedded development project back on track.

When I started my career in embedded programming, before JTAG, we relied on ‘black box’ programming: we couldn’t see what the code was doing internally. It was a bit like Schrödinger’s cat — we didn’t know whether the code was alive or dead until we opened the door. That need to make the code’s behaviour visible also arose whilst working with a recent client. A communications port provided the door: once it was working, we had a way to get debug information from the code.

The client was investigating a device that stopped responding after a period of time.  As the issue had appeared only after a hardware change, their investigation initially focused on the hardware. With tight timescales and the cause still unresolved, they contacted us to support the investigation. During the initial consultation, I identified the embedded software as another possible source of the problem.

The following day, I met with the development team to review the symptoms. It became clear that the system entered a state from which it could recover only by being powered off and on again so therefore there was no smoking gun. The cause remained unclear, so the investigation needed evidence of how the code reached that state.

To support the team’s software investigation, I suggested they implement a debug function that could be switched on and off to show the flow through the code.  I also agreed to investigate the hardware and report on my findings. This would help distinguish a hardware issue from a software fault.

I couldn’t see a problem with the hardware, although the lack of test points made the investigation difficult.  This highlights the value of including test points during layout so they can be manually installed during development; these can be removed from the component list when the board goes into production to reduce costs.

The hardware got a clean bill of health, and within a few days of implementing the suggested debug logging in the embedded code, the client’s development team found the problem.

Working alongside the client’s development team, we supported an investigation that brought hardware checks and software debug logging together. In under a week, their development project was back on track towards a production run.

The Takeaway

Sometimes old-school debugging basics can help uncover a problem which slips through gaps in modern debugging tools.