Meeting strict real-time requirements is a common challenge in embedded development. Whether you must sample a sensor every 500 µs with sub µs jitter or acknowledge an input within 20 µs, verifying those during testing is essential. Traditional instrumentation can alter timing behavior and hide root causes. Hardware trace combined with automated analysis provides a non-intrusive, repeatable way to both verify timing and find why a requirement was missed.
Why instrumentation can be misleading
Instrumenting application code and using cycle counters or timers to measure time gives direct timing numbers, but it also changes the code path and timing characteristics. When you have timing requirements in the order of µs, adding instrumentation can affect the system enough to mask real problems or create false positives.
Two common problems with code instrumentation:
- It affects the real-time behavior of the application, potentially breaking the very timing you want to measure.
- It rarely shows why a timing requirement failed, only that it did.
Hardware trace is non-intrusive and informative
Many microcontrollers and processors include hardware trace blocks that capture execution information without modifying software timing. For Cortex-M, A and R cores, this is done by the Embedded Trace Macrocell (ETM). ETM generates instruction trace (and sometimes data trace) of the microcontroller core, so you can record what has been executed with timestamps, without instrumentation overhead.
Using hardware trace you get two benefits:
-
Accurate timing with no software-induced jitter.
-
Context, you can see exactly what code executed, in which order, and which interrupts or routines interfered with timing.
Automating trace capture and analysis with TRACE32
Lauterbach TRACE32 is a flexible toolset for debugging and tracing embedded systems. It provides a rich API that can be driven from automation scripts, making it suitable for integration into test environments.
A typical automated workflow looks like this:
-
Start TRACE32 from a test script (for example a Python script).
-
Configure debugger connection and download the binary and symbols to the target.
-
Run tests on the target while TRACE32 captures trace data.
-
Read timing data from the trace information.
-
Verify timing requirements programmatically. If a test fails, save the full trace for post-mortem analysis.
Saving the trace capture when a test fail is powerful. You get the full execution context and can replay the trace in a simulator to investigate the failure.
Below is a simplified Python script showing how a test script can start TRACE32, connect, run tests, and evaluate timing.
import lauterbach.trace32.rcl as t32
start_trace32()
dbg = t32.connect(node='localhost', port=20000, protocol="TCP", timeout=40.0)
run_test()
get_timing_data()
check_timing()
if test_failed > 0:
dbg.cmm('save_trace.cmm')
macro = dbg.practice.get_macro('&folder_name')
print(f'Trace saved to {macro.value}')
dbg.cmd('quit')
dbg.disconnect()
Practical analysis example: finding the root cause
Imagine you have an ISR, sample_sensor (expected every 500 µs). After a test, an automated trace analysis shows occasional intervals exceeding the 2 µs jitter requirement.
Using TRACE32 features like distance analysis you can:
- Count how many times the ISR ran and measure the time between consecutive entries.
-
Visualize a histogram of intervals to quickly spot deviations.
-
Jump directly to any deviations for more details.
In the example, the trace shows sample_temp_data executing for 5.3 µs right between two sample_sensor invocations. Sample_temp_data is another ISR and it blocks sample_sensor ISR from running and caused the ISR to miss its timing window.
Common fixes once you find the cause
-
Increase interrupt priority for the critical ISR if the CPU supports nested interrupts.
-
Reduce work done inside the long running ISR, move processing to a task/thread.
Conclusion
When µs-level timing matters, rely on hardware trace rather than intrusive instrumentation. Combining hardware trace like ETM with a powerful toolset such as TRACE32 and automating capture and analysis enables accurate verification of timing requirements and quick identification of root causes.
Integrate trace capture into your test automation to save development time and reduce costly timing bugs.
For information or support, contact support@nohau.se