ESP32 Interrupt Response

It is fairly well documented in the ESP forums that the interrupt response time of the ESP32 is 2 to 3us which is in the same range as an Arduino Nano. I was expecting a lot better than that considering the clock rate of the ESP32’s processor is 240 MHz compared to 16 MHz of the Nano. It seems that the reason for this is there is a lot of overhead to be able to use an interrupt handler written in C instead of assembler. Also user interrupt handlers are at the lowest priority (which probably explains the +/- 8us jitter as well).

To investigate the ESP32s interrupt response I setup a test system with PPS generating an interrupt and then the interrupt handler setting an output pin high and setting a flag. In the main loop I checked for the flag and then set the output pin low again after about 100ms. That gives an output pulse very similar to PPS and I was able to compare the two on the oscilloscope using PPS as the reference. With that test system it was easy to see the interrupt latency and the jitter but not so easy to measure it. My Tektronix ’scope has the ability to trigger on a pulse width exceeding a preset value but I had two pulses and I wanted to measure the difference between them.


Pulse Width Measurement

This is where I got really sidetracked on this project! I had already been experimenting with simple logic development with FPGAs using the Lattice iCEstick evaluation board and the IceStudio open source development tools and I was looking for something interesting to do with it. With that equipment already sitting on my workbench I had a system implemented in a few minutes to generate a single pulse who’s width was the difference between the two input pulses. It is just an XOR gate, and Inverter and an AND gate which is really simple however the alternative to using the FPGA was to buy some components and make a circuit on a breadboard to do the same which needless to say wasn’t going to happen in a few minutes. I was hooked!

With my revised test system I could then use the oscilloscope's pulse width trigger function to roughly measure the maximum pulse width. But that seemed not very satisfactory so I ended up switching to the Lattice iCE40-HX8K Breakout Board mostly because it has a row of 8 LEDs that I could use to display the pulse width value. To accomplish what I wanted I had to learn some Verilog so this was a bit more involved but after a few days I had a really decent pulse width measurement tool implemented which more than met my objectives.

Now that I had a good way to measure the ESP32 interrupt response time I tried a number of things to improve the system. The ESP32 has two cores and uses FreeRTOS “under the hood” to manage the system. One core is left for the user’s application and this is supported with both the ESP Arduino framework and the ESP-IDF framework. I tried everything I could think of to improve the interrupt response time including moving the interrupt handler to the system core (had no effect) and then trying the ESP-IDF instead of the Arduino framework. That did give some improvement especially with regard to the jitter however it wasn’t what I was hoping for. I was able to determine however that if the users application uses WIFI that it doesn’t seem to affect the interrupt handling which is good.


Moving the PPS Logic to the FPGA

At this point I realized I was a big part of the way towards using the FPGA to do all the timing and pulse generation instead of the ESP32 so I decided to continue with that approach. [NOTE: Once I started pursuing using an external counter with an FPGA I did not continue to try and improve the ESP32’s interrupt response, perhaps by implementing an interrupt handler in assembly language and then changing the interrupt priority. I now think that it would be possible to implement a decent system with just the ESP32 using a simple NTP server that one of my friends developed. I will try this at some point in the future when I find some spare time].

In continuing the development with the FPGA however I really ended up going down a "rabbit hole". First of all I needed a better way to get information out of the FPGA and into the ESP32. After a lot of investigation into what the possibilities were I decided to implement a UART on the FPGA and use one of the ESP32’s serial ports (it has 3) to receive the data. Fortunately there is lots of freely available example Verilog code to implement the UART and I had no difficulty with that. In IceStudio I use a code block and strip off the interface section of the Verilog code and that works really well.

What I ended up with for the timing logic is a 32 bit counter that counts down to 0. It is then loaded with a constant that represents a one second count for the counter and that constant is determined by calibration.

If the PPS is exactly one second and if the clock frequency for the FPGA was exactly 12 MHz and it didn't take any time for the logic to execute then loading 11999999 into the counter everytime would result in the counter hitting 0 exactly when the PPS occurs. But the reality is that PPS varies up to 40 ns, the logic takes some time to execute (at least one clock cycle) and the clock frequency is not all that precise or stable.

As a result when PPS comes in the clock may not have reached 0 yet (clock too slow so the count will be some small value) or the clock may have already been loaded (clock too fast so count will be near the loaded value). That value in the counter at the time of PPS is what I wanted to see to measure performance.

The UART sends 8 bits of data at a time and I wanted it to be human readable to display on a terminal, at least to start with. I ended up finding the Verilog code for a binary to BCD sequencer that takes the counter value and sends it to the UART as 8 bit ASCII characters (I actually made this all 64 bits just in case 32 bits wasn’t going to be enough).

This was starting to get quite complex however I did get it working without too much difficulty and I was really getting comfortable with Verilog. There is nothing like learning by working on something interesting.


FPGA Hell

I then ran into a strange problem in that my counter logic appeared to be missing PPS pulses. After a lot of experimenting and then research I came to find out that this is often referred to as “FPGA Hell” where even simple logic sometimes doesn’t appear to work the way it is supposed too! In this case my problem ended up being that the PPS input wasn’t synchronized with the FPGAs clock. This is something that I’m sure that every experienced FPGA developer would know but it wasn’t something that was in any of the tutorials or documentation that I read and it sure is something that can stump a beginner!


Looking for Better Stability

At this point comparing the FPGA solution to the ESP32 I determined that I had solved the jitter problem but not the drift. Both systems drifted significantly at around 2ms per day at best. I tried various ways to calibrate the FPGA timer constant but that didn’t appear to be very helpful. I started wondering about the stability of the oscillators for both the FPGA boards and the ESP32. I found this really helpful chart that explains what is possible:

 I decided that it would be worth trying a TCXO to supply an external clock to the FPGA.

Next page, OpenPPS Oscillators...