chSchDoReschedule() with empty rlist queue

[iggarpe]

Hi everyone,

I'm randomly experiencing an unhandled exception inside the ChibiOS scheduler which is caused by a call to chSchDoReschedule() when the ready thread queue is empty (ch.rlist.queue.next == &ch.rlist). Yeah, I know. That should not be happeing.

This is on a custom design around a STM32F072. It is random in the sense that it never happens at the same point in time and that disabling some parts of the code alter the occurence causes the bug to vanish in a way that points to them doing so indirectly by modifying the execution timing. In particular I'm using the USB serial port code and just disconnecting the USB port fixes the problem (not using the CDC USB port, but if connected the device enumerates and I'm guessing SOF interrupts alter execution timing of the other code).

Important: the bug only happens with compiler optimizations enabled (-O2) and I'm usintg gcc-arm-none-eabi-7-2017-q4-major toolchain.

AFIK this is the normal sequence:

1- _port_irq_epilogue

chSchIsPreemptionRequired() is called and returns true. If ch.rlist.queue is empty at this point chSchIsPreemptionRequired() would return false by a clever trick: &ch.rlist is used as end of list marker and ch.rlist has a prio member at the same offset than the thread_t structure. So when the "end of list" marker &ch.rlist is typecasted to (thread_t *) and the prio member is checked, it is actually ch.rlist.prio being checked. ch.rlist.prio is initialized with 0 which would cause the current thread priority to be always higher and thus preemption not being required.

(note: I feel this requirement that ready_list_t and thread_t both have queue and prio as their first two members should be documented somewhere)

2- _port_switch_from_isr

This calls chSchDoReschedule()

3- chSchDoReschedule()

I breakpoint here when ch.rlist.queue is empty, which is the condition that later causes the unhandled exception.

The problem is that I also set a breakpoint on the same condition at (1) when chSchIsPreemptionRequired() is called, and this breakpoint is never triggered before the one at (3).

So either:

a) Execution somehow jumps to (2) or (3) without going through (1).
b) Execution goes as expected (1) --> (2) --> (3) but something happens in between (1) and (3) that empties ch.rlist.queue.

Basically my problem is that ch.rlist.queue is empty in a point in code which is not executed if ch.rlist.queue is empty.

- Usual suspects like stack overflow have been discarded.
- Most if not all ChibiOS checks are enabled.
- I don't see any evident corruption in the ch structure at (3) above. ch.rlist.prio is 0, ch.rlist.queue.next == &ch.rlist and ch.rlist.queue.prev == &ch.rlist as expected.

Any help and/or suggestions are welcome. Buying an expensive probe with code execution trace capabilities is not off the table, but that would force me to use a commercial IDE like uVision or EWARM and I'm afraid just changing compiler may hide the bug (as using -O0 apparently does).

Thanks.

[Giovanni]

Hi,

You need to use GCC version 5.4.1 or below, all newer releases have a bug impacting Cortex-M0.

Which ChibiOS version are you using? latest versions warn about this during build.

Giovanni

[iggarpe]

I'm using ChibiOS 18.2.2.

gcc is the one coming in toolchain gcc-arm-none-eabi-7-2017-q4-major:

gcc version 7.2.1 20170904

I'll look into that warning you mention which I guess is in ChibiOS 19.x.x.

In any case, can you point me to more info about this bug? It feels odd that it has not been fixed and we need to resort to use ancient gcc versions...

[Giovanni]

It has been discussed here: viewtopic.php?f=35&t=4894

The bug has been reported and it is solved in GCC 9.

Giovanni

[iggarpe]

Thank you. You made my day. At some point I was pretty sure I was debugging either a ChibiOS or compiler bug (more likely the latter since -O0 solved the issue). And shame on me because a search on the forum should have taken me to that thread. Sorry about that. I only have sleep deprivation as excuse.