chSchDoReschedule() with empty rlist queue
Posted: Mon Aug 12, 2019 2:26 pm
Hi everyone,
I'm randomly experiencing an unhandled exception inside the ChibiOS scheduler which is caused by a call to chSchDoReschedule() when the ready thread queue is empty (ch.rlist.queue.next == &ch.rlist). Yeah, I know. That should not be happeing.
This is on a custom design around a STM32F072. It is random in the sense that it never happens at the same point in time and that disabling some parts of the code alter the occurence causes the bug to vanish in a way that points to them doing so indirectly by modifying the execution timing. In particular I'm using the USB serial port code and just disconnecting the USB port fixes the problem (not using the CDC USB port, but if connected the device enumerates and I'm guessing SOF interrupts alter execution timing of the other code).
Important: the bug only happens with compiler optimizations enabled (-O2) and I'm usintg gcc-arm-none-eabi-7-2017-q4-major toolchain.
AFIK this is the normal sequence:
1- _port_irq_epilogue
chSchIsPreemptionRequired() is called and returns true. If ch.rlist.queue is empty at this point chSchIsPreemptionRequired() would return false by a clever trick: &ch.rlist is used as end of list marker and ch.rlist has a prio member at the same offset than the thread_t structure. So when the "end of list" marker &ch.rlist is typecasted to (thread_t *) and the prio member is checked, it is actually ch.rlist.prio being checked. ch.rlist.prio is initialized with 0 which would cause the current thread priority to be always higher and thus preemption not being required.
(note: I feel this requirement that ready_list_t and thread_t both have queue and prio as their first two members should be documented somewhere)
2- _port_switch_from_isr
This calls chSchDoReschedule()
3- chSchDoReschedule()
I breakpoint here when ch.rlist.queue is empty, which is the condition that later causes the unhandled exception.
The problem is that I also set a breakpoint on the same condition at (1) when chSchIsPreemptionRequired() is called, and this breakpoint is never triggered before the one at (3).
So either:
a) Execution somehow jumps to (2) or (3) without going through (1).
b) Execution goes as expected (1) --> (2) --> (3) but something happens in between (1) and (3) that empties ch.rlist.queue.
Basically my problem is that ch.rlist.queue is empty in a point in code which is not executed if ch.rlist.queue is empty.
- Usual suspects like stack overflow have been discarded.
- Most if not all ChibiOS checks are enabled.
- I don't see any evident corruption in the ch structure at (3) above. ch.rlist.prio is 0, ch.rlist.queue.next == &ch.rlist and ch.rlist.queue.prev == &ch.rlist as expected.
Any help and/or suggestions are welcome. Buying an expensive probe with code execution trace capabilities is not off the table, but that would force me to use a commercial IDE like uVision or EWARM and I'm afraid just changing compiler may hide the bug (as using -O0 apparently does).
Thanks.
I'm randomly experiencing an unhandled exception inside the ChibiOS scheduler which is caused by a call to chSchDoReschedule() when the ready thread queue is empty (ch.rlist.queue.next == &ch.rlist). Yeah, I know. That should not be happeing.
This is on a custom design around a STM32F072. It is random in the sense that it never happens at the same point in time and that disabling some parts of the code alter the occurence causes the bug to vanish in a way that points to them doing so indirectly by modifying the execution timing. In particular I'm using the USB serial port code and just disconnecting the USB port fixes the problem (not using the CDC USB port, but if connected the device enumerates and I'm guessing SOF interrupts alter execution timing of the other code).
Important: the bug only happens with compiler optimizations enabled (-O2) and I'm usintg gcc-arm-none-eabi-7-2017-q4-major toolchain.
AFIK this is the normal sequence:
1- _port_irq_epilogue
chSchIsPreemptionRequired() is called and returns true. If ch.rlist.queue is empty at this point chSchIsPreemptionRequired() would return false by a clever trick: &ch.rlist is used as end of list marker and ch.rlist has a prio member at the same offset than the thread_t structure. So when the "end of list" marker &ch.rlist is typecasted to (thread_t *) and the prio member is checked, it is actually ch.rlist.prio being checked. ch.rlist.prio is initialized with 0 which would cause the current thread priority to be always higher and thus preemption not being required.
(note: I feel this requirement that ready_list_t and thread_t both have queue and prio as their first two members should be documented somewhere)
2- _port_switch_from_isr
This calls chSchDoReschedule()
3- chSchDoReschedule()
I breakpoint here when ch.rlist.queue is empty, which is the condition that later causes the unhandled exception.
The problem is that I also set a breakpoint on the same condition at (1) when chSchIsPreemptionRequired() is called, and this breakpoint is never triggered before the one at (3).
So either:
a) Execution somehow jumps to (2) or (3) without going through (1).
b) Execution goes as expected (1) --> (2) --> (3) but something happens in between (1) and (3) that empties ch.rlist.queue.
Basically my problem is that ch.rlist.queue is empty in a point in code which is not executed if ch.rlist.queue is empty.
- Usual suspects like stack overflow have been discarded.
- Most if not all ChibiOS checks are enabled.
- I don't see any evident corruption in the ch structure at (3) above. ch.rlist.prio is 0, ch.rlist.queue.next == &ch.rlist and ch.rlist.queue.prev == &ch.rlist as expected.
Any help and/or suggestions are welcome. Buying an expensive probe with code execution trace capabilities is not off the table, but that would force me to use a commercial IDE like uVision or EWARM and I'm afraid just changing compiler may hide the bug (as using -O0 apparently does).
Thanks.