Getting stuck with tickless mode

[kulve]

Hi,

My system gets stuck within a few seconds and I'm not able to pin point the reason. I'm using LSM9DS0 (9dof chip) and it's firing interrupts on three interrupt lines at 100Hz, 100Hz and 95Hz. I have one ISR handler for all three lines and I'm calling chBSemSignalI() from it. In a separate thread I have chBSemWaitTimeout(xxx, 50).

It doesn't get stuck if I disable tickless kernel. Nor does it get stuck if I disable 2 of 3 interrupt lines. Nor does it get stuck if I replace chBSemWaitTimeout() with chThdSleep(MS2ST(3)).

My ISR handler:
https://github.com/snowcap-electronics/ ... 9ds0.c#L85

My chBSemWaitTimeout() code:
https://github.com/snowcap-electronics/ ... ds0.c#L251

The backtrace when it's stuck:

Code: Select all

(gdb) thread apply all backtrace

Thread 14 (Thread 536877552):
#0  chSchGoSleepS () at ChibiOS/os/rt/src/chschd.c:270
#1  0x08002cde in chThdSuspendS () at ChibiOS/os/rt/src/chthreads.c:518
#2  0x0800858e in osalThreadSuspendS () at ChibiOS/os/hal/osal/rt/osal.h:657
#3  0x08009a70 in usb_lld_pump () at ChibiOS/os/hal/ports/STM32/LLD/OTGv1/usb_lld.c:1311
#4  0x080002ea in _port_thread_start ()
#5  0x080002ea in _port_thread_start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 13 (Thread 536880656):
#0  chSchGoSleepS () at ChibiOS/os/rt/src/chschd.c:270
#1  0x080022f8 in chSchGoSleepTimeoutS () at ChibiOS/os/rt/src/chschd.c:344
#2  0x080032be in chSemWaitTimeoutS () at ChibiOS/os/rt/src/chsem.c:281
#3  0x08003b3e in chMBFetchS () at ChibiOS/os/rt/src/chmboxes.c:388
#4  0x08003afa in chMBFetch () at ChibiOS/os/rt/src/chmboxes.c:357
#5  0x0800c772 in eventLoopThread () at src/sc_event.c:112
#6  0x080002ea in _port_thread_start ()
#7  0x080002ea in _port_thread_start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 12 (Thread 536887088):
#0  chSchGoSleepS () at ChibiOS/os/rt/src/chschd.c:270
---Type <return> to continue, or q <return> to quit---
#1  0x080022f8 in chSchGoSleepTimeoutS () at ChibiOS/os/rt/src/chschd.c:344
#2  0x08002dca in chThdEnqueueTimeoutS () at ChibiOS/os/rt/src/chthreads.c:652
#3  0x08003ff4 in chOQWriteTimeout () at ChibiOS/os/rt/src/chqueues.c:420
#4  0x08004e4e in write () at ChibiOS/os/hal/src/serial_usb.c:59
#5  0x0800cd7c in scSduSendThread () at src/sc_sdu.c:455
#6  0x080002ea in _port_thread_start ()
#7  0x080002ea in _port_thread_start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 11 (Thread 536886400):
#0  chSchGoSleepS () at ChibiOS/os/rt/src/chschd.c:270
#1  0x080022d2 in chSchGoSleepTimeoutS () at ChibiOS/os/rt/src/chschd.c:338
#2  0x08002dca in chThdEnqueueTimeoutS () at ChibiOS/os/rt/src/chthreads.c:652
#3  0x08003d8a in chIQGetTimeout () at ChibiOS/os/rt/src/chqueues.c:182
#4  0x08004f26 in gett () at ChibiOS/os/hal/src/serial_usb.c:86
#5  0x0800ccb8 in scSduReadThread () at src/sc_sdu.c:415
#6  0x080002ea in _port_thread_start ()
#7  0x080002ea in _port_thread_start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 10 (Thread 536876712):
#0  chSchDoRescheduleAhead () at ChibiOS/os/rt/src/chschd.c:504
#1  0x080024b8 in chSchDoReschedule () at ChibiOS/os/rt/src/chschd.c:534
---Type <return> to continue, or q <return> to quit---
#2  0x080002fa in _port_switch_from_isr ()
#3  0x080002fa in _port_switch_from_isr ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 9 (Thread 536876552):
#0  chSchGoSleepS () at ChibiOS/os/rt/src/chschd.c:270
#1  0x080022d2 in chSchGoSleepTimeoutS () at ChibiOS/os/rt/src/chschd.c:338
#2  0x0800295c in chThdSleepS () at ChibiOS/os/rt/include/chthreads.h:268
#3  0x08002b82 in chThdSleep () at ChibiOS/os/rt/src/chthreads.c:315
#4  0x0800f202 in main () at src/main-ahrs.c:92

Thread 8 (Thread 536888160):
#0  chSysHalt () at ChibiOS/os/rt/src/chsys.c:171
#1  0x080017f2 in chVTDoTickI () at ChibiOS/os/rt/include/chvt.h:480
#2  0x0800197c in chSysTimerHandlerI () at ChibiOS/os/rt/src/chsys.c:311
#3  0x0800a188 in osalOsTimerHandlerI () at ChibiOS/os/hal/osal/rt/osal.h:554
#4  0x0800a1aa in Vector108 () at ChibiOS/os/hal/ports/STM32/LLD/TIMv1/st_lld.c:160
#5  0xffffffec in ?? ()
#6  0xffffffec in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)


Any hints or suggestions are most welcome.

Thanks.

[Giovanni]

Hi,

Are assertions and state checker enabled?

Giovanni

[kulve]

Yes, I have enabled assertions and the following checks:

#define CH_DBG_STATISTICS TRUE
#define CH_DBG_SYSTEM_STATE_CHECK TRUE
#define CH_DBG_ENABLE_CHECKS TRUE
#define CH_DBG_ENABLE_ASSERTS TRUE
#define CH_DBG_ENABLE_TRACE TRUE
#define CH_DBG_ENABLE_STACK_CHECK TRUE
#define CH_DBG_FILL_THREADS TRUE
#define CH_DBG_THREADS_PROFILING FALSE

Compiler optimisations are off.

LTO is disabled as OpenOCD didn't work with it.

[Giovanni]

Could you try to increase the delta parameter and see if it makes a difference?

Giovanni

[kulve]

I tried increasing it from 2 to 4 or 8 but still stucks within seconds.

[Giovanni]

Try comment out the chDbgAssert() at line 480 in chvt.h and see the result please.

Giovanni

[kulve]

EDIT: I only now noticed your latest comment. Will try without the assert in a bit.

os/rt/include/chvt.h:480

Code: Select all

 chDbgAssert(chVTIsTimeWithinX(ch.vtlist.vt_lasttime + vtp->vt_delta,
                                ch.vtlist.vt_lasttime,
                                chVTGetSystemTimeX() + 1),
              "out of time window");


Barthess mentioned in the IRC that with tickless kernel those backtraces might not be always accurate as the timer might not get stopped correctly. I don't know anything about the topic.

[Giovanni]

Sorry, I edited my previous message, could you remove that assert? the fact that is triggered is already "interesting".

Giovanni

[kulve]

At least it didn't stuck withing the first minute. I'll leave it running for an hour before doing more tests as I have other things to do now.

[Giovanni]

Thanks, I will give it more time tomorrow, there is something there...

Giovanni