SMT32G474- i2cAcquireBus not getting Mutex when built with -O0 compiler option Topic is solved

[Giovanni]

Hi,

You need to enable stack check in chconf.h. In addition you can also add PORT_ENABLE_GUARD_PAGES at TRUE in order to add an HW barrier to stacks.

If a stack overflows then the system stops in chSysHalt() and you can do a stack trace.

Giovanni

[gp7]

Apologies to kick this ticket again. I am following the advice on the debugging guide to "never give up"

I have implemented the Lock class as suggested by alex31 and placed breakpoints on the constructor and destructor.

#include "osal.h"

class Lock
{
public:
//Lock(mutex_t &_mtx) : mtx(_mtx) {chMtxLock(&mtx);};
//~Lock() {chMtxUnlock(&mtx);};
Lock(mutex_t &_mtx) : mtx(_mtx)
{
osalMutexLock(&mtx);
};
~Lock()
{
osalMutexUnlock(&mtx);
};
private:
mutex_t &mtx;
};

Now for the most part everything happens as expected, Lock gets called, then ~Lock.
If I debug I can step through and calls are Lock, ~Lock 9 times until on one occasion the destructor is NOT called before the next call to Lock().
If I continue stepping through the following calls are made before the main thread resumes
(L, ~L) x 9, (L x 2), (L,~L) x 3

After this both my worker threads are locked in state WTMTX and never recover..
Any suggestions for debugging? So far I am only removing code one line at a time to see where things fail but this is arduous

I have turned off recursive mutex support but that didn't appear to change things.

[Giovanni]

Hi,

Is the mutex initialized somewhere?

Anyway, there is a C++ wrapper under /os/various/cpp_wrappers, you may want to give it a try or use it as a reference.

Giovanni

[gp7]

Thanks Giovanni,

The cpp wrapper files are included in the Makefile
I have found that the Mutex objects were initialised in the main loop:

static mutex_t mux_mtx;
chMtxObjectInit(&EmbeddedToolsN::mux_mtx);

In the end of the worker thread (main function) I send data via I2C bus then I have the following:

m_prev = chThdSleepUntilWindowed(m_prev, chTimeAddX(m_prev, TIME_MS2I(5000))); //give other threads a chance
}

In the main thread, program I am just sleeping:

systime_t next = chVTGetSystemTime();
while (true) {
next = chThdSleepUntilWindowed(next, chTimeAddX(next, TIME_MS2I(1000)));
continue;

When I step through the code one of my worker threads locks mux_mtx and the other thread attempts to Lock the mutex but fails (as I would expect, it's already locked. )

I can debug through the following line in chmtx.c : chSchGoSleepS(CH_STATE_WTMTX)
After this the one worker thread is in WTMTX state and the other one SLEEPING but the worker thread that is in WTMTX never gets windowed again even though I can step into the chMtxUnlock immediately after the waiting thread is locked.



I'm wondering if the code in chmtx.c has been changed and whether there is a way I can check this against the original Chibios 21.6.0 release?

[Giovanni]

I am not sure to understand.

Just a point, the thread unlocking a mutex -must- be the same thread that locked it, is this the case?

Giovanni

[gp7]

Thank you Giovanni,

I found that one of the Chibios 21.6.0 files had been altered. I have returned it to it's original state.
The following line within chMtxLockS was commented out:

ch_sch_prio_insert(&mp->queue, &currtp->hdr.queue);

When uncommented I get an unhandled exception within ch_sch_prio_insert in the file chschd.h.

void ch_sch_prio_insert(ch_queue_t *qp, ch_queue_t *tp) {

  ch_queue_t *cp = qp;
  do {
    cp = cp->next;
  } while ((cp != qp) &&
           (((thread_t *)cp)->hdr.pqueue.prio >= ((thread_t *)tp)->hdr.pqueue.prio));
  tp->next       = cp;
  tp->prev       = cp->prev;
  tp->prev->next = tp;
  cp->prev       = tp;
}
#endif /* CH_CFG_OPTIMIZE_SPEED */

If I debug into

static inline void ch_sch_prio_insert(ch_queue_t *qp, ch_queue_t *tp)

then I see that

qp                                                0x200037a0
qp->next                                            0x0
qp->next->next                                  0x20000400
qp->next->next->next                        0x55555555
qp->next->next->prev                        0x55555555
qp->next->prev                                  0x80002c9 <Reset_Handler>
qp->next->prev->next                        0xf000e7
qp->next->prev->prev                        0x2de7fef8
qp->prev                                            0x0
qp->prev->next                                  0x20000400
qp->prev->next->next                        0x55555555
qp->prev->next->prev                        0x55555555
qp->prev->prev                                  0x80002c9 <Reset_Handler>
qp->prev->prev->next                        0xf000e7
qp->prev->prev->prev                        0x2de7fef8

I have set CH_CFG_OPTIMIZE_SPEED to FALSE but this makes no noticeable difference. I am surprised we need to recurse over the next and prev members of the queue when they are 0? In any case I will see if I can identify where the queue gets setup to see if I can find any issue there.

Also I can confirm that the thread unlocking the mutex is the same thread that locked it.

[Giovanni]

Hi,

That is a ready list corruption, it can happen because a stack overflow, because invalid use of I and S APIs or because an invalid priority has been assigned to some IRQ.

Are state checker and assertions enabled in your chconf.h?

Note that 21.6.x has been discontinued, you should work with 20.3.x or 21.11.x.

Giovanni

[gp7]

Thank you Giovanni,

I performed a diff -qr on my chibios folder and reverted all files back to originals.
Finally I still had an unhandled exception:

I am still receiving an unhandled exception in my code in the function __sch_ready_behind(thread_t *tp)
If I put a breakpoint on ch_pqueue_insert_behind I see it gets called with the following threads (expression is tp->name):

0x801de24 "idle"
0x801de34 <ch_debug> "main"
0x801de34 <ch_debug> "main"
0x80002cb <VectorF4> "ð"

Not sure why main is passed twice? Strangely when I look at the stack there is an anomaly.

My stack trace finally looks like this:

Thread #1 [main] 1 [core: 0] (Suspended : Signal : SIGTRAP:Trace/breakpoint trap)	
	VectorF4() at vectors.S:1,021 0x80002ca	
	<signal handler called>() at 0xfffffffd	
	ch_pqueue_insert_behind() at chlists.h:379 0x800fa7e	
	__sch_ready_behind() at chschd.c:80 0x800fbca	
	chSchReadyI() at chschd.c:289 0x800fd6c	
	chMtxUnlock() at chmtx.c:386 0x8010da8	
	osalMutexUnlock() at osal.h:1,082 0x801420c	
	Lock::~Lock() at Lock.hh:24 0x8014260	
	EmbeddedToolsN::I2CMux::setup() at I2CMux.cc:46 0x8014b0a	
	main() at main.cc:252 0x8013dae

When I click on I2CMux::setup() I see that the constructor for lock should be getting called, the call stack shows the destructor instead..?:

uint32_t I2CMux::setup()
{
Lock lock(mux_mtx); (excecution pointer is here)
if (m_bIsSetup == true)

I will try with one of the other versions to see if that solves this issue. I have also enabled state checker and assertions.

[gp7]

Hi All,

I have checked my stack sizes, removed everything but main thread and I am still able to create an unhandled exception.
I have also changed to Chibios v21.11

When I look at main and idle threads there are 2840 bytes available on main and 64 in idle under StkUnused before ch_queue_fifo_remove

qp->next->prev (0x55555555) = qp (0x20002f1c <_ZN14EmbeddedToolsNL7mux_mtxE.lto_priv.1>)

After this StkUnused on main drops to 4. Finally I see an unhandled exception occurs:

Thread #1 [main] 1 [core: 0] (Suspended : Signal : SIGTRAP:Trace/breakpoint trap)
VectorF4() at vectors.S:1,021 0x80002ca
<signal handler called>() at 0xfffffffd
ch_pqueue_insert_behind() at chlists.h:379 0x8010534
__sch_ready_behind.isra.0() at chschd.c:80 0x8010534
chSchReadyI() at chdebug.c:254 0x800e110
chMtxUnlock() at chmtx.c:386 0x800e110
EmbeddedToolsN::I2CMux::setup() at I2CMux.cc:46 0x800f7e2
main() at main.cc:192 0x800f7e2

My main function looks like this:
[[noreturn]] int main(void) {
halInit();
chSysInit();
SetInitialState();
chMtxObjectInit(&EmbeddedToolsN::mux_mtx);
EmbeddedToolsN::g_param_FlowMeter1 = EmbeddedToolsN::g_i2cMux.setup();

My mutex as follows: static mutex_t mux_mtx;

I'm really not sure where to focus my efforts to find the source of the error. From what I can see the stack sizes are OK. I'm not sure if my code uses I or S APIs but I haven't called these directly. I have looked for some IRQ but can't see this defined in the code. Would this come from the .ioc file? My NVIC sheet in CubeMX has the following enabled: Non maskable interrupt, Hard fault interrupt, Memory management fault, Prefetch fault, memory access fault, Undefined instruction or illegal state, System service call via SWI instruction, Debug monitor, Pendable request for system service and Time base: System tick timer. All these have pre-emption priority of 0 with exception of Time base which is priority 15. Sub priorities are all set to 0.

[Giovanni]

Check for stack overflows, -O0 increases stack usage, a lot.

Giovanni