ChibiOS Free Embedded RTOS

Posted: **Wed Oct 10, 2012 10:09 am**

Hi saraben,

most of your outputs look great. What is really interesting, is that your backtrace goes further back than then _port_thread_start functions:

Code: Select all

      Thread [4] 536877952 (shell :  : WTQUEUE) (Suspended : Container)   
         chSchGoSleepS() at chschd.c:122 0x8001664   
         chSchGoSleepTimeoutS() at chschd.c:195 0x8001768   
         qwait() at chqueues.c:69 0x8003c40   
         chIQReadTimeout() at chqueues.c:232 0x8003e6c   
         read() at serial.c:63 0x800574e   
         shellGetLine() at shell.c:261 0x80134c6   
         shell_thread() at shell.c:155 0x801328a   
         _port_thread_start() at chcore_v7m.c:255 0x8000d32   
         chSchGoSleepS() at chschd.c:121 0x8001664   
         0xaa52432a

This should not be. For me it looks like this:

Code: Select all

Thread 8 (Thread 536901664):
#0  0x080009f2 in chSchGoSleepTimeoutS (newstate=<value optimized out>, time=<value optimized out>) at ../../os/kernel/src/chschd.c:195
#1  0x08001b3e in chMBFetchS (mbp=0x20008c28, msgp=0x20007c94, time=<value optimized out>) at ../../os/kernel/src/chmboxes.c:335
#2  0x0800a480 in sys_arch_mbox_fetch (mbox=<value optimized out>, msg=<value optimized out>, timeout=<value optimized out>) at ../../os/various/lwip_bindings/arch/sys_arch.c:169
#3  0x080110ea in netconn_accept (conn=0x20004d48, new_conn=0x20007cb0) at ../../ext/lwip/src/api/api_lib.c:304
#4  0x08014876 in http_server (p=<value optimized out>) at web/web.c:115
#5  0x0800068a in _port_thread_start () at ../../os/ports/GCC/ARMCMx/chcore_v7m.c:255
Backtrace stopped: frame did not save the PC

I have no idea though how gdb figures out that the PC was not saved. Maybe it is a toolchain error. Maybe we need to clean the lr register on _port_thread_start? Giovanni, do you have any idea what could cause this? I also tried with -O3 and it still worked for me.

saraben: You could also try to set a break point into one of those threads, so that it is the current thread. Check if the backtrace still contains these function calls before _port_thread_start.

Posted: **Wed Oct 10, 2012 10:18 am**

Not sure, does backtrace stop on LR==0? you could easily test it. Back trace should never go past _port_thread_start anyway.

Giovanni

Posted: **Wed Oct 10, 2012 7:10 pm**

Hi mabl

feanor wrote:With Option "USE_FPU = no" the threaded debugging works. Thanks for your help.

Apparently I was a bit hasty with this statement. I get all the threads running in ChibiOS, but it is not possible for me to debug them correctly. Hitting a breakpoint does not open the corresponding thread but it ends every time in chSchGoSleepS(). Also single-stepping does not work, it stays also in chSchGoSleepS() and produces this output of openocd

Code: Select all

Open On-Chip Debugger 0.7.0-dev-gcb69d06 (2012-10-07-15:02)
Licensed under GNU GPL v2
For bug reports, read
   http://openocd.sourceforge.net/doc/doxygen/bugs.html
adapter speed: 1000 kHz
srst_only separate srst_nogate srst_open_drain
Info : clock speed 1000 kHz
Info : stm32f4x.cpu: hardware has 6 breakpoints, 4 watchpoints
Info : accepting 'gdb' connection from 3333
Info : stm32f4x errata detected - fixing incorrect MCU_IDCODE
Info : device id = 0x10006413
Info : flash size = 1024kbytes
Warn : acknowledgment received, but no packet pending
undefined debug reason 6 - target needs reset
Info : stm32f4x errata detected - fixing incorrect MCU_IDCODE
Info : device id = 0x10006413
Info : flash size = 1024kbytes
flash 'stm32f2x' found at 0x08000000
ERROR: last sector must be <= 11
in procedure 'flash'
target state: halted
target halted due to debug-request, current mode: Thread 
xPSR: 0x01000000 pc: 0x08002670 msp: 0x20000400
background polling: on
TAP: stm32f4x.cpu (enabled)
target state: halted
target halted due to debug-request, current mode: Thread 
xPSR: 0x01000000 pc: 0x08002670 msp: 0x20000400
stm32x mass erase complete
target state: halted
target halted due to breakpoint, current mode: Thread 
xPSR: 0x61000000 pc: 0x20000042 msp: 0x20000400
wrote 84080 bytes from file ../stm32-mf-chibios/src/build/chibi-mf.elf in 2.188476s (37.519 KiB/s)
Warn : negative acknowledgment, but no packet pending
ERROR: last sector must be <= 11
in procedure 'flash'
Info : dropped 'gdb' connection
Info : accepting 'gdb' connection from 3333
Warn : acknowledgment received, but no packet pending
requesting target halt and executing a soft reset
Info : should no be called
Auto-detected RTOS: ChibiOS

Info : Successfully loaded memory map of ChibiOS/RT target running version 2.5.1
Error: ChibiOS registry integrity check failed, NULL pointer
Info : Only showing current execution because of a broken ChibiOS thread registry.
Error: ChibiOS registry integrity check failed, NULL pointer
Info : Only showing current execution because of a broken ChibiOS thread registry.
Error: address + size wrapped(0xffffffff, 0x00000004)
Error: ChibiOS registry integrity check failed, double linked list violation
Info : Only showing current execution because of a broken ChibiOS thread registry.
Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Info : halted: PC: 0x08010b12
Error: null stack pointer in thread

Error: null stack pointer in thread

Info : halted: PC: 0x08010b14
Error: null stack pointer in thread

Info : halted: PC: 0x08010a90
Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Info : halted: PC: 0x08010b1a
Error: null stack pointer in thread

Info : halted: PC: 0x08010b1e
Error: null stack pointer in thread

Info : halted: PC: 0x08010b22
Error: null stack pointer in thread

Info : halted: PC: 0x08010b24
Error: null stack pointer in thread

Info : halted: PC: 0x080098c0
Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Info : halted: PC: 0x08010b12
Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Error: null stack pointer in thread

Info : halted: PC: 0x08010b10
Info : halted: PC: 0x08010b10
Info : halted: PC: 0x08010b10
Info : halted: PC: 0x08010b10
Info : halted: PC: 0x08010b10
Info : halted: PC: 0x08010b10
Info : halted: PC: 0x08010b10
Info : halted: PC: 0x08010b10
Info : halted: PC: 0x08010b10
Info : halted: PC: 0x08010b10
Info : halted: PC: 0x08010b10
Info : halted: PC: 0x08010b10
Info : halted: PC: 0x08010b10
Info : halted: PC: 0x08010b10
Info : halted: PC: 0x08010b10

Posted: **Thu Oct 11, 2012 4:31 am**

Mabl,
To run the code yesterday, I downloaded the latest Chibios from GitHub and added a test application I was using. It appeared to be running OK.
The code has a shell, and number of GPIO interrupts set up so perhaps it was potentially confusing, adding problems with new kernal/application integration.
I went back to the demo code ARMCM3-STM32F103 with some i/o changes to match my board in the new kernal. The selftest code all ran fine.

This may be helpful for others trying to test the code as well a check.
I then tried that with your openOCD and it looks much better. At startup.

Code: Select all

openOCD_test Default [GDB Hardware Debugging]   
   ch.elf   
      Thread [3] 536875472 (blinker :  : SLEEPING) (Suspended : Container)   
         chSchGoSleepS() at chschd.c:122 0x8000990   
         chSchGoSleepTimeoutS() at chschd.c:190 0x8000a6a   
         chThdSleep() at chthreads.c:290 0x8000ed8   
         Thread1() at main.c:37 0x8009ea6   
         _port_thread_start() at chcore_v7m.c:255 0x80003cc   
      Thread [2] 536873192 (idle :  : CURRENT) (Suspended : Container)   
         _idle_thread() at chsys.c:61 0x8000508   
         _port_thread_start() at chcore_v7m.c:255 0x80003cc   
      Thread [1] 536873384 (main :  : SLEEPING) (Suspended : User Request)   
         0x8000508   
         chSchGoSleepTimeoutS() at chschd.c:190 0x8000a6a   
         chThdSleep() at chthreads.c:290 0x8000ed8   
         main() at main.c:77 0x8009f14   
   gdb

The last entry in Thread 1 does not look complete
After Resume Pause

Code: Select all

openOCD_test Default [GDB Hardware Debugging]   
   ch.elf   
      Thread [3] 536875472 (blinker :  : SLEEPING) (Suspended : Signal : SIGINT:Interrupt)   
         chSchGoSleepS() at chschd.c:122 0x8000990   
         chSchGoSleepTimeoutS() at chschd.c:190 0x8000a6a   
         chThdSleep() at chthreads.c:290 0x8000ed8   
         Thread1() at main.c:35 0x8009e90   
         _port_thread_start() at chcore_v7m.c:255 0x80003cc   
      Thread [2] 536873192 (idle :  : CURRENT) (Suspended : Container)   
         _idle_thread() at chsys.c:61 0x8000508   
         _port_thread_start() at chcore_v7m.c:255 0x80003cc   
      Thread [1] 536873384 (main :  : SLEEPING) (Suspended : Container)   
         chSchGoSleepS() at chschd.c:122 0x8000990   
         chSchGoSleepTimeoutS() at chschd.c:190 0x8000a6a   
         chThdSleep() at chthreads.c:290 0x8000ed8   
         main() at main.c:77 0x8009f14   
   gdb

All looks as expected
After resuming to a breakpoint in blinker

Code: Select all

openOCD_test Default [GDB Hardware Debugging]   
   ch.elf   
      Thread [4] 536873472 (\250\t :  : READY) (Suspended : Container)   
         Thread1() at main.c:36 0x8009e90   
         _port_thread_start() at chcore_v7m.c:255 0x80003cc   
      Thread [3] 536875472 (blinker :  : CURRENT) (Suspended : Signal : SIGTRAP:Trace/breakpoint trap)   
         chSchGoSleepS() at chschd.c:122 0x8000990   
         chSchGoSleepTimeoutS() at chschd.c:190 0x8000a6a   
         chThdSleep() at chthreads.c:290 0x8000ed8   
         Thread1() at main.c:35 0x8009e90   
         _port_thread_start() at chcore_v7m.c:255 0x80003cc   
      Thread [2] 536873192 (idle :  : READY) (Suspended : Container)   
         chSchDoRescheduleBehind() at chschd.c:309 0x8000b88   
         chSchDoReschedule() at chschd.c:363 0x8000c36   
         _port_switch_from_isr() at chcore_v7m.c:203 0x8000394   
   gdb

So far so good.
Then removing the breakpoint and resume then pause

Code: Select all

openOCD_test Default [GDB Hardware Debugging]   
   ch.elf   
      Thread [5] 536873384 (main :  : SLEEPING) (Suspended : Container)   
         chSchGoSleepS() at chschd.c:122 0x8000990   
         chSchGoSleepTimeoutS() at chschd.c:190 0x8000a6a   
         chThdSleep() at chthreads.c:290 0x8000ed8   
         main() at main.c:77 0x8009f14   
      Thread [3] 536875472 (blinker :  : SLEEPING) (Suspended : Signal : SIGINT:Interrupt)   
         chSchGoSleepS() at chschd.c:122 0x8000990   
         chSchGoSleepTimeoutS() at chschd.c:190 0x8000a6a   
         chThdSleep() at chthreads.c:290 0x8000ed8   
         Thread1() at main.c:35 0x8009e90   
         _port_thread_start() at chcore_v7m.c:255 0x80003cc   
      Thread [2] 536873192 (idle :  : CURRENT) (Suspended : Container)   
         _idle_thread() at chsys.c:61 0x8000508   
         _port_thread_start() at chcore_v7m.c:255 0x80003cc   
   gdb

Looks good.
Now there is one issue I do not understand and it does not appear correct.
When running openOCD with gdb in the past, going Resume Pause virtually always ended in _idle_thread as expected as the processor is doing very little.
Now however doing the same ends in the chSchGoSleepS in the main loop every time I try.
Does the openOCD RTOS code modify the idle thread?
I also notice that I cannot break in the idle thread.
I setup to run the old openOCD with the same code and openOCD configuration files. It stops in the loop in _idle_thread every time I try.

Now comparing with my application, I have a number of Debug flags set in chconfig.h, so I trided the same flags in ths simple code and setting

Code: Select all

CH_DBG_SYSTEM_STATE_CHECK        TRUE

results in the same behaviour as my application code.
Just for completeness I tested the following

Code: Select all

CH_DBG_SYSTEM_STATE_CHECK        TRUE    NO GOOD
CH_DBG_ENABLE_CHECKS         TRUE   OK
CH_DBG_ENABLE_ASSERTS         TRUE   OK
CH_DBG_ENABLE_TRACE             TRUE   OK
CH_DBG_ENABLE_STACK_CHECK       TRUE   OK
CH_DBG_FILL_THREADS             TRUE   OK
CH_DBG_THREADS_PROFILING        TRUE   OK

When running in the state with CH_DBG_SYSTEM_STATE_CHECK set, if I continue to run to breakpoints, it keeps creating new threads, like Thread 4 in the breakpoint case above, but a new one every time.
I do not know if having that flag set is valid when running openOCD.

I hope this helps.
Regards

Posted: **Thu Oct 11, 2012 5:05 am**

Mabl,
Another problem which may also relate to the info above.
When a breakpoint is set, it sometimes stops at the end of one of the chSchGoSleepS routines. Not always the same one.
It appears to break at the correct time, due to external input or regular events, but indicates this routine.
It will sometimes do this once then the correct break, sometimes it gets stuck repetatively.

Regards saraben

Posted: **Sun Oct 14, 2012 1:12 pm**

Hi saraben,

thanks for this great report! Today I did some real work debuging work in eclipse (i was using gdb directly before) and I was also not happy with the results shown in eclipse. It's hard to tell what exactly is wrong - but threads tend to display "wrongly".

I've discovered a workaround though: There are TWO debug frameworks in eclipse. I think you are currently using the "GDB (DSF) Hardware Debugging Launcher" while Giovannis ChibiOS plugin also only works with "Standard GDB Hardware Debugging Launcher"! The "Standard Launcher" gives me FAR better results - also on breakpoints! One problem though: Thread names are not displayed with it *grrrr* :twisted:

This also explains the "unresolved symbols"! You will have to check "Load Image" and "Load Symbols" checkboxes and in addition use the file command with the FULL path of the ELF. Here is an example screenshot:

: eclipse_debug_setups.png (93.62 KiB) Viewed 9502 times

Note, the selected debug launcher and the "file" command as well as the checked "Load image/symbols".

The other stuff:

saraben wrote:Mabl,

Code: Select all

openOCD_test Default [GDB Hardware Debugging]   
   ch.elf   
      Thread [3] 536875472 (blinker :  : SLEEPING) (Suspended : Container)   
         chSchGoSleepS() at chschd.c:122 0x8000990   
         chSchGoSleepTimeoutS() at chschd.c:190 0x8000a6a   
         chThdSleep() at chthreads.c:290 0x8000ed8   
         Thread1() at main.c:37 0x8009ea6   
         _port_thread_start() at chcore_v7m.c:255 0x80003cc   
      Thread [2] 536873192 (idle :  : CURRENT) (Suspended : Container)   
         _idle_thread() at chsys.c:61 0x8000508   
         _port_thread_start() at chcore_v7m.c:255 0x80003cc   
      Thread [1] 536873384 (main :  : SLEEPING) (Suspended : User Request)   
         0x8000508   
         chSchGoSleepTimeoutS() at chschd.c:190 0x8000a6a   
         chThdSleep() at chthreads.c:290 0x8000ed8   
         main() at main.c:77 0x8009f14   
   gdb

The last entry in Thread 1 does not look complete

Thread1 is the main thread and special. It is ok the way it is displayed.

saraben, could you give it a try again with the Default debug launcher?

Posted: **Sun Oct 14, 2012 4:22 pm**

Nice work mabl. I managed to get it to work with SublimeGDB:

Now I must hack SublimeGDB some more so that it actually works nicely for embedded development!

Posted: **Mon Oct 15, 2012 9:26 pm**

Ok, I checked out a lot of different IDEs: Netbeans, Code::Blocks, codelite - all had issues. But I think I just fell in love with qt creator:

: qtcreator_ide.gif (166.24 KiB) Viewed 9466 times

: qtcreator_toolchain.png (88.1 KiB) Viewed 9466 times

: qtcreator_buildsettings1.png (116.37 KiB) Viewed 9466 times

Posted: **Mon Oct 15, 2012 9:27 pm**

: qtcreator_startdebugger.png (42.23 KiB) Viewed 9466 times

Probably I still need a right startup file here. But it worked magically for now.

Posted: **Mon Oct 15, 2012 9:41 pm**

I had no idea it could be used for embedded.

Giovanni

ChibiOS Free Embedded RTOS

ChibiOS OpenOCD support

Re: ChibiOS OpenOCD support

Re: ChibiOS OpenOCD support

Re: ChibiOS OpenOCD support

Re: ChibiOS OpenOCD support

Re: ChibiOS OpenOCD support

Re: ChibiOS OpenOCD support

Re: ChibiOS OpenOCD support

Re: ChibiOS OpenOCD support

Re: ChibiOS OpenOCD support

Re: ChibiOS OpenOCD support