STM32G473 test fail

[alexblack]

Hi.
I tried to use the new stable Chibios Release 21.6.x and have troubles (the MCU goes to the Hard Fault) and decided to check it by running TEST.
The test hangs up in "Test Case 5.2" in infinity loop of the function chRegFindThreadByWorkingArea.
The state of the MCU is in picture.

Code: Select all

*** ChibiOS/RT Test Suite
***
*** Compiled:     Jun 24 2021 - 08:48:36
*** Platform:     STM32G4 Performance Line

============================================================================
=== Test Sequence 1 (Information)
----------------------------------------------------------------------------
--- Test Case 1.1 (Port Info)
--- Architecture:                       ARMv7E-M
--- Core Variant:                       Cortex-M4F
--- Compiler:                           RVCT
--- Port Info:                          Advanced kernel mode
--- Natural alignment:                  4
--- Stack alignment:                    8
--- Working area alignment:             8
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 1.2 (Kernel Info)
--- Product:                            ChibiOS/RT
--- Stable Flag:                        0
--- Version String:                     7.0.0
--- Major Number:                       7
--- Minor Number:                       0
--- Patch Number:                       0
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 1.3 (Kernel Settings)
--- CH_CFG_ST_RESOLUTION:               32
--- CH_CFG_ST_FREQUENCY:                10000
--- CH_CFG_INTERVALS_SIZE:              32
--- CH_CFG_TIME_TYPES_SIZE:             32
--- CH_CFG_ST_TIMEDELTA:                2
--- CH_CFG_TIME_QUANTUM:                0
--- CH_CFG_MEMCORE_SIZE:                0
--- CH_CFG_NO_IDLE_THREAD:              0
--- CH_CFG_OPTIMIZE_SPEED:              1
--- CH_CFG_USE_TM:                      0
--- CH_CFG_USE_REGISTRY:                1
--- CH_CFG_USE_WAITEXIT:                1
--- CH_CFG_USE_SEMAPHORES:              1
--- CH_CFG_USE_SEMAPHORES_PRIORITY:     0
--- CH_CFG_USE_MUTEXES:                 0
--- CH_CFG_USE_MUTEXES_RECURSIVE:       0
--- CH_CFG_USE_CONDVARS:                0
--- CH_CFG_USE_CONDVARS_TIMEOUT:        0
--- CH_CFG_USE_EVENTS:                  1
--- CH_CFG_USE_EVENTS_TIMEOUT:          1
--- CH_CFG_USE_MESSAGES:                1
--- CH_CFG_USE_MESSAGES_PRIORITY:       0
--- CH_CFG_USE_DYNAMIC:                 0
--- CH_DBG_STATISTICS:                  0
--- CH_DBG_SYSTEM_STATE_CHECK:          1
--- CH_DBG_ENABLE_CHECKS:               1
--- CH_DBG_ENABLE_ASSERTS:              1
--- CH_DBG_TRACE_MASK:                  255
--- CH_DBG_TRACE_BUFFER_SIZE:           16
--- CH_DBG_ENABLE_STACK_CHECK:          1
--- CH_DBG_FILL_THREADS:                1
--- CH_DBG_THREADS_PROFILING:           0
--- Result: SUCCESS
============================================================================
=== Test Sequence 2 (System layer and port interface)
----------------------------------------------------------------------------
--- Test Case 2.1 (System integrity functionality)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 2.2 (Critical zones functionality)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 2.3 (Interrupts handling functionality)
--- Result: SUCCESS
============================================================================
=== Test Sequence 3 (Time and Intervals Functionality)
----------------------------------------------------------------------------
--- Test Case 3.1 (System Tick Counter functionality)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 3.2 (Time ranges functionality)
--- Result: SUCCESS
============================================================================
=== Test Sequence 4 (Time Stamps Functionality)
----------------------------------------------------------------------------
--- Test Case 4.1 (Time Stamps functionality)
--- Result: SUCCESS
============================================================================
=== Test Sequence 5 (Threads Functionality)
----------------------------------------------------------------------------
--- Test Case 5.1 (Thread Sleep functionality)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 5.2 (Ready List functionality, threads priority order)

[Giovanni]

Hi,

It looks like that the registry list is corrupted somehow, probably you are observing the effect not the cause.

Start with checking for stack overflows.

Giovanni

[alexblack]

It can't be the stack problem because it is huge: 0x1000 for main and 0x1000 for irq, 512 bytes for iddle.

The situation is: I used the trunk copy of the 20.x branch from 02/04/2021 and my projects worked.
Now I tried to switch to the current stable 20.3.x or 21.6.x and it hangs up in Hard Fault. If I debug the code by steps and it pass the problem and hangs up later. I can't catch the place where it hangs. I decided to start from start and run the TEST - it was fail. Is it possible that it is some configuration problems? I tried configurations from DEMO and the all tests passed.

Code: Select all

*** ChibiOS/RT Test Suite
***
*** Compiled:     Jun 24 2021 - 11:03:52
*** Platform:     STM32G4 Performance Line

============================================================================
=== Test Sequence 1 (Information)
----------------------------------------------------------------------------
--- Test Case 1.1 (Port Info)
--- Architecture:                       ARMv7E-M
--- Core Variant:                       Cortex-M4F
--- Compiler:                           RVCT
--- Port Info:                          Advanced kernel mode
--- Natural alignment:                  4
--- Stack alignment:                    8
--- Working area alignment:             8
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 1.2 (Kernel Info)
--- Product:                            ChibiOS/RT
--- Stable Flag:                        0
--- Version String:                     7.0.0
--- Major Number:                       7
--- Minor Number:                       0
--- Patch Number:                       0
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 1.3 (Kernel Settings)
--- CH_CFG_ST_RESOLUTION:               32
--- CH_CFG_ST_FREQUENCY:                10000
--- CH_CFG_INTERVALS_SIZE:              32
--- CH_CFG_TIME_TYPES_SIZE:             32
--- CH_CFG_ST_TIMEDELTA:                2
--- CH_CFG_TIME_QUANTUM:                0
--- CH_CFG_MEMCORE_SIZE:                0
--- CH_CFG_NO_IDLE_THREAD:              0
--- CH_CFG_OPTIMIZE_SPEED:              1
--- CH_CFG_USE_TM:                      1
--- CH_CFG_USE_REGISTRY:                1
--- CH_CFG_USE_WAITEXIT:                1
--- CH_CFG_USE_SEMAPHORES:              1
--- CH_CFG_USE_SEMAPHORES_PRIORITY:     0
--- CH_CFG_USE_MUTEXES:                 1
--- CH_CFG_USE_MUTEXES_RECURSIVE:       0
--- CH_CFG_USE_CONDVARS:                1
--- CH_CFG_USE_CONDVARS_TIMEOUT:        1
--- CH_CFG_USE_EVENTS:                  1
--- CH_CFG_USE_EVENTS_TIMEOUT:          1
--- CH_CFG_USE_MESSAGES:                1
--- CH_CFG_USE_MESSAGES_PRIORITY:       0
--- CH_CFG_USE_DYNAMIC:                 1
--- CH_DBG_STATISTICS:                  0
--- CH_DBG_SYSTEM_STATE_CHECK:          0
--- CH_DBG_ENABLE_CHECKS:               0
--- CH_DBG_ENABLE_ASSERTS:              0
--- CH_DBG_TRACE_MASK:                  255
--- CH_DBG_TRACE_BUFFER_SIZE:           128
--- CH_DBG_ENABLE_STACK_CHECK:          0
--- CH_DBG_FILL_THREADS:                0
--- CH_DBG_THREADS_PROFILING:           0
--- Result: SUCCESS
============================================================================
=== Test Sequence 2 (System layer and port interface)
----------------------------------------------------------------------------
--- Test Case 2.1 (System integrity functionality)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 2.2 (Critical zones functionality)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 2.3 (Interrupts handling functionality)
--- Result: SUCCESS
============================================================================
=== Test Sequence 3 (Time and Intervals Functionality)
----------------------------------------------------------------------------
--- Test Case 3.1 (System Tick Counter functionality)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 3.2 (Time ranges functionality)
--- Result: SUCCESS
============================================================================
=== Test Sequence 4 (Time Stamps Functionality)
----------------------------------------------------------------------------
--- Test Case 4.1 (Time Stamps functionality)
--- Result: SUCCESS
============================================================================
=== Test Sequence 5 (Threads Functionality)
----------------------------------------------------------------------------
--- Test Case 5.1 (Thread Sleep functionality)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 5.2 (Ready List functionality, threads priority order)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 5.3 (Priority change test)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 5.4 (Priority change test with Priority Inheritance)
--- Result: SUCCESS
============================================================================
=== Test Sequence 6 (Suspend/Resume)
----------------------------------------------------------------------------
--- Test Case 6.1 (Suspend and Resume functionality)
--- Result: SUCCESS
============================================================================
=== Test Sequence 7 (Counter Semaphores)
----------------------------------------------------------------------------
--- Test Case 7.1 (Semaphore primitives, no state change)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 7.2 (Semaphore enqueuing test)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 7.3 (Semaphore timeout test)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 7.4 (Testing chSemAddCounterI() functionality)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 7.5 (Testing chSemWaitSignal() functionality)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 7.6 (Testing Binary Semaphores special case)
--- Result: SUCCESS
============================================================================
=== Test Sequence 8 (Mutexes, Condition Variables and Priority Inheritance)
----------------------------------------------------------------------------
--- Test Case 8.1 (Priority enqueuing test)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 8.2 (Priority return verification)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 8.3 (Repeated locks, non recursive scenario)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 8.4 (Condition Variable signal test)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 8.5 (Condition Variable broadcast test)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 8.6 (Condition Variable priority boost test)
--- Result: SUCCESS
============================================================================
=== Test Sequence 9 (Synchronous Messages)
----------------------------------------------------------------------------
--- Test Case 9.1 (Messages Server loop)
--- Result: SUCCESS
============================================================================
=== Test Sequence 10 (Event Sources and Event Flags)
----------------------------------------------------------------------------
--- Test Case 10.1 (Events registration)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 10.2 (Event Flags dispatching)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 10.3 (Events Flags wait using chEvtWaitOne())
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 10.4 (Events Flags wait using chEvtWaitAny())
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 10.5 (Events Flags wait using chEvtWaitAll())
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 10.6 (Events Flags wait timeouts)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 10.7 (Broadcasting using chEvtBroadcast())
--- Result: SUCCESS
============================================================================
=== Test Sequence 11 (Dynamic threads)
----------------------------------------------------------------------------
--- Test Case 11.1 (Threads creation from Memory Heap)
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 11.2 (Threads creation from Memory Pool)
--- Result: SUCCESS
============================================================================
=== Test Sequence 12 (Benchmarks)
----------------------------------------------------------------------------
--- Test Case 12.1 (Messages performance #1)
--- Score : 138540 msgs/S, 277080 ctxswc/S
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 12.2 (Messages performance #2)
--- Score : 118248 msgs/S, 236496 ctxswc/S
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 12.3 (Messages performance #3)
--- Score : 118258 msgs/S, 236516 ctxswc/S
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 12.4 (Context Switch performance)
--- Score : 589232 ctxswc/S
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 12.5 (Threads performance, full cycle)
--- Score : 88761 threads/S
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 12.6 (Threads performance, create/exit only)
--- Score : 117292 threads/S
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 12.7 (Mass reschedule performance)
--- Score : 42151 reschedules/S, 252906 ctxswc/S
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 12.8 (Round-Robin voluntary reschedule)
--- Score : 340240 ctxswc/S
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 12.9 (Virtual Timers set/reset performance)
--- Score : 218110 timers/S
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 12.10 (Semaphores wait/signal performance)
--- Score : 463984 wait+signal/S
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 12.11 (Mutexes lock/unlock performance)
--- Score : 383476 lock+unlock/S
--- Result: SUCCESS
----------------------------------------------------------------------------
--- Test Case 12.12 (RAM Footprint)
--- OS    : 136 bytes
--- Thread: 72 bytes
--- Timer : 24 bytes
--- Semaph: 12 bytes
--- Mutex : 16 bytes
--- CondV.: 8 bytes
--- EventS: 4 bytes
--- EventL: 20 bytes
--- MailB.: 40 bytes
--- Result: SUCCESS
----------------------------------------------------------------------------

Final result: SUCCESS

[Giovanni]

Hi,

Try increasing test stack sizes in ch_test_root.h, try 256 instead of 128.

Giovanni

[alexblack]

Hi.
I took another PCB and the prohect start to work. After detailed analyse, I found that the PCBs has differnet Chip Revisions.
Thats works well is the Z Revision, while another is Y Revision hangs in hard fault after project started..

I'll check the Errata but can't anderstand what module is wrong. Common scenario is it hangs in spiStart, but if I switch off gptStart which is called before it does not hang.

[Giovanni]

Try compiling with -O0 and see where it crashes exactly.

Giovanni

[alexblack]

Hi.
If I set CH_CFG_USE_DYNAMIC to TRUE, then the test 5.2 passed, if I set this macro to FALSE the test 5.2 hangs up!
But this option does not effect on the Hard fault effect.

p.s. of course I compile the project with the option -O0.

[Giovanni]

Hi.
If I set CH_CFG_USE_DYNAMIC to TRUE, then the test 5.2 passed, if I set this macro to FALSE the test 5.2 hangs up!
But this option does not effect on the Hard fault effect.

Let me check this.

Giovanni

[alexblack]

About Hard Fault: it seems it is related to the PCB because I checked several PCBs with the same MCU but different decisions and found that the problem is in the one type PCB only. I will check the hardware ...

[alexblack]

I cheked the hardware and even reflowed MCU with different revisions: MCU revisions do not effect for the Hard Fault problem.

I sequently checked last revisions of the SVN Repositiry and was found that last working revision is 14324 of trunk branch.
I do not understand how the OS implies to the problem but I did not found any hardware problems on PCB.

When HardFault happned I coud'nt determine the reason of the Exception: the return address is pointing to invalid addresses, stack do not have needed information. If Step In the code, where the exception happened - the problem goes to another place when I run the MCU.

All optimization of the compiller is off, the state checker and other debug options of the ChibiOS is activeted.
The nature and place of the the Exception is changed if I change some code.