ChibiOS Free Embedded RTOS

Posted: **Wed Apr 01, 2020 3:29 pm**

Sorry for the vague topic, but I haven't been able to narrow this problem down to a specific system.

Since my last post(viewtopic.php?f=16&t=5441), I've started to get my devices up and running. I've got three serial ports, an SPI device (MPU6500), and two I2C devices (MS5611, QMC5883L) online. There are problems though. I suspect they're all related, but here are the symptoms I'm seeing:

One I2C device often repeats the same set of samples many times in a row, and the other toggles between good data and some fixed and totally wrong data
One I2C device only runs at 100khz, though the device itself is capable of high-speed mode. At 400khz, I get system lock-ups where the processor goes into an infinite loop of trying to service an I2C interrupt
One I2C device gets a constant trickle of errors returned from i2cMasterTransmitTimeout. 1-2% of requests seem to generate something other than MSG_OK. Whenever this happens I stop the bus, sleep, reconfigure, start the bus again, and then run my device configuration.
Sometimes after some minutes of operation, this will change to 100% errors. Resetting the MCU fixes this, even though the I2C device remains powered the whole time.
I have a brittle set of memory buffers for the various devices. If I change the memory location for any of the communication buffers, either the error rate changes or I start getting lock-ups. (detail below)

Maybe the common thread in these is the DMA system? I have very little understanding of how that works. In my mcuconf.h, all DMA streams are set to STM32_DMA_STREAM_ID_ANY.

Any ideas on what might be going on? My basic task is to get device communications working reliably, without any possibility of lock-ups.

Things I've done:

Verified through an o-scope that the I2C signals appear to be healthy and at the correct rate.
Tried both the clock configuration from the H7 demo, and from ArduPilot. (Modified to use HSI rather than HSE)
Swapped the physical devices.
Swapped the board/processor that I'm developing on.
Putting sleep(1) between all of the successive i2c calls.
Increased the timeout value in i2cMasterTransmitTimeout to 20ms - way bigger than necessary.
Followed the instructions in http://www.chibios.org/dokuwiki/doku.ph ... _dma_guide, having 32*x byte buffers in-32-byte aligned memory, and flushing TX and invalidating RX before/after communicating.
Implemented basic load and stack-space monitoring. All debug options are turned on as well.
Electrically, there is only one device per i2c bus. All SCL and SDA lines have 2.2k external pullup resistors. Internal pullups/pulldowns are disabled.

My architecture:

Buffers are defined as globals in main.cpp:

Code: Select all

__attribute__((__section__(".ram2"))) __attribute__((aligned (32))) static uint8_t mpu6500_tx_buffer[32];
__attribute__((__section__(".ram2"))) __attribute__((aligned (32))) static uint8_t mpu6500_rx_buffer[32];
__attribute__((__section__(".ram2"))) __attribute__((aligned (32))) static uint8_t ms5611_tx_buffer[32];
__attribute__((__section__(".ram2"))) __attribute__((aligned (32))) static uint8_t ms5611_rx_buffer[32];
__attribute__((__section__(".ram1"))) __attribute__((aligned (32))) static uint8_t qmc5883l_tx_buffer[32];
__attribute__((__section__(".ram1"))) __attribute__((aligned (32))) static uint8_t qmc5883l_rx_buffer[32];

Objects are defined as globals in main.cpp. Objects extend chibios_rt::BaseStaticThread. Each object creates one thread, which handles all of its communications to its device:

Code: Select all

__attribute__((__section__(".ram0"))) MPU6500 mpu;
__attribute__((__section__(".ram0"))) MS5611 baro;
__attribute__((__section__(".ram0"))) QMC5883L mag;

Objects are configured and started in main():

Code: Select all

  mag.configure(&BUS_I2C_MAG);
  mag.setI2CBuffers(qmc5883l_tx_buffer, qmc5883l_rx_buffer);
  mag.start(NORMALPRIO);

Within the device driver, the actual communication code looks like the following:

Code: Select all

void QMC5883L::read_registers(uint8_t first_register, uint8_t read_byte_count) {
  tx_buffer[0] = first_register;
  cacheBufferFlush(this->tx_buffer, 32);
  msg_t res = i2cMasterTransmitTimeout(this->i2c, QMC5883L_ADDRESS, this->tx_buffer, 1, this->rx_buffer, read_byte_count, TIME_MS2I(20));
  cacheBufferInvalidate(this->rx_buffer, 32);
  if (res != MSG_OK) restart_bus();
}

Posted: **Wed Apr 01, 2020 3:50 pm**

Hi,

STM32_DMA_STREAM_ID_ANY means that on xxxStart() a DMA channel is allocated among those available and taken until xxxStop(). Two drivers cannot take the same channel, the allocation can fail (with an assertion) but only in xxxStart(). I don't think the problem is here. You may try to assign static channels if you want using STM32_DMA1_STREAM0 etc.

Do you specify a timeout on I2C transfers? what is this returned code other than MSG_OK?

Something you could try is to isolate the problem, just test one device at time, is this something that happens only when all devices are being polled?

Could you dump the I2C status registers when the system goes into that infinite loop? Is something not being serviced or there is a continuously triggered condition? This has already been reported and supposedly fixed, look inside this megatopic: viewtopic.php?f=16&t=4140

Giovanni

Posted: **Thu Apr 02, 2020 2:11 am**

So I've narrowed the problem down a bit. Part of it was my fault - I was mishandling the data in my MS5611 driver, which creating the stuttering issue I described.

But using your suggestion to isolate the problem, I think I've zeroed in on something that might be an actual issue.

I've turned off all of the other drivers except for one serial port, which I'm sending a tiny amount of data through. So I now have that, and a QMC5883L compass attached to I2C3. If I use ram2 as the buffer location, it works for a few minutes. If I use anything else, like ram4, I get lots of errors constantly. (I'm using ram4 for the SPI device and other device on I2C1, when they're turned on, without issue). This, I think, is the easily-reproducible problem to focus on.

When using ram4 buffers, I almost always (say 85% of the time) get a MSG_RESET from i2cMasterTransmitTimeout(). When that happens, here's what I see in the debugger (I added a breakpoint in i2cMasterTransmitTimeout() immediately after the call to i2c_lld_master_transmit_timeout(), that triggers when the return value isn't 0):

Code: Select all

(gdb) p *i2cp
$1 = {state = I2C_ACTIVE_TX, config = 0x2401a774 <mag+604>, errors = 4, mutex = {queue = {next = 0x24000584 <I2CD3+12>, prev = 0x24000584 <I2CD3+12>}, owner = 0x0, next = 0x0}, thread = 0x0, txbytes = 0, rxbytes = 1, 
  rxdmamode = 197654, txdmamode = 197718, is_bdma = false, rx = {dma = 0x800f470 <_stm32_dma_streams>, bdma = 0x800f470 <_stm32_dma_streams>}, tx = {dma = 0x800f480 <_stm32_dma_streams+16>, 
    bdma = 0x800f480 <_stm32_dma_streams+16>}, i2c = 0x40005c00}
(gdb) p/x *i2cp->i2c
$2 = {CR1 = 0xc091, CR2 = 0x11c7d, OAR1 = 0x0, OAR2 = 0x0, TIMINGR = 0x20404768, TIMEOUTR = 0x0, ISR = 0x21, ICR = 0x0, PECR = 0x0, RXDR = 0x22, TXDR = 0x0}

For reference, here's the same info from a cycle where the i2c transaction completes successfully. This was for a command to read a register, so write 1 byte read 1 byte.

Code: Select all

(gdb) p *i2cp
$1 = {state = I2C_ACTIVE_RX, config = 0x2401a774 <mag+604>, errors = 0, mutex = {queue = {next = 0x24000584 <I2CD3+12>, prev = 0x24000584 <I2CD3+12>}, owner = 0x0, next = 0x0}, thread = 0x0, txbytes = 0, rxbytes = 0, 
  rxdmamode = 197654, txdmamode = 197718, is_bdma = false, rx = {dma = 0x800f470 <_stm32_dma_streams>, bdma = 0x800f470 <_stm32_dma_streams>}, tx = {dma = 0x800f480 <_stm32_dma_streams+16>, 
    bdma = 0x800f480 <_stm32_dma_streams+16>}, i2c = 0x40005c00}
(gdb) p/x *i2cp->i2c
$2 = {CR1 = 0xc091, CR2 = 0x1041a, OAR1 = 0x0, OAR2 = 0x0, TIMINGR = 0x20404768, TIMEOUTR = 0x0, ISR = 0x21, ICR = 0x0, PECR = 0x0, RXDR = 0x5, TXDR = 0x6}

Posted: **Thu Apr 02, 2020 6:25 am**

Hi,

H7 is a very complex device, RAM4 is on a separate matrix (D3 domain) and is seen as a single slave by all masters, see the figure 1 in the STM32H7 RM "2.1 System Architecture". Perhaps you have a bandwidth problem with all DMAs and CPU doing heavy traffic there, it is also possible there is a cost for crossing the domain (additional wait states).

Note that RAM4 has a dedicated DMA (BDMA), probably you should use that for buffers in RAM4. I would:

1) Use RAM3 for buffers.
2) Make RAM3 non-cached using MPU, this would avoid the need to flush/invalidate manually increasing performance.
3) Use RAM4 only for those peripherals attached to BDMA or other non-critical things.

Giovanni

ChibiOS Free Embedded RTOS

STM32H7 Instability?

STM32H7 Instability?

Re: STM32H7 Instability?

Re: STM32H7 Instability?

Re: STM32H7 Instability?