[DONE] Serial port losing data when several threads are acti

[trepidacious]

Hi,

Sorry for the brief reply here - phpBB loves to lose my replies, so I'm typing this out for the second time

I'd like to use the UART driver, and it works well, but unfortunately on the 407 with the DCMI errata I can't use DMA2, so I've run out of DMA streams...

I've been doing some more debugging, and (as expected) it doesn't seem to be the event listener directly causing/fixing any problems. If I compile the same code with -O2 instead of -O3, it will run fine without the event listener.

To simplify things, I tried a loopback with a jumper from PA2 to PA3, and a simple loop to test it:

Code: Select all

  sdStart(&SD2, NULL);
  palSetPadMode(GPIOA, 2, PAL_MODE_ALTERNATE(7));
  palSetPadMode(GPIOA, 3, PAL_MODE_ALTERNATE(7));

  //Listen for serial events
  chThdCreateStatic(wa_serial_event_thread, sizeof(wa_serial_event_thread), NORMALPRIO -1, serial_event_thread, NULL);

  uint8_t i = 0;
  while (TRUE) {
    sdPut(&SD2, i);
    msg_t r = sdGetTimeout(&SD2, MS2ST(100));
    chprintf((BaseSequentialStream *)&SDU1, "Got %d.\r\n", r);
    chThdSleepMilliseconds(100);
    i++;
  }

This code can also give me timeouts. In this case it times out regularly if the event listener thread is running, but not if it is commented out. It times out on about 80% of gets if compiled with -O1. This seems a bit like a timing problem, but it seems like it should work, and with the 100ms timeouts and delays it's not really very demanding? I guess otherwise it could be something like uninitialised structures, I've been bitten by that before, but I can't find anything like that in the onewire code, and there isn't really anywhere for it to happen at all in the loopback code.

[trepidacious]

In case it's of interest, I reduced it to a project with no onewire code, just the serial loopback. This gives me repeated timeouts, almost no data gets through. Enabling the event listener thread, OR going from -O1 to -O3, will fix it to run completely reliably. This just needs a jumper from PA2 to PA3 on a discovery F4 board.

https://www.dropbox.com/s/kgo4xh9s7r2f69t/discovery-serial-loopback.tar.gz

The length of the timeout is now 10ms, but even with 1s timeout, it still fails the same way, similarly with a delay between put and get, or more delay between repeats of the loop.

[Giovanni]

The optimizations should have no effects on this, it is a symptom of something wrong in the code.

That loopback should never timeout, it does not even depends on external code. Do you have all debug options activated in chconf.h?

I suggest you make a "naked" project, just put that loopback in the main and see if you get the same problems. BTW, you should check for RDY_TIMEOUT after sdGetTimeout() returns, you print it like it was a received character.

Try something like this:

Code: Select all

      while (TRUE) {
        sdPut(&SD2, 'X');
        msg_t r = sdGetTimeout(&SD2, MS2ST(100));
        if (RDY_TIMEOUT != r)
          chprintf((BaseSequentialStream *)&SDU1, "Got %d.\r\n", r);
        else
          chprintf((BaseSequentialStream *)&SDU1, "Timeout.\r\n", r);
        chThdSleepMilliseconds(100);
      }

Giovanni

[Giovanni]

In case it's of interest, I reduced it to a project with no onewire code, just the serial loopback. This gives me repeated timeouts, almost no data gets through. Enabling the event listener thread, OR going from -O1 to -O3, will fix it to run completely reliably. This just needs a jumper from PA2 to PA3 on a discovery F4 board.

https://www.dropbox.com/s/kgo4xh9s7r2f69t/discovery-serial-loopback.tar.gz

The length of the timeout is now 10ms, but even with 1s timeout, it still fails the same way, similarly with a delay between put and get, or more delay between repeats of the loop.

I am going to try this over the weekend.

Giovanni

[trepidacious]

Thanks
Just in case it is compiler specific, I'm using Ubuntu 12.04.3 and the toolchain here: https://launchpad.net/~terry.guo/+archive/gcc-arm-embedded, sending .bin using texane stlink, hopefully none of that will make any difference though. I'm on ChibiOS 2.6.1 from SVN. I also dug out another discovery board and it behaves the same as the first. The ch.bin/elf should be in that zip as well in case you want to try it directly.

[Giovanni]

Hi,

I wrote this test (using ChibiOS 3.0 but it should be the same):

Code: Select all

static void cmd_loop(BaseSequentialStream *chp, int argc, char *argv[]) {
  event_listener_t el;

  (void)argv;
  if (argc > 0) {
    chprintf(chp, "Usage: loop\r\n");
    return;
  }

  chEvtRegister(chnGetEventSource(&SD2), &el, 0);
  while (chnGetTimeout((BaseAsynchronousChannel *)chp,
                       TIME_IMMEDIATE) == MSG_TIMEOUT) {
    msg_t r;
    eventflags_t flags;

    sdPut(&SD2, 'X');
    r = sdGetTimeout(&SD2, MS2ST(100));

    flags = chEvtGetAndClearFlags(&el);
    if ((flags & (SD_PARITY_ERROR   | SD_FRAMING_ERROR |
                  SD_OVERRUN_ERROR  | SD_NOISE_ERROR   |
                  SD_BREAK_DETECTED)) != 0)
      chprintf(chp, "%8d Error %4x.\r\n", chVTGetSystemTimeX(), flags);

    if (MSG_TIMEOUT == r)
      chprintf(chp, "%8d Timeout\r\n", chVTGetSystemTimeX(), r);
    else
      chprintf(chp, "%8d Got %d.\r\n", chVTGetSystemTimeX(), r);

    chThdSleepMilliseconds(100);
  }
  chprintf(chp, "Done\r\n");
  chEvtUnregister(&((BaseAsynchronousChannel *)chp)->event, &el);
}

It is written as a shell command, note that some type names are different but the code should be the same on 2.6.

Running the command there are no errors nor timeouts, detaching the loopback the timeout is detected. Could you run the same code?

Giovanni

[trepidacious]

Hi Giovanni,

Thanks for the code - I adapted to 2.6.1, and I still get timeouts, the output is a little odd, mostly timeouts, occasionally successfully reading 88:

Code: Select all

ChibiOS/RT Shell
ch> loop
Got 88.
Timeout
Timeout
Timeout
Timeout
Got 88.
Timeout
Timeout
Timeout
Timeout
Timeout
Got 88.
Timeout
Timeout
Timeout
Timeout
Timeout
Timeout
Timeout
Timeout
Timeout
Timeout
Timeout
Timeout
Timeout
Timeout
Timeout

Changing to -O3, I get mostly success, some timeouts:

Code: Select all

ch> loop
Got 88.
Got 88.
Got 88.
Got 88.
Got 88.
Got 88.
Got 88.
Timeout
Got 88.
Got 88.
Got 88.
Got 88.
Got 88.
Got 88.
Got 88.
Got 88.
Got 88.
Timeout
Got 88.
Got 88.
Got 88.
Got 88.

Here's the code in case I didn't adapt correctly, I used this in the fairly standard project I was using for the one-wire and my original loopback test, which I attached in the zip before. It's based on the disco F4 demo with USB shell, but with debug enabled:

Code: Select all

#include "ch.h"
#include "hal.h"

#include "chprintf.h"
#include "shell.h"

#include "usbcfg.h"

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <ctype.h>
#include <math.h>

/* Virtual serial port over USB.*/
SerialUSBDriver SDU1;

#define SHELL_WA_SIZE   THD_WA_SIZE(2048)

static void cmd_loop(BaseSequentialStream *chp, int argc, char *argv[]) {
  EventListener el;

  (void)argv;
  if (argc > 0) {
    chprintf(chp, "Usage: loop\r\n");
    return;
  }

  chEvtRegister(&((BaseAsynchronousChannel *)chp)->event, &el, 0);
  while (chnGetTimeout((BaseAsynchronousChannel *)chp,
                       TIME_IMMEDIATE) == Q_TIMEOUT) {
    msg_t r;
    flagsmask_t flags;

    sdPut(&SD2, 'X');
    r = sdGetTimeout(&SD2, MS2ST(100));

    flags = chEvtGetAndClearFlags(&el);
    if ((flags & (SD_PARITY_ERROR   | SD_FRAMING_ERROR |
                  SD_OVERRUN_ERROR  | SD_NOISE_ERROR   |
                  SD_BREAK_DETECTED)) != 0)
      chprintf(chp, "Error %4x.\r\n", flags);

    if (Q_TIMEOUT == r)
      chprintf(chp, "Timeout\r\n", r);
    else
      chprintf(chp, "Got %d.\r\n", r);

    chThdSleepMilliseconds(100);
  }
  chprintf(chp, "Done\r\n");
  chEvtUnregister(&((BaseAsynchronousChannel *)chp)->event, &el);
}

static const ShellCommand commands[] = {
  {"loop", cmd_loop},
  {NULL, NULL}
};

static const ShellConfig shell_cfg1 = {
  (BaseSequentialStream *)&SDU1,
  commands
};

/*===========================================================================*/
/* Initialization and main thread.                                           */
/*===========================================================================*/

/*
 * Application entry point.
 */
int main(void) {

  /*
   * System initializations.
   * - HAL initialization, this also initializes the configured device drivers
   *   and performs the board-specific initializations.
   * - Kernel initialization, the main() function becomes a thread and the
   *   RTOS is active.
   */
  halInit();
  chSysInit();

  /*
   * Shell manager initialization.
   */
  shellInit();

  /*
   * Initializes a serial-over-USB CDC driver.
   */
  sduObjectInit(&SDU1);
  sduStart(&SDU1, &serusbcfg);

  /*
   * Activates the USB driver and then the USB bus pull-up on D+.
   * Note, a delay is inserted in order to not have to disconnect the cable
   * after a reset.
   */
  usbDisconnectBus(serusbcfg.usbp);
  chThdSleepMilliseconds(1000);
  usbStart(serusbcfg.usbp, &usbcfg);
  usbConnectBus(serusbcfg.usbp);

  /*
   * Activates the serial driver 2 using the driver default configuration.
   * PA2(TX) and PA3(RX) are routed to USART2.
   */
  sdStart(&SD2, NULL);
  palSetPadMode(GPIOA, 2, PAL_MODE_ALTERNATE(7));
  palSetPadMode(GPIOA, 3, PAL_MODE_ALTERNATE(7));

  /*
   * Normal main() thread activity, in this demo it just performs
   * a shell respawn upon its termination.
   */
  while (TRUE) {
    if (!shelltp) {
      if (SDU1.config->usbp->state == USB_ACTIVE) {
        /* Spawns a new shell.*/
        shelltp = shellCreate(&shell_cfg1, SHELL_WA_SIZE, NORMALPRIO);
      }
    }
    else {
      /* If the previous shell exited.*/
      if (chThdTerminated(shelltp)) {
        /* Recovers memory of the previous shell.*/
        chThdRelease(shelltp);
        shelltp = NULL;
      }
    }
    chThdSleepMilliseconds(500);
  }

}

[Giovanni]

The code looks OK.

I can offer no explanation, it works perfectly on my discovery 4. It could be something related to your HW, injected noise for example, could you try it on another F4 board?

Giovanni

[trepidacious]

I've just tried on another F4 board, and the results are the same.

I also tried adding an event listener on SD2, no errors are reported by this, and just adding this code makes the serial run reliably again.

I was wondering about trying a different toolchain, although it's tricky because with my current toolchain the code only seems to break for some combinations of event listener present/absent and optimisation level, so it will be hard to prove that a new toolchain fixes things, it could always break if something unrelated changes in the code

Here's the modified code with event listener on SD2. I'll let you know if I can work out anything more. It's good to know that the code itself is ok, I did worry that I was just using the driver incorrectly. Also useful to see how the event listener is used, I will add one to my onewire driver in the same way you show in the shell command.

Code: Select all

static void cmd_loop(BaseSequentialStream *chp, int argc, char *argv[]) {
  EventListener el;
  EventListener el2;

  (void)argv;
  if (argc > 0) {
    chprintf(chp, "Usage: loop\r\n");
    return;
  }

  chEvtRegister(&((BaseAsynchronousChannel *)chp)->event, &el, 0);
  chEvtRegister(chnGetEventSource(&SD2), &el2, 0);

  while (chnGetTimeout((BaseAsynchronousChannel *)chp,
                       TIME_IMMEDIATE) == Q_TIMEOUT) {
    msg_t r;
    flagsmask_t flags;

    sdPut(&SD2, 'X');
    r = sdGetTimeout(&SD2, MS2ST(100));

    flags = chEvtGetAndClearFlags(&el);
    if ((flags & (SD_PARITY_ERROR   | SD_FRAMING_ERROR |
                  SD_OVERRUN_ERROR  | SD_NOISE_ERROR   |
                  SD_BREAK_DETECTED)) != 0)
      chprintf(chp, "SDU1 Error %4x.\r\n", flags);

    flags = chEvtGetAndClearFlags(&el2);
    if ((flags & (SD_PARITY_ERROR   | SD_FRAMING_ERROR |
                  SD_OVERRUN_ERROR  | SD_NOISE_ERROR   |
                  SD_BREAK_DETECTED)) != 0)
      chprintf(chp, "SD2 Error %4x.\r\n", flags);

    if (Q_TIMEOUT == r)
      chprintf(chp, "Timeout\r\n", r);
    else
      chprintf(chp, "Got %d.\r\n", r);

    chThdSleepMilliseconds(100);
  }

  chprintf(chp, "Done\r\n");
  chEvtUnregister(&((BaseAsynchronousChannel *)chp)->event, &el);
  chEvtUnregister(chnGetEventSource(&SD2), &el2);
}

[Giovanni]

LOL, I didn't notice I was checking errors on the USB serial channel, there is no need for that. I edited the code I posted.

If I put that code in the demo ARMCM4-STM32F4-DISCOVERY-MEMS, it works perfectly regardless optimizations. I never get a timeout unless I remove the jumper.

Try ChibiStudio, question, does the test suite pass using your toolchain? also try running the IRQ_STORM for a while.

Giovanni