STM32 MAC driver optimization
Posted: Mon Dec 24, 2012 7:27 pm
Hi all,
I'm building an application which has the quite restrictive requirement of being able to cope with network stall conditions that result in burst packet arrival (while in normal conditions they arrive one every 20ms). In this scenario available RAM is the most restrictic factor, and as of now, the ChibiOS STM32 MAC driver implementation does not cut it.
Problem has been commented in a thread in the support board, but I'll summarize it here:
- mac_lld buffering is wasteful: 1522 bytes (actually 1524 because of round up to alignment) per packet. If packets are small (as they are in my case) a lot of RAM is wasted.
- lwipthread is copied from mac_lld buffers to lwIP pbufs allocated from own lwIP pool. Packets are either in mac_lld buffers or in lwIP pbufs, but never in both. In order to cope with a burst of, say, 100 packets, I must allocate 100x1522 in mac_lld AND 100 pbufs in lwIP. Not enough SRAM.
I've studied the STM32 MAC peripheral and after giving it some though I've come up with a solution that involves:
1- Reduce the mac_ldd buffer size to, say, 256 bytes. The STM32 MAC is capable of storing a frame in multiple DMA descriptors, and with a granularity of 256 bytes (instead of 1522) much less memory would be wasted.
2- Implement zero copy on reception (see note at the end on transmission).
While It's quite clear to me how to implement this, I'm not so sure about how to do it in the less disruptive way so it can be easily integrated in ChibiOS and not break other MAC drivers, so I'm hereby requesting your advice and comments.
(I think (2) can be implemented without touching the mac_lld receive API)
Currently the mac_lld API for reception comprises three functions:
(a) mac_lld_get_receive_descriptor
(b) mac_lld_read_receive_descriptor
(c) mac_lld_release_receive_descriptor
The receive descriptor is an opaque structure belonging to mac_lld. (a) finds a received packet, (b) reads from it in several steps (necessary because data is to be placed in a pbuf chain of probably several pbufs) and (c) releases all buffers used for this frame so they're reverted to DMA for storage of new incoming packets.
As I see it, I think the only way to implement zero copy is to, instead of imposing mac_lld a read to a provided buffer (mac_lld_read_receive_descriptor), make repeated calls to a mac_lld function that will return chunks of memory as pairs of pointer/size until it returns telling there are no more chunks for this packet. lwipthread would allocate PBUF_ROM (or PBUF_REF, not sure) instead of PBUF_POOL and fill them in with the retrieved pointer/size pairs. Finally, lwipthread WOULD NOT call mac_lld_release_descriptor but instead set up a callback mechanism such that when the pbufs are freed later, we tell mac_lld to reuse those buffers.
lwIP already supports "custom" pbufs that include a deallocation callback, though they are only enabled when IP_FRAG is. A tiny patch would allow their enable independent of IP_FRAG.
Also, lwipthread would need some rewriting, because I understand that some architectures may not be able to provide a pointer/size pair, and the mac_lld_read_receive_descriptor may be more appropriate in their case.
Finally, note that there is an important distinction between the current and the proposed buffer release mechanism. Current works on the descriptor (which mac_lld internally interprets to release one or many buffers, whatever it represents), but proposed would work at a buffer level, because once each memory buffer goes into a pbuf they become basically independent to the lwIP memory management mechanism (though we know they're not because they all belong to the same packet pbuf chain).
I find it quite difficult to reconciliate both mechanism with a single API. Also (correct me if I'm wrong), imposing a MAC API that requires mac_lld to provide pointer/size pairs may mean some drivers would need quite a bit of work, particularly those not supporting DMA: the driver would need to manage it self buffer allocation and copy.
Maybe we could use #defines to choose between a DMA zero-copy oriented API and the current one.
Thoughs?
NOTE: I have less of a problem with transmission, where there is no problem if we use only two transmit buffers and let the outgoing packets pile up in the lwIP layer. Nonetheless, I think it is also quite easy to implement zero copy transmit, just not so sure if it is worth the trouble. The main catch is that the STM32 MAC DMA cannot read from ROM (flash), only from SRAM (excluding the CCM area). That means that outgoing pbufs would have to be searched for buffers in "offending" addresses and then allocate some RAM and copy those contents there so DMA can read them.
I'm building an application which has the quite restrictive requirement of being able to cope with network stall conditions that result in burst packet arrival (while in normal conditions they arrive one every 20ms). In this scenario available RAM is the most restrictic factor, and as of now, the ChibiOS STM32 MAC driver implementation does not cut it.
Problem has been commented in a thread in the support board, but I'll summarize it here:
- mac_lld buffering is wasteful: 1522 bytes (actually 1524 because of round up to alignment) per packet. If packets are small (as they are in my case) a lot of RAM is wasted.
- lwipthread is copied from mac_lld buffers to lwIP pbufs allocated from own lwIP pool. Packets are either in mac_lld buffers or in lwIP pbufs, but never in both. In order to cope with a burst of, say, 100 packets, I must allocate 100x1522 in mac_lld AND 100 pbufs in lwIP. Not enough SRAM.
I've studied the STM32 MAC peripheral and after giving it some though I've come up with a solution that involves:
1- Reduce the mac_ldd buffer size to, say, 256 bytes. The STM32 MAC is capable of storing a frame in multiple DMA descriptors, and with a granularity of 256 bytes (instead of 1522) much less memory would be wasted.
2- Implement zero copy on reception (see note at the end on transmission).
While It's quite clear to me how to implement this, I'm not so sure about how to do it in the less disruptive way so it can be easily integrated in ChibiOS and not break other MAC drivers, so I'm hereby requesting your advice and comments.
(I think (2) can be implemented without touching the mac_lld receive API)
Currently the mac_lld API for reception comprises three functions:
(a) mac_lld_get_receive_descriptor
(b) mac_lld_read_receive_descriptor
(c) mac_lld_release_receive_descriptor
The receive descriptor is an opaque structure belonging to mac_lld. (a) finds a received packet, (b) reads from it in several steps (necessary because data is to be placed in a pbuf chain of probably several pbufs) and (c) releases all buffers used for this frame so they're reverted to DMA for storage of new incoming packets.
As I see it, I think the only way to implement zero copy is to, instead of imposing mac_lld a read to a provided buffer (mac_lld_read_receive_descriptor), make repeated calls to a mac_lld function that will return chunks of memory as pairs of pointer/size until it returns telling there are no more chunks for this packet. lwipthread would allocate PBUF_ROM (or PBUF_REF, not sure) instead of PBUF_POOL and fill them in with the retrieved pointer/size pairs. Finally, lwipthread WOULD NOT call mac_lld_release_descriptor but instead set up a callback mechanism such that when the pbufs are freed later, we tell mac_lld to reuse those buffers.
lwIP already supports "custom" pbufs that include a deallocation callback, though they are only enabled when IP_FRAG is. A tiny patch would allow their enable independent of IP_FRAG.
Also, lwipthread would need some rewriting, because I understand that some architectures may not be able to provide a pointer/size pair, and the mac_lld_read_receive_descriptor may be more appropriate in their case.
Finally, note that there is an important distinction between the current and the proposed buffer release mechanism. Current works on the descriptor (which mac_lld internally interprets to release one or many buffers, whatever it represents), but proposed would work at a buffer level, because once each memory buffer goes into a pbuf they become basically independent to the lwIP memory management mechanism (though we know they're not because they all belong to the same packet pbuf chain).
I find it quite difficult to reconciliate both mechanism with a single API. Also (correct me if I'm wrong), imposing a MAC API that requires mac_lld to provide pointer/size pairs may mean some drivers would need quite a bit of work, particularly those not supporting DMA: the driver would need to manage it self buffer allocation and copy.
Maybe we could use #defines to choose between a DMA zero-copy oriented API and the current one.
Thoughs?
NOTE: I have less of a problem with transmission, where there is no problem if we use only two transmit buffers and let the outgoing packets pile up in the lwIP layer. Nonetheless, I think it is also quite easy to implement zero copy transmit, just not so sure if it is worth the trouble. The main catch is that the STM32 MAC DMA cannot read from ROM (flash), only from SRAM (excluding the CCM area). That means that outgoing pbufs would have to be searched for buffers in "offending" addresses and then allocate some RAM and copy those contents there so DMA can read them.
