[NEWS] Introducing MFS

This forum is dedicated to feedback, discussions about ongoing or future developments, ideas and suggestions regarding the ChibiOS projects are welcome. This forum is NOT for support.
User avatar
Giovanni
Site Admin
Posts: 14444
Joined: Wed May 27, 2009 8:48 am
Location: Salerno, Italy
Has thanked: 1074 times
Been thanked: 921 times
Contact:

[NEWS] Introducing MFS

Postby Giovanni » Thu Nov 16, 2017 7:39 pm

Hi,

A new module has been added to the HAL, it is called MFS (Managed Flash Storage).

This module uses two banks of a flash memory to implement a persistent storage with the following features:
- Data is organized in records.
- Records can have variable size.
- Records are accessed by index 1..N.
- Record can be written, rewritten and retrieved, size can change on rewrite.
- Stored data is checked using a CRC16.
- The overhead per-record is just 12 bytes.
- MFS performs auto-initialization of flash on mfsStart().
- MFS is able to self-repair the flash storage on mfsStart().
- Wear leveling is ensured.
- MFS is power loss resistant, the lost information is only the record being written, the previous instance is restored on next boot, data is never overwritten, only appended, it should prove very safe.
- MFS uses two banks for swapping so if you need to store 64KB you need 128KB total. A bank can be composed of multiple sectors.
- MFS sits on top of the HAL flash model, any memory implementing that interface can be used.
- The API is extremely simple.

It is not a full file system but I think it is very good for:
- Configuration data.
- User preferences.
- Network MAC and IP.
- Storage of keys.
- Etc

It is not ideal in situations where data is appended to existing data, for example a logger, that would require rewriting a whole record limiting the maximum size to the memory buffer size used for the operation.

Probably I will add encryption of stored data at a later point, when the crypto subsystem will be stable.

The demo is under /testhal/STM32/multi/QSPI-MFS in trunk.

Giovanni

steved
Posts: 823
Joined: Fri Nov 09, 2012 2:22 pm
Has thanked: 12 times
Been thanked: 135 times

Re: [NEWS] Introducing MFS

Postby steved » Thu Nov 16, 2017 9:25 pm

Looks very interesting - I may have an imminent use for this.

The name suggests that it's intended for use with QSPI flash, which limits it to the F7 family ATM, I think. Is this so, or will it work with ordinary SPI flash as well?

User avatar
Giovanni
Site Admin
Posts: 14444
Joined: Wed May 27, 2009 8:48 am
Location: Salerno, Italy
Has thanked: 1074 times
Been thanked: 921 times
Contact:

Re: [NEWS] Introducing MFS

Postby Giovanni » Thu Nov 16, 2017 9:27 pm

The flash driver can work on both SPI and QSPI.

Giovanni

faisal
Posts: 374
Joined: Wed Jul 19, 2017 12:44 am
Has thanked: 44 times
Been thanked: 60 times

Re: [NEWS] Introducing MFS

Postby faisal » Fri Nov 17, 2017 12:02 am

The macros PAIR and RET_ON_ERROR have generic names. They should be prefixed appropriately, so as not to pollute the global C preprocessor namespace. PAIR and RET_ON_ERROR sound like macros that user/application code could define.

Also, it was previously mentioned in another thread that there should one place system wide which has CRC routines.

User avatar
Giovanni
Site Admin
Posts: 14444
Joined: Wed May 27, 2009 8:48 am
Location: Salerno, Italy
Has thanked: 1074 times
Been thanked: 921 times
Contact:

Re: [NEWS] Introducing MFS

Postby Giovanni » Fri Nov 17, 2017 8:52 am

hi,

Those macros are local to mfs.c, other modules cannot see those.

CRC work is in progress.

Giovanni

faisal
Posts: 374
Joined: Wed Jul 19, 2017 12:44 am
Has thanked: 44 times
Been thanked: 60 times

Re: [NEWS] Introducing MFS

Postby faisal » Sat Nov 18, 2017 1:39 am

Giovanni wrote:hi,

Those macros are local to mfs.c, other modules cannot see those.

Giovanni


Preprocessor macro definitions are in the global namespace ... the preprocessor runs before the compiler does. If PAIR, or RET_ON_ERROR is defined anywhere else, won't those macros be replaced by the last defined one?

User avatar
Giovanni
Site Admin
Posts: 14444
Joined: Wed May 27, 2009 8:48 am
Location: Salerno, Italy
Has thanked: 1074 times
Been thanked: 921 times
Contact:

Re: [NEWS] Introducing MFS

Postby Giovanni » Sat Nov 18, 2017 6:33 am

Hi,

I believe not, nothing can redefine those names after definition in mfs.c. Macros are global but only in the compiled module, the preprocessor is run for each module.

Giovanni

User avatar
Giovanni
Site Admin
Posts: 14444
Joined: Wed May 27, 2009 8:48 am
Location: Salerno, Italy
Has thanked: 1074 times
Been thanked: 921 times
Contact:

Re: [NEWS] Introducing MFS

Postby Giovanni » Mon Nov 20, 2017 12:11 pm

Question!

How valuable would it be to be able to use FatFS (or others) on top of a MFS partition? a thin block-device emulation layer would be needed between the twos. It would enable creating a full FS on top of any flash, even internal ones (performance would not be necessarily great).

This emulation layer could also include a RAM buffers cache at 512 bytes block level for cached access and, optionally, delayed writes.

The interesting thing is that even a FAT-based file system would have ensured wear leveling by the underlying MFS.

Giovanni

steved
Posts: 823
Joined: Fri Nov 09, 2012 2:22 pm
Has thanked: 12 times
Been thanked: 135 times

Re: [NEWS] Introducing MFS

Postby steved » Mon Nov 20, 2017 1:03 pm

Giovanni wrote:How valuable would it be to be able to use FatFS (or others) on top of a MFS partition? a thin block-device emulation layer would be needed between the twos. It would enable creating a full FS on top of any flash, even internal ones (performance would not be necessarily great).

This emulation layer could also include a RAM buffers cache at 512 bytes block level for cached access and, optionally, delayed writes.

Excellent idea!
AIUI, this would solve the problem of power fail during write - worst case you just lose the latest write. Plus it makes better use of flash if you have lots of small files (the quick way to implement FatFS on QSPI seems to be to use a 4096-byte block size, which must consume a relatively large amount of RAM).

(I've just started doing a board with 8M of QSPI flash, which I'm intending to use for a file system in some way TBD).

User avatar
Giovanni
Site Admin
Posts: 14444
Joined: Wed May 27, 2009 8:48 am
Location: Salerno, Italy
Has thanked: 1074 times
Been thanked: 921 times
Contact:

Re: [NEWS] Introducing MFS

Postby Giovanni » Mon Nov 20, 2017 1:55 pm

I am evaluating several approaches for this, some are time-efficient others space-efficient, undecided which one would be better.

Memory-inefficient ones are generally faster because garbage collection operations (involving flash erase of half space) are more sporadic. Other schemes would assign disk blocks to "buckets", GC would only require one extra bucket, not double space like MFS does, those schemes are less good at wear leveling, it would not be perfectly uniform.

Some notes, some could not make immediately sense, I am used to write random thoughts and then try to reorganize:

Code: Select all

Solution 1:
* 512 bytes disk blocks are written sequentially in the same bank.
- Locating a block requires scanning.
  - Storing potitions of all blocks would take a lot of RAM.
    + It could be mitigated by a cache of block positions.
- GC would be slow because it would involve copying a whole flash bank
  into the other.
- Space inefficient, 50+% of space is used for GC in addition to metadata.
+ Would reuse the MFS code and general logic.
+ Wear leveling is optimal, the whole surface is guaranteed to be used.

Solution 2:
* 512 bytes disk blocks are written is specific positions in buckets.
* Each bucket is known to contain a specific range of disk blocks.
- Each block re-write requires copying a whole bucket and erase the previous
  instance.
  + This can be mitigated using small buckets, many flash devices can work
    with small sectors, this can be used.
- Wear leveling would not be uniform, some buckets would see more activity
  than others, some could never be touched after writing. The FAT structure
  would be an example of critical blocks.
  + This can be mitigated by "spreading" block indexes using a 1->1 function,
    this way, adjacent, frequently-updated block ranges would be spread in
    multiple buckets.
+ Read access is fast because positions are known.
+ GC would be fast because it would involve only a bucket.
+ Space efficient, only one extra bucket is required for GC.
  + There could be spare buckets in case of damage.

Solution 3:
* 512 bytes disk blocks are written sequentially in buckets.
* Each bucket is known to contain a specific range of disk blocks.
* Each bucket has a % of unallocated space to allow for several writes
  before needing to GC the bucket.
* There is one empty bucket per disk for GC on copy.
- Wear leveling would not be uniform, some buckets would see more activity
  than others, some could never be touched after writing. The FAT structure
  would be an example of critical blocks.
  + This can be mitigated by "spreading" block indexes using a 1->1 function,
    this way, adjacent, frequently-updated block ranges would be spread in
    multiple buckets.
+ GC only happens on block write when a bucket is filled.
+ GC would be fast because it would involve only a bucket.
+ Space efficient, only one extra bucket is required for GC.
  + There could be spare buckets in case of damage.
 
Solution 4:
* Create the disk storage on top of MFS, records would be the buckets.
* Requires (easy) changes to MFS to allow writing parts of records, additional
  API.
* Partial writes are incompatible with CRC handled by MFS, CRC should become
  optional. CRC would be handled by the block handling layer per-block.
* Per-bucket strategy could be the same of solutions 2 or 3.
  * Solution 2:
    + Easier to implement.
    - Requires an MFS rewrite for each block written.
      + This can be mitigated using small buckets.
    - Increases frequency of general GCs because more frequent rewrites.   
  * Solution 3:
    - More complex to implement.
    - Requires extra buffer space in each bucket for rewrites.
    + Would reduce MFS rewrites and GC proportionally to the buffer space
      in buckets.
    + Solution 2 can be seen as a special case of this solution when there is
      no buffer space in a bucket, it could be a parameter.
- General GCs would be slow because it would involve the whole flash bank.
- Space inefficient, especially sub-solution 3.
+ MFS would take care of wear leveling and repair.
+ Buckets implemented in MFS records could have a size which is not a power
  of two saving some fragmented space.


Lots of choices: Space vs speed vs wear leveling vs complexity vs RAM usage....

Giovanni


Return to “Development and Feedback”

Who is online

Users browsing this forum: No registered users and 12 guests