px-lib  0.8.3 dev
Cross-platform embedded library and documentation for 8/16/32-bit microcontrollers
px_log_fs.h : Record-based file system for Serial Flash

Description

Record-based file system for Serial Flash.

File(s):

1. Introduction

px_log_fs is a basic but resilient record-based file system to store sequential log data on Serial Flash, for example Adesto AT25S Serial Flash. It is designed with the following fundamental characteristics in mind:

File system features:

A file is defined as a collection of sequential records that contain fixed-size log data. The records can be navigated forwards or backwards to read data, while still being able to append new data. New records will overwrite oldest records when the file is full (PX_LOG_FS_CFG_STOP_WR_WHEN_FULL = 0) or record writing will stop when the file is full (PX_LOG_FS_CFG_STOP_WR_WHEN_FULL = 1).

The size of each record is configured with PX_LOG_FS_CFG_REC_DATA_SIZE. See 'px_log_fs_cfg_template.h'

The erase block size (number of pages) is configurable with PX_LOG_FS_CFG_ERASE_BLOCK_SIZE. For example a single page of the Adesto AT45DB041E can be erased at a time (very convenient), but the AT25SF041 has a minimum erase block size of 4 KB or 8 pages (a much cheaper option).

The file system must start at the beginning of an erase block and end at the end of an erase block. It must be at least two erase blocks in size, but more is better.

First a code example to demonstrate the API, followed by implementation details...

2. Code example

#include <stdio.h>
#include "px_board.h"
#include "px_spi.h"
#include "px_at25s.h"
#include "px_log_fs.h"
// Define record data content
typedef struct
{
int16_t temperature;
} px_log_fs_data_t;
static px_spi_handle_t px_at25s_spi_handle;
static px_log_fs_handle_t px_log_fs_handle;
static px_log_fs_data_t px_log_fs_data;
int main(void)
{
uint8_t i;
px_log_fs_err_t px_log_fs_err;
// Initialise modules
px_spi_open2(&px_at25s_spi_handle, PX_SPI_PER_1, PX_BOARD_SPI_CS_AT25S,
px_at25s_init(&px_at25s_spi_handle);
// Initialise file system
px_log_fs_init(&px_log_fs_handle, 0, PX_AT25S_PAGES - 1);
// Write 8 records
px_log_fs_data.temperature = 160; // 16.0 deg Celsius
for(i=0; i<8; i++)
{
// Write record
px_log_fs_record_wr(&px_log_fs_handle,
&px_log_fs_data,
sizeof(px_log_fs_data));
// Increment temperature by 1.3 deg
px_log_fs_data.temperature += 13;
}
// Read first record
px_log_fs_err = px_log_fs_rd_first(&px_log_fs_data, sizeof(px_log_fs_data));
while(px_log_fs_err == PX_LOG_FS_ERR_NONE)
{
// Display record data
printf("%d\n", px_log_fs_data.temp);
// Read next record
px_log_fs_err = px_log_fs_rd_next(&px_log_fs_data, sizeof(px_log_fs_data));
}
}

3. Implementation details

The Serial Flash pages are treated as one big circular buffer. Records are stored sequentially and record writing will stop (or wrap to overwrite oldest records) when the start of the oldest record is reached. This simplified scheme implicitly allows for wear leveling, because each Serial Flash page is erased once per pass through the circular buffer.

The file system leverages a characteristic of Serial Flash. When a block of pages are erased, all the data bits are reset to 1. When writing a byte, bits can be cleared (0), but not reset to 1 again. The following 8-bit marker values are used:

markers.png
8-bit markers

By inspecting the bits, it can be observed that a marker can be changed from FREE to USED (or RECORD) to BAD, but not the other way around without erasing the whole block.

Each written page starts with a header (px_log_fs_header_t) which consists of an 8-bit marker, a 16-bit incrementing number and an 8-bit CRC. Only the first 4 bytes of each page is read to quickly index the whole file system during px_log_fs_init(). The incrementing number is used to figure out which is the oldest USED page and which is the newest during px_log_fs_init().

page_header.png
Page header structure

Pages containing records are marked as USED. Each record (px_log_fs_record_t) contains an 8-bit RECORD marker, the record data (fixed size set with PX_LOG_FS_CFG_REC_DATA_SIZE) and an 8-bit CRC.

record.png
Record structure

A record cannot span across pages, so there may be unused space at the end of each page. For example, if a page is 256 bytes and the page header takes up 4 bytes, then there are 252 bytes left over for records. If 8 data bytes are stored in each record, then the total record size is 10 bytes (including overhead). This means that 25 records can be stored in each page and 2 bytes will be unused (252 - 25*10 = 2).

4. How to find the FIRST and LAST page

To figure out which page contains the oldest records (FIRST page) and which page contains the newest records (LAST page), a rolling (or wrapping) number is assigned to each page. This means that each consecutive page is labelled with an incrementing number. If the number exceeds the maximum, it starts again at zero (rolls over).

To simplify these examples, the rolling number starts at 00, increments up to 99 and then rolls over to 00.

Each page's rolling number is compared with the next page's rolling number. The largest difference in the rolling numbers indicates that the LAST page has been found. The next valid page is per implication the FIRST.

To be exact, modulo unsigned integer arithmetic is used:

Diff = (rolling_number_next - rolling_number) mod 100

Here are a few select test cases to demonstrate that it works under various conditions:

first_last_01.png
1. Page 5 is LAST and page 0 is FIRST (largest diff is 95)
first_last_02.png
2. Page 9 is LAST and page 4 is FIRST (largest diff = 95)
first_last_03.png
3. Page 10 is LAST and page 12 is FIRST (largest diff = 86)
first_last_04.png
4. Page 14 is LAST and page 0 is FIRST (largest diff = 86)
first_last_05.png
5. Page 2 is LAST and page 3 is FIRST (largest diff = 85)
first_last_06.png
6. Page 3 is LAST and page 4 is FIRST (largest diff = 85)
first_last_07.png
7. Page 6 is LAST and page 7 is FIRST (largest diff = 85)

Data Structures

struct  px_log_fs_adr_t
 Specification of data address in Serial Flash. More...
 
struct  px_log_fs_handle_t
 

Macros

#define PX_LOG_FS_CFG_PAGE_SIZE   PX_AT25S_PAGE_SIZE
 Page size for file system. More...
 
#define PX_LOG_FS_CFG_ERASE_BLOCK_SIZE   PX_AT25S_PAGES_PER_BLOCK_4KB
 Erase block size (in pages) for file system. More...
 
#define PX_LOG_FS_CFG_REC_DATA_SIZE   13
 Record data size (total record size = PX_LOG_FS_CFG_REC_DATA_SIZE + 3 bytes overhead) More...
 
#define PX_LOG_FS_CFG_STOP_WR_WHEN_FULL   0
 Stop writing when full (1) or erase oldest records and continue writing (0) More...
 
#define PX_LOG_FS_CFG_PAGE_SIZE   PX_AT25S_PAGE_SIZE
 Page size for file system. More...
 
#define PX_LOG_FS_CFG_ERASE_BLOCK_SIZE   PX_AT25S_PAGES_PER_BLOCK_4KB
 Erase block size (in pages) for file system. More...
 
#define PX_LOG_FS_CFG_REC_DATA_SIZE   13
 Record data size (total record size = PX_LOG_FS_CFG_REC_DATA_SIZE + 3 bytes overhead) More...
 
#define PX_LOG_FS_CFG_STOP_WR_WHEN_FULL   0
 Stop writing when full (1) or erase oldest records and continue writing (0) More...
 

Enumerations

enum  px_log_fs_err_t {
  PX_LOG_FS_ERR_NONE = 0, PX_LOG_FS_ERR_EMPTY, PX_LOG_FS_ERR_FULL, PX_LOG_FS_ERR_WRITE_FAIL,
  PX_LOG_FS_ERR_NO_RECORD, PX_LOG_FS_ERR_FATAL
}
 Error codes. More...
 

Functions

px_log_fs_err_t px_log_fs_init (px_log_fs_handle_t *handle, uint16_t fs_page_start, uint16_t fs_page_end)
 Initialise log file system. More...
 
px_log_fs_err_t px_log_fs_reset (px_log_fs_handle_t *handle, uint16_t fs_page_start, uint16_t fs_page_end)
 Reset log file system. More...
 
px_log_fs_err_t px_log_fs_rd_first (px_log_fs_handle_t *handle, void *data, size_t nr_of_bytes)
 Read first (oldest) record. More...
 
px_log_fs_err_t px_log_fs_rd_next (px_log_fs_handle_t *handle, void *data, size_t nr_of_bytes)
 Read next (newer) record. More...
 
px_log_fs_err_t px_log_fs_rd_last (px_log_fs_handle_t *handle, void *data, size_t nr_of_bytes)
 Read last (newest) record. More...
 
px_log_fs_err_t px_log_fs_rd_previous (px_log_fs_handle_t *handle, void *data, size_t nr_of_bytes)
 Read previous (older) record. More...
 
px_log_fs_err_t px_log_fs_wr (px_log_fs_handle_t *handle, const void *data, size_t nr_of_bytes)
 Write a record to the file. More...
 
void px_log_fs_dbg_report_info (px_log_fs_handle_t *handle)
 Report log file system info. More...
 
void px_log_fs_glue_rd (void *buffer, uint16_t page, uint16_t start_byte_in_page, uint16_t nr_of_bytes)
 Read data from Serial Flash. More...
 
void px_log_fs_glue_wr (const void *buffer, uint16_t page, uint16_t start_byte_in_page, uint16_t nr_of_bytes)
 Writes data to Serial Flash. More...
 
void px_log_fs_glue_erase_block (uint16_t page)
 Erases data in Serial Flash. More...
 

Data Structure Documentation

◆ px_log_fs_adr_t

struct px_log_fs_adr_t

Specification of data address in Serial Flash.

Definition at line 217 of file px_log_fs.h.

Data Fields
uint16_t page Page.
uint16_t offset Offset inside page.

◆ px_log_fs_handle_t

struct px_log_fs_handle_t

Definition at line 223 of file px_log_fs.h.

Data Fields
uint16_t fs_page_start First page in file system (must be on the start of an erase block)
uint16_t fs_page_end Last page in file system (must be on the end of an erase block)
uint16_t page_first First page with records (PX_LOG_FS_PAGE_INVALID if empty)
uint16_t page_last Last page with records (PX_LOG_FS_PAGE_INVALID if empty)
uint16_t page_nr_next Next page number to use (starts at 0)
px_log_fs_adr_t adr_rd Current read address.
px_log_fs_adr_t adr_wr Next write address (open position)

Macro Definition Documentation

◆ PX_LOG_FS_CFG_PAGE_SIZE [1/2]

#define PX_LOG_FS_CFG_PAGE_SIZE   PX_AT25S_PAGE_SIZE

Page size for file system.

Definition at line 34 of file px_log_fs_cfg.h.

◆ PX_LOG_FS_CFG_ERASE_BLOCK_SIZE [1/2]

#define PX_LOG_FS_CFG_ERASE_BLOCK_SIZE   PX_AT25S_PAGES_PER_BLOCK_4KB

Erase block size (in pages) for file system.

Definition at line 37 of file px_log_fs_cfg.h.

◆ PX_LOG_FS_CFG_REC_DATA_SIZE [1/2]

#define PX_LOG_FS_CFG_REC_DATA_SIZE   13

Record data size (total record size = PX_LOG_FS_CFG_REC_DATA_SIZE + 3 bytes overhead)

Definition at line 40 of file px_log_fs_cfg.h.

◆ PX_LOG_FS_CFG_STOP_WR_WHEN_FULL [1/2]

#define PX_LOG_FS_CFG_STOP_WR_WHEN_FULL   0

Stop writing when full (1) or erase oldest records and continue writing (0)

Definition at line 43 of file px_log_fs_cfg.h.

◆ PX_LOG_FS_CFG_PAGE_SIZE [2/2]

#define PX_LOG_FS_CFG_PAGE_SIZE   PX_AT25S_PAGE_SIZE

Page size for file system.

Definition at line 34 of file px_log_fs_cfg_template.h.

◆ PX_LOG_FS_CFG_ERASE_BLOCK_SIZE [2/2]

#define PX_LOG_FS_CFG_ERASE_BLOCK_SIZE   PX_AT25S_PAGES_PER_BLOCK_4KB

Erase block size (in pages) for file system.

Definition at line 37 of file px_log_fs_cfg_template.h.

◆ PX_LOG_FS_CFG_REC_DATA_SIZE [2/2]

#define PX_LOG_FS_CFG_REC_DATA_SIZE   13

Record data size (total record size = PX_LOG_FS_CFG_REC_DATA_SIZE + 3 bytes overhead)

Definition at line 40 of file px_log_fs_cfg_template.h.

◆ PX_LOG_FS_CFG_STOP_WR_WHEN_FULL [2/2]

#define PX_LOG_FS_CFG_STOP_WR_WHEN_FULL   0

Stop writing when full (1) or erase oldest records and continue writing (0)

Definition at line 43 of file px_log_fs_cfg_template.h.

Enumeration Type Documentation

◆ px_log_fs_err_t

Error codes.

Enumerator
PX_LOG_FS_ERR_NONE 

No error.

PX_LOG_FS_ERR_EMPTY 

File is empty.

PX_LOG_FS_ERR_FULL 

File is full.

PX_LOG_FS_ERR_WRITE_FAIL 

Failed to write.

PX_LOG_FS_ERR_NO_RECORD 

No record found.

PX_LOG_FS_ERR_FATAL 

Fatal file system error.

Definition at line 206 of file px_log_fs.h.

Function Documentation

◆ px_log_fs_init()

px_log_fs_err_t px_log_fs_init ( px_log_fs_handle_t handle,
uint16_t  fs_page_start,
uint16_t  fs_page_end 
)

Initialise log file system.

Parameters
handlePointer to file system handle structure
fs_page_startFirst page in file system (must be on the start of an erase block)
fs_page_endLast page in file system (must be on the end of an erase block)
Return values
PX_LOG_FS_ERR_NONEFile system was succesfully indexed

Definition at line 360 of file px_log_fs.c.

◆ px_log_fs_reset()

px_log_fs_err_t px_log_fs_reset ( px_log_fs_handle_t handle,
uint16_t  fs_page_start,
uint16_t  fs_page_end 
)

Reset log file system.

Parameters
handlePointer to file system handle structure
fs_page_startFirst page in file system (must be on the start of an erase block)
fs_page_endLast page in file system (must be on the end of an erase block)
Return values
PX_LOG_FS_ERR_NONEFile system was succesfully resetted

Definition at line 458 of file px_log_fs.c.

◆ px_log_fs_rd_first()

px_log_fs_err_t px_log_fs_rd_first ( px_log_fs_handle_t handle,
void *  data,
size_t  nr_of_bytes 
)

Read first (oldest) record.

Parameters
handlePointer to file system handle structure
dataPointer to buffer where record data must be copied to
nr_of_bytesNumber of bytes to copy from record
Return values
PX_LOG_FS_ERR_NONEValid record data found and copied into structure
PX_LOG_FS_ERR_NO_RECORDNo record found

Definition at line 513 of file px_log_fs.c.

◆ px_log_fs_rd_next()

px_log_fs_err_t px_log_fs_rd_next ( px_log_fs_handle_t handle,
void *  data,
size_t  nr_of_bytes 
)

Read next (newer) record.

Parameters
handlePointer to file system handle structure
dataPointer to buffer where record data must be copied to
nr_of_bytesNumber of bytes to copy from record
Return values
PX_LOG_FS_ERR_NONEValid record data found and copied into structure
PX_LOG_FS_ERR_NO_RECORDNo record found

Definition at line 530 of file px_log_fs.c.

◆ px_log_fs_rd_last()

px_log_fs_err_t px_log_fs_rd_last ( px_log_fs_handle_t handle,
void *  data,
size_t  nr_of_bytes 
)

Read last (newest) record.

Parameters
handlePointer to file system handle structure
dataPointer to buffer where record data must be copied to
nr_of_bytesNumber of bytes to copy from record
Return values
PX_LOG_FS_ERR_NONEValid record data found and copied into structure
PX_LOG_FS_ERR_NO_RECORDNo record found

Definition at line 583 of file px_log_fs.c.

◆ px_log_fs_rd_previous()

px_log_fs_err_t px_log_fs_rd_previous ( px_log_fs_handle_t handle,
void *  data,
size_t  nr_of_bytes 
)

Read previous (older) record.

Parameters
handlePointer to file system handle structure
dataPointer to buffer where record data must be copied to
nr_of_bytesNumber of bytes to copy from record
Return values
PX_LOG_FS_ERR_NONEValid record data found and copied into structure
PX_LOG_FS_ERR_NO_RECORDNo record found

Definition at line 599 of file px_log_fs.c.

◆ px_log_fs_wr()

px_log_fs_err_t px_log_fs_wr ( px_log_fs_handle_t handle,
const void *  data,
size_t  nr_of_bytes 
)

Write a record to the file.

A new record is appended to the end of the list of records. The maximum number of bytes that can be stored in the record is specified by PX_LOG_FS_CFG_DATA_SIZE.

Parameters
handlePointer to file system handle structure
dataPointer to buffer containing data that must be stored in the record
nr_of_bytesNumber of bytes that must be written
Return values
PX_LOG_FS_ERR_NONESuccess
PX_LOG_FS_ERR_FULLFile (or file system) is full
PX_LOG_FS_ERR_WRITE_FAILFailed to write record (verify failed)

Definition at line 651 of file px_log_fs.c.

◆ px_log_fs_dbg_report_info()

void px_log_fs_dbg_report_info ( px_log_fs_handle_t handle)

Report log file system info.

Parameters
handlePointer to file system handle structure

Definition at line 750 of file px_log_fs.c.

◆ px_log_fs_glue_rd()

void px_log_fs_glue_rd ( void *  buffer,
uint16_t  page,
uint16_t  start_byte_in_page,
uint16_t  nr_of_bytes 
)

Read data from Serial Flash.

This glue function reads data from Serial Flash and stores it in the specified buffer.

Parameters
bufferBuffer to store read data
pagePX_LOG_FS_CFG_PAGE_START to PX_LOG_FS_CFG_PAGE_END
start_byte_in_page0 to PX_LOG_FS_CFG_PAGE_SIZE - 1
nr_of_bytes1 to PX_LOG_FS_CFG_PAGE_SIZE

Definition at line 41 of file px_log_fs_glue_at25s.c.

◆ px_log_fs_glue_wr()

void px_log_fs_glue_wr ( const void *  buffer,
uint16_t  page,
uint16_t  start_byte_in_page,
uint16_t  nr_of_bytes 
)

Writes data to Serial Flash.

This glue function write data from Serial Flash and stores it in the specified buffer.

Parameters
bufferBuffer to store read data
pagePX_LOG_FS_CFG_PAGE_START to PX_LOG_FS_CFG_PAGE_END
start_byte_in_page0 to PX_LOG_FS_CFG_PAGE_SIZE - 1
nr_of_bytes1 to PX_LOG_FS_CFG_PAGE_SIZE

Definition at line 54 of file px_log_fs_glue_at25s.c.

◆ px_log_fs_glue_erase_block()

void px_log_fs_glue_erase_block ( uint16_t  page)

Erases data in Serial Flash.

This glue function erases a block of pages in Serial Flash.

Parameters
pagePX_LOG_FS_CFG_PAGE_START to PX_LOG_FS_CFG_PAGE_END

Definition at line 67 of file px_log_fs_glue_at25s.c.