px-fwlib 0.10.0
Cross-platform embedded library and documentation for 8/16/32-bit microcontrollers generated with Doxygen 1.9.2
px_log_fs.h : Record-based file system for Serial Flash

Description

Record-based file system for Serial Flash.

File(s):

1. Introduction

px_log_fs is a basic but resilient record-based file system to store sequential log data on Serial Flash, for example Adesto AT25S Serial Flash. It is designed with the following fundamental characteristics in mind:

File system features:

A file is defined as a collection of sequential records that contain fixed-size log data. The records can be navigated forwards or backwards to read data, while still being able to append new data. New records will overwrite oldest records when the file is full (PX_LOG_FS_CFG_STOP_WR_WHEN_FULL = 0) or record writing will stop when the file is full (PX_LOG_FS_CFG_STOP_WR_WHEN_FULL = 1).

The size of each record is configured with PX_LOG_FS_CFG_REC_DATA_SIZE. See 'px_log_fs_cfg_template.h'

The erase block size (number of pages) is configurable with PX_LOG_FS_CFG_ERASE_BLOCK_SIZE. For example a single page of the Adesto AT45DB041E can be erased at a time (very convenient), but the AT25SF041 has a minimum erase block size of 4 KB or 8 pages (a much cheaper option).

The file system must start at the beginning of an erase block and end at the end of an erase block. It must be at least two erase blocks in size, but more is better.

First a code example to demonstrate the API, followed by implementation details...

2. Code example

#include <stdio.h>
#include "px_board.h"
#include "px_spi.h"
#include "px_at25s.h"
#include "px_log_fs.h"
// Define record data content
typedef struct
{
int16_t temperature;
} px_log_fs_data_t;
static px_spi_handle_t px_at25s_spi_handle;
static px_log_fs_handle_t px_log_fs_handle;
static px_log_fs_data_t px_log_fs_data;
int main(void)
{
uint8_t i;
px_log_fs_err_t px_log_fs_err;
// Initialise modules
px_spi_open2(&px_at25s_spi_handle, PX_SPI_NR_1, PX_BOARD_SPI_CS_AT25S,
px_at25s_init(&px_at25s_spi_handle);
// Initialise file system
px_log_fs_init(&px_log_fs_handle, 0, PX_AT25S_PAGES - 1);
// Write 8 records
px_log_fs_data.temperature = 160; // 16.0 deg Celsius
for(i = 0; i < 8; i++)
{
// Write record
px_log_fs_wr(&px_log_fs_handle, &px_log_fs_data, sizeof(px_log_fs_data));
// Increment temperature by 1.3 deg
px_log_fs_data.temperature += 13;
}
// Read first record
px_log_fs_err = px_log_fs_rd_first(&px_log_fs_handle, &px_log_fs_data, sizeof(px_log_fs_data));
while(px_log_fs_err == PX_LOG_FS_ERR_NONE)
{
// Display record data
printf("%d\n", px_log_fs_data.temp);
// Read next record
px_log_fs_err = px_log_fs_rd_next(&px_log_fs_handle, &px_log_fs_data, sizeof(px_log_fs_data));
}
}
void px_board_init(void)
Initialise the board hardware.
Definition: px_board.c:168
void px_at25s_init(px_spi_handle_t *handle)
Initialise driver.
Definition: px_at25s.c:95
#define PX_AT25S_SPI_DATA_ORDER
SPI Data order.
Definition: px_at25s.h:129
#define PX_AT25S_SPI_MODE
SPI Clock / Data phase.
Definition: px_at25s.h:127
#define PX_AT25S_MAX_SPI_CLOCK_HZ
Maximum SPI Clock rate.
Definition: px_at25s.h:125
px_log_fs_err_t px_log_fs_init(px_log_fs_handle_t *handle, uint16_t fs_page_start, uint16_t fs_page_end)
Initialise log file system.
Definition: px_log_fs.c:452
px_log_fs_err_t px_log_fs_rd_next(px_log_fs_handle_t *handle, void *data, size_t nr_of_bytes)
Read next (newer) record.
Definition: px_log_fs.c:675
px_log_fs_err_t px_log_fs_rd_first(px_log_fs_handle_t *handle, void *data, size_t nr_of_bytes)
Read first (oldest) record.
Definition: px_log_fs.c:607
px_log_fs_err_t px_log_fs_wr(px_log_fs_handle_t *handle, const void *data, size_t nr_of_bytes)
Write a record to the file.
Definition: px_log_fs.c:897
px_log_fs_err_t
Error codes.
Definition: px_log_fs.h:236
@ PX_LOG_FS_ERR_NONE
No error.
Definition: px_log_fs.h:237
void px_spi_init(void)
Initialise SPI driver.
Definition: px_spi.c:162
bool px_spi_open2(px_spi_handle_t *handle, px_spi_nr_t spi_nr, uint8_t cs_id, px_spi_baud_t baud, px_spi_mode_t mode, px_spi_dord_t data_order, uint8_t mo_dummy_byte)
Open SPI peripheral using specified parameters.
Definition: px_spi.c:252
Define SPI handle.
Definition: px_spi.h:126

3. Implementation details

The Serial Flash pages are treated as one big circular buffer. Records are stored sequentially and record writing will stop (or wrap to overwrite oldest records) when the start of the oldest record is reached. This simplified scheme implicitly allows for wear leveling, because each Serial Flash page is erased once per pass through the circular buffer.

The file system leverages a characteristic of Serial Flash. When a block of pages are erased, all the data bits are reset to 1. When writing a byte, bits can be cleared (0), but not reset to 1 again. The following 8-bit marker values are used:

By inspecting the bits, it can be observed that a marker can be changed from FREE to PAGE to PAGE+A to BAD, but not the other way around without erasing the whole block.

Each written page starts with a header (px_log_fs_header_t) which consists of an 8-bit marker, a 16-bit incrementing number and an 8-bit CRC. Only the first 4 bytes of each page is read to quickly index the whole file system during px_log_fs_init(). The incrementing number is used to figure out which is the oldest page and which is the newest during px_log_fs_init().

Pages containing records are marked with PAGE (0x5F). Each record (px_log_fs_record_t) contains an 8-bit RECORD marker (0xAF), the record data (fixed size set with PX_LOG_FS_CFG_REC_DATA_SIZE) and an 8-bit CRC.

A record cannot span across pages, so there may be unused space at the end of each page. For example, if a page is 256 bytes and the page header takes up 4 bytes, then there are 252 bytes left over for records. If 8 data bytes are stored in each record, then the total record size is 10 bytes (including overhead). This means that 25 records can be stored in each page and 2 bytes will be unused (252 - 25*10 = 2).

4. How to find the FIRST and LAST page

To figure out which page contains the oldest records (FIRST page) and which page contains the newest records (LAST page), a rolling (or wrapping) number is assigned to each page. This means that each consecutive page is labelled with an incrementing number. If the number exceeds the maximum, it starts again at zero (rolls over).

To simplify these examples, the rolling number starts at 00, increments up to 99 and then rolls over to 00.

Each page's rolling number is compared with the next page's rolling number. The largest difference in the rolling numbers indicates that the LAST page has been found. The next valid page is per implication the FIRST.

To be exact, modulo unsigned integer arithmetic is used:

Diff = (rolling_number_next - rolling_number) mod 100

Here are a few select test cases to demonstrate that it works under various conditions:

5. Archiving records

Oldest records can be marked as archived (RECORD+A marker = 0xAA) for example when the record has been succesfully uploaded to a server. This makes it possible to find the next (unarchived) record that must be uploaded to the server after a reset or power interruption.

If all the records on a page is marked as archived then the page is marked as archived too (PAGE+A marker = 0x55). This makes it faster to find the first unarchived record, because each page marker is inspected until the first unarchived page (PAGE marker = 0x5F) is found. Then each record on that page is inspected until the first unarchived record is found (RECORD marker =0xAF).

Data Structures

struct  px_log_fs_adr_t
 Specification of data address in Serial Flash. More...
 
struct  px_log_fs_handle_t
 

Macros

#define PX_LOG_FS_CFG_VER   2
 Config version. More...
 
#define PX_LOG_FS_CFG_PAGE_SIZE   PX_AT25S_PAGE_SIZE
 Page size for file system. More...
 
#define PX_LOG_FS_CFG_ERASE_BLOCK_SIZE   PX_AT25S_PAGES_PER_BLOCK_4KB
 Erase block size (in pages) for file system. More...
 
#define PX_LOG_FS_CFG_REC_DATA_SIZE   13
 Record data size (total record size = PX_LOG_FS_CFG_REC_DATA_SIZE + 3 bytes overhead) More...
 
#define PX_LOG_FS_CFG_STOP_WR_WHEN_FULL   0
 Stop writing when full (1) or erase oldest records and continue writing (0) More...
 

Enumerations

enum  px_log_fs_err_t {
  PX_LOG_FS_ERR_NONE = 0 , PX_LOG_FS_ERR_EMPTY , PX_LOG_FS_ERR_FULL , PX_LOG_FS_ERR_WRITE_FAIL ,
  PX_LOG_FS_ERR_NO_RECORD , PX_LOG_FS_ERR_FATAL
}
 Error codes. More...
 

Functions

px_log_fs_err_t px_log_fs_init (px_log_fs_handle_t *handle, uint16_t fs_page_start, uint16_t fs_page_end)
 Initialise log file system. More...
 
px_log_fs_err_t px_log_fs_reset (px_log_fs_handle_t *handle, uint16_t fs_page_start, uint16_t fs_page_end)
 Reset log file system. More...
 
px_log_fs_err_t px_log_fs_rd_first (px_log_fs_handle_t *handle, void *data, size_t nr_of_bytes)
 Read first (oldest) record. More...
 
px_log_fs_err_t px_log_fs_rd_first_unarchived (px_log_fs_handle_t *handle, void *data, size_t nr_of_bytes)
 Read first (oldest) unarchived record. More...
 
px_log_fs_err_t px_log_fs_rd_next (px_log_fs_handle_t *handle, void *data, size_t nr_of_bytes)
 Read next (newer) record. More...
 
px_log_fs_err_t px_log_fs_rd_last (px_log_fs_handle_t *handle, void *data, size_t nr_of_bytes)
 Read last (newest) record. More...
 
px_log_fs_err_t px_log_fs_rd_previous (px_log_fs_handle_t *handle, void *data, size_t nr_of_bytes)
 Read previous (older) record. More...
 
bool px_log_fs_rd_rec_is_archived (px_log_fs_handle_t *handle)
 See if currently read record is archived or not. More...
 
px_log_fs_err_t px_log_fs_rd_rec_set_archive (px_log_fs_handle_t *handle)
 Set currently read record as archived. More...
 
px_log_fs_err_t px_log_fs_wr (px_log_fs_handle_t *handle, const void *data, size_t nr_of_bytes)
 Write a record to the file. More...
 
void px_log_fs_dbg_report_info (px_log_fs_handle_t *handle)
 Report log file system info. More...
 
void px_log_fs_glue_rd (void *buffer, uint16_t page, uint16_t start_byte_in_page, uint16_t nr_of_bytes)
 Read data from Serial Flash. More...
 
void px_log_fs_glue_wr (const void *buffer, uint16_t page, uint16_t start_byte_in_page, uint16_t nr_of_bytes)
 Writes data to Serial Flash. More...
 
void px_log_fs_glue_erase_block (uint16_t page)
 Erases data in Serial Flash. More...
 

Data Structure Documentation

◆ px_log_fs_adr_t

struct px_log_fs_adr_t

Specification of data address in Serial Flash.

Definition at line 246 of file px_log_fs.h.

Data Fields
uint16_t page Page.
uint16_t offset Offset inside page.

◆ px_log_fs_handle_t

struct px_log_fs_handle_t

Definition at line 252 of file px_log_fs.h.

Data Fields
uint16_t fs_page_start First page in file system (must be on the start of an erase block)
uint16_t fs_page_end Last page in file system (must be on the end of an erase block)
uint16_t page_first First page with records (PX_LOG_FS_PAGE_INVALID if empty)
uint16_t page_last Last page with records (PX_LOG_FS_PAGE_INVALID if empty)
uint16_t page_nr_next Next page number to use (starts at 0)
px_log_fs_adr_t adr_wr Next write address (open position)
px_log_fs_adr_t adr_rd Current read address.
bool archive_flag Flag is set if current read record is archived.

Macro Definition Documentation

◆ PX_LOG_FS_CFG_VER

#define PX_LOG_FS_CFG_VER   2

Config version.

Definition at line 35 of file px_log_fs_cfg_template.h.

◆ PX_LOG_FS_CFG_PAGE_SIZE

#define PX_LOG_FS_CFG_PAGE_SIZE   PX_AT25S_PAGE_SIZE

Page size for file system.

Definition at line 38 of file px_log_fs_cfg_template.h.

◆ PX_LOG_FS_CFG_ERASE_BLOCK_SIZE

#define PX_LOG_FS_CFG_ERASE_BLOCK_SIZE   PX_AT25S_PAGES_PER_BLOCK_4KB

Erase block size (in pages) for file system.

Definition at line 41 of file px_log_fs_cfg_template.h.

◆ PX_LOG_FS_CFG_REC_DATA_SIZE

#define PX_LOG_FS_CFG_REC_DATA_SIZE   13

Record data size (total record size = PX_LOG_FS_CFG_REC_DATA_SIZE + 3 bytes overhead)

Definition at line 44 of file px_log_fs_cfg_template.h.

◆ PX_LOG_FS_CFG_STOP_WR_WHEN_FULL

#define PX_LOG_FS_CFG_STOP_WR_WHEN_FULL   0

Stop writing when full (1) or erase oldest records and continue writing (0)

Definition at line 47 of file px_log_fs_cfg_template.h.

Enumeration Type Documentation

◆ px_log_fs_err_t

Error codes.

Enumerator
PX_LOG_FS_ERR_NONE 

No error.

PX_LOG_FS_ERR_EMPTY 

File is empty.

PX_LOG_FS_ERR_FULL 

File is full.

PX_LOG_FS_ERR_WRITE_FAIL 

Failed to write.

PX_LOG_FS_ERR_NO_RECORD 

No record found.

PX_LOG_FS_ERR_FATAL 

Fatal file system error.

Definition at line 235 of file px_log_fs.h.

Function Documentation

◆ px_log_fs_init()

px_log_fs_err_t px_log_fs_init ( px_log_fs_handle_t handle,
uint16_t  fs_page_start,
uint16_t  fs_page_end 
)

Initialise log file system.

Parameters
handlePointer to file system handle structure
fs_page_startFirst page in file system (must be on the start of an erase block)
fs_page_endLast page in file system (must be on the end of an erase block)
Return values
PX_LOG_FS_ERR_NONEFile system was succesfully indexed

Definition at line 452 of file px_log_fs.c.

◆ px_log_fs_reset()

px_log_fs_err_t px_log_fs_reset ( px_log_fs_handle_t handle,
uint16_t  fs_page_start,
uint16_t  fs_page_end 
)

Reset log file system.

Parameters
handlePointer to file system handle structure
fs_page_startFirst page in file system (must be on the start of an erase block)
fs_page_endLast page in file system (must be on the end of an erase block)
Return values
PX_LOG_FS_ERR_NONEFile system was succesfully resetted

Definition at line 550 of file px_log_fs.c.

◆ px_log_fs_rd_first()

px_log_fs_err_t px_log_fs_rd_first ( px_log_fs_handle_t handle,
void *  data,
size_t  nr_of_bytes 
)

Read first (oldest) record.

Parameters
handlePointer to file system handle structure
dataPointer to buffer where record data must be copied to
nr_of_bytesNumber of bytes to copy from record
Return values
PX_LOG_FS_ERR_NONEValid record data found and copied into structure
PX_LOG_FS_ERR_NO_RECORDNo record found

Definition at line 607 of file px_log_fs.c.

◆ px_log_fs_rd_first_unarchived()

px_log_fs_err_t px_log_fs_rd_first_unarchived ( px_log_fs_handle_t handle,
void *  data,
size_t  nr_of_bytes 
)

Read first (oldest) unarchived record.

Parameters
handlePointer to file system handle structure
dataPointer to buffer where record data must be copied to
nr_of_bytesNumber of bytes to copy from record
Return values
PX_LOG_FS_ERR_NONEValid record data found and copied into structure
PX_LOG_FS_ERR_NO_RECORDNo record found

Definition at line 624 of file px_log_fs.c.

◆ px_log_fs_rd_next()

px_log_fs_err_t px_log_fs_rd_next ( px_log_fs_handle_t handle,
void *  data,
size_t  nr_of_bytes 
)

Read next (newer) record.

Parameters
handlePointer to file system handle structure
dataPointer to buffer where record data must be copied to
nr_of_bytesNumber of bytes to copy from record
Return values
PX_LOG_FS_ERR_NONEValid record data found and copied into structure
PX_LOG_FS_ERR_NO_RECORDNo record found

Definition at line 675 of file px_log_fs.c.

◆ px_log_fs_rd_last()

px_log_fs_err_t px_log_fs_rd_last ( px_log_fs_handle_t handle,
void *  data,
size_t  nr_of_bytes 
)

Read last (newest) record.

Parameters
handlePointer to file system handle structure
dataPointer to buffer where record data must be copied to
nr_of_bytesNumber of bytes to copy from record
Return values
PX_LOG_FS_ERR_NONEValid record data found and copied into structure
PX_LOG_FS_ERR_NO_RECORDNo record found

Definition at line 739 of file px_log_fs.c.

◆ px_log_fs_rd_previous()

px_log_fs_err_t px_log_fs_rd_previous ( px_log_fs_handle_t handle,
void *  data,
size_t  nr_of_bytes 
)

Read previous (older) record.

Parameters
handlePointer to file system handle structure
dataPointer to buffer where record data must be copied to
nr_of_bytesNumber of bytes to copy from record
Return values
PX_LOG_FS_ERR_NONEValid record data found and copied into structure
PX_LOG_FS_ERR_NO_RECORDNo record found

Definition at line 755 of file px_log_fs.c.

◆ px_log_fs_rd_rec_is_archived()

bool px_log_fs_rd_rec_is_archived ( px_log_fs_handle_t handle)
inline

See if currently read record is archived or not.

NB! Record MUST have been succesfully read first.

Parameters
handlePointer to file system handle structure
Return values
truerecord is marked as archived
falserecord is NOT marked as archived

Definition at line 382 of file px_log_fs.h.

◆ px_log_fs_rd_rec_set_archive()

px_log_fs_err_t px_log_fs_rd_rec_set_archive ( px_log_fs_handle_t handle)

Set currently read record as archived.

NB! Record MUST have been succesfully read first.

As a side effect, if this is the last record in the page, then the page marker is set as archived for quick indexing.

It is important that the records are marked as archived starting from the oldest to the newest in sequence otherwise the algorithm that mark a whole page as archived will not work correctly.

Parameters
handlePointer to file system handle structure
Return values
PX_LOG_FS_ERR_NONESuccess. Record is set as archived.
PX_LOG_FS_ERR_WRITE_FAILFailed to set record as archived.

Definition at line 818 of file px_log_fs.c.

◆ px_log_fs_wr()

px_log_fs_err_t px_log_fs_wr ( px_log_fs_handle_t handle,
const void *  data,
size_t  nr_of_bytes 
)

Write a record to the file.

A new record is appended to the end of the list of records. The maximum number of bytes that can be stored in the record is specified by PX_LOG_FS_CFG_DATA_SIZE.

Parameters
handlePointer to file system handle structure
dataPointer to buffer containing data that must be stored in the record
nr_of_bytesNumber of bytes that must be written
Return values
PX_LOG_FS_ERR_NONESuccess
PX_LOG_FS_ERR_FULLFile (or file system) is full
PX_LOG_FS_ERR_WRITE_FAILFailed to write record (verify failed)

Definition at line 897 of file px_log_fs.c.

◆ px_log_fs_dbg_report_info()

void px_log_fs_dbg_report_info ( px_log_fs_handle_t handle)

Report log file system info.

Parameters
handlePointer to file system handle structure

Definition at line 1000 of file px_log_fs.c.

◆ px_log_fs_glue_rd()

void px_log_fs_glue_rd ( void *  buffer,
uint16_t  page,
uint16_t  start_byte_in_page,
uint16_t  nr_of_bytes 
)

Read data from Serial Flash.

This glue function reads data from Serial Flash and stores it in the specified buffer.

Parameters
bufferBuffer to store read data
pagePX_LOG_FS_CFG_PAGE_START to PX_LOG_FS_CFG_PAGE_END
start_byte_in_page0 to PX_LOG_FS_CFG_PAGE_SIZE - 1
nr_of_bytes1 to PX_LOG_FS_CFG_PAGE_SIZE

Definition at line 41 of file px_log_fs_glue_at25s.c.

◆ px_log_fs_glue_wr()

void px_log_fs_glue_wr ( const void *  buffer,
uint16_t  page,
uint16_t  start_byte_in_page,
uint16_t  nr_of_bytes 
)

Writes data to Serial Flash.

This glue function write data from Serial Flash and stores it in the specified buffer.

Parameters
bufferBuffer to store read data
pagePX_LOG_FS_CFG_PAGE_START to PX_LOG_FS_CFG_PAGE_END
start_byte_in_page0 to PX_LOG_FS_CFG_PAGE_SIZE - 1
nr_of_bytes1 to PX_LOG_FS_CFG_PAGE_SIZE

Definition at line 54 of file px_log_fs_glue_at25s.c.

◆ px_log_fs_glue_erase_block()

void px_log_fs_glue_erase_block ( uint16_t  page)

Erases data in Serial Flash.

This glue function erases a block of pages in Serial Flash.

Parameters
pagePX_LOG_FS_CFG_PAGE_START to PX_LOG_FS_CFG_PAGE_END

Definition at line 67 of file px_log_fs_glue_at25s.c.