Description

Record-based file system for Serial Flash.

File(s):

1. Introduction

px_log_fs is a basic but resilient record-based file system to store sequential log data on Serial Flash, for example Adesto AT25S Serial Flash. It is designed with the following fundamental characteristics in mind:

The Serial Flash is evenly divided up into pages
A block of pages is erased at a time (less is better)
An erase will reset all bits in a block back to ones (1)
A write selectively set bits in a page to zero (0)
A bit that is zero can only be restored to one with a block erase

File system features:

Basic wear levelling by spreading block erases and page writes sequentially over the whole Serial Flash
Serial Flash bit fault tolerance without complete file corruption (only one record or extreme worst case one page will be lost)
Power disruption or unplanned reset tolerance (only one record may be lost)
Oldest records can be marked as archived.

A file is defined as a collection of sequential records that contain fixed-size log data. The records can be navigated forwards or backwards to read data, while still being able to append new data. New records will overwrite oldest records when the file is full (PX_LOG_FS_CFG_STOP_WR_WHEN_FULL = 0) or record writing will stop when the file is full (PX_LOG_FS_CFG_STOP_WR_WHEN_FULL = 1).

The size of each record is configured with PX_LOG_FS_CFG_REC_DATA_SIZE. See 'px_log_fs_cfg_template.h'

The erase block size (number of pages) is configurable with PX_LOG_FS_CFG_ERASE_BLOCK_SIZE. For example a single page of the Adesto AT45DB041E can be erased at a time (very convenient), but the AT25SF041 has a minimum erase block size of 4 KB or 8 pages (a much cheaper option).

The file system must start at the beginning of an erase block and end at the end of an erase block. It must be at least two erase blocks in size, but more is better.

First a code example to demonstrate the API, followed by implementation details...

2. Code example

#include <stdio.h>
#include "px_board.h"
#include "px_spi.h"
#include "px_at25s.h"
#include "px_log_fs.h"
 
// Define record data content
typedef struct
{
    int16_t temperature;
} px_log_fs_data_t;
 
static px_spi_handle_t    px_at25s_spi_handle;
static px_log_fs_handle_t px_log_fs_handle;
static px_log_fs_data_t   px_log_fs_data;
 
int main(void)
{
    uint8_t         i;
    px_log_fs_err_t px_log_fs_err;
 
    // Initialise modules
    px_board_init();
    px_spi_init();
    px_spi_open2(&px_at25s_spi_handle, PX_SPI_NR_1, PX_BOARD_SPI_CS_AT25S,
                 PX_AT25S_MAX_SPI_CLOCK_HZ, PX_AT25S_SPI_MODE, PX_AT25S_SPI_DATA_ORDER, 0x00);
    px_at25s_init(&px_at25s_spi_handle);
 
    // Initialise file system
    px_log_fs_init(&px_log_fs_handle, 0, PX_AT25S_PAGES - 1);
 
    // Write 8 records
    px_log_fs_data.temperature = 160;   // 16.0 deg Celsius
    for(i = 0; i < 8; i++)
    {
        // Write record
        px_log_fs_wr(&px_log_fs_handle, &px_log_fs_data, sizeof(px_log_fs_data));
        // Increment temperature by 1.3 deg
        px_log_fs_data.temperature += 13;
    }
 
    // Read first record
    px_log_fs_err = px_log_fs_rd_first(&px_log_fs_handle, &px_log_fs_data, sizeof(px_log_fs_data));
    while(px_log_fs_err == PX_LOG_FS_ERR_NONE)
    {
        // Display record data
        printf("%d\n", px_log_fs_data.temp);
        // Read next record
        px_log_fs_err = px_log_fs_rd_next(&px_log_fs_handle, &px_log_fs_data, sizeof(px_log_fs_data));
    }
}

3. Implementation details

The Serial Flash pages are treated as one big circular buffer. Records are stored sequentially and record writing will stop (or wrap to overwrite oldest records) when the start of the oldest record is reached. This simplified scheme implicitly allows for wear leveling, because each Serial Flash page is erased once per pass through the circular buffer.

The file system leverages a characteristic of Serial Flash. When a block of pages are erased, all the data bits are reset to 1. When writing a byte, bits can be cleared (0), but not reset to 1 again. The following 8-bit marker values are used:

By inspecting the bits, it can be observed that a marker can be changed from FREE to PAGE to PAGE+A to BAD, but not the other way around without erasing the whole block.

Each written page starts with a header (px_log_fs_header_t) which consists of an 8-bit marker, a 16-bit incrementing number and an 8-bit CRC. Only the first 4 bytes of each page is read to quickly index the whole file system during px_log_fs_init(). The incrementing number is used to figure out which is the oldest page and which is the newest during px_log_fs_init().

Pages containing records are marked with PAGE (0x5F). Each record (px_log_fs_record_t) contains an 8-bit RECORD marker (0xAF), the record data (fixed size set with PX_LOG_FS_CFG_REC_DATA_SIZE) and an 8-bit CRC.

A record cannot span across pages, so there may be unused space at the end of each page. For example, if a page is 256 bytes and the page header takes up 4 bytes, then there are 252 bytes left over for records. If 8 data bytes are stored in each record, then the total record size is 10 bytes (including overhead). This means that 25 records can be stored in each page and 2 bytes will be unused (252 - 25*10 = 2).

4. How to find the FIRST and LAST page

To figure out which page contains the oldest records (FIRST page) and which page contains the newest records (LAST page), a rolling (or wrapping) number is assigned to each page. This means that each consecutive page is labelled with an incrementing number. If the number exceeds the maximum, it starts again at zero (rolls over).

To simplify these examples, the rolling number starts at 00, increments up to 99 and then rolls over to 00.

Each page's rolling number is compared with the next page's rolling number. The largest difference in the rolling numbers indicates that the LAST page has been found. The next valid page is per implication the FIRST.

To be exact, modulo unsigned integer arithmetic is used:

Diff = (rolling_number_next - rolling_number) mod 100

Here are a few select test cases to demonstrate that it works under various conditions:

5. Archiving records

Oldest records can be marked as archived (RECORD+A marker = 0xAA) for example when the record has been succesfully uploaded to a server. This makes it possible to find the next (unarchived) record that must be uploaded to the server after a reset or power interruption.

If all the records on a page is marked as archived then the page is marked as archived too (PAGE+A marker = 0x55). This makes it faster to find the first unarchived record, because each page marker is inspected until the first unarchived page (PAGE marker = 0x5F) is found. Then each record on that page is inspected until the first unarchived record is found (RECORD marker =0xAF).

Data Structures
struct	px_log_fs_adr_t
	Specification of data address in Serial Flash. More...

struct	px_log_fs_handle_t

Macros
#define	PX_LOG_FS_CFG_VER 2
	Config version. More...

#define	PX_LOG_FS_CFG_PAGE_SIZE PX_AT25S_PAGE_SIZE
	Page size for file system. More...

#define	PX_LOG_FS_CFG_ERASE_BLOCK_SIZE PX_AT25S_PAGES_PER_BLOCK_4KB
	Erase block size (in pages) for file system. More...

#define	PX_LOG_FS_CFG_REC_DATA_SIZE 13
	Record data size (total record size = PX_LOG_FS_CFG_REC_DATA_SIZE + 3 bytes overhead) More...

#define	PX_LOG_FS_CFG_STOP_WR_WHEN_FULL 0
	Stop writing when full (1) or erase oldest records and continue writing (0) More...

Enumerations
enum	px_log_fs_err_t { PX_LOG_FS_ERR_NONE = 0 , PX_LOG_FS_ERR_EMPTY , PX_LOG_FS_ERR_FULL , PX_LOG_FS_ERR_WRITE_FAIL , PX_LOG_FS_ERR_NO_RECORD , PX_LOG_FS_ERR_FATAL }
	Error codes. More...

Functions
px_log_fs_err_t	px_log_fs_init (px_log_fs_handle_t *handle, uint16_t fs_page_start, uint16_t fs_page_end)
	Initialise log file system. More...

px_log_fs_err_t	px_log_fs_reset (px_log_fs_handle_t *handle, uint16_t fs_page_start, uint16_t fs_page_end)
	Reset log file system. More...

px_log_fs_err_t	px_log_fs_rd_first (px_log_fs_handle_t handle, void data, size_t nr_of_bytes)
	Read first (oldest) record. More...

px_log_fs_err_t	px_log_fs_rd_first_unarchived (px_log_fs_handle_t handle, void data, size_t nr_of_bytes)
	Read first (oldest) unarchived record. More...

px_log_fs_err_t	px_log_fs_rd_next (px_log_fs_handle_t handle, void data, size_t nr_of_bytes)
	Read next (newer) record. More...

px_log_fs_err_t	px_log_fs_rd_last (px_log_fs_handle_t handle, void data, size_t nr_of_bytes)
	Read last (newest) record. More...

px_log_fs_err_t	px_log_fs_rd_previous (px_log_fs_handle_t handle, void data, size_t nr_of_bytes)
	Read previous (older) record. More...

bool	px_log_fs_rd_rec_is_archived (px_log_fs_handle_t *handle)
	See if currently read record is archived or not. More...

px_log_fs_err_t	px_log_fs_rd_rec_set_archive (px_log_fs_handle_t *handle)
	Set currently read record as archived. More...

px_log_fs_err_t	px_log_fs_wr (px_log_fs_handle_t handle, const void data, size_t nr_of_bytes)
	Write a record to the file. More...

void	px_log_fs_dbg_report_info (px_log_fs_handle_t *handle)
	Report log file system info. More...

void	px_log_fs_glue_rd (void *buffer, uint16_t page, uint16_t start_byte_in_page, uint16_t nr_of_bytes)
	Read data from Serial Flash. More...

void	px_log_fs_glue_wr (const void *buffer, uint16_t page, uint16_t start_byte_in_page, uint16_t nr_of_bytes)
	Writes data to Serial Flash. More...

void	px_log_fs_glue_erase_block (uint16_t page)
	Erases data in Serial Flash. More...

Data Structure Documentation

◆ px_log_fs_adr_t

struct px_log_fs_adr_t

Specification of data address in Serial Flash.

Definition at line 246 of file px_log_fs.h.

Data Fields
uint16_t	page	Page.
uint16_t	offset	Offset inside page.

◆ px_log_fs_handle_t

struct px_log_fs_handle_t

Definition at line 252 of file px_log_fs.h.

Data Fields
uint16_t	fs_page_start	First page in file system (must be on the start of an erase block)
uint16_t	fs_page_end	Last page in file system (must be on the end of an erase block)
uint16_t	page_first	First page with records (PX_LOG_FS_PAGE_INVALID if empty)
uint16_t	page_last	Last page with records (PX_LOG_FS_PAGE_INVALID if empty)
uint16_t	page_nr_next	Next page number to use (starts at 0)
px_log_fs_adr_t	adr_wr	Next write address (open position)
px_log_fs_adr_t	adr_rd	Current read address.
bool	archive_flag	Flag is set if current read record is archived.

Macro Definition Documentation

◆ PX_LOG_FS_CFG_VER

#define PX_LOG_FS_CFG_VER 2

Config version.

Definition at line 35 of file px_log_fs_cfg_template.h.

◆ PX_LOG_FS_CFG_PAGE_SIZE

#define PX_LOG_FS_CFG_PAGE_SIZE PX_AT25S_PAGE_SIZE

Page size for file system.

Definition at line 38 of file px_log_fs_cfg_template.h.

◆ PX_LOG_FS_CFG_ERASE_BLOCK_SIZE

#define PX_LOG_FS_CFG_ERASE_BLOCK_SIZE PX_AT25S_PAGES_PER_BLOCK_4KB

Erase block size (in pages) for file system.

Definition at line 41 of file px_log_fs_cfg_template.h.

◆ PX_LOG_FS_CFG_REC_DATA_SIZE

#define PX_LOG_FS_CFG_REC_DATA_SIZE 13

Record data size (total record size = PX_LOG_FS_CFG_REC_DATA_SIZE + 3 bytes overhead)

Definition at line 44 of file px_log_fs_cfg_template.h.

◆ PX_LOG_FS_CFG_STOP_WR_WHEN_FULL

#define PX_LOG_FS_CFG_STOP_WR_WHEN_FULL 0

Stop writing when full (1) or erase oldest records and continue writing (0)

Definition at line 47 of file px_log_fs_cfg_template.h.

Enumeration Type Documentation

◆ px_log_fs_err_t

enum px_log_fs_err_t

Error codes.

Enumerator
PX_LOG_FS_ERR_NONE	No error.
PX_LOG_FS_ERR_EMPTY	File is empty.
PX_LOG_FS_ERR_FULL	File is full.
PX_LOG_FS_ERR_WRITE_FAIL	Failed to write.
PX_LOG_FS_ERR_NO_RECORD	No record found.
PX_LOG_FS_ERR_FATAL	Fatal file system error.

Definition at line 235 of file px_log_fs.h.

Function Documentation

◆ px_log_fs_init()

px_log_fs_err_t px_log_fs_init	(	px_log_fs_handle_t *	handle,
		uint16_t	fs_page_start,
		uint16_t	fs_page_end
	)

Initialise log file system.

Parameters

handle	Pointer to file system handle structure
fs_page_start	First page in file system (must be on the start of an erase block)
fs_page_end	Last page in file system (must be on the end of an erase block)

Return values

PX_LOG_FS_ERR_NONE File system was succesfully indexed

Definition at line 452 of file px_log_fs.c.

◆ px_log_fs_reset()

px_log_fs_err_t px_log_fs_reset	(	px_log_fs_handle_t *	handle,
		uint16_t	fs_page_start,
		uint16_t	fs_page_end
	)

Reset log file system.

Parameters

handle	Pointer to file system handle structure
fs_page_start	First page in file system (must be on the start of an erase block)
fs_page_end	Last page in file system (must be on the end of an erase block)

Return values

PX_LOG_FS_ERR_NONE File system was succesfully resetted

Definition at line 550 of file px_log_fs.c.

◆ px_log_fs_rd_first()

px_log_fs_err_t px_log_fs_rd_first	(	px_log_fs_handle_t *	handle,
		void *	data,
		size_t	nr_of_bytes
	)

Read first (oldest) record.

Parameters

handle	Pointer to file system handle structure
data	Pointer to buffer where record data must be copied to
nr_of_bytes	Number of bytes to copy from record

Return values

PX_LOG_FS_ERR_NONE	Valid record data found and copied into structure
PX_LOG_FS_ERR_NO_RECORD	No record found

Definition at line 607 of file px_log_fs.c.

◆ px_log_fs_rd_first_unarchived()

px_log_fs_err_t px_log_fs_rd_first_unarchived	(	px_log_fs_handle_t *	handle,
		void *	data,
		size_t	nr_of_bytes
	)

Read first (oldest) unarchived record.

Parameters

handle	Pointer to file system handle structure
data	Pointer to buffer where record data must be copied to
nr_of_bytes	Number of bytes to copy from record

Return values

PX_LOG_FS_ERR_NONE	Valid record data found and copied into structure
PX_LOG_FS_ERR_NO_RECORD	No record found

Definition at line 624 of file px_log_fs.c.

◆ px_log_fs_rd_next()

px_log_fs_err_t px_log_fs_rd_next	(	px_log_fs_handle_t *	handle,
		void *	data,
		size_t	nr_of_bytes
	)

◆ px_log_fs_rd_last()

px_log_fs_err_t px_log_fs_rd_last	(	px_log_fs_handle_t *	handle,
		void *	data,
		size_t	nr_of_bytes
	)

Read last (newest) record.

Parameters

handle	Pointer to file system handle structure
data	Pointer to buffer where record data must be copied to
nr_of_bytes	Number of bytes to copy from record

Return values

PX_LOG_FS_ERR_NONE	Valid record data found and copied into structure
PX_LOG_FS_ERR_NO_RECORD	No record found

Definition at line 739 of file px_log_fs.c.

◆ px_log_fs_rd_previous()

px_log_fs_err_t px_log_fs_rd_previous	(	px_log_fs_handle_t *	handle,
		void *	data,
		size_t	nr_of_bytes
	)

Read previous (older) record.

Parameters

handle	Pointer to file system handle structure
data	Pointer to buffer where record data must be copied to
nr_of_bytes	Number of bytes to copy from record

Return values

PX_LOG_FS_ERR_NONE	Valid record data found and copied into structure
PX_LOG_FS_ERR_NO_RECORD	No record found

Definition at line 755 of file px_log_fs.c.

◆ px_log_fs_rd_rec_is_archived()

bool px_log_fs_rd_rec_is_archived ( px_log_fs_handle_t * handle )

inline

See if currently read record is archived or not.

NB! Record MUST have been succesfully read first.

Parameters

handle Pointer to file system handle structure

Return values

true	record is marked as archived
false	record is NOT marked as archived

Definition at line 382 of file px_log_fs.h.

◆ px_log_fs_rd_rec_set_archive()

px_log_fs_err_t px_log_fs_rd_rec_set_archive ( px_log_fs_handle_t * handle )

Set currently read record as archived.

NB! Record MUST have been succesfully read first.

As a side effect, if this is the last record in the page, then the page marker is set as archived for quick indexing.

It is important that the records are marked as archived starting from the oldest to the newest in sequence otherwise the algorithm that mark a whole page as archived will not work correctly.

Parameters

handle Pointer to file system handle structure

Return values

PX_LOG_FS_ERR_NONE	Success. Record is set as archived.
PX_LOG_FS_ERR_WRITE_FAIL	Failed to set record as archived.

Definition at line 818 of file px_log_fs.c.

◆ px_log_fs_wr()

px_log_fs_err_t px_log_fs_wr	(	px_log_fs_handle_t *	handle,
		const void *	data,
		size_t	nr_of_bytes
	)

Write a record to the file.

A new record is appended to the end of the list of records. The maximum number of bytes that can be stored in the record is specified by PX_LOG_FS_CFG_DATA_SIZE.

Parameters

handle	Pointer to file system handle structure
data	Pointer to buffer containing data that must be stored in the record
nr_of_bytes	Number of bytes that must be written

Return values

PX_LOG_FS_ERR_NONE	Success
PX_LOG_FS_ERR_FULL	File (or file system) is full
PX_LOG_FS_ERR_WRITE_FAIL	Failed to write record (verify failed)

Definition at line 897 of file px_log_fs.c.

◆ px_log_fs_dbg_report_info()

void px_log_fs_dbg_report_info ( px_log_fs_handle_t * handle )

Report log file system info.

Parameters

handle Pointer to file system handle structure

Definition at line 1000 of file px_log_fs.c.

◆ px_log_fs_glue_rd()

void px_log_fs_glue_rd	(	void *	buffer,
		uint16_t	page,
		uint16_t	start_byte_in_page,
		uint16_t	nr_of_bytes
	)

Read data from Serial Flash.

This glue function reads data from Serial Flash and stores it in the specified buffer.

Parameters

buffer	Buffer to store read data
page	PX_LOG_FS_CFG_PAGE_START to PX_LOG_FS_CFG_PAGE_END
start_byte_in_page	0 to PX_LOG_FS_CFG_PAGE_SIZE - 1
nr_of_bytes	1 to PX_LOG_FS_CFG_PAGE_SIZE

Definition at line 41 of file px_log_fs_glue_at25s.c.

◆ px_log_fs_glue_wr()

void px_log_fs_glue_wr	(	const void *	buffer,
		uint16_t	page,
		uint16_t	start_byte_in_page,
		uint16_t	nr_of_bytes
	)

Writes data to Serial Flash.

This glue function write data from Serial Flash and stores it in the specified buffer.

Parameters

buffer	Buffer to store read data
page	PX_LOG_FS_CFG_PAGE_START to PX_LOG_FS_CFG_PAGE_END
start_byte_in_page	0 to PX_LOG_FS_CFG_PAGE_SIZE - 1
nr_of_bytes	1 to PX_LOG_FS_CFG_PAGE_SIZE

Definition at line 54 of file px_log_fs_glue_at25s.c.

◆ px_log_fs_glue_erase_block()

void px_log_fs_glue_erase_block ( uint16_t page )

Erases data in Serial Flash.

This glue function erases a block of pages in Serial Flash.

Parameters

page	PX_LOG_FS_CFG_PAGE_START to PX_LOG_FS_CFG_PAGE_END

Definition at line 67 of file px_log_fs_glue_at25s.c.

Description

1. Introduction

2. Code example

3. Implementation details

4. How to find the FIRST and LAST page

5. Archiving records

Data Structures

Macros

Enumerations

Functions

Data Structure Documentation

◆ px_log_fs_adr_t

◆ px_log_fs_handle_t

Macro Definition Documentation

◆ PX_LOG_FS_CFG_VER

◆ PX_LOG_FS_CFG_PAGE_SIZE

◆ PX_LOG_FS_CFG_ERASE_BLOCK_SIZE

◆ PX_LOG_FS_CFG_REC_DATA_SIZE

◆ PX_LOG_FS_CFG_STOP_WR_WHEN_FULL

Enumeration Type Documentation

◆ px_log_fs_err_t

Function Documentation

◆ px_log_fs_init()

◆ px_log_fs_reset()

◆ px_log_fs_rd_first()

◆ px_log_fs_rd_first_unarchived()

◆ px_log_fs_rd_next()

◆ px_log_fs_rd_last()

◆ px_log_fs_rd_previous()

◆ px_log_fs_rd_rec_is_archived()

◆ px_log_fs_rd_rec_set_archive()

◆ px_log_fs_wr()

◆ px_log_fs_dbg_report_info()

◆ px_log_fs_glue_rd()

◆ px_log_fs_glue_wr()

◆ px_log_fs_glue_erase_block()