In NVS, allocation table entries (ATEs) are written backwards within
each sector. Under delete-only or delete-heavy workloads, a sector may
contain only delete ATEs, causing the ATE write pointer to approach the
sector boundary.
Without an explicit boundary check, ATE writes may occur at offset 0 of
the current sector, allowing the write pointer to underflow into the
previous sector and corrupt unrelated data or metadata.
Fix this by disallowing ATE writes when the write pointer is at the
sector boundary. This ensures that ATE writes remain confined to the
current sector and prevents pointer underflow across sectors.
Signed-off-by: Lingao Meng <menglingao@xiaomi.com>
Replace some manually-defined DT_COMPAT_<> Kconfig macro variables with
their automatically generated counterparts. In most cases, this is
straightforward as the manually defined macro is named identically to the
one generated by the build system.
Signed-off-by: Mathieu Choplain <mathieu.choplain-ext@st.com>
The initial implementation had a couple of flaws, including:
* Generally overcounting the free space by ATE_SIZE bytes per sector.
* Counting multiple ATEs of the same ID if placed in the same sector.
The correct behavior is to only count the most recent entry.
* When a sector was mostly or completely filled with small data sizes,
there was a correction term applied in the wrong location.
* The same correction term would be mistakenly applied when a sector
was filled with only delete ATEs, which resulted in undercounting
the free space in that sector.
This is addressed by rewriting the API according to a simple principle:
the total free space in a filesystem should equal the sum of free space
in every GC'd sector (minus 1 sector reserved for GC itself).
This should work because during garbage collection, ATEs are not moved
between sectors arbitrarily. Only one sector gets GC'd at a time and
every entry in sector N gets either removed or copied to sector N-1.
This means that any two valid entries that occupy a single sector would
still occupy a (different) single sector before and after the operation.
This property lets us simplify the total free space calculation by
considering one sector at a time and reapplying the calculation which
was just introduced for `zms_active_sector_free_space()`.
Signed-off-by: Grzegorz Swiderski <grzegorz.swiderski@nordicsemi.no>
The initial implementation had a flaw in it: when an active sector was
almost full, the naïve calculation could easily underflow and return a
negative value, which would be misinterpreted as an errno.
Rewrite this API to satisfy the following, reasonable expectations:
* Always return a non-negative value, provided ZMS is initialized.
* Always return 0 if no more data can be written into the sector
without triggering garbage collection.
* Never return a value less than the actual number of bytes that can
still be written into the sector.
This requires awareness of a few special cases in ZMS. These are now
captured in a helper function, which will be reused later.
Signed-off-by: Grzegorz Swiderski <grzegorz.swiderski@nordicsemi.no>
Remove duplicated #include directives within the same
preprocessor scope across the Zephyr tree.
Duplicates inside different #ifdef branches are preserved
as they may be intentional.
Signed-off-by: Sylvio Alves <sylvio.alves@espressif.com>
Avoid polluting every build that includes this Kconfig with EXT2
logging config unless the ext2 filesystem is actually enabled.
Signed-off-by: Luca Burelli <l.burelli@arduino.cc>
For devices that need an erase before a write, ZMS do not verify that
the next available location in the open sector is filled with the
erase_value.
Fix this by adding a check at init.
Signed-off-by: Riadh Ghaddab <rghaddab@baylibre.com>
Lfs provide lfs_fs_gc function for some time now.
Function allow offloading of expensive block allocation scan.
Introduce lfs_fs_gc via api call fc_gc.
Signed-off-by: Maciej Zagrabski <mzi@trackunit.com>
Adds cast to int when ret is assigned error code returned by
functions that return int64_t. The cast has been added to indicate
that assignment with truncation is here intentional.
Signed-off-by: Dominik Ermel <dominik.ermel@nordicsemi.no>
REQUEST_QUEUE was passed as queue_num to virtio_init_virtqueues
where REQUEST_QUEUE+1 was required
Signed-off-by: Jakub Michalski <jmichalski@antmicro.com>
Allow the ZMS API to optionally accept 64 bit IDs. A typedef `zms_id_t`
is added, so that the maximum ID width can be controlled using Kconfig.
The current ATE structure is already large enough that it is possible to
reserve 64 bits for IDs without increasing its total size (128 bits).
This makes the feature a natural, low footprint alternative to Settings,
for cases where the supported key namespace must be larger than 32 bit
but not arbitrarily large.
The ATE format does have to be altered to accommodate larger IDs, but
the default "32 bit" format is left as is. Now, the `struct zms_ate`
describes one of two supported formats, selected by an `#if` condition.
In the future, it may be possible to support multiple ATE formats at
runtime, in which case the structure can be turned into a union.
In the new, "64 bit" ATEs, the `offset` and `metadata` fields are moved
into a union, because they are found to be mutually exclusive. With the
old format, the same fields are in different locations, but one of them
always gets filled with a dummy value, depending on the given ATE type.
To cover both cases, a `memset` is used, which should be optimized away
by the compiler when appropriate.
The only limitation is that the new ATE format has no room for data CRC,
but an alternative integrity check can be implemented by the caller.
Signed-off-by: Grzegorz Swiderski <grzegorz.swiderski@nordicsemi.no>
Support using FUSE v3, instead of FUSE v2.
FUSE v3 has been out for a couple of years now, so most distributions
have it.
But note the FUSE library (and its v3 branch) is currently unmaintained,
and older distributions do not ship it.
Some Linux distros have transitioned their packages away from fuse2.
Which results in less atention in these distros to fuse2, and therefore
a higher likelyhood that there would be distribution issues with the
corresponding packages.
There is also some likelihood the fuse2 packages may be eventually
droped by some of these.
So let's support either version, with a kconfig adapting to either API
and their quirks.
Note that both the fuse2 and fuse3 library code have quite a few
sideeffects on the process that uses it.
By now we continue defaulting to fuse2 as it is the most common.
But users can select to use the v3 if they have an issue w v2 or want
to start trying out using v3.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Avoid a possible race between the FUSE thread and the Zephyr threads
during init.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
FUSE_INCLUDE_DIRS should be passed to the native_simulator build.
If there is several fuse libraries installed, we need to ensure the
bottom side of the driver is built with the correct include paths.
(These includes are irrelevant for the Zephyr side build)
Support having a list of include paths instead of single one.
Support having a list of libraries to link to instead of a single one.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Fix deleting a key that doesn't exist from writing a new entry to the
filesystem when `CONFIG_ZMS_NO_DOUBLE_WRITE` and
`CONFIG_ZMS_LOOKUP_CACHE` are enabled.
Signed-off-by: Jordan Yates <jordan@embeint.com>
List the file size helps in developer debugging experience. Provide
a config to disable new behavior in case any users depended on
command output to be a list.
Signed-off-by: Utsav Munendra <utsavm@meta.com>
Add checks on the fs pointer passed through the api before using to
avoid causing an exception
Signed-off-by: Theis Mejnertsen <theismejnertsen@gmail.com>
zms_active_sector_free_space is documented to return -EACCES on fs not
mounted error but currently returns an usigned type size_t. This fixes
that by changing the return value into a ssize_t
Signed-off-by: Theis Mejnertsen <theismejnertsen@gmail.com>
Allows NVS to work with flash device configured to
use only 64KB block erase. Due to how addresses
are encoded internally in NVS, 64KB is the maximum
sector size. Add a test for this during mount.
Add a native_sim unit test case for 64kb erase block size
Signed-off-by: Mike J. Chen <mjchen@google.com>
When fs_stat() queries the root / mountpoint it should return its root
i_node but instead it tries to return the parent i_node which does not
exist. Fix this by checking if parent is set otherwise return the root
i_node.
Fixes https://github.com/zephyrproject-rtos/zephyr/issues/94000.
Signed-off-by: Bas van Loon <bas@arch-embedded.com>
Device tree and/or automount support was not present and might be useful
when for example the settings module needs early access to a filesystem.
This patch adds support to mount an EXT2 filesystem automatically. The
implementation chosen is to provide the drive name matching a disk
drive specified in the device tree ie on an (e)MMC or SD card.
Signed-off-by: Bas van Loon <bas@arch-embedded.com>
ZMS does not work without this and gives compilation errors, fix
this by adding the correct dependency
Signed-off-by: Jamie McCrae <spam@helper3000.net>
Fix FS shell using fixed 'storage_partition' nodelabel instead of
accessing the DT defined partition from the zephyr,fstab,littlefs
node partition property.
Signed-off-by: Joakim Andersson <joerchan@gmail.com>
This commit adds virtiofs functions implementing fuse operations required
to enable zephyr filesystem support
Signed-off-by: Jakub Michalski <jmichalski@antmicro.com>
It is not possible to build anymore in that mode, so we do not
need to support it in this driver cmake files.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Instead of relaying on stddef.h being included by other headers
let's include it explicitly, following a report of it being
missing for Ubuntu 22.04 (glibc 2.35)
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
When nvs_write is called, the nvs_flash_block_cmp is used to check if
the new data to be written matches the data already on flash. This check
always fail when CONFIG_NVS_DATA_CRC is enabled, caused by the
NVS_DATA_CRC_SIZE being added to the len parameter. The pointer to the
new data does not already have the CRC part added, while the data on
flash does, and the size to be compared includes CRC section.
By removing the addition of NVS_DATA_CRC_SIZE to the compare size, only
the data without CRC is compared, which will make the compare work in
both cases.
Signed-off-by: Yonas Alizadeh <yonas.alizadeh@alfalaval.com>
Settings subsystem is storing the name ID and the linked list node ID
with only one bit difference at BIT(0).
Settings subsystem is also storing the name ID and the data ID in two
different ZMS entries at an exact offset of ZMS_DATA_ID_OFFSET.
Using the regular lookup function could result in many cache misses.
Therefore, to assure the least number of collisions in the lookup cache,
the BIT(0) of the hash indicates whether the given ZMS ID represents a
linked list entry or not, the BIT(1) indicates whether the ZMS ID is a
name or data and the remaining bits of the hash are set to a truncated
part of the original hash generated by Settings.
Signed-off-by: Riadh Ghaddab <rghaddab@baylibre.com>
When the CONFIG_ZMS_NO_DOUBLE_WRITE is not enabled there is no need to
search in the cache for matching ID
Signed-off-by: Riadh Ghaddab <rghaddab@baylibre.com>
When power cuts during a GC operation, the sector is erased again in the
next reboot cycle and the cycle_cnt of the empty ATE is incremented.
If the same power cut happens 255 times in a row, the empty ATE cycle_cnt
will become equal to the close ATE which causes a memory corruption.
Fix this by checking the close ATE cycle_cnt before incrementing the
empty ATE cycle_cnt.
Fixes: #84874
Signed-off-by: Riadh Ghaddab <rghaddab@baylibre.com>
Previously, when detecting the GC_done ATE the gc_done_marker boolean
variable was not set.
Fix this by setting it to true if the GC_done ATE is found.
Signed-off-by: Riadh Ghaddab <rghaddab@baylibre.com>
In order to use fstab with fatfs in addition to
littlefs a new driver fstab,fatfs was introduced
containing the required logic to detect and if
needed automount fatfs devices specified in fstab.
Additionally as the partition phandle is not needed
by fatfs and currently is specific to littlefs it
was moved from the common dts binding into the
littlefs binding.
Signed-off-by: Carlo Kirchmeier <carlo.kirchmeier@zuehlke.com>
Co-authored-by: Carlo Kirchmeier <carlo.kirchmeier@zuehlke.com>
Co-authored-by: Dominik Ermel <dominik.ermel@nordicsemi.no>
Split the fuse FS driver into 2 parts: A top built in the embedded side,
with the embedded libC, and a bottom built in the runner side with the
host libC.
The error returns are converted to match the host libC.
Also, before the host FUSE thread, which is asynchronous to Zephyr was
calling directly into the Zephyr filesystem code, which resulted quite
often if catastrophic failures or corruption of the Zephyr state.
This is now fixed by having the FUSE thread queue requests to a Zephyr
thread, which will be handled in the embedded side in a coherent way.
This adds a slightly noticeable overhead, but the performance is still
acceptable.
Signed-off-by: Alberto Escolar Piedras <alberto.escolar.piedras@nordicsemi.no>
Add missing Copyright from derived files and fix the Copyright year for
some files to keep the original Copyright notice
Signed-off-by: Riadh Ghaddab <rghaddab@baylibre.com>
`FLASH_PAGE_LAYOUT` has a hardware dependency on `FLASH_HAS_PAGE_LAYOUT`
which is not present for all boards. Forcing this symbol to `y` when
the hardware doesn't support it results in build errors at the Kconfig
stage.
`FLASH_PAGE_LAYOUT` is enabled by default when `FLASH_HAS_PAGE_LAYOUT`
is true, so this change will not require any user changes.
Signed-off-by: Jordan Yates <jordan@embeint.com>
This resolves some addressed comments in this PR zephyrproject-rtos#77930
as well as this PR zephyrproject-rtos#80407
Signed-off-by: Riadh Ghaddab <rghaddab@baylibre.com>
