README.md
1libfiemap
2=============
3
4`libfiemap` is a library for creating block-devices that are backed by
5storage in read-write partitions. It exists primary for gsid. Generally, the
6library works by using `libfiemap_writer` to allocate large files within
7filesystem, and then tracks their extents.
8
9There are three main uses for `libfiemap`:
10 - Creating images that will act as block devices. For example, gsid needs to
11 create a `system_gsi` image to store Dynamic System Updates.
12 - Mapping the image as a block device while /data is mounted. This is fairly
13 tricky and is described in more detail below.
14 - Mapping the image as a block device during first-stage init. This is simple
15 because it uses the same logic from dynamic partitions.
16
17Image creation is done through `SplitFiemap`. Depending on the file system,
18a large image may have to be split into multiple files. On Ext4 the limit is
1916GiB and on FAT32 it's 4GiB. Images are saved into `/data/gsi/<name>/`
20where `<name>` is chosen by the process requesting the image.
21
22At the same time, a file called `/metadata/gsi/<name>/lp_metadata` is created.
23This is a super partition header that allows first-stage init to create dynamic
24partitions from the image files. It also tracks the canonical size of the image,
25since the file size may be larger due to alignment.
26
27Mapping
28-------
29
30It is easy to make block devices out of blocks on `/data` when it is not
31mounted, so first-stage init has no issues mapping dynamic partitions from
32images. After `/data` is mounted however, there are two problems:
33 - `/data` is encrypted.
34 - `/dev/block/by-name/data` may be marked as in-use.
35
36We break the problem down into three scenarios.
37
38### FDE and Metadata Encrypted Devices
39
40When FDE or metadata encryption is used, `/data` is not mounted from
41`/dev/block/by-name/data`. Instead, it is mounted from an intermediate
42`dm-crypt` or `dm-default-key` device. This means the underlying device is
43not marked in use, and we can create new dm-linear devices on top of it.
44
45On these devices, a block device for an image will consist of a single
46device-mapper device with a `dm-linear` table entry for each extent in the
47backing file.
48
49### Unencrypted and FBE-encrypted Devices
50
51When a device is unencrypted, or is encrypted with FBE but not metadata
52encryption, we instead use a loop device with `LOOP_SET_DIRECT_IO` enabled.
53Since `/data/gsi` has encryption disabled, this means the raw blocks will be
54unencrypted as well.
55
56### Split Images
57
58If an image was too large to store a single file on the underlying filesystem,
59on an FBE/unencrypted device we will have multiple loop devices. In this case,
60we create a device-mapper device as well. For each loop device it will have one
61`dm-linear` table entry spanning the length of the device.
62
63State Tracking
64--------------
65
66It's important that we know whether or not an image is currently in-use by a
67block device. It could be catastrophic to write to a dm-linear device if the
68underlying blocks are no longer owned by the original file. Thus, when mapping
69an image, we create a property called `gsid.mapped_image.<name>` and set it to
70the path of the block device.
71
72Additionally, we create a `/metadata/gsi/<subdir>/<name>.status` file. Each
73line in this file denotes a dependency on either a device-mapper node or a loop
74device. When deleting a block device, this file is used to release all
75resources.
76