This release gets apfs-label and mkapfs to run on systems with 16K
pages. Similar changes are also needed in the fsck, but that can wait
because it's only a testing tool for my own use.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Following what was done to apfs-label, get rid of mmap in the mkfs to
avoid alignment issues.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
There are obvious memory leaks in read_latest_super(), plug them. Part
of the problem also exists in the driver, so I'll need to fix that next.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Apparently apfs-label does not work for 16K page sizes:
https://github.com/linux-apfs/apfsprogs/issues/21
The problem is that the offsets need to be page-aligned. Just get rid of
mmap entirely, for a tool like this there is no performance cost in just
reading everything into a buffer.
The same problem exists in the mkfs and fsck though. I will have to deal
with it next.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I was hoping that this commit would include all layout checks for fusion
drives, but I couldn't quite figure out tenant records, or why there are
multiple different cache mechanisms. I'm having a hard time getting back
into this research, and I'm not even sure if there's any point to it at
all, so I'll just leave things as they are for now.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I've never made any releases for apfsprogs before because it was mostly
intended for my own use, to help me test the driver. But at this point a
few of the tools might actually be useful to others, and even the mkfs
is probably mature enough to be safe, so I guess I should start. I'm
using 0.2.0 because all of the man pages have been set to 0.1 until now.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
When I mistakenly run the fsck on a whole disk, instead of running it on
the apfs partition, I get a confusing "block size is too small" error
message. To avoid this noise, run no checks at all until we have
confirmed that we are working with an apfs filesystem. If not, explain
this clearly to the user.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
The mkapfs has a fatal() function, but I keep forgetting about it and
calling fprintf() and exit() directly. Tidy this up.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Back when I was only working with relatively small disks I was under the
impression that ip_bm_free_next was just a fixed-length 256-bit field.
In reality it's an array of 16-bit fields, with 16 times as many entries
as the number of bitmap blocks needed to map the internal pool. Fix the
boundary checks in spaceman_256_from_off() and rename it accordingly.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
The -v flag to print the version is mentioned when usage() gets called
for mkapfs and apfsck, but not for apfs-label and apfs-snap. Be
consistent about this.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I now understand how free_next actually works, so fix an outdated
comment from the mkfs that seems to imply that the tail block is in use.
Also use the macro for the invalid index instead of the magical number
0xFFFF.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Instead of forcing the tier 2 device to fill the whole disk, allow the
user to specify a block count, same as the main device. This may be
useful for ENOSPC tests at some point.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
While implementing the correct ip bitmap list in the driver, I realized
that expecting all entries between two consecutive free bitmaps to be
used made everything stupidly complicated for no reason. So I just
implemented the list in the way that made sense: allocating new bitmaps
in the head and freeing the old ones in the tail, without worrying about
the direction of the jump for each link. The resulting images were
accepted by the official implementations, so I guess that was indeed the
right way.
My fsck does complain of course, so get rid of this silly check.
Instead, just count the free blocks so that we can later confirm that
every used block is marked as invalid.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I just noticed that the fsck reports corruption whenever my mkfs creates
a filesystem that doesn't use the whole disk. I don't test this much
because I don't need it right now, and I don't even know if the official
implementations accept it. But it's important to have it working for the
ENOSPC tests of xfstests, which will be important once I actually start
checking for ENOSPC before starting transactions.
Anyway, fix this to the best of my understanding right now, though I may
need to make more changes in the future once I finally get around to
testing these devices on macos.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
The used_bitmaps array was allocated with one slot for each used ip
bitmap block, but it should also have a slot for the free tail. This
isn´t serious because I don't expect anyone to use this fsck other than
me (lots of similar issues probably exist), but it's causing the mkfs
tests to fail, so fix it.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I never fully understood how the ip bitmap list was supposed to work,
but now I'm noticing that the difference between what I do and what the
official driver does are bad enough to quickly corrupt the list when a
container moves back and forth. Of course this is only an issue for
filesystems above 1.32 TiB or so, but it's still pretty serious.
So, I thought about it harder and I'm pretty sure I finally figured it
out: the array entries for each free bitmap are actually kept in a
linked list, that starts in sm_ip_bm_free_head and ends in
sm_ip_bm_free_tail. All entries outside the list (that is, the entries
for the used blocks) are marked with the invalid index 0xFFFF. The tail
itself also gets 0xFFFF even though it's free, because it has no next
entry - so that accounts for the constant off-by-one weirdness in the
number of free blocks.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
As I explained before in commit 19b8d6d8 ("apfsck: verify order of
checkpoint mappings"), ephemeral objects (or at least the spaceman) can
wrap around. Add support for reading such objects to the fsck.
This is of course not possible using only mmap as we do now, so
ephemeral objects will need to be copied to allocated memory, and some
hacky changes are needed to free such memory correctly. Also a few
checks inside free_cpoint_map() have become redundant, so make sure they
happen in full inside parse_cpoint_map() and parse_cpoint_map_blocks()
instead.
For the record, this was all tested against an image created by my
driver after applying patch dfdafd2a07cf ("Wrap ephemeral objects around
if there is no room") and rigging it so that transactions would fail
after a wrapped-around object was discovered.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
The spaceman includes the arrays of cib addresses inside its own block
or blocks, and for that purpose the spaceman struct reports the offset
of each array (one for the main device and one for tier 2). I always
assumed that the offset had to actually be within the boundaries of the
spaceman object, so even regular non-fusion drives would have to store a
single "null" entry in the tier 2 array. Now that I've actually
implemented fusion drives I accidentally got rid of that extra entry and
it turns out that the official implementation doesn't mind.
That said, my mkfs tests claim to work with images of the exact sizes
where the spaceman becomes multiblock, but this is no longer correct
because the spaceman has become slightly smaller. Adjust those two image
sizes by a single cib.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
After commit f523ce46 ("mkapfs: run tests on fusion drives as well"), I
noticed that a few fusion drive tests are crashing with sigbus, but I
got sidetracked by another issue and I only got to it now. The problem
is, of course, that with a large tier 2 disk the metadata may not fit in
the main device. This is not a bug, but it's a bit ugly to fail with
sigbus, so put some coarse space checks in place and fail cleanly when
the size difference is too crazy.
Also update the fusion drive tests so that they start checking for clean
failure in the odd cases.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
As explained in the previous patch, ephemeral objects must be
consecutive and in the same order as their mappings. Since the spaceman
has a variable size and some objects are not needed for all kinds of
drives, stop hardcoding the block numbers and find them on runtime
instead.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I have recently noticed that Apple refuses to acknowledge drives that
were formatted by my mkfs, unless my driver wrote to them at least once.
The official fsck does not report any corruption, but after some
experiments it seems that the problem is that checkpoint mappings are
supposed to be in order, and not leave any holes.
For containers with a multiblock spaceman, the object itself is supposed
to wrap around, again leaving no holes. When the official driver mounts
a 7T container that wrapped around precisely when my driver unmounted
it, the last transaction gets silently discarded. This should be
extremely rare, but I need to fix it.
Start by putting the proper checks in place for the mappings.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Ernesto A. Fernández
Integral