This release gets apfs-label and mkapfs to run on systems with 16K
pages. Similar changes are also needed in the fsck, but that can wait
because it's only a testing tool for my own use.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I was hoping that this commit would include all layout checks for fusion
drives, but I couldn't quite figure out tenant records, or why there are
multiple different cache mechanisms. I'm having a hard time getting back
into this research, and I'm not even sure if there's any point to it at
all, so I'll just leave things as they are for now.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I've never made any releases for apfsprogs before because it was mostly
intended for my own use, to help me test the driver. But at this point a
few of the tools might actually be useful to others, and even the mkfs
is probably mature enough to be safe, so I guess I should start. I'm
using 0.2.0 because all of the man pages have been set to 0.1 until now.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
When I mistakenly run the fsck on a whole disk, instead of running it on
the apfs partition, I get a confusing "block size is too small" error
message. To avoid this noise, run no checks at all until we have
confirmed that we are working with an apfs filesystem. If not, explain
this clearly to the user.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Back when I was only working with relatively small disks I was under the
impression that ip_bm_free_next was just a fixed-length 256-bit field.
In reality it's an array of 16-bit fields, with 16 times as many entries
as the number of bitmap blocks needed to map the internal pool. Fix the
boundary checks in spaceman_256_from_off() and rename it accordingly.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
While implementing the correct ip bitmap list in the driver, I realized
that expecting all entries between two consecutive free bitmaps to be
used made everything stupidly complicated for no reason. So I just
implemented the list in the way that made sense: allocating new bitmaps
in the head and freeing the old ones in the tail, without worrying about
the direction of the jump for each link. The resulting images were
accepted by the official implementations, so I guess that was indeed the
right way.
My fsck does complain of course, so get rid of this silly check.
Instead, just count the free blocks so that we can later confirm that
every used block is marked as invalid.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I just noticed that the fsck reports corruption whenever my mkfs creates
a filesystem that doesn't use the whole disk. I don't test this much
because I don't need it right now, and I don't even know if the official
implementations accept it. But it's important to have it working for the
ENOSPC tests of xfstests, which will be important once I actually start
checking for ENOSPC before starting transactions.
Anyway, fix this to the best of my understanding right now, though I may
need to make more changes in the future once I finally get around to
testing these devices on macos.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
The used_bitmaps array was allocated with one slot for each used ip
bitmap block, but it should also have a slot for the free tail. This
isn´t serious because I don't expect anyone to use this fsck other than
me (lots of similar issues probably exist), but it's causing the mkfs
tests to fail, so fix it.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I never fully understood how the ip bitmap list was supposed to work,
but now I'm noticing that the difference between what I do and what the
official driver does are bad enough to quickly corrupt the list when a
container moves back and forth. Of course this is only an issue for
filesystems above 1.32 TiB or so, but it's still pretty serious.
So, I thought about it harder and I'm pretty sure I finally figured it
out: the array entries for each free bitmap are actually kept in a
linked list, that starts in sm_ip_bm_free_head and ends in
sm_ip_bm_free_tail. All entries outside the list (that is, the entries
for the used blocks) are marked with the invalid index 0xFFFF. The tail
itself also gets 0xFFFF even though it's free, because it has no next
entry - so that accounts for the constant off-by-one weirdness in the
number of free blocks.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
As I explained before in commit 19b8d6d8 ("apfsck: verify order of
checkpoint mappings"), ephemeral objects (or at least the spaceman) can
wrap around. Add support for reading such objects to the fsck.
This is of course not possible using only mmap as we do now, so
ephemeral objects will need to be copied to allocated memory, and some
hacky changes are needed to free such memory correctly. Also a few
checks inside free_cpoint_map() have become redundant, so make sure they
happen in full inside parse_cpoint_map() and parse_cpoint_map_blocks()
instead.
For the record, this was all tested against an image created by my
driver after applying patch dfdafd2a07cf ("Wrap ephemeral objects around
if there is no room") and rigging it so that transactions would fail
after a wrapped-around object was discovered.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I have recently noticed that Apple refuses to acknowledge drives that
were formatted by my mkfs, unless my driver wrote to them at least once.
The official fsck does not report any corruption, but after some
experiments it seems that the problem is that checkpoint mappings are
supposed to be in order, and not leave any holes.
For containers with a multiblock spaceman, the object itself is supposed
to wrap around, again leaving no holes. When the official driver mounts
a 7T container that wrapped around precisely when my driver unmounted
it, the last transaction gets silently discarded. This should be
extremely rare, but I need to fix it.
Start by putting the proper checks in place for the mappings.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Multiblock ephemeral objects have been supported for a while, but I just
noticed a line that claims otherwise. Get rid of it.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Add checks for all the fusion-related structures and fields that show up
in an empty fusion drive. This requires a few changes to the way blocks
are read, so that the fsck knows which drive holds the data. Nothing
major though.
With all this in place, the allocation bitmaps add up, which is always
a good sign. I was hoping to split this into multiple patches, but I got
lazy.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
As a first step towards supporting fusion drives, add a command line
option to specify the tier 2 device. We now have two global fd's in use,
so give them names that are less confusing. As a simple sanity check,
fix the block count check, which is the first to fail if the fsck gets
run against the main device alone.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I have a check in place to make sure that snapshot xids come strictly
before the current xid, but if a crash happens right after the snapshot
commit they could actually be equal, and that isn't corruption. Allow
it.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I've just encountered a sealed volume with snapshots, probably for the
first time. I have a check in place to confirm that all snapshots xids
are older than the snapshot, but I had already identified an exception
in the extent reference tree. It seems that more exceptions exist for
sealed volumes. I haven't looked into it in depth, I don't really care
that much because I don't intend to implement sealing. Just ignore this
problem for sealed volumes.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I've just run into an image that uses a new 0x10 xattr flag. I have not
tried to figure it out yet, but document it so that I don't forget.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
File extent records have two undocumented flags. One is the crypto flag,
which we don't support yet. As for the other one, I've only seen it once
so far, on an extent that was entirely allocated beyond the end of a
file. I'm guessing that's what it means, but it only seems to be valid
when the volume has the 0x80 incompatible feature. This is all hard to
confirm because I don't know how to preallocate like that in macos, and
the official fsck and driver won't complain at all if you flip this
extent bit.
Put checks in place to see if we trip on something in the future.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Add to raw.h a few recently noticed flags for the volume superblock. I
don't know much about them yet, but put the usual basic checks in place
in the fsck.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
So far I have mostly ignored this "cloneinfo id epoch" in the fsck, but
I notice now some issues with one of my older images. I guess it was
created before the bugfix, so some inodes have the WAS_CLONED flag, but
not WAS_EVER_CLONED. Since this is technically allowed stop reporting it
as corruption, but leave a warning message in place anyway.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Containers with smaller than 128M use a different value for the minimum
size of the data area. Change this.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
My driver now splits orphan deletion among multiple transactions, but it
doesn't bother to update the inode record for half-deleted files. This
is reasonable because the orphans are not real files, and we don't want
to waste time on them. My fsck is still checking one field though: the
sparse byte count. Stop that.
By the way, the official driver behaves like us in all cases I've
encountered so far, but I have never seen what it does to sparse files.
I have confirmed, however, that the official fsck has no problem with
this, so it's probably the same.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Ernesto A. Fernández
Integral