Introduce compression, using a modified version of Stan's code from the
driver, plus my new implementation of LZBITMAP. This is mainly needed
for sealed volumes, since for some reason Apple calculates hashes on the
decompressed contents of files instead of the xattrs themselves. At this
point, the hash checks are all passing. Very few checks on the
compression metadata were put in place; I would like to revisit this in
the future, at least to check the inode's reported decompressed size,
but it's not a priority right now.
These are a lot of messy changes and I was forced to interrupt the work
halfway through to reverse engineer LZBITMAP, so I don't recall all the
details and I'll keep the commit message brief.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Sealed volumes store hashes to the contents of each file in info records
kept in the catalog, and store hashes of each catalog node in the index
record values. They also keep file extents in a separate "fext" tree,
for reasons I ignore. Add checks for all this stuff.
One problem remaining is that, for compressed files, the contents hashed
are the actual file contents, and not the compressed data in the xattr
that matches the info oid. To verify these hashes I will first have to
support compression inside the fsck. I'll try to do that next.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
One of the apfs images I use for testing has a single allocated block
that I have struggled to account for. Now I finally understand where it
comes from: it's the block for a volume keybag, and those are listed
encrypted inside the contaner's keybag. So the only way to finally get
compare_container_bitmaps() to pass is to actually decrypt and parse the
keybag.
The official reference claims that RFC 3394 is in use here, but that's
not true: it actually uses AES under the XTS mode of operation. I got
that info from the apfs-fuse <https://github.com/sgan81/apfs-fuse>
source code. This patch includes an implementation of AES which I ripped
from wpa_supplicant, so it's under the (3-clause) BSD license. I've also
included their implementation of RFC3394, even though it's not currently
needed.
As for the XTS implementation, I intended to write an original one in
case I ever need to avoid the GPL license from apfs-fuse. This did not
work out, because I had that version too fresh in my mind, so I ended up
mostly doing a C port of that code.
Anyway, once the container keybag is decrypted, run superficial checks
on the few tags I've encountered so far and mark the volume keybag
blocks as used. Do not actually parse and check the volume keybag for
now.
Strangely enough, I've encountered at least one container keybag that
doesn't appear to be encrypted at all. Put provisions in place for that
case as well.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
The formula for ip_fq_node_limit() currently considers containers with
fewer than 376 chunks as a special case, even though they follow the
same linear function as the others. The intention was to avoid dealing
with negative numbers, since all the variables are unsigned and rounding
behaves differently. Instead, rearrange the linear formula to avoid
these problems.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
The Apple implementation refuses to mount any container that doesn't
have a specific tree node limit set in the free queues. It took me a
while, but I think I figured out the formulas they use. Implement them
in a new common library file, and use it both for the fsck to check the
value and for the mkfs to set it correctly in new filesystems.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Clean up our hash table implementation. Get rid of that nonsense with
unions and just cast the headers to the whole entry when needed. Embed
the actual header structure in the entries instead of reproducing it,
and use a macro to allow access to the object ids in the old fashion.
Get rid of the redundant TABLE_BUCKETS macros, since all hash tables use
the same value.
I haven't given any thought to the potential strict aliasing issues
here, so just add the -fno-strict-aliasing flag to all Makefiles.
Signed-off-by: Ernesto A. Fernández <ernesto.mnd.fernandez@gmail.com>
There is no real point in declaring the names of files in their headers,
so get rid of them. This is just something I copied from ext2, but the
kernel developers do not seem to like it these days.
Signed-off-by: Ernesto A. Fernández <ernesto.mnd.fernandez@gmail.com>
Instead of linking each library file by itself, assemble an archive with
all the common library code. Use this to simplify the fsck Makefile,
which no longer needs to relink every single time.
Signed-off-by: Ernesto A. Fernández <ernesto.mnd.fernandez@gmail.com>
Move the Fletcher-64 implementation into the crc32.c library file, now
renamed checksum.c, with the intention of calling it from the mkfs as
well. Update authorship information accordingly.
Signed-off-by: Ernesto A. Fernández <ernesto.mnd.fernandez@gmail.com>
Move the crc32c code to the lib directory, and its headers to include.
The intention is to be able to link this file to the mkfs as well.
Tweak the fsck Makefile one more time, to prevent the recursive call to
make from happening once for each library file. From now on, always
relink apfsck, even if the libraries weren't updated. This is not ideal
but it's good enough for now.
Signed-off-by: Ernesto A. Fernández <ernesto.mnd.fernandez@gmail.com>
Create a new 'lib' directory outside of 'apfsck', and move all the
unicode routines in there. Also move their headers to include/apfs.
The intention is to be able to link this file to the mkfs as well.
Write a Makefile for the new library directory, and call it from the
original Makefile for the fsck.
Signed-off-by: Ernesto A. Fernández <ernesto.mnd.fernandez@gmail.com>
Ernesto A. Fernández
Nick Cao
Ernesto A. Fernández