I was hoping that this commit would include all layout checks for fusion
drives, but I couldn't quite figure out tenant records, or why there are
multiple different cache mechanisms. I'm having a hard time getting back
into this research, and I'm not even sure if there's any point to it at
all, so I'll just leave things as they are for now.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Add checks for all the fusion-related structures and fields that show up
in an empty fusion drive. This requires a few changes to the way blocks
are read, so that the fsck knows which drive holds the data. Nothing
major though.
With all this in place, the allocation bitmaps add up, which is always
a good sign. I was hoping to split this into multiple patches, but I got
lazy.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I've just run into an image that uses a new 0x10 xattr flag. I have not
tried to figure it out yet, but document it so that I don't forget.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
File extent records have two undocumented flags. One is the crypto flag,
which we don't support yet. As for the other one, I've only seen it once
so far, on an extent that was entirely allocated beyond the end of a
file. I'm guessing that's what it means, but it only seems to be valid
when the volume has the 0x80 incompatible feature. This is all hard to
confirm because I don't know how to preallocate like that in macos, and
the official fsck and driver won't complain at all if you flip this
extent bit.
Put checks in place to see if we trip on something in the future.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Add to raw.h a few recently noticed flags for the volume superblock. I
don't know much about them yet, but put the usual basic checks in place
in the fsck.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Like we did in the driver, add the latest fields discovered for the
volume superblock to the raw.h header.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Now that I've actually implemented snapshots in the driver, I can see
that the way extent reference counting is handled in the fsck makes no
sense at all. Update extents mean that the number of references can be
different for each block of a physical extent. So now, the extents
listed in the table will get split when it becomes necessary, and each
fraction will have its own refcnt.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
It seems that I forgot to actually implement be32_to_cpu() when I added
the compression code. Fix it.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Introduce compression, using a modified version of Stan's code from the
driver, plus my new implementation of LZBITMAP. This is mainly needed
for sealed volumes, since for some reason Apple calculates hashes on the
decompressed contents of files instead of the xattrs themselves. At this
point, the hash checks are all passing. Very few checks on the
compression metadata were put in place; I would like to revisit this in
the future, at least to check the inode's reported decompressed size,
but it's not a priority right now.
These are a lot of messy changes and I was forced to interrupt the work
halfway through to reverse engineer LZBITMAP, so I don't recall all the
details and I'll keep the commit message brief.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Some recently defined object types are missing the "APFS_" prefix. I
noticed this while working on the previous patch, which makes use of a
few of them. Fix it now.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Sealed volumes store hashes to the contents of each file in info records
kept in the catalog, and store hashes of each catalog node in the index
record values. They also keep file extents in a separate "fext" tree,
for reasons I ignore. Add checks for all this stuff.
One problem remaining is that, for compressed files, the contents hashed
are the actual file contents, and not the compressed data in the xattr
that matches the info oid. To verify these hashes I will first have to
support compression inside the fsck. I'll try to do that next.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
One of the apfs images I use for testing has a single allocated block
that I have struggled to account for. Now I finally understand where it
comes from: it's the block for a volume keybag, and those are listed
encrypted inside the contaner's keybag. So the only way to finally get
compare_container_bitmaps() to pass is to actually decrypt and parse the
keybag.
The official reference claims that RFC 3394 is in use here, but that's
not true: it actually uses AES under the XTS mode of operation. I got
that info from the apfs-fuse <https://github.com/sgan81/apfs-fuse>
source code. This patch includes an implementation of AES which I ripped
from wpa_supplicant, so it's under the (3-clause) BSD license. I've also
included their implementation of RFC3394, even though it's not currently
needed.
As for the XTS implementation, I intended to write an original one in
case I ever need to avoid the GPL license from apfs-fuse. This did not
work out, because I had that version too fresh in my mind, so I ended up
mostly doing a C port of that code.
Anyway, once the container keybag is decrypted, run superficial checks
on the few tags I've encountered so far and mark the volume keybag
blocks as used. Do not actually parse and check the volume keybag for
now.
Strangely enough, I've encountered at least one container keybag that
doesn't appear to be encrypted at all. Put provisions in place for that
case as well.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
One of the first images I encountered had a number of inodes with the
"finder info" flag set, but none of them had a finder info xfield, like
the official reference requires. What I just noticed is that they do
have a finder info xattr, so I'm guessing older implementations used
that instead. Allow it.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
There appears to be an undocumented flag that gets set in inodes with a
purgeable flags xfield. Allow this flag, and add a check for it.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I have encountered a second dentry flag that purgeable files may have. I
don't understand what makes it different from the original, but then
again I don't understand purgeable files, so it's not surprising. I do
notice that the reported filetype value changes depending on which flag
is in use, so add checks for that.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I've encountered a directory that has the INO_EXT_TYPE_PURGEABLE_FLAGS
xfield. The existence of this xfield type is documented, but the value
is claimed to be reserved, and no details are given. The reference seems
to imply that only purgeable files should use this, but the directory I
saw was not purgeable itself (though it did include mostly purgeable
files).
Anyway, for now just add superficial size checks for this xfield type,
and also for INO_EXT_TYPE_ORIG_SYNC_ROOT_ID. Both had been missing from
the original reference and were added later on.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I just noticed that INODE_FLAG_UNUSED is not actually listed as one of
the VALID_INTERNAL_INODE_FLAGS in the offcial documentation. So this
flag is banned, but I have no actual checks in place for that. Fix it.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
It seems that I missed a change in the layout of the container
superblock, with two new fields added to the end of it. One of those
fields, the wrapped media key, refers to an allocated block, so take it
into consideration for the spaceman checks.
This is finally enough to get those checks to pass for at least one of
my testing images.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Implement some basic checks for crypto state records, which were the
only remaining unsupported record kind. For now we focus only on file
encryption and ignore metadata encryption, which I'm yet to encounter.
While testing this patch I found that some cloned dstreams don't have a
crypto id of their own: they set it to -1 and apparently use the one for
the original file. I'm not sure how any of this works yet.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
A purgeable file has a dentry under an imaginary purgeable directory.
This dentry has two strange features: an undocumented flag which I have
nicknamed APFS_DREC_PURGEABLE, and a type of '4' (directory, I guess?)
which is not shared with its inode or other hardlinks. Add checks for
these features.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
To finally account for all blocks allocated by volumes with snapshots,
parse the extended snapshot metadata object. I don't fully understand
how it works either, but it seems to only be set for the most recent
snapshot and for the current volume.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
While trying to track down the source of two extra blocks in a volume's
apfs_fs_alloc_count, I noticed that there is an extra snapshot tree
related to the object map. I'm not sure why it's there, but it seems
simple enough so add checks for it.
This only accounts for one of the two extra blocks though, so I'll have
to keep looking.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Snapshots are not perfectly documented, so I'll start adding checks in
small chunks to see what I find. In this first patch, go through each
snapshot tree and run some simple checks on each record. Also count the
number of snapshots and check the one reported by the volume superblock.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Update records for physical extents are not supposed to have a valid
owner. I intend to fully support records of this kind soon, but for now
at least avoid immediate failure when they are encountered.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I don't know much about the reaper yet, but some images with an empty
reaper still have an ephemeral object for the reap list. If this is
ignored, its ephemeral mapping will be reported as dangling. So read the
block and run some superficial checks.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
