This release gets apfs-label and mkapfs to run on systems with 16K
pages. Similar changes are also needed in the fsck, but that can wait
because it's only a testing tool for my own use.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Following what was done to apfs-label, get rid of mmap in the mkfs to
avoid alignment issues.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
There are obvious memory leaks in read_latest_super(), plug them. Part
of the problem also exists in the driver, so I'll need to fix that next.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Apparently apfs-label does not work for 16K page sizes:
https://github.com/linux-apfs/apfsprogs/issues/21
The problem is that the offsets need to be page-aligned. Just get rid of
mmap entirely, for a tool like this there is no performance cost in just
reading everything into a buffer.
The same problem exists in the mkfs and fsck though. I will have to deal
with it next.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I was hoping that this commit would include all layout checks for fusion
drives, but I couldn't quite figure out tenant records, or why there are
multiple different cache mechanisms. I'm having a hard time getting back
into this research, and I'm not even sure if there's any point to it at
all, so I'll just leave things as they are for now.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I've never made any releases for apfsprogs before because it was mostly
intended for my own use, to help me test the driver. But at this point a
few of the tools might actually be useful to others, and even the mkfs
is probably mature enough to be safe, so I guess I should start. I'm
using 0.2.0 because all of the man pages have been set to 0.1 until now.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
When I mistakenly run the fsck on a whole disk, instead of running it on
the apfs partition, I get a confusing "block size is too small" error
message. To avoid this noise, run no checks at all until we have
confirmed that we are working with an apfs filesystem. If not, explain
this clearly to the user.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
The mkapfs has a fatal() function, but I keep forgetting about it and
calling fprintf() and exit() directly. Tidy this up.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Back when I was only working with relatively small disks I was under the
impression that ip_bm_free_next was just a fixed-length 256-bit field.
In reality it's an array of 16-bit fields, with 16 times as many entries
as the number of bitmap blocks needed to map the internal pool. Fix the
boundary checks in spaceman_256_from_off() and rename it accordingly.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
The -v flag to print the version is mentioned when usage() gets called
for mkapfs and apfsck, but not for apfs-label and apfs-snap. Be
consistent about this.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I now understand how free_next actually works, so fix an outdated
comment from the mkfs that seems to imply that the tail block is in use.
Also use the macro for the invalid index instead of the magical number
0xFFFF.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Instead of forcing the tier 2 device to fill the whole disk, allow the
user to specify a block count, same as the main device. This may be
useful for ENOSPC tests at some point.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
While implementing the correct ip bitmap list in the driver, I realized
that expecting all entries between two consecutive free bitmaps to be
used made everything stupidly complicated for no reason. So I just
implemented the list in the way that made sense: allocating new bitmaps
in the head and freeing the old ones in the tail, without worrying about
the direction of the jump for each link. The resulting images were
accepted by the official implementations, so I guess that was indeed the
right way.
My fsck does complain of course, so get rid of this silly check.
Instead, just count the free blocks so that we can later confirm that
every used block is marked as invalid.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I just noticed that the fsck reports corruption whenever my mkfs creates
a filesystem that doesn't use the whole disk. I don't test this much
because I don't need it right now, and I don't even know if the official
implementations accept it. But it's important to have it working for the
ENOSPC tests of xfstests, which will be important once I actually start
checking for ENOSPC before starting transactions.
Anyway, fix this to the best of my understanding right now, though I may
need to make more changes in the future once I finally get around to
testing these devices on macos.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
The used_bitmaps array was allocated with one slot for each used ip
bitmap block, but it should also have a slot for the free tail. This
isn´t serious because I don't expect anyone to use this fsck other than
me (lots of similar issues probably exist), but it's causing the mkfs
tests to fail, so fix it.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I never fully understood how the ip bitmap list was supposed to work,
but now I'm noticing that the difference between what I do and what the
official driver does are bad enough to quickly corrupt the list when a
container moves back and forth. Of course this is only an issue for
filesystems above 1.32 TiB or so, but it's still pretty serious.
So, I thought about it harder and I'm pretty sure I finally figured it
out: the array entries for each free bitmap are actually kept in a
linked list, that starts in sm_ip_bm_free_head and ends in
sm_ip_bm_free_tail. All entries outside the list (that is, the entries
for the used blocks) are marked with the invalid index 0xFFFF. The tail
itself also gets 0xFFFF even though it's free, because it has no next
entry - so that accounts for the constant off-by-one weirdness in the
number of free blocks.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
As I explained before in commit 19b8d6d8 ("apfsck: verify order of
checkpoint mappings"), ephemeral objects (or at least the spaceman) can
wrap around. Add support for reading such objects to the fsck.
This is of course not possible using only mmap as we do now, so
ephemeral objects will need to be copied to allocated memory, and some
hacky changes are needed to free such memory correctly. Also a few
checks inside free_cpoint_map() have become redundant, so make sure they
happen in full inside parse_cpoint_map() and parse_cpoint_map_blocks()
instead.
For the record, this was all tested against an image created by my
driver after applying patch dfdafd2a07cf ("Wrap ephemeral objects around
if there is no room") and rigging it so that transactions would fail
after a wrapped-around object was discovered.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
The spaceman includes the arrays of cib addresses inside its own block
or blocks, and for that purpose the spaceman struct reports the offset
of each array (one for the main device and one for tier 2). I always
assumed that the offset had to actually be within the boundaries of the
spaceman object, so even regular non-fusion drives would have to store a
single "null" entry in the tier 2 array. Now that I've actually
implemented fusion drives I accidentally got rid of that extra entry and
it turns out that the official implementation doesn't mind.
That said, my mkfs tests claim to work with images of the exact sizes
where the spaceman becomes multiblock, but this is no longer correct
because the spaceman has become slightly smaller. Adjust those two image
sizes by a single cib.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
After commit f523ce46 ("mkapfs: run tests on fusion drives as well"), I
noticed that a few fusion drive tests are crashing with sigbus, but I
got sidetracked by another issue and I only got to it now. The problem
is, of course, that with a large tier 2 disk the metadata may not fit in
the main device. This is not a bug, but it's a bit ugly to fail with
sigbus, so put some coarse space checks in place and fail cleanly when
the size difference is too crazy.
Also update the fusion drive tests so that they start checking for clean
failure in the odd cases.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
As explained in the previous patch, ephemeral objects must be
consecutive and in the same order as their mappings. Since the spaceman
has a variable size and some objects are not needed for all kinds of
drives, stop hardcoding the block numbers and find them on runtime
instead.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I have recently noticed that Apple refuses to acknowledge drives that
were formatted by my mkfs, unless my driver wrote to them at least once.
The official fsck does not report any corruption, but after some
experiments it seems that the problem is that checkpoint mappings are
supposed to be in order, and not leave any holes.
For containers with a multiblock spaceman, the object itself is supposed
to wrap around, again leaving no holes. When the official driver mounts
a 7T container that wrapped around precisely when my driver unmounted
it, the last transaction gets silently discarded. This should be
extremely rare, but I need to fix it.
Start by putting the proper checks in place for the mappings.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Multiblock ephemeral objects have been supported for a while, but I just
noticed a line that claims otherwise. Get rid of it.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Expand the existing test script to run tests on fusion drives as well,
going through all possible combinations of the size array. Of course
these specific sizes are not as relevant for fusion drives, because they
were picked as the critical points where the layout changes for regular
drives, but hopefully it will be enough to cover everything.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Instead of repeatedly calling the test_size() function, put all sizes in
an array and loop over it. This will make it easier to add tests for
fusion drives in the next patch.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Multiblock ip bitmaps have been supported for a while, so remove a few
comments that claim otherwise.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Add support for creating empty fusion drives, with a minimal writeback
cache of a single block. This is enough to pass my own fsck (at least
for the few filesystem sizes I've tried), but I haven't checked if it
fools apple's. And I should also write some automated tests.
By the way, I suspect that the changes I had to make to cib/cab creation
may have also added support for regular filesystems of more than 113 TB,
but I haven't tried that yet.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Add checks for all the fusion-related structures and fields that show up
in an empty fusion drive. This requires a few changes to the way blocks
are read, so that the fsck knows which drive holds the data. Nothing
major though.
With all this in place, the allocation bitmaps add up, which is always
a good sign. I was hoping to split this into multiple patches, but I got
lazy.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
As a first step towards supporting fusion drives, add a command line
option to specify the tier 2 device. We now have two global fd's in use,
so give them names that are less confusing. As a simple sanity check,
fix the block count check, which is the first to fail if the fsck gets
run against the main device alone.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Luflosi reports that the build of apfs-label is broken in a clean
checkout:
https://github.com/linux-apfs/apfsprogs/issues/13
I mostly copied the Makefile from apfs-snap, but apfs-label is different
because it uses some shared library code (for the fletcher checksum). So
make sure that the library is getting built as well.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
On a corrupted filesystem, a volume label may not be properly
null-terminated. I guess passing that to printf may be mildly unsafe.
Anyway, put a check in place.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Since the driver only allows you to mount volumes based on their index,
sometimes it's hard to remember which volume goes by which number:
https://github.com/linux-apfs/linux-apfs-rw/issues/69
Implement a simple tool to list the volumes in a container that isn't
mounted.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I have a check in place to make sure that snapshot xids come strictly
before the current xid, but if a crash happens right after the snapshot
commit they could actually be equal, and that isn't corruption. Allow
it.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I've just encountered a sealed volume with snapshots, probably for the
first time. I have a check in place to confirm that all snapshots xids
are older than the snapshot, but I had already identified an exception
in the extent reference tree. It seems that more exceptions exist for
sealed volumes. I haven't looked into it in depth, I don't really care
that much because I don't intend to implement sealing. Just ignore this
problem for sealed volumes.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I've just run into an image that uses a new 0x10 xattr flag. I have not
tried to figure it out yet, but document it so that I don't forget.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
File extent records have two undocumented flags. One is the crypto flag,
which we don't support yet. As for the other one, I've only seen it once
so far, on an extent that was entirely allocated beyond the end of a
file. I'm guessing that's what it means, but it only seems to be valid
when the volume has the 0x80 incompatible feature. This is all hard to
confirm because I don't know how to preallocate like that in macos, and
the official fsck and driver won't complain at all if you flip this
extent bit.
Put checks in place to see if we trip on something in the future.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Add to raw.h a few recently noticed flags for the volume superblock. I
don't know much about them yet, but put the usual basic checks in place
in the fsck.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Like we did in the driver, add the latest fields discovered for the
volume superblock to the raw.h header.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
So far I have mostly ignored this "cloneinfo id epoch" in the fsck, but
I notice now some issues with one of my older images. I guess it was
created before the bugfix, so some inodes have the WAS_CLONED flag, but
not WAS_EVER_CLONED. Since this is technically allowed stop reporting it
as corruption, but leave a warning message in place anyway.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
There are two very old comments in the functions that create the free
queues. It seems that, at the time, I had doubts about the proper value
for sfq_oldest_xid on an empty free queue. Using zero has caused no
trouble at all so it's time to clean this up.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
For containers of 3G, mkfs is currently failing with sigbus. This is a
result of the previous patch, which uses an unnecessarily convoluted
formula for the number of data blocks in filesystems between 1G and 4G,
as part of a senseless (and failed) attempt to get the exact same values
as the offcial newfs_apfs. The problem is that a term of the formula can
go negative for some sizes, and C integer promotion rules turn this into
huge values. I should really simplify the formula, but I'm really tired
of this so for now just cast the dangerous term to a big enough signed
type.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
The sizes of the checkpoint descriptor and data areas have always been
hardcoded here. This is probably a bad idea in general, but currently it
makes it impossible to support tiny filesystems. So calculate the sizes
from the container block count instead.
I tried to test the official newfs_apfs to figure out their formula,
which has probably been somewhat optimized. It's a bit messy though, and
I doubt the exact values matter, so I ended up with a rough
approximation.
With this change the minimum supported container size drops to 512K,
which is as far as I intend to go. Update the test script as well.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
When we map the spaceman blocks into memory with get_zeroed_blocks(), we
are supposed to pass the spaceman size in blocks as a parameter. Instead
I mistakenly passed the size in bytes, as returned by spaceman_size().
This has not been a problem so far, but it will trigger sigbus for tiny
filesystems. Fix it.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
When I first wrote this mkfs I hardcoded a lot of block numbers to
senseless values. Some of them are too high, limiting my ability to
support tiny filesystems. Pack them closer together to push our size
limit down to 24M.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I remember reading in the official reference that containers under 128M
were a special case. At the time I assumed this was because the whole
disk could get mapped in a single chunk, and so I expected some
complications. But I've been looking at a tiny image for the first time
today, and there's actually no real work that needs to be done. Just set
the proper value in the ephemeral info field.
With this patch, the lower size limit is pushed to 83M. I can't make it
smaller yet because of the hardcoded block numbers, but I'll look into
that next.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Containers with smaller than 128M use a different value for the minimum
size of the data area. Change this.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
My driver now splits orphan deletion among multiple transactions, but it
doesn't bother to update the inode record for half-deleted files. This
is reasonable because the orphans are not real files, and we don't want
to waste time on them. My fsck is still checking one field though: the
sparse byte count. Stop that.
By the way, the official driver behaves like us in all cases I've
encountered so far, but I have never seen what it does to sparse files.
I have confirmed, however, that the official fsck has no problem with
this, so it's probably the same.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
If both an inode and its parent directory get deleted, and then the
parent gets evicted from the orphan list before the child, the child
will continue to report the same parent id referring to the now
nonexistent inode. New changes to my driver are making this possible,
and as a result collect_dirstats() creates an empty inode struct for the
old parent, and later reports that it has "no name for primary link".
Anyway, the problem here is that collect_dirstats() is considering
orphan files at all. The parent id they report is always stale, and the
stats for their old parent should have already been updated with the
deletion, so just ignore them entirely.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I don't know what went wrong with my previous mkapfs tests that made me
think we could support filesystems up to 3.6 PiB. In reality, the limit
is around 113 TiB. The code is still correct because it doesn't use any
hardcoded value for the size check, but fix the comments and the error
message.
By the way, the limit is much lower than I would have expected because
the metadata for the ip bitmaps takes up much more space than the CAB
addresses, and I had forgotten to take those into account. Mention it in
the comments.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Make all tools report the current git commit id (or tag, if any) when
called with the "-v" flag. This will be useful to confirm I'm not
running an old version by mistake.
The makefile changes here are a bit awkward. In general this repo's
structure isn't great. I should look into that at some point.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
The filesystem superblock has an apfs_formatted_by field to report which
mkfs implementation was responsible for creating the container in the
first place. Improve the version information provided, following the
recent driver changes from commit e0b1eeb29d2a ("Include release info in
apfs_modified_by field").
One small difference to note with the driver case is that we need to use
the "--always" flag with "git describe", or else it fails. The reason is
that I've never made a tagged release here. I should get to that soon.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
The space manager of an apfs container changes a lot depending on the
size of the filesystem. I fear that I will forget which size ranges are
of special interest for testing so, as a way to document this, put a
simple test script in place here. Just check that the fsck and mkfs are
consistent for all interesting size ranges I support so far.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
As explained in commit daa19ddc6f ("apfsck: implement checks for cib
address blocks"), containers with more than 8574096900096 bytes list
cib address blocks (CABs) in the spaceman instead of CIBs. Support this
in the mkfs.
With this change, I believe we've finally removed all userland hurdles
to support large containers, and I can move on to implement them in the
driver. I did leave an upper size limit of around 3.6 PB, so that all
CAB addresses would continue to fit in a single spaceman block. This
would be easy to fix (and the official mkfs does it fine), but I don't
think there is any point.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
As explained in the previous patch for the fsck, filesystems within a
certain size range require more than one block to fit the spaceman.
Implement this in the mkfs.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
For small filesystems, the addresses of the CIBs are kept as an array in
the spaceman block. For huge filesystems, the spaceman instead lists the
addresses of CABs, and those in turn list the CIBs. One might expect the
boundary between the two to be before the point where the CIBs overrun
the spaceman block, but that's not correct.
I used the official newfs_apfs to create a filesystem with 8574096900096
bytes (~7.8T), which is the maximum size that would fit in a single CAB.
As it turns out, this filesystem still keeps the CIB list inside the
spaceman, but the length of the spaceman object is now two blocks
instead of one.
I had never encountered a multiblock ephemeral object before, but I
always suspected they were possible because the object size is reported
in checkpoint mappings. This is a bit annoying because pretty much all
objects are a single block in size, and now I need to pass the object
size around everywhere just for this one crazy spaceman case. There is
no way around it though, so just do it.
By the way, filesystems bigger that 8574096900096 bytes do use CABs as
expected. That's where the limit is: containers that would fit in a
single CAB don't use it at all. This had been my guess in the previous
patch, I just failed to realize that the CIB list would overrun the
spaceman block.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Large containers don't list CIBs directly in the spaceman, they instead
list something called a cib address block (CAB) which itself contains a
list of CIBs. Add checks for this case. It's straightforward because the
logic is more or less the same as for CIBs.
This has all been tested on a single fresh 10Tb image created with the
official newfs_apfs. Much more testing will be necessary, but I'll
probably wait until CABs are implemented in the driver.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I just attempted to create an image with a size of 6T, so that it would
need 5 bitmaps to map the internal pool. I haven't checked it with the
official fsck yet, but mine chokes on it:
Internal pool: non-zeroed bitmap.
This comes from a silly check I have in place, in a function called
check_ip_bitmap_blocks(). This goes through all the stale ip bitmaps in
the ring, and confirms that none of them has bits set that map to blocks
after the end of the internal pool. I guess this was intended as a
little sanity check, to make sure I understood what the old bitmaps were
doing.
But this doesn't make any sense for multiblock bitmaps, because bitmap
blocks that don't map the tail blocks of the ip will be used in full.
When they go stale, there is no way to tell them apart from the tails.
Just refuse to run this check for the multiblock case.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
As mentioned before in 50eb66c184 ("apfsck: fix check for multiblock
ip_bitmap_offset"), the field pointed to by sm_ip_bitmap_offset is not
actually a single 64-bit number, but an array of 16-bit numbers, one for
each bitmap block in use for the internal pool. In our case, we are just
using consecutive blocks (the first few in the ring), so set the array
to consecutive numbers starting from 0.
Also allow the array to take more than 8 bytes, since I imagine that
will be needed for bigger images (I haven't tested this yet). Keep the
following field aligned to 8 bytes though.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
The spaceman field whose offset is given by sm_ip_bitmap_offset is
actually not a single 64-bit number, but an array of 16-bit numbers,
one for each ip bitmap block. This is all very similar to the previous
problem with xids, fixed in e3926bbf56 ("apfsck: check xids for
multiblock ip bitmaps").
In this case, the array tells us which of the bitmap blocks from the
nonfree region are actually in use, and in what order they should be
considered. So handle the ip bitmap properly, reading the 16-bit offsets
one by one.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Testing my multiblock ip bitmaps against the official fsck, I got a
report about overlapping spaceman fields. It seems that the xids for the
ip bitmaps are actually an array, so each bitmap block can have its own.
I guess that makes some sense, though I still don't understand all the
details.
So, set this field correctly, or as close as I can. To this end, stop
hardcoding the offset of the following two fields.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Testing multiblock ip bitmaps on my own mkfs against the official fsck,
I found out that the ip bitmap xid in the spaceman is actually an array.
I guess it's one xid for each bitmap block, though I don't know how they
get rotated yet. Extend to the whole array the only simple xid check I
have in place here.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
My recent patch 0d61bafe ("apfsck: don't assume consecutive volume
numbers") worked fine for the affected new image, but broke some old
ones I had. My mistake was leaving the volume count check in place, so
now it only checks the total size of the array (including NULL volumes)
and may fail for small containers.
Move the check so that it only verifies the highest number for actual
volumes.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Images above a certain size need more than one block for their internal
pool bitmaps. Implement this following what I've observed in a test
image.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Images above a certain size need more than one block for their internal
pool bitmaps. Before implementing this in the mkfs, put the proper
checks in place in the fsck.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
While researching huge filesystem images, I accidentally created one
with the official newfs_apfs that only has one volume, but the volume
number is 2. I have no idea why this happened in this case, but I guess
I was wrong to assume that volume numbers would be consecutive. From now
on check the whole volume array.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
We store the cib address inside the spaceman, right after
ip_bm_free_next. I don't think this is mandatory, but I have never tried
to put it anywhere else. Right now we have a problem, because the size
of ip_bm_free_next depends on the number of ib bitmap blocks, so for big
disks we overwrite the cib address when ip_bm_free_next is generated. So
stop hardcoding the cib address offset and calculate it from the ip
size.
By the way, the apfsck doesn't report this problem correctly. Instead,
it throws SIGBUS because we are mmapping a huge block number for the
cib. I don't think this is very important right now, but it could be
improved.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Calculate the size of the ip bitmaps from the size of the ip itself.
Note that we don't actually support multiblock bitmaps yet, but I wanted
to make a separate commit for this so that I could test that I wasn't
breaking anything for the small ip case.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
The driver no longer has a size limit. I should get rid of it here soon,
but for now just increase it a lot.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Update the README with information about the latest tool: apfs-snap.
Also take this opportunity to improve the explanation a little.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
The fsck tool is currently of no value to regular users of apfs, it's
only for development. Warn people about this in the man page, especially
because there is some risk that the tool could be vulnerable to attacks
with crafted filesystems.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Fresh empty containers seem to have free queues with no nodes at all.
Allow this here.
As long as this never shows up in containers with actual volumes, there
is no reason to support it in the driver.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Empty containers are perfectly valid, even if there aren't many reasons
to encounter them in the wild. They have an empty container omap, which
may report zero as its longest key/value. Allow this.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
In my testing of the driver, I ran into a situation where a single block
had two NEW extent records. The fsck reported this as a bug in its own
code, but that's not correct of course: it was a bug in the driver.
Report it correctly from now on.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I'm working with an image that doesn't set any fields in its volume's
apfs_meta_crypto struct. The reference seems to claim that the major
version should always be 5 or more, but here it's 0. The error reports
are getting annoying, so allow this.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
On a clone, the resulting file gets the APFS_INODE_WAS_CLONED flag set.
The APFS_INODE_WAS_EVER_CLONED sounds like it should only get set on the
original file, but it actually gets set on both. So, every file with
APFS_INODE_WAS_CLONED should also have APFS_INODE_WAS_EVER_CLONED. I'm
not entirely sure about this, but add a check for it and wait to see if
something breaks.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
After making some changes to one of my images with the official driver,
the official fsck complains that the reported number of freed blocks in
a volume became higher than the reported number of alloced blocks. This
is not considered an error, just a warning, but still add the same check
here. I think it's the only check that makes sense for these two fields.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Now that I've actually implemented snapshots in the driver, I can see
that the way extent reference counting is handled in the fsck makes no
sense at all. Update extents mean that the number of references can be
different for each block of a physical extent. So now, the extents
listed in the table will get split when it becomes necessary, and each
fraction will have its own refcnt.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Sigh, one more of these. I forgot that nodes with fixed key/value sizes
still have different value lengths in the index nodes, and that matters
when calculating the maximum number of records. Fix it.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
While free queues have fixed key/value sizes, they are different than
other such trees in that they also use "ghost records", which don't have
a value at all. The point of this is to save space, so it makes no sense
if we force the nodes to have a fixed record count. The driver always
did this correctly, but this fsck has started complaining about it with
the previous patch. Fix it.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
After the driver creates a snapshot, the official fsck complains about
the size of a node's table:
** Checking the snapshot metadata.
error: btn: invalid btn_table_space (0, 64), given btn_flags (0x7)
This fsck doesn't seem to notice. There are two bugs that conspire to
hide this kind of corruption. One is that nodes with fixed key/value
sizes are supposed to have an index of the exact size to map the whole
block, but we currently treat this number only as a minimum. The other
is that I forgot to add support for OMAP_SNAPSHOT and FEXT tree types
to node_min_table_size(), so those trees are treated as variable-length.
Fix both problems.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Apparently the "inum" field of a snapshot is just an identifier which is
picked from the same pool as regular inode numbers. At the moment, my
driver just leaves it as 0, and the official fsck complains about it.
Check that it's set to a valid, non-reused inode number.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Some images with compressed files are failing with "oid used twice for
same snapshot". The reason is that read_whole_dstream() will inevitably
go through some objects a second time during the queries. I already have
the 'ongoing_query' global to deal with these situations, but I forgot
to use it when adding the snapshot checks. Fix it.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
It seems that I forgot to actually implement be32_to_cpu() when I added
the compression code. Fix it.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I don't care much about leaks in the fsck, at least for now. It's just
intended as a hacky tool for testing the driver, and some of the leaks
would be tricky to fix without big rewrites. This one though, is very
dumb, and it's so brutal that we are actually hitting vm.max_map_count
for some images. Fix it.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I'm running some tests with a filesystem image where the decmpfs xattrs
themselves are not embedded. I don't why this is, and it's done for both
inline and resource compression. So, add support for this.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
While adding a few missing incompatible feature flags to the driver, I
noticed that this one check was missing from the fsck as well. It's good
to have it, so that I can confirm that I've never encountered an image
with this feature.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
For compressed files in sealed volumes, we keep a list of hashes of the
contents inside the xattr's dstream struct. Since some of those xattrs
are inline, they use a fake dstream struct with a d_inline field set.
When I implemented this, I forgot to cover the case of nonsealed
volumes, which won't go through any of this, so now they are all broken.
Fix it.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Remove a check that was copied from Stan's zlib code, but isn't needed
for the LZBITMAP case.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Introduce compression, using a modified version of Stan's code from the
driver, plus my new implementation of LZBITMAP. This is mainly needed
for sealed volumes, since for some reason Apple calculates hashes on the
decompressed contents of files instead of the xattrs themselves. At this
point, the hash checks are all passing. Very few checks on the
compression metadata were put in place; I would like to revisit this in
the future, at least to check the inode's reported decompressed size,
but it's not a priority right now.
These are a lot of messy changes and I was forced to interrupt the work
halfway through to reverse engineer LZBITMAP, so I don't recall all the
details and I'll keep the commit message brief.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Some recently defined object types are missing the "APFS_" prefix. I
noticed this while working on the previous patch, which makes use of a
few of them. Fix it now.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Sealed volumes store hashes to the contents of each file in info records
kept in the catalog, and store hashes of each catalog node in the index
record values. They also keep file extents in a separate "fext" tree,
for reasons I ignore. Add checks for all this stuff.
One problem remaining is that, for compressed files, the contents hashed
are the actual file contents, and not the compressed data in the xattr
that matches the info oid. To verify these hashes I will first have to
support compression inside the fsck. I'll try to do that next.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
In the past I've encountered extended snapshot metadata blocks listed in
the omap, but unreachable from any of the snapshot volumes. I added some
checks to allow this in commit 12731e61 ("apfsck: allow a single leaked
block in the omap"), but I had no idea why it was happening.
While I was implementing this stuff in the driver, I finally figured it
out. The extended snapshot metadata was a late addition to apfs, and the
official reference allows old implementations to simply ignore it when
they find it. So, when a snapshot is created by a recent implementation
and then deleted by an old one, this block would leak. This is fine and
not a form of corruption.
Now that I understand this, update the comments on the matter. Also
allow any number of leaked blocks of this type: so far I've only ever
seen one, but there's no limit here.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
Write a very simple tool that requests a new snapshot to the driver. I
still need to implement snapshots over there, but it's better to have
this ready for testing.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I just noticed this while copying this code to the apfs-snap tool. I
guess this was a copy-paste error from when I first created the mkfs.
Not important of course, but better keep comments updated.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I've always had an image where the reported regular file count seems to
be off by one. After I ran into a similar issue with another image, I
fixed it with commit 8b1575bd ("apfsck: ignore orphans in the volume
file counts") and I thought that was it. But no, the original image is
still failing and I have no idea why.
Make this failure into a "weird" report again, like it was before that
commit.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I've had some trouble getting allocated block counts to add up in the
presence of snapshots. On images produced by a recent iOS release, the
check would pass if there was a single snapshot, but each new snapshot
added would make the calculated count drift by one.
I think I've figured this out, and it's a combination of two problems.
First, there is the leaked block I mentioned in 12731e61 ("apfsck: allow
a single leaked block in the omap"). Even if it doesn't seem to be
reachable, it still needs to be counted as part of its volume. Ignoring
this block made the check appear to pass for the one-snapshot case.
The second problem are the volume superblocks. I've always known that
they were not counted, but the previous confusion made me think that
they behaved differently inside snapshots. They don't though, and they
always need to be ignored.
Fix both issues.
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
I've encountered an image that has an unused omap record marked as
deleted. As usual, I'm not really sure what the point of this is, but
take it into consideration for the checks in free_omap_record_list().
Signed-off-by: Ernesto A. Fernández <ernesto@corellium.com>
