Please help! Unknown error when apfs-inspect

[zhanchengqian]

Hi,

My file system got damaged when I was trying to copy large trunk of files from the drive (apfs encrypted MacBook Pro ssd, I put it inside a third-party ssd-to-usb external case). When I was copying, suddenly it says the directory doesn’t exist, and soon I found out the folder I was copying disappeared.

I tried to do ls in terminal, but no result, it disappeared. Then that night, I downloaded disk drill. Disk drill shows the missing folder but after I recovered the files I found out 50% of the files are garbled, not opening. The other 50% is okay, but anything before November 2017 is non openable.

Then I ran across this GitHub page and I tried to use apfs-inspect and apfs-read, but no success. I think both of these returned ABORT unknown error. I don’t know what happened because the message offers little detail.

I mean these files are not something too important, but are memorable things that I kept during college. I really want to recover these data. The garbled portion is around 35GB. Please help me diagnose the issue. Is it possible to get more detail of why apfs-inspect failed?

Also just FYI, the recovered non-openable files are the same size as the good files. I also have very small portion of the garbled data on another drive (known to be good and can be opened), like 1GB out of the garbled 35GB. I compared the folders that are in common. for example, file A (from the other drive) is 35MB, and the garbled file A (what I recovered) is also 35MB. I compared the hex, the hex has nothing in common, totally garbled in the recovered file.

I was reading the drive on another Mac machine, High Sierra, and the ssd is connected thru ssd-usb3.0 external dongle. This MacBook ssd is extracted out of my MacBook, since it got damaged in a recent incident.

Please help me! I don’t know what caused the data damage, and I don’t understand why Disk Drill recovered some good files and some garbled files. Also, why the apfs-inspect and apfs-read didn’t work at all? I will provide screenshots if needed.

Let me know what I should do next.

Thanks, God bless you all

[zhanchengqian]

[zhanchengqian]

also, when I ran fsck it says Object map is invalid.

[jivanpal]

Output of diskutil list and diskutil apfs list would be helpful just to make sure we're not missing anything trivial.

There are a couple of possible (but exceedingly unlikely) error conditions for fopen() that I didn't handle, but have now added code for. Can you checkout the unstable branch, recompile, and try again? That may give us some info as to why you can't open /dev/disk4s1.

The actual "ABORT: Unknown error" message you are seeing appears when fopen() cannot open the file (/dev/disk4s1) for some unknown reason. I don't know what to do with that info (or rather, lack of info). My best guess is that your SATA-to-USB adapter (or something along the way, such as your USB controller/bus) is not working properly, and it may have caused the damage in the first place. Can you attach the drive directly to a machine's SATA port and try again?

I've been meaning to sort out this project for a long time now, but hey, life gets in the way. Hopefully I'll have a week or two in November to dedicate to working on this.

[zhanchengqian]

Thanks for the reply. I tried the unstable branch (on another mac), and still got the "ABORT unknown error". I assume that maybe it has something to do with the USB adapter. The thing is Apple uses its own proprietary hardware interface, so I have no way of directly attach the SSD to a machine's SATA port. It means I can only access it thru an adapter of some sort. I will try to get another adapter from another brand, and see if any thing different will appear.

Also, here are the outputs from diskutil: (shows as disk3s1 on this machine)

Thanks once again for your help, hope all the best!

[zhanchengqian]

I know it's kinda dumb but I ran the repair in disk utility and it gave me this. I don't know whether this would give some more info or not. But I wanna post anyway.

I also tried to unmount other volumes on the same SSD, and I also tried to do this in recovery mode. It always give me the same error message.

[jivanpal]

here are the outputs from diskutil: (shows as disk3s1 on this machine)

disk3s1 is not the APFS partition/container itself; it is the Data volume within your APFS container. The container itself is disk2s2 (see "Physical store" in the output of both commands). Try that first.

I ran the repair in disk utility and it gave me this.

This is telling you that a repair cannot be done whilst you are running macOS normally, since one of the volumes is mounted. I would clone the partition or whole disk to another disk (in case the repair actually worsens things) and then attempt repair from macOS Recovery. hold CMD+R when booting, then open Terminal and find the physical store ID via diskutil list (e.g. as disk2s2 is currently). Unmount all mounted volumes belonging to the APFS container with diskutil unmountDisk diskXsY, and then run e.g fsck_apfs -y /dev/rdiskXsY, where X and Y are from the physical store ID. Note, that's rdisk in the latter command, not just disk.

[zhanchengqian]

Hi jivanpal, thanks for the reply.

Now I have a new adapter, and now I understand that it's actually drive disk2s2. Here is what happened when I ran diskutil verifyVolume:

~ diskutil verifyVolume /dev/disk2s2
Started file system verification on disk2s2
Verifying storage system
Performing fsck_apfs -n -x /dev/disk2s2
Checking the container superblock
Checking the EFI jumpstart record
Checking the space manager
Checking the space manager free queue trees
Checking the object map
error: (oid 0xd55eb) om: invalid o_xid (0x6a1c29)
error: (oid 0xd55eb) om: invalid o_type (0x40000003, expected 0x4000000b)
error: verification/reading of the omap object failed: Illegal byte sequence
The volume /dev/disk2s2 could not be verified completely
Storage system check exit code is 8
Error: -69716: Storage system verify or repair failed
Underlying error: 8

Then I ran these commands, for you reference. (I am a CS grad student, but please forgive me that I don't have a good understanding of what's going on in the Apple APFS system, please point out any problem here)

➜  bin sudo ./apfs-list /dev/disk2s2 0 /Users/Encholl
Opening file at `/dev/disk2s2` in read-only mode ... OK.
Simulating a mount of the APFS container.
Validating checksum of block 0x0 ... OK.
Locating the checkpoint descriptor area:
- Its length is 280 blocks.
- It is contiguous.
- The address of its first block is 0x10647c.
Loading the checkpoint descriptor area into memory ... OK.
Locating the most recent well-formed container superblock in the checkpoint descriptor area:
- It lies at index 201 within the checkpoint descriptor area.
- The corresponding checkpoint starts at index 200 within the checkpoint descriptor area, and spans 2 blocks.

Loading the corresponding checkpoint ... OK.
- There are 68 checkpoint-mappings in this checkpoint.

Reading the Ephemeral objects used by this checkpoint ... OK.
Validating the Ephemeral objects ... OK.
The container superblock states that the container object map has Physical OID 0xd6f38.
Loading the container object map ... OK.
Validating the container object map ... OK.
Reading the root node of the container object map B-tree ... OK.
Validating the root node of the container object map B-tree ... OK.
The container superblock lists 5 APFS volumes, whose superblocks have the following Virtual OIDs:
- 0x403
- 0x54961
- 0x54963
- 0x5ee18
- 0x3cc230

Reading the APFS volume superblocks ... OK.
Validating the APFS volume superblocks ... OK.

 Volume list
================
 0: Macintosh HD - 数据
 1: Preboot
 2: Recovery
 3: VM
 4: Macintosh HD
The volume object map has Physical OID 0xd6edc.
Reading the volume object map ... OK.
Validating the volume object map ... OK.
Reading the root node of the volume object map B-tree ... OK.
Validating the root node of the volume object map B-tree ... OK.
The file-system tree root for this volume has Virtual OID 0x404.
Looking up this Virtual OID in the volume object map ... corresponding block address is 0x6250.
Reading ... validating ... FAILED.
Going back to look at the previous checkpoint instead.
➜  bin

[zhanchengqian]

The output when running apfs-inspect:

inspect-output.txt

Also I tried to read somewhere around 0xa4bf0 ~ 0xa4bf2, which is the address I found some of the missing files via Disk Drill. And I got these two results, and it seems something wrong happened around these addresses, I assume?

➜  bin sudo ./apfs-read /dev/disk2s2 0xa4bf0

Opening file at `/dev/disk2s2` in read-only mode ... OK.

Reading block 0xa4bf0 ... validating ... FAILED.
The specified block may contain file-system data or be free space.

Details of block 0xa4bf0:
--------------------------------------------------------------------------------
Stored checksum:    0xbbfcbad22880ea15
OID:                0x5a217bf9d6cd0011
XID:                0x71f860f74621dcab
Storage type:       Ephemeral
Type flags:         Non-persistent (should never appear on disk --- if it does, file a bug against the APFS implementation that created this object)
Type:               Unknown type (0x%08x) --- perhaps this type was introduced in a later version of APFS than that published on 2019-02-27.
Subtype:            Unknown subtype (0x00003c3f) --- perhaps this subtype was introduced in a later version of APFS than that published on 2019-02-27.

--- This tool cannot currently display more details about this type of block ---
--------------------------------------------------------------------------------

END: All done.
➜  bin sudo ./apfs-read /dev/disk2s2 0xa4bf1

Opening file at `/dev/disk2s2` in read-only mode ... OK.

Reading block 0xa4bf1 ... validating ... FAILED.
The specified block may contain file-system data or be free space.

Details of block 0xa4bf1:
--------------------------------------------------------------------------------
Stored checksum:    0x0000000000000000
OID:                0x0
XID:                0x0
Storage type:       Virtual
Type flags:         (none)
Type:               (none/invalid)
Subtype:            (none/invalid)

--- This tool cannot currently display more details about this type of block ---
--------------------------------------------------------------------------------

END: All done.
➜  bin

[zhanchengqian]

Running the fsck gives me this:

fsck_apfs -y -x /dev/rdisk3s1
Checking the container superblock.
Checking the EFI jumpstart record.
Checking the space manager.
Checking the space manager free queue trees.
Checking the object map.
error: (oid 0xd55eb) om: invalid o_xid (0x6a1c29)
error: (oid 0xd55eb) om: invalid o_type (0x40000003, expected 0x4000000b)
error: verification/reading of the omap object failed: Illegal byte sequence
The volume /dev/rdisk3s1 could not be verified completely.

In all, I know there is /Users/Encholl folder somewhere near 0xa4bf2, (I got that address from Disk Drill), and what kind of "next step" should I take? I hope I've provided enough info for you to find what the problem is. I will provide more info as you request. I lack some knowledge about this kind of file system. I will be very grateful if you could give me some more insight into how I could recover some of the files near that address in the drive.

Thanks for your help again, I think I cannot find another knowledgeable guy like you on the Internet.
Hope you have a great day!

[jivanpal]

Your volume's object map block is at address 0xd6edc. This block only stores metadata about the object map; it isn't the object map B-tree itself. However, it tells you where the B-tree is. Reading that block with apfs-read should reveal the address of the object map B-tree's root node. Substitute the address you find for <omap root node address> in the commands below.

---

We want to use apfs-recover to grab your files from /Users/Encholl. Unfortunately, apfs-recover always traverses from / down the tree, but the address that virtual OID 0x404 (the root of the volume) resolves to is an invalid block, so apfs-recover will currently encounter an error because it cannot find /.

We can either:

(a) Check whether virtual OID 0xa4bf2 is really /Users/Encholl, after which we can modify apfs-recover to scan from there rather than from /; or

(b) Fix the blocks which represent the filesystem root, /, /Users, and /Users/Encholl, then use apfs-recover in its current form.

---

For (a), the way forward is:

1. Check block 0xa4bf2 with apfs-read. Please provide this output, as it seems to be missing from the output you pasted in your previous comment.
2. If it really is a filesystem B-tree node, check its contents with apfs-explore-fs-tree /dev/disk2s2 0xa4bf2 <omap root node address>.
3. If it really contains entries from /Users/Encholl, modify apfs-recover to fetch files using this info.
4. Use apfs-recover to grab your files.

---

For (b), the way forward would be:

1. Correct the block with virtual OID 0x404, which represents the root of your Data volume. This block is distinct from the root of your filesystem, /; in fact, this block lists the virtual OID of / (usually 0x2) and another entity called private-data (usually 0x1).
2. Correct the blocks which represent /, /Users, and /Users/Encholl, if necessary.
3. Use apfs-recover to grab your files from /Users/Encholl.

Ideally, steps (1) and (2) would be automated, but these tools are not sophisticated enough to do this (implementing this is the main thing I would like to work on right now), as this can be quite case-specific. The apfs-explore-* commands allow you to explore B-trees (object maps and filesystem records), and apfs-search allows you to scan the disk to look for blocks with certain properties (e.g. if you are hunting for a specific dentry or virtual OID) in order to find the relevant info to do these steps manually instead. Manual fixing of blocks is done with the aid of apfs-modify (use with extreme caution, it WILL irrevocably overwrite data on disk). This whole process can be extremely time-consuming.

The first thing to do would be to see whether a valid instance of the block with virtual OID 0x404 can be found. You can explore the object map B-tree in order to look for the physical address of blocks with a given virtual OID. Do this with apfs-explore-omap-tree /dev/disk2s2 <omap root node address>.

---

Best of luck!

[zhanchengqian]

Thanks for the reply, here is the 0xa4bf2 apfs-read result:

➜  bin sudo ./apfs-read /dev/disk2s2 0xa4bf2

Opening file at `/dev/disk2s2` in read-only mode ... OK.

Reading block 0xa4bf2 ... validating ... FAILED.
The specified block may contain file-system data or be free space.

Details of block 0xa4bf2:
--------------------------------------------------------------------------------
Stored checksum:    0x0000000000000000
OID:                0x0
XID:                0x0
Storage type:       Virtual
Type flags:         (none)
Type:               (none/invalid)
Subtype:            (none/invalid)

--- This tool cannot currently display more details about this type of block ---
--------------------------------------------------------------------------------

END: All done.

[zhanchengqian]

To be specific, I tried to read multiple addresses, and arrived at these results:

0xa4bf0 (checksum OID XID looks good)
0xa4bf1 (checksum OID XID All zeroes)
0xa4bf2 (checksum OID XID All zeroes)
...
...
0xd4155 (checksum OID XID All zeroes)
0xd4156 (checksum OID XID All zeroes)
0xd4157 (checksum OID XID looks good)

By "looks good" I mean they have some values instead of zeroes.

[zhanchengqian]

Here is the object map info:

➜  bin sudo ./apfs-read /dev/disk2s2 0xd6edc

Opening file at `/dev/disk2s2` in read-only mode ... OK.

Reading block 0xd6edc ... validating ... OK.

Details of block 0xd6edc:
--------------------------------------------------------------------------------
Stored checksum:    0x5184bf83edf6462b
OID:                0xd6edc
XID:                0x6a1c6d
Storage type:       Physical
Type flags:         (none)
Type:               Object map
Subtype:            (none/invalid)
Flags:
- No flags are set.
Object mappings tree:
- Storage type:         Physical
- Type flags:           (none)
- Type:                 B-tree (root node)
- Object ID:            0xd6ef8
Snapshots tree:
- Storage type:         Physical
- Type flags:           (none)
- Type:                 B-tree (root node)
- Object ID:            0x0
- Number of snapshots:  0 snapshots
- Latest snapshot XID:  0x0
In-progress revert:
- Minimum XID:          0x0
- Maximum XID:          0x0
--------------------------------------------------------------------------------

END: All done.
➜  bin

[zhanchengqian]

So from the following result, I assume that 0xa4bf2 is not really the address I want?
It does not bring any useful result and thus it seems it's not the file-system tree root node.

➜  bin sudo ./apfs-explore-fs-tree /dev/disk2s2 0xa4bf2 0xd6ef8

Opening file at `/dev/disk2s2` in read-only mode ... OK.

Reading the file-system tree root node (block 0xa4bf2) ... validating ... FAILED.

Reading the object map root node (block 0xd6ef8) ... validating ... OK.


Node details:
--------------------------------------------------------------------------------
Stored checksum:    0x0000000000000000
OID:                0x0
XID:                0x0
Storage type:       Virtual
Type flags:         (none)
Type:               (none/invalid)
Subtype:            (none/invalid)
Flags:                          (none)
Number of child levels:         0
Number of keys in this node:    0
Location of table of contents:
- Offset from start of node data area:  0x0 = 0
- Length (bytes):                       0x0 = 0
Location of key–value shared free space:
- Offset from start of keys area:       0x0 = 0
- Length (bytes):                       0x0 = 0
--------------------------------------------------------------------------------


Node has 0 entries, as follows:
Assertion failed: (node->btn_nkeys > 0), function main, file src/apfs-explore-fs-tree.c, line 160.
[1]    20545 abort      sudo ./apfs-explore-fs-tree /dev/disk2s2 0xa4bf2 0xd6ef8