irods-sync-verify

Three small tools for checking that a local directory tree and an iRODS collection contain the same data.

They are intended for cold-storage migration checks. After copying a tree to iRODS, generate a timestamped inventory on each side and compare the two. The resulting reports provide a record of what was checked and any differences found.

irods_audit.sh   ──►  irods__<collection>__<timestamp>.tsv  ──┐
                                                              ├──►  compare_audits.py  ──►  diff__<a>__vs__<b>.tsv
local_audit.sh   ──►  local__<path>__<timestamp>.tsv       ──┘

The audit scripts can run independently, including on different machines and at different times. Because comparison is separate, you can compare the same pair of reports again without reading or hashing the data another time.

---

Requirements

Tool	Needs
irods_audit.sh	bash, awk (mawk or gawk), iRODS icommands (iquest), an authenticated session (iinit)
local_audit.sh	bash, find, Python 3.6+
compare_audits.py	Python 3.6+, no third-party packages

There are no other dependencies. irods_audit.sh calculates dates in awk instead of using strftime(), which keeps it compatible with both mawk (the default awk on Debian/Ubuntu) and gawk.

---

Usage

The audit scripts do not take options. They always record every available field, so their reports use the same schema and no important column can be left out by accident.

# 1. inventory the iRODS side
iinit                                          # authenticate first
sh irods_audit.sh /tempZone/home/labA/data

# 2. inventory the local side
sh local_audit.sh /data/labA

# 3. compare
python compare_audits.py irods__*.tsv local__*.tsv

Each audit writes <kind>__<path slug>__<UTC timestamp>.tsv to the current directory. The comparison tool reads the report kind from its header, so the input order does not matter.

An optional third argument to compare_audits.py sets the diff filename; otherwise it is derived from the two input names.

Exit codes

compare_audits.py:

Code	Meaning
0	clean — every object matched on presence, size and checksum
1	differences found
2	could not run (bad input, unreadable report)

This is suitable for use as a migration gate in a pipeline. Both audit scripts exit non-zero when they fail. They also do not write a report if the query or read fails, which avoids leaving an empty report that could be mistaken for a successful audit of an empty collection.

---

Example

$ ./compare_audits.py irods__tempZone_home_labA_data__20260922T095304Z.tsv \
                      local__data_labA__20260922T095304Z.tsv

iRODS report : irods__tempZone_home_labA_data__20260922T095304Z.tsv
  root       : /tempZone/home/labA/data
  objects    : 6 (7 replica rows)
local report : local__data_labA__20260922T095304Z.tsv
  root       : /data/labA
  files      : 6

  verified (size + checksum match) : 3
  present in both, unverifiable    : 0

  missing locally  (iRODS only)    : 1
  missing in iRODS (local only)    : 1
  size mismatches                  : 1
  checksum mismatches              : 1
  iRODS-internal replica conflicts : 0

size mismatches:
  run2/empty.log
    irods=999B (999)  local=1B (1)

checksum mismatches (same size, different content):
  run2/big file with spaces.fastq.gz
    sha256 irods=3dbb3963d11aa418de8b61f846c3dbd5af43b40d252842adb823f90936fe6920
           local=b3c3a85ca630aa5791aa8097167726e1bbd24a26545216047aff2eab22528906

diff written to diff__irods__...__vs__local__....tsv

The diff TSV is the machine-readable form, one row per problem:

issue                relpath                              detail
missing_locally      run2/orphan_irods.txt                in iRODS, not on disk
missing_in_irods     run2/orphan_local.txt                on disk, not in iRODS
size_mismatch        run2/empty.log                       irods=999 local=1
checksum_mismatch    run2/big file with spaces.fastq.gz   sha256 irods=3dbb... local=b3c3...

---

Report format

Both audit scripts use the same tab-separated schema. Metadata lines begin with #, followed by the column header and then one row per local file or iRODS replica.

Column	iRODS	Local	Notes
relpath	✔	✔	path relative to that report's root — this is the join key
size_bytes	✔	✔	compared
mtime_utc	✔	✔	recorded, not compared (see below)
owner	✔	✔	recorded, not compared (see below)
checksum_raw	✔	—	verbatim iRODS checksum, in whatever encoding the zone uses
md5_hex	—	✔
sha256_hex	—	✔
repl_num	✔	—
repl_status	✔	—	1 = current replica
resource	✔	—	omitted if the zone's GenQuery doesn't expose RESC_NAME

Paths are relative to the report root, recorded in the # root header. This allows /tempZone/home/labA/data/run1/x.bam to match /data/labA/run1/x.bam. Tabs and newlines in filenames are backslash-escaped to keep the TSV valid.

---

Design notes

These details affect how the results should be interpreted.

mtime and owner are recorded but never compared

iRODS DATA_MODIFY_TIME is when the replica was written into iRODS, not the original file's mtime. DATA_OWNER_NAME is an iRODS username, not a POSIX account. After any migration both differ for essentially every object. Comparing either field would mark nearly every migrated file as different. They remain in the report as useful context, but only presence, size, and checksum are used as integrity checks.

iRODS checksums are not hex

Depending on the zone's configured hash scheme, DATA_CHECKSUM is one of:

• sha2:<base64> — base64 of the raw digest bytes, not a hex string
• <32 hex chars> — a bare legacy MD5
• md5:<hex> — an explicitly prefixed MD5

compare_audits.py converts each form to lowercase hexadecimal. Since the local audit does not know which scheme the zone uses, it calculates both MD5 and SHA-256 in one pass over each file. The hashing loop is therefore in Python instead of an md5sum + sha256sum pipeline. One Python process handles the whole tree, avoiding a second pass over large datasets.

"unverifiable" is not "verified"

DATA_CHECKSUM is only populated if a checksum was ever computed for that replica. Objects present on both sides with no stored iRODS checksum are counted and reported as unverifiable, rather than included in the verified total. If many objects are unverifiable, the run is mostly a size comparison. Populate checksums first:

ichksum -r -K /tempZone/home/labA/data

Replicas are collapsed, and disagreements flagged

A replicated object appears once per replica in the iRODS report. compare_audits.py reduces these to one logical object for comparison and reports any replicas that disagree on size or checksum. Such a disagreement is an iRODS issue regardless of the local copy.

GenQuery LIKE wildcards

irods_audit.sh uses a broad LIKE prefix, then applies an exact prefix check in awk. GenQuery treats _ and % as wildcards, so without the second check, a collection such as lab_data could include sibling collections.

Unicode and non-UTF-8 filenames

Relative paths are NFC-normalised before matching, so NFD filenames (as produced by macOS) match their NFC equivalents. Filenames that are not valid UTF-8 are handled with surrogateescape, so they do not stop the local audit.

Other robustness details

• iquest output is parsed with \x1f (unit separator) as the field delimiter, not | — a pipe can legitimately appear in an iRODS path.
• iquest's exit status on an empty result set is inconsistent across versions, so CAT_NO_ROWS_FOUND is treated as "empty, fine" and anything else non-zero as a genuine failure.
• RESC_NAME is spelled differently across iRODS versions. The richer query is attempted first and falls back silently to the base field set, so you get the resource column when the zone supports it and a working report when it doesn't.
• Reports are written to a temp file and moved into place only on success.

---

Limitations

• The local audit reads every byte. Runtime depends on storage throughput. Test a small tree before scheduling a large run: hashing a multi-TB collection can take a substantial part of the verification window.
• No resume. An interrupted local audit starts over.
• Regular files only. Symlinks are not followed (a followed symlink would double-count data and could escape the tree) and are counted separately in the run summary. Empty directories are not represented, since iRODS collections and POSIX directories do not correspond one-to-one.
• Hashing is single-threaded. This is generally sufficient when I/O is the bottleneck.
• Collection paths containing a single quote are rejected, rather than guessing at the escaping rules of a particular iRODS version.

---

Verify the semantics against your own zone first

Two assumptions require a live zone to verify. Confirm both with a few known files before using these reports for migration sign-off:

1. That DATA_REPL_STATUS == 1 means "current, good replica" in your iRODS version.
2. That DATA_MODIFY_TIME behaves as described above in your zone's configuration.

Cross-check a few rows against ils -l and ichksum directly.