fuse: deliver fdclose to the migrated fd before dropping it

[ThalesBarretto]

fuse: deliver fdclose to the migrated fd before dropping it

Problem

fd_close() (added with the open-behind rewrite, db95388706) tells the xlators of the graph an fd belongs
to that the application's handle is closing; open-behind uses it to cancel a deferred open and drop the fd
reference and the open stub it keeps for it. fuse_release() calls it on the base fd only. After a graph
switch the base fd's ctx owns activefd, an fd created on the new graph by fuse_migrate_fd_open() whose
open went through the new graph's open-behind and was deferred (lazy-open is the default; a small-file read
is served by quick-read above it, and the close-time FLUSH is answered locally, so nothing triggers it).
fuse_fd_ctx_destroy() then just fd_unref()s activefd: open-behind's two references keep it alive, and
with it the fuse fd ctx and the new-graph inode, until the file is opened again through that graph or the
process exits. Every open file migrated across a volume set with performance.open-behind on (the default)
and then closed without a further fd fop leaks this way.

Fix

• fuse_fd_ctx_destroy(): fd_close(activefd) before fd_unref(activefd). fd_close() walks the fd's own
  graph, so the new graph's ob_fdclose() runs, cancels the deferred open and drops both references; the
  following unref reaches zero and fd_destroy() runs the normal .release cbks.
• fuse_migrate_fd_open(): likewise fd_close(old_activefd) before the fd_unref() that abandons the
  previous active fd on a re-migration (the fsync in fuse_migrate_fd() already prevents a leak there; this
  keeps the protocol uniform and balances open-behind's open_count on the abandoned graph).

Composes with #4490, which moves fuse_fd_ctx_destroy() into the .release/.releasedir cbks and keeps
calling it — verified on a stock arm carrying #4490 on top of this change: 10 files across a switch, close, 0 live
fds / 0 fuse ctxs / itable active 1.

Reproduction and A/B

Stock builds of devel a482a8578a and of this branch (default ./configure: tcmalloc, no mem-pools, no debug),
volume with performance.open-behind on; N files held open O_RDWR on a FUSE mount, one volume set, the mount
poked with stat (FUSE switches graphs on the next request), the handles closed, then a client statedump. On a
tcmalloc build there is no fd_t mem-pool, so the count is the fuse xlator's memory accounting (one
gf_common_mt_fd_ctx array per live fd_t), plus the inode table fuse dumps natively.

scenario	live fds before / after	pinned inodes (itable active_size − root) before / after
1 switch, plain close, N = 10	10 / 0	10 / 0
1 switch, plain close, N = 100	100 / 0	100 / 0
2 switches, plain close	10 / 0 (not 20: the re-migration fsync releases the previous fd)	10 / 0
1 switch, pread immediately (quick-read cache), close	10 / 0	10 / 0
1 switch, pread 3 s later (reaches open-behind), close	0 / 0	0 / 0
no switch (control)	0 / 0	0 / 0
reopen of one file after the leak (before only)	10 → 9	10 → 9

Statedump of the devel client after all ten handles were closed:

[mount/fuse.fuse - usage-type gf_common_mt_fd_ctx memusage]
num_allocs=10
[mount/fuse.fuse - usage-type gf_fuse_mt_fd_ctx_t memusage]
num_allocs=10
[xlator.mount.fuse.itable]
xlator.mount.fuse.itable.active_size=11
[xlator.mount.fuse.itable.active.1]
gfid=3f960812-7d76-4236-aae1-2bc4f36e6d5a
nlookup=0
fd-count=1
active-fd-count=1
ref=2
ref_by_xl:.fuse=1
ref_by_xl:.vol4490-open-behind=1

With this change the same run gives num_allocs absent (0) for both types and active_size=1.

Test

tests/bugs/fuse/fd-migration-open-behind-leak.t: five files held open, one switch (poked), closed without
any other fop, expects the fuse xlator's gf_common_mt_fd_ctx allocation count (live fds; exists on every build,
unlike the fd_t mem-pool) and its gf_fuse_mt_fd_ctx_t count to be 0 in the mount statedump. Fails on stock
devel on exactly those two assertions (Got "5" instead of "0"), passes with the change.

The gfapi side of the same gap (gfapi never calls fd_close() at all) is #3977; a separate PR addresses it.

Fixes: #4836