feat(pi-fff): search explicitly specified ignored paths via a scoped auxiliary index

[RunMintOn]

Summary

fffind and ffgrep currently cannot search a Git-ignored file or directory when it is explicitly supplied through the path parameter and remains inside the current workspace.

This is consistent with FFF's current indexing design: ignored paths are omitted during the initial scan, and path narrows the existing index rather than reading from disk. However, it is inconvenient for agent workflows that need to inspect a specifically named dependency, generated file, local configuration, or build artifact.

This is a pi-fff-specific feature proposal, narrower than a general --no-ignore mode.

Example

Given:

node_modules/

and a real file at:

project/node_modules/example-package/package.json

an explicit search such as:

fffind(path="node_modules/example-package/", pattern="package.json")
ffgrep(path="node_modules/example-package/package.json", pattern="some-key")

currently returns no result when the main workspace index was used.

Proposed behavior

When the caller explicitly supplies a concrete existing path:

• For a directory, create a scoped auxiliary finder rooted at that directory.
• For a file, create a scoped auxiliary finder rooted at its parent directory and constrain the query to that file.
• Search that scoped finder, allowing the explicitly requested path to be discovered even if it was ignored by the parent workspace scan.
• Keep the current ignore-aware behavior for searches without an explicit path.
• Keep the scoped finder short-lived or bounded by the existing auxiliary-finder cache; do not index all ignored files eagerly.

The implementation could generalize the existing AuxFinderPool path used for paths outside the workspace. The internal lookup should require the requested root to match exactly rather than reusing a broader finder that may already have excluded the target subtree.

No change to the default workspace index is proposed, and this should not require a core FFF --no-ignore mode.

Acceptance criteria

• Normal fffind / ffgrep calls without path behave exactly as they do today.
• An explicitly specified ignored directory can be searched by both tools.
• An explicitly specified ignored file can be searched by both tools.
• Existing tracked paths continue to work.
• Broad globs and nonexistent paths retain their current behavior unless deliberately included in a later change.
• Tests cover Git-ignored directories, Git-ignored files, normal files, and both tools.

Why this is useful

Agents often need to inspect files that are intentionally not part of the project index, including installed packages under node_modules, generated outputs, local configuration, and temporary tool results. Requiring a separate shell fallback makes the tool behavior inconsistent and loses FFF's normal path/search interface.

Related discussions:

• Does it support --no-ignore args? #108 — Does it support --no-ignore args?
• feat: option to search git ignored files #406 — feat: option to search git ignored files
• Feature: Allow grep/glob outside search index #412 — Feature: Allow grep/glob outside search index
• API to control what is ignored and --no-ignore ability to index everything #530 — API to control what is ignored and --no-ignore ability to index everything

This proposal is intentionally narrower: explicit path selection acts as the opt-in, while ordinary searches remain ignore-aware.

[gustav-fff]

[triage-bot] FEATURE REQUEST: needs maintainer decision.

Not a bug. Related open discussion: #108 (maintainer previously declined general --no-ignore on design grounds — separate indexing conflicts with FFF's incremental-scan model). Also related: #406, #412 (closed as dup of #108), #530.

This proposal is narrower — it's scoped to pi-fff (packages/pi-fff/src/index.ts) and reuses the existing AuxFinderPool in packages/pi-fff/src/aux-finders.ts that already handles out-of-workspace paths. Sketch:

• routePathConstraint (aux-finders.ts:142) currently returns null for absolute paths inside cwd. Would need to instead route to an aux finder when the explicit path is inside cwd AND the workspace picker would have ignored it.
• AuxFinderPool.acquire (aux-finders.ts:44) would need exact: true (or a similar mechanism) to avoid reusing a broader covering picker that already excluded the target subtree.
• Ignore-detection: probably resolve via git check-ignore or by probing the main picker.

Not opening a PR — this is a design call, not a defect. @dmtrKovalenko please confirm whether the pi-fff-scoped variant is acceptable given the constraints raised in #108, or if the answer remains no.

Honk-Honk 🪿

[dmtrKovalenko]

Wait but we have implemented exactly this in 0.10 it works exactly like that now, no?

[RunMintOn]

Thanks! To clarify, I’m specifically referring to an ignored path inside the active workspace, for example:

cwd=/project
path=/project/node_modules/foo

The existing auxiliary-finder support appears to cover paths outside the workspace, which is already very useful. The narrower question here is whether an explicitly specified path inside the workspace should also use a scoped auxiliary finder when the main workspace index has excluded it because of .gitignore.

In the current pi-fff implementation, routePathConstraint() still returns no auxiliary route for paths inside cwd, so those searches use the main finder and cannot see ignored files. If v0.10 already handles this in-workspace case too, a test or implementation reference would help me understand the intended behavior.

[dmtrKovalenko]

Now I get it - you mean ignored path inside a workspace. Im nkt sure I like it cause it could be pretty heavy especially if AI tries to open something like node_modules

I need to think more about it

[vlsi]

I have the same issue with fff-mcp when working on pgjdbc/pgjdbc#3062

Repro

Real project (pgjdbc, Gradle). pgjdbc-jazzer-test/build/generated/sources/fuzzTargets/java/org/postgresql/test/jazzer/GeneratedScalarDecodeRobustnessFuzzTest.java exists on disk (19 KB, emitted by a build task) and is referenced by name from src/.

• find_files("GeneratedScalarDecodeRobustnessFuzzTest.java") returns the two generator classes that emit the file, not the file itself.
• grep("GeneratedScalarDecodeRobustnessFuzzTest") returns 18 matching lines across 8 files, all under src/** or *.md. The generated file, which declares that class and 71 test methods, never appears.
• Same for JUnit XML under build/test-results/: no match by name or content, so reading a test's captured stdout means dropping to a shell.

Why it matters

Generated code is first-class in many stacks: protobuf and gRPC stubs, OpenAPI clients, annotation-processor output, Gradle codegen. Debugging means reading the generated file, not the generator that wrote it. Today every such lookup forces a fallback to grep/find in Bash, which is exactly what my agent instructions tell the agent not to do while FFF is available.

Proposal

Index everything and filter out binaries, instead of using .gitignore as the filter:

• skip files that look binary: a NUL byte in the first few KB, or a binary attribute from .gitattributes;
• keep .git/ and other VCS metadata directories excluded.

fff might sort gitignored files after non-ignored ones (e.g. to show source code first, and test results later).

I would gladly trade a longer initial scan for full coverage. --max-cached-files and the temporary-mmap path already bound memory for files outside the cache, so coverage and RAM look like separable concerns.

I don't have multi-gigabyte node_modules yet, so I'm not sure how it would work/fail.

If that is too aggressive as a default, an opt-out flag or a per-query includeIgnored: true would already unblock the workflow. The one option I would rather avoid is a manual path whitelist: the interesting directories differ per build tool and per module, and maintaining that list by hand defeats the purpose of a fast finder.

Environment

fff-mcp 0.10.1 (686a849), macOS aarch64 (Darwin 25.5.0), Claude Code as the MCP client.

[dmtrKovalenko]

Unfortunately indexing everything is not an option. Our index is lightweight but standard rust or node js project can easily have millions of files ignored.

[vlsi]

On the filter itself: .gitignore is a poor proxy for "not worth indexing".

Can fff search without index over "ignored" files?
Generated output changes only when a build runs, and a stale hit that the agent then reads from disk is far better than no hit at all.
Looks like the current issue asks for exactly this.

[dmtrKovalenko]

.gitignore is a poor proxy for "not worth indexing"

I disagree, existing tools like ripgrep will never found the files in ignored folders. I've never seen AI successfully figure out how to use rg to find something in node modules cause it will always find the top level git root and use that ignore file