feature: ignore outputs in trycmd, only check successful exit

[bryango]

This would be useful when dealing with non-reproducible commands (where the output may change frequently). In this case, checking against the output is unrealistic, but it is still useful to validate that e.g. all the examples in README run successfully.

Context: I am writing test cases for a web scraper.

[bryango]

A proposal: turn on the "check status only" mode through the env var TRYCMD=status. I've provided a quick realization of this proposal in #376 (pending discussions).

[epage]

Can the output be made reproducible through redactions, including globs?

If the output shouldn't be checked, shouldn't that be a default state, rather than relying on an env variable to opt-out?

Are you wanting the ability to force an overwrite of the output?

[bryango]

If the output shouldn't be checked, shouldn't that be a default state, rather than relying on an env variable to opt-out?

Are you wanting the ability to force an overwrite of the output?

Yes precisely! I would like TRYCMD=status for continuous testing, but also TRYCMD=overwrite to update the snapshot once in a while.

For my specific use case: I would like CI to ensure that the examples in my README continue to run successfully, but also reserve the ability to update them one in a while. AFAIK trycmd is the only library that can do this to a README file magically.

Can the output be made reproducible through redactions, including globs?

I do use some redactions, but since it's a user facing README, too much redactions would lead to a poor demonstration of the program (a web scraper).

[epage]

If we go forward with this, it seems like a state that should be put as a code fence attribute or in the trycmd API.

I have considered adding a "force update" value that would write output independent of test status. That seems like it'd be a good fit with this. The problem I've had is overwrite would be a good name for this (with the current behavior being update or bless) but I'm concerned about deviating from trybuild and the user disruption that renaming the standard value would have.