trycmd: Document `fs.sandbox` and working directory behavior

[kpreid]

The documentation of trycmd makes a couple of references to fs.sandbox = true, but does not explain what that option does, nor what is done instead if it is false. It mentions the “full schema” but that has no specific documentation for fs.sandbox. The documentation doesn't answer these questions:

• If fs.sandbox is not set and *.out/ does not exist, what is the CWD? From the misbehavior I was observing in my own tests, it almost seems like it is nondeterministic (perhaps inheriting a previous test's output directory?).
• Does it mean just creating a temporary directory, or does it mean using actual OS features to constrain or redirect writes in some manner?

I think I now correctly understand that if the binary under test writes an output file to a relative path, and that output file is to be ignored rather than compared, then I should set fs.sandbox = true. But that was not obvious when I was setting up some of my tests, particularly since the output file usually went apparently nowhere.

[epage]

If fs.sandbox is not set and *.out/ does not exist, what is the CWD? From the misbehavior I was observing in my own tests, it almost seems like it is nondeterministic (perhaps inheriting a previous test's output directory?).

From the docs:

*.in/
When present, this will automatically be picked as the CWD for the command.

We don't specify the default outside of that condition though I beleive it should be the same CWD as your tests generally, CARGO_MANIFEST_DIR.

Does it mean just creating a temporary directory, or does it mean using actual OS features to constrain or redirect writes in some manner?

Creating a temp directory

I think I now correctly understand that if the binary under test writes an output file to a relative path, and that output file is to be ignored rather than compared, then I should set fs.sandbox = true. But that was not obvious when I was setting up some of my tests, particularly since the output file usually went apparently nowhere.

The docs about this are assuming you have a *.in/ to specify files specifically for that test. It says

Tests are assumed to not modify files in *.in/ unless an *.out/ is provided or fs.sandbox = true is set in the .toml file.

(having a *.out/ sets fs.sandbox = true by default)

btw a similar discussion about these interactions is happening in #64.

I thought we had an issue for the fact that *.out/ is only ever a subset of your output, even on first write. You need dump to help set things up.

[kpreid]

We don't specify the default outside of that condition

I mean to claim that it should be at least documented as unspecified, for clarity.

I beleive it should be the same CWD as your tests generally, CARGO_MANIFEST_DIR.

If that were the whole story, I wouldn't be seeing the inconsistent behavior I saw. (I don't mean to make this a bug report, though.)

The docs about this are assuming you have a *.in/ to specify files specifically for that test.

FYI, in my case, I was testing commands that read no files and only wrote files. So, there was no *.in/. I can see that the existing behavior is less surprising if use of *.in/ is common, though.