tests/int: revamp (drop runc(), use run -N, add asserts)

[kolyshkin]

This supersedes #4946, taking a different approach to the problems
discussed there (see this comment and below).

The end result is that integration tests call runc the way any other
command is called, and check its output with helpers that say what went
wrong:

run -0 runc start ctr
assert_output --partial "something"

instead of the current

runc start ctr
[ "$status" -eq 0 ]
[[ "$output" == *"something"* ]]

where runc is a bash function hiding a call to bats' run helper, and
a failed comparison only shows the expression, not the values.

What a failure looks like now

To see this in practice, four deliberate bugs were introduced (runc -v
printing spec :, runc list -q adding a trailing space to the ID, a
reworded hook error, and runc delete --force failing on a missing
container), and the tests were run with and without this PR:

Bug	Before	Now
list -q adds a trailing space	20 lines of every runc call made so far, ending with an output that looks exactly right	expected: 'test_box1' actual: 'test_box1 '
spec : instead of spec:	the failed expression, plus the whole runc -v output to eyeball	expected: 'spec: [0-9]+\.[0-9]+\.[0-9]+' actual: 'spec : 1.3.0'
reworded hook error	ditto, with the message to compare by hand	expected: 'error running createRuntime hook #1:' actual: '...failure in createRuntime hook #1...'
delete --force fails	[ "$status" -eq 0 ]' failed -- but after which call?	run -0 runc delete --force notexists' failed, expected exit code 0, got 1

The trailing space one is the point of the whole exercise. Before:

not ok 1 list
# (in test file tests/integration/list.bats, line 37)
#   `[ "${lines[0]}" = "test_box1" ]' failed
# runc spec --rootless (status=0)
#
# runc run -d --console-socket /tmp/.../tty/sock test_box1 (status=0)
#
[... 10 more lines of the preceding commands and their output ...]
# runc list -q (status=0)
# test_box1
# test_box2
# test_box3

The expected value is not shown, the actual one looks correct, and the
difference is invisible. Now:

not ok 1 list
# (in test file tests/integration/list.bats, line 32)
#   `assert_line --index 0 "test_box1"' failed
# -- line 0 does not match (exact) --
# expected: 'test_box1'
#   actual: 'test_box1 '
# --

[kolyshkin]

@AkihiroSuda @cyphar @thaJeztah @lifubang @rata PTAL (this changes our bats tests considerably, and so I have to rebase this manually every time we merge something involving bats tests). If you agree with the concept and the implementation, can we please merge this to reduce my load?

One note on reimplementing asserts vs using bats' own -- unfortunately, they don't come with bats-core and the installation is a real PITA, on the other hand the re-implementation is pretty straightforward and probably wouldn't change much. Yet if you don't like it, I can remove the last few commits which add/use asserts.

[thaJeztah]

Sorry, looks like I missed the previous ping; are the failures expected, or something broken in the current PR?

[kolyshkin]

Sorry, looks like I missed the previous ping; are the failures expected, or something broken in the current PR?

@thaJeztah It got broken because since it's been written we have added other tests (for example, in commit 5f76d4d), and they all use the old syntax (like runc ... ; [ $status -eq 1 ] etc instead of run -1 runc).

I am fixing this right now, but this is basically why I keep asking for reviews -- this changes the tests style slightly, and thus I need to maintain it until it's merged.

[kolyshkin]

I'd love this to be included into 1.6 (but not setting the milestone to not block 1.6-rc1 release); while it does touch runc directly (it's just CI), it will reshape the way we're writing the tests and I'd rather have it sooner than later (it's a big patch set and hard to keep updated as we move along).

PTAL @cyphar @thaJeztah @AkihiroSuda

[thaJeztah]

LGTM

[kolyshkin]

A somewhat heavy rebase after merging #5455. I'd rather have this one merged (unless there are any objections) to avoid further rebase work. PTAL @lifubang @AkihiroSuda