yes: support invalid unicode on Windows

[oech3]

One of a common usecase of yes is generating periodic binary data which is blocked by

coreutils/src/uu/yes/src/yes.rs

Lines 73 to 82 in 1717352

	// But, on Windows, we must hop through a String.
	#[cfg(not(any(unix, target_os = "wasi")))]
	{
	for part in itertools::intersperse(i.map(|a| a.to_str()), Some(" ")) {
	let bytes = part
	.ok_or_else(|| USimpleError::new(1, translate!("yes-error-invalid-utf8")))?
	.as_bytes();
	buf.extend_from_slice(bytes);
	}
	}

currently.

[xtqqczze]

But does Windows stdio support non-UTF-8 byte sequences? Maybe not in console mode?

[oech3]

I think stdout converted to File can do that as same as dd.

[xtqqczze]

I think stdout converted to File can do that as same as dd.

Uh right, well I guess we can't do that unless the console's code page is set to UTF-8.

[xtqqczze]

Related:

• Reading from and writing to the Windows console should be lossy rust-lang/rust#116871
• Windows: Don't error on broken non UTF-8 output rust-lang/rust#134534

[oech3]

If yes > file generates valid binary. It is useful enough.

[xtqqczze]

So we just need to careful if we are in console mode with the code page set to UTF-16.

[ChrisDenton]

My suggestion would be to handle the simpleyes > file case first and leave the harder console case until later. A Windows console can be detected by calling GetConsoleMode on a handle and seeing if it succeeds.

Though tbh I don't understand the utility of outputting binary data directly to any console (without piping through something else to make it human-readable). While it would be nice to have full compatibility, I don't think it's a big priority here.

[oech3]

Which content should we write to_string_lossy or as_encoded_bytes when stdout().is_terminal()?

[ChrisDenton]

Which content should we write to_string_lossy or as_encoded_bytes when stdout().is_terminal()?

At the moment is should use to_string_lossy because std may error for non-unicode bytes. In the future this may be fixed so we can use as_encoded_bytes.

[oech3]

If std already validate and reject it with proper error message, we canuse as_encoded_bytes now.