Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/TestingCI.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ env:
# The crates that build and test on Windows. Utilities are ported a whole
# crate at a time, because `cargo test -p` builds every binary in the
# crate; add a crate here once all of its binaries and tests are portable.
WINDOWS_CRATES: -p gettext-rs -p plib -p posixutils-xform -p posixutils-text
WINDOWS_CRATES: -p gettext-rs -p plib -p posixutils-xform -p posixutils-text -p posixutils-datetime -p posixutils-calc

jobs:
lint:
Expand Down
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

41 changes: 5 additions & 36 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,42 +98,11 @@ Note that `cargo install` copies the declared binaries and nothing else, so the

### Windows

Two crates build and are tested on Windows, natively with MSVC (no MinGW or
Cygwin runtime): `xform` (`cksum`, `compress`, `uuencode`, `uudecode`) and
`text` (`asa`, `comm`, `csplit`, `cut`, `diff`, `expand`, `fold`, `grep`,
`head`, `join`, `nl`, `paste`, `patch`, `pr`, `sed`, `sort`, `tail`, `tr`,
`tsort`, `unexpand`, `uniq`, `wc`).

```sh
cargo build --release -p posixutils-xform -p posixutils-text
```

Building the whole workspace on Windows does not work: most other utilities
are inherently Unix (users, terminals, signals, file modes and ownership).
What behaves differently on Windows:

- a file's POSIX mode is its read-only attribute, read as the owner-write bit
(`0444` or `0644`); setting a mode without owner write makes it read-only;
- `compress` restores permissions and times but not ownership, and does not
warn about hard links; the `zcat` and `uncompress` aliases do not exist
(use `compress -c -d` and `compress -d`);
- `LC_ALL`, `LC_*` and `LANG` set to `C` or `POSIX` select the C locale:
ASCII-only character classes and case mapping, one byte per character, and
byte-order collation; `C.UTF-8` (or any `C`/`POSIX` name with a codeset)
collates in byte order but reads UTF-8 and classifies and case-maps by
Unicode; any other value, or none, is the user's locale with UTF-8 input and
Unicode characters;
- POSIX regular expressions are musl's (vendored in `plib/vendor/musl-regex`)
and do not support characters above U+FFFF;
- `sort -n` always takes `.` as the decimal point, with no thousands
separator;
- `diff` reports anything that is neither a file nor a directory as a
special file, and recognises a directory loop by its canonical path;
- `pr -p` and `patch` prompt on the console; `csplit` removes its files on
Ctrl-C, Ctrl-Break and termination, Windows having no hangup or quit
signal.

[WINDOWS.md](WINDOWS.md) is the guide to porting another crate.
Some crates build and are tested on Windows, natively with MSVC (no MinGW or
Cygwin runtime); building the whole workspace there does not work, as most
other utilities are inherently Unix. [WINDOWS.md](WINDOWS.md) lists the
crates and utilities that build there, how they behave differently, and how
another crate is ported.

### Container image

Expand Down
104 changes: 85 additions & 19 deletions WINDOWS.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,64 @@
# Porting a crate to Windows
# Windows

How a workspace crate is made to build, and its tests to pass, on Windows
(`x86_64-pc-windows-msvc`), and how it then joins CI. Follow it in order; each
step names what to check before going on.
What builds on Windows (`x86_64-pc-windows-msvc`) and how it behaves there,
then how another workspace crate is ported.

## Rules
## Using the utilities on Windows

These crates build and are tested on Windows, natively with MSVC (no MinGW or
Cygwin runtime):

| Crate | Utilities |
|---|---|
| `calc` | `bc`, `expr` |
| `datetime` | `cal`, `date`, `sleep`, `time` |
| `text` | `asa`, `comm`, `csplit`, `cut`, `diff`, `expand`, `fold`, `grep`, `head`, `join`, `nl`, `paste`, `patch`, `pr`, `sed`, `sort`, `tail`, `tr`, `tsort`, `unexpand`, `uniq`, `wc` |
| `xform` | `cksum`, `compress`, `uuencode`, `uudecode` |

```sh
cargo build --release -p posixutils-calc -p posixutils-datetime -p posixutils-text -p posixutils-xform
```

What behaves differently on Windows:

- a file's POSIX mode is its read-only attribute, read as the owner-write bit
(`0444` or `0644`); setting a mode without owner write makes it read-only;
- `compress` restores permissions and times but not ownership, and does not
warn about hard links; the `zcat` and `uncompress` aliases do not exist
(use `compress -c -d` and `compress -d`);
- `LC_ALL`, `LC_*` and `LANG` set to `C` or `POSIX` select the C locale:
ASCII-only character classes and case mapping, one byte per character, and
byte-order collation; `C.UTF-8` (or any `C`/`POSIX` name with a codeset)
collates in byte order but reads UTF-8 and classifies and case-maps by
Unicode; any other value, or none, is the user's locale with UTF-8 input and
Unicode characters;
- POSIX regular expressions are musl's (vendored in `plib/vendor/musl-regex`)
and do not support characters above U+FFFF;
- `sort -n` always takes `.` as the decimal point, with no thousands
separator;
- `diff` reports anything that is neither a file nor a directory as a
special file, and recognises a directory loop by its canonical path;
- `pr -p` and `patch` prompt on the console; `csplit` removes its files on
Ctrl-C, Ctrl-Break and termination, Windows having no hangup or quit
signal;
- `date` and `cal` name months and days as the POSIX locale does, whatever
`LC_TIME` says; `TZ` takes the C runtime's forms (`UTC0`, `EST5EDT`: a
three-letter name, an offset, an optional daylight name) and not zoneinfo
names such as `America/New_York`; unset, it is the system time zone;
- `date` sets the clock only with the system-time privilege, and reads the
local time it is given in the system time zone, not `TZ`;
- `time` reports its own CPU time plus the utility's, Windows keeping no
totals for a process's descendants;
- `expr` operands are text: a Windows command line cannot carry bytes that
are not UTF-8.

## Porting a crate

How a workspace crate is made to build, and its tests to pass, on Windows,
and how it then joins CI. Follow it in order; each step names what to check
before going on.

### Rules

- **A whole crate at a time.** `cargo test -p <crate>` builds every binary in
the crate, so a crate is ported when *all* of its binaries and tests compile
Expand All @@ -24,7 +78,7 @@ step names what to check before going on.
(`mode_of`, `set_mode`, `remove_file`, `link_count`), called from shared
code. Never scatter `cfg` through a function body twice for the same rule.

## Windows meaning of Unix concepts
### Windows meaning of Unix concepts

| Unix | Windows |
|---|---|
Expand Down Expand Up @@ -52,10 +106,15 @@ step names what to check before going on.
| FIFOs, devices, sockets | none: neither a file nor a directory is "special" |
| `SIGQUIT` | `SIGBREAK` (Ctrl-Break) where a quit key is meant |
| absolute paths | a root (`\x`), a drive prefix (`C:x`) or `..` must all be refused where only relative names are allowed |
| `localtime_r`, `gmtime_r`, `TZ` | the C runtime's `localtime_s` / `gmtime_s`, which read `TZ` in its own forms (no zoneinfo); the zone name is its `strftime("%Z")` |
| `strftime`, `LC_TIME` | `plib::timefmt`: the POSIX locale's names and formats, `LC_TIME` ignored |
| `clock_settime` | `SetSystemTime` (needs the system-time privilege) |
| `times()` children's CPU | `GetProcessTimes` on the waited-for child's handle, added to the process's own |
| argv as bytes (`OsStrExt::as_bytes`) | `OsStr::as_encoded_bytes`, portable: the same bytes on Unix, UTF-8 on Windows |

## Steps
### Steps

### 1. Survey
#### 1. Survey

```sh
cargo check --target x86_64-pc-windows-msvc -p <crate> --all-targets 2>&1 | grep -E '^error'
Expand All @@ -67,26 +126,26 @@ Sort the errors into: missing `plib` modules (step 2), the crate's own sources
Windows meaning; a binary that cannot be ported keeps the whole crate off
Windows (or the crate is split), never stubbed.

### 2. plib
#### 2. plib

`plib` is the base of every crate. Its modules with no Windows meaning are
`#[cfg(unix)]` in `plib/src/lib.rs`; a crate that needs one ports that module
(or the part it uses) first, as its own commit, with the module's unit tests
running on Windows. Ported so far: `diag`, `io` (including the terminal for
prompts), `locale` (characters and case), `lzw`, `regex`, `testing`,
prompts), `locale` (characters, case and `strftime`), `lzw`, `regex`, `testing`,
`archive`, `cscan`, `linediff`. Still
Unix-only: `curuser`, `exec`, `group`, `modestr`, `platform`, `priority`,
`projectdir`, `sccsfile`, `syslog`, `test_expr`, `tmp`, `tty`, `user`, `utmpx`.

### 3. The crate's sources
#### 3. The crate's sources

Apply the table above through small cfg'd helpers. Prefer a portable std API
that serves both platforms when it is exactly equivalent on Unix. Read every
`libc::` and `std::os::unix` use; `cargo check --target ...` finds them, but
not the ones that compile and mean something else (a path with `/dev/stdout`,
a `:`-separated list).

### 4. Tests
#### 4. Tests

- Integration tests find binaries with `plib::testing::get_binary_path`, which
already handles `.exe` and `--target` layouts.
Expand All @@ -101,7 +160,7 @@ a `:`-separated list).
- Windows will not delete a read-only file under Wine: clear the attribute
before cleanup.

### 5. Verify
#### 5. Verify

Every commit, never two cargo commands at once:

Expand All @@ -128,17 +187,24 @@ A change to `plib::testing` or anything every crate uses gets the full
Wine's limits: the `-gnu` target links `msvcrt`, which refuses the UTF-8
locale, so UTF-8 multibyte regex behaviour is exercised only by CI's MSVC
build; and Wine refuses to delete a read-only file even where current Windows
does, which is the stricter behaviour to be correct against.
does, which is the stricter behaviour to be correct against. Three
`datetime` tests fail under Wine 9 and pass only on Windows: its `msvcrt`
parses `TZ` wrongly (`TZ=UTC` names the zone `UT`), failing `date`'s
`test_tz_utc` and `test_default_format_utc`; and its `GetProcessTimes`
answers the caller's own times for another process, failing `time`'s
`cpu_bound_child_reports_nonzero_cpu_time`. Wine maps `/dev/full` to the
Linux device, so the tests that write to it run there and not on Windows.

### 6. CI and docs
#### 6. CI and docs

Add `-p <crate>` to `WINDOWS_CRATES` in `.github/workflows/TestingCI.yml`; the
`windows` job, the lint job's Windows clippy and the `msrv` job's Windows
check all read it. List the crate's utilities in README's "Windows" section,
with anything that behaves differently there, and add any new Unix→Windows
meaning to the table above. Ported so far: `xform`, `text`.
check all read it. Add the crate and its utilities to the table under "Using
the utilities on Windows" and to its build command, anything that behaves
differently to the list there, and any new Unix→Windows meaning to the table
above. README only points here: it is not edited per crate.

### 7. Commits
#### 7. Commits

A bisectable series, each commit building and passing clippy on both targets
on its own: plib modules first, then the crate's sources, then its tests,
Expand Down
19 changes: 12 additions & 7 deletions calc/bc.rs
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ fn main() {
}
Err(e) => {
diag::init("bc");
diag::error(&format!("{}", e));
diag::error(&diag::io_error_text(&e));
std::process::exit(1);
}
}
Expand All @@ -92,6 +92,11 @@ fn report(e: impl std::fmt::Display) -> bool {
true
}

/// Report a failed write as the system names it. Returns true, as [`report`].
fn report_io(e: std::io::Error) -> bool {
report(diag::io_error_text(&e))
}

/// Report each diagnostic of a parse failure at its own position.
fn report_parse_error(e: &ParseError) -> bool {
for (line, col, message) in e.diagnostics() {
Expand Down Expand Up @@ -164,14 +169,14 @@ fn run() {
had_error |= report(e);
}
if let Err(e) = out.flush() {
had_error |= report(e);
had_error |= report_io(e);
}
}
Err(e) => had_error |= report_parse_error(&e),
}
if interpreter.has_quit() {
if let Err(e) = out.flush() {
had_error |= report(e);
had_error |= report_io(e);
}
std::process::exit(if had_error { 1 } else { 0 });
}
Expand All @@ -180,7 +185,7 @@ fn run() {
let mut repl = match DefaultEditor::new() {
Ok(repl) => repl,
Err(e) => {
diag::error(&format!("{}", e));
diag::error(&diag::error_text(&e));
std::process::exit(1);
}
};
Expand All @@ -201,7 +206,7 @@ fn run() {
failed = report(e);
}
if let Err(e) = out.flush() {
failed = report(e);
failed = report_io(e);
}
had_error |= failed && !interactive;
line_buffer.clear();
Expand All @@ -218,7 +223,7 @@ fn run() {
// End of input (Ctrl-D) or interrupt (Ctrl-C): exit silently.
Err(ReadlineError::Eof) | Err(ReadlineError::Interrupted) => break,
Err(e) => {
had_error |= report(format!("{:?}", e));
had_error |= report(diag::error_text(&e));
break;
}
}
Expand All @@ -234,7 +239,7 @@ fn run() {
}

if let Err(e) = out.flush() {
had_error |= report(e);
had_error |= report_io(e);
}
std::process::exit(if had_error { 1 } else { 0 });
}
2 changes: 1 addition & 1 deletion calc/bc_util/interpreter.rs
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ impl From<&'static str> for ExecutionError {
impl From<io::Error> for ExecutionError {
fn from(e: io::Error) -> Self {
Self {
message: format!("cannot write output: {e}"),
message: format!("cannot write output: {}", plib::diag::io_error_text(&e)),
call_stack: Vec::new(),
}
}
Expand Down
8 changes: 4 additions & 4 deletions calc/expr.rs
Original file line number Diff line number Diff line change
Expand Up @@ -152,13 +152,13 @@ fn parse_token(arg: Vec<u8>) -> Token {

// tokenize the command line arguments, all in a single pass
fn tokenize() -> Vec<Token> {
use std::os::unix::ffi::OsStrExt;

// POSIX operands are byte strings: a pathname or a compared string need
// not be text, and decoding argv as UTF-8 aborts on one that is not.
// not be text, and decoding argv as UTF-8 aborts on one that is not. On
// Unix these are the argument's own bytes; Windows arguments are UTF-16,
// read as UTF-8.
let mut args: Vec<Vec<u8>> = std::env::args_os()
.skip(1)
.map(|arg| arg.as_bytes().to_vec())
.map(|arg| arg.as_encoded_bytes().to_vec())
.collect();

// POSIX / XBD 12.2 Guideline 10: a leading "--" delimits the end of
Expand Down
Loading
Loading