Skip to content

Fix the macOS test hang: send every byte of virtual masks and XY arrays - #60

Merged
CSSFrancis merged 2 commits into
directelectron:mainfrom
CSSFrancis:fix/socket-partial-sends
Sep 24, 2026
Merged

CSSFrancis merged 2 commits into
directelectron:mainfrom
CSSFrancis:fix/socket-partial-sends

Conversation

@CSSFrancis

@CSSFrancis CSSFrancis commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Fixes the macOS CI hang: macos-latest-py3.12 sometimes stalls in test_client.py, then times out every remaining test until the job reaches its 20-minute limit.

Cause

  • Client.__sendToSocket could drop data. It sent virtual masks and XY scan arrays in 4096-byte chunks with sock.send(chunk) and ignored the return value. Once a socket has a timeout, send can accept only part of a chunk. macOS's loopback does this regularly. The unsent bytes were dropped, and after a send timeout the loop also skipped the rest of that chunk. The server then waited for mask bytes that never arrived.
  • The fake server kept broken connections open. After an error it went back to accept() without closing the connection. The session-wide test client sent every later command into that dead connection, and each test waited out --timeout=60.

In CI this showed up as a hang at test_set_virtual_mask, which sends a 1 MB mask.

Changes

  • Client: __sendToSocket uses sendall, with the same timeout as before. The command packets in set_virtual_mask and set_xy_array also use sendall, like the other commands.
  • Simulated server: closes a connection after an error, so the client gets an error instead of waiting.
  • Test suite: close_port now stops only deapi's own simulated server. Before, running the tests on a machine with DE-Server listening on 13240 terminated DE-Server. Now the run stops with a message instead.

Tests

  • New test_fake_server/test_partial_sends.py: wraps the client's socket so send accepts at most 1000 bytes per call, which reproduces the hang on any OS. It covers a virtual mask round trip (followed by one more command, to check the connection is still usable) and the shared send helper over a socket pair.
  • Old client: both new tests hang, even on Windows.
  • This branch: they pass in about 2 s. The full suite, run locally on Windows with --timeout=60, gives 51 passed and 128 skipped.

Also fixed

test_start_acquisition_scan_disabled also failed intermittently on macOS, and the hang used to hide it. With scan off the fake acquisition is one frame, which takes 1 ms at the suite's 1000 fps. It could finish before the test read acquiring. The test now runs at 2 fps, so the acquisition lasts 0.5 s.

@codecov-commenter

codecov-commenter commented Sep 24, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 86.30137% with 10 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
deapi/tests/conftest.py 0.00% 8 Missing ⚠️
deapi/client.py 66.66% 1 Missing ⚠️
deapi/simulated_server/initialize_server.py 0.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

…n connection

socket.send may accept only part of a buffer, and does on macOS's loopback once
the socket has a timeout. __sendToSocket ignored the count (and skipped a chunk
on timeout), so the server waited for mask bytes that never came. The fake
server then went back to accept() with the connection still open, so every
later command hung: one lost chunk became 60 s per remaining test until the
job hit its 20-minute limit.

- __sendToSocket uses sendall; set_xy_array and set_virtual_mask send their
  command packets with sendall too
- the simulated server closes a connection after an error
- close_port stops only deapi's simulated server; a real DE-Server on port 13240
  ends the run with a message instead of being terminated
- tests: a socket that sends at most 1000 bytes per call reproduces the hang on
  any OS
Without a scan it is one frame, 1 ms at 1000 fps, and could finish before the
test read acquiring (the other macOS py3.12 failure).
@CSSFrancis
CSSFrancis force-pushed the fix/socket-partial-sends branch from 7d27745 to d3cb5e2 Compare September 24, 2026 21:29
@CSSFrancis
CSSFrancis merged commit 204b6d6 into directelectron:main Sep 24, 2026
8 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants