fix(input): reliable synthetic pointer engine — single activation, timestamps, drag hold, hittable flag, screenshot geometry

[pierre-cheneau]

Problem

Emulated input is unreliable in ways that make agent-driven testing untrustworthy. Reproduced on Windows (Flutter 3.44.9) with a probe app whose state is mirrored into a Text, so every check below is asserted against the app's own state, not screenshots (tools/harness_interaction.py in the linked repro):

1. Taps activate twice. tap(key|text|ref) dispatches a synthetic pointer tap and then invokes the widget's onTap/onPressed directly as a "fallback". Any hittable target fires twice — a counter goes +2, an expand/collapse toggle visibly does nothing. tap(text) only looked right because a Text has no callback to re-fire.
2. Pointer events carry no timestamps. All emitted PointerDown/Move/Up had timeStamp = 0. Flutter's VelocityTracker and scroll physics derive velocities from timestamps, so swipe on a PageView/TabBarView either snaps back (< half page) or is left mid-transition; on a real app the stuck drag then swallowed every later tap until a restart. Same in drag, swipe_coordinates, long press. The pointer was also never released if a handler threw.
3. Actions return on fixed sleeps, not frames. Future.delayed(300ms) after the input; on a heavier app the state query issued right after still saw the pre-tap tree.
4. Hidden window = hangs. Actions and screenshots await endOfFrame; a minimized/occluded window renders no frames on Windows, so calls hung for minutes.
5. drag cannot do long-press drags (LongPressDraggable, long-press-to-select lists): it moved immediately after pointer-down. The live schema (engine/tool_registry.dart) had no coordinate form; tool_definitions.dart is a stale second copy.
6. visible ≠ reachable. inspect/inspect_interactive reported cards under a bottom bar / FAB, or scrolled off-screen, as visible: true; a drag started there landed on the bar.
7. Screenshots carry no geometry. Images are scaled by max_width while tap_at/swipe take logical coordinates — an easy trap for agents (I fell into it).
8. Indicator particle painter asserted every frame (Color.withOpacity with a negative alpha): 1000+ entries polluting get_errors.

Fix

SDK (lib/flutter_skill.dart)

• One synthetic-pointer engine, _emitGesture(from, to, hold, travel), used by tap, long press, swipe, swipe_coordinates, drag and double tap: monotonically increasing timeStamps from a Stopwatch, ~one move per frame, PointerUp in finally, then _settle() — two frames (the one that runs the callbacks and the one that paints the result), each bounded to 500 ms. When no frame arrives (window minimized/hidden) the action returns promptly and its response carries warning: "No frame was rendered after this action; the app window is probably minimized or hidden…".
• tap(key|text|ref): _isHittableAt(element, center) (RendererBinding.hitTestInView) decides — reachable → pointer only; covered/pointer-ignored → direct callback. Exactly one activation.
• inspect / inspect_interactive gain hittable (hit test at the element's center); visible keeps its old meaning ("has a size").
• drag and swipeCoordinates accept hold (ms).
• Screenshot ext returns imageWidth/Height, logicalWidth/Height, devicePixelRatio.
• Particle painter alpha clamped.

Server

• drag tool: start_x/start_y/end_x/end_y, hold_ms, duration_ms in the live registry (and the legacy copy); numeric args arriving as strings are accepted.
• screenshot adds image_size, logical_size, device_pixel_ratio, scale and a coordinates_hint.
• tap/tap_at/swipe/drag responses pass the SDK warning through.

Verification

Probe app (TabBar/TabBarView, ListView with a toggling group, counter button, LongPressDraggable → DragTarget, root Listener recording pointer-down positions) driven over MCP stdio; indicators on and off.

Scenario	stock 0.9.36	this PR
tap_at accuracy at 4 screen points (incl. under the indicator bar)	8/8	8/8
single activation: text / key / ref / coordinates	2/6	8/8
tab switch by text and by coordinates	4/4	4/4
swipe → next tab, settled	0/2	2/2
state read right after an action is fresh	4/4	4/4
long-press drag onto DragTarget (keys, coordinates)	n/a	4/4
screenshot geometry	0/2	2/2
total	20/28	32/32

Also on a real app (Strat·It Playbook, every move undone): template → group header, template → section header, strategy → folder, strategy → strategy (opens create-folder dialog), batch drag in selection mode, edge auto-scroll up/down. Actions got 3–4× faster (frame-settled instead of 300 ms sleeps). Minimized-window case verified: actions return in ~1 s with the warning instead of hanging.

dart analyze clean on the touched files.

Notes for reviewers

• Behaviour change: the direct-callback fallback now only runs when the target is not reachable by a pointer. If a caller relied on double invocation, that was the bug.
• SDK edits to registerExtension handlers need a hot restart in the running app (hot reload swaps method bodies but not the registered closures).
• Stacked on fix(hot-reload): drive Flutter Tools' reloadSources/hotRestart VM services so hot_reload/hot_restart actually apply changes #58 (hot reload/restart via Flutter Tools services); both touch flutter_driver.dart. Happy to rebase onto main once fix(hot-reload): drive Flutter Tools' reloadSources/hotRestart VM services so hot_reload/hot_restart actually apply changes #58 lands.

[pierre-cheneau]

Pushed a second commit (7e911fe) from a sweep of the remaining tool families against the probe app (54 checks, all passing): real get_current_route/get_navigation_stack (the SDK tested element.widget is ModalRoute, never true; the server read the null result as "Unknown tool"), working press_key (KeyData was queued in KeyEventManager waiting for a raw-key message; keys now go through HardwareKeyboard + the KeyMessage handler with correct physical keys), Flutter implementations for type_text/focus/blur/set_checkbox/hover, scroll_to for lazily built rows, scroll_until_visible direction semantics + ensureVisible, and a settle between execute_batch steps.