Skip to content

Fix configNUMBER_OF_CORES never being defined, silently disabling SMP - #20

Open
insolace wants to merge 2 commits into
midi2-dev:masterfrom
Muse-Kinetics:fix-smp-core-count
Open

insolace wants to merge 2 commits into
midi2-dev:masterfrom
Muse-Kinetics:fix-smp-core-count

Conversation

@insolace

@insolace insolace commented Aug 24, 2026

Copy link
Copy Markdown

Summary

FreeRTOSConfig.h only defined configNUM_CORES, which comes from an older pre-merge FreeRTOS SMP branch. The vendored FreeRTOS kernel and RP2040 SMP port both use configNUMBER_OF_CORES instead (FreeRTOS.h, portmacro.h), so the system has been silently defaulting to single-core execution.

This means the codebase has likely never actually been running in SMP mode, despite clearly intending to. main.c already contains a multicore_launch_core1() call and an "on both cores" log message, but both are gated behind portSUPPORT_SMP, another leftover macro from the old SMP branch that isn't defined anywhere in the current kernel.

Defining configNUMBER_OF_CORES enables the actual SMP path. Once it's set to portMAX_CORE_COUNT, the RP2040 FreeRTOS port launches the second core from xPortStartScheduler() automatically, so the dead portSUPPORT_SMP path in main.c is never used.

Enabling SMP also requires two additional configuration options:

  • configUSE_PASSIVE_IDLE_HOOK, which FreeRTOS requires whenever configNUMBER_OF_CORES > 1
  • configUSE_CORE_AFFINITY, which is required by pico_flash when PICO_FLASH_SAFE_EXECUTE_SUPPORT_FREERTOS_SMP is enabled so it can safely determine task placement before pausing all cores for flash operations

Testing

Verified on real dual-core RP2040 hardware using the UUT_FreeRTOS/USB_MIDI_Echo target. Both cores were confirmed running over SWD, and USB enumeration completed normally.

The remaining targets that share this configuration were verified to build successfully, but were not flashed or tested on hardware as part of this change. Since this re-enables an SMP code path that appears to have never actually been exercised in this repository, I'd recommending we get more testing before merging.

FreeRTOSConfig.h only defined configNUM_CORES, an older name from a
pre-merge FreeRTOS SMP branch. The vendored FreeRTOS-Kernel SMP port and
RP2040 port both check configNUMBER_OF_CORES (see FreeRTOS.h:96,
portmacro.h), which silently defaulted to 1 without it being set. This
codebase has been running single-core the entire time despite clearly
intending dual-core -- main.c already has a multicore_launch_core1 call
and an "on both cores" printf, both gated on the equally-stale
portSUPPORT_SMP macro (also from the same pre-merge branch, not defined
anywhere in this vendored kernel).

Defining configNUMBER_OF_CORES correctly reactivates genuine dual-core
execution: FreeRTOS-Kernel's own xPortStartScheduler() calls
multicore_launch_core1() automatically once configNUMBER_OF_CORES ==
portMAX_CORE_COUNT, independent of main.c's now-dead portSUPPORT_SMP
branch. Two more defines are newly required once real SMP is active and
had to be added alongside it: configUSE_PASSIVE_IDLE_HOOK (FreeRTOS.h
requires it when configNUMBER_OF_CORES > 1) and configUSE_CORE_AFFINITY
(required by pico_flash's PICO_FLASH_SAFE_EXECUTE_SUPPORT_FREERTOS_SMP,
which needs a way to know how tasks are pinned before it can safely pause
every core for a flash write).

Testing scope: verified booting and running correctly on real dual-core
hardware (both cores confirmed running via SWD, clean USB enumeration)
on the UUT_FreeRTOS/USB_MIDI_Echo target. The other targets sharing this
header were confirmed to still build, but have not been flashed/verified
running on real dual-core hardware as part of this change -- worth a
broader smoke test before relying on this for other targets, since this
reactivates a code path (genuine SMP) that appears to have never
actually executed in this codebase's history.
@insolace
insolace marked this pull request as ready for review August 24, 2026 06:22
@AmeNote-Michael

Copy link
Copy Markdown
Contributor

Reviewed this — root cause and fix location are right (configNUMBER_OF_CORES is indeed what FreeRTOS.h/the RP2040 port check, configNUM_CORES is a dead pre-merge SMP-branch name), and configUSE_PASSIVE_IDLE_HOOK/configUSE_CORE_AFFINITY are genuinely required once configNUMBER_OF_CORES > 1 (confirmed against FreeRTOS.h's own #errors). One gap worth resolving before merge, plus two minor nits:

1. Fix scope is incomplete — ProtoZOA_Main isn't actually patched (confirmed).
This PR only touches Common/include/FreeRTOSConfig.h, but ProtoZOA_Main has its own separate FreeRTOSConfig.h that shadows it: ProtoZOA_Main/CMakeLists.txt's target_include_directories lists . before ../Common (and it's ../Common, not ../Common/include, so the Common copy isn't even reachable from there). So #include "FreeRTOSConfig.h" in that target resolves to ProtoZOA_Main/FreeRTOSConfig.h, which still only defines configNUM_CORES — no configNUMBER_OF_CORES.

The catch: ProtoZOA_Main/main.c has the exact same dead portSUPPORT_SMP / multicore_launch_core1 pattern described in this PR's writeup as the symptom of the bug. So after merging, the flagship ProtoZOA_Main firmware keeps silently running single-core, while the PR reads like it fixes the SMP bug repo-wide. Worth either patching ProtoZOA_Main/FreeRTOSConfig.h the same way (and testing on real hardware, since that's the main board), or explicitly scoping the PR title/description to the targets it actually covers.

(ProtoZOA_PicoProbe's two FreeRTOSConfig.h copies are separate too, but those look intentionally single-core already — configNUM_CORES=1 — so no concern there.)

2. Nit: alignment. configUSE_PASSIVE_IDLE_HOOK's value sits one column off from every other #define in the file (extra space).

3. Nit: comment slightly inaccurate. The comment says configNUM_CORES is kept "only because main.c still references it in a dead #if branch" — but pico-sdk's pico_async_context_freertos.c/.h also gate SMP behavior on configNUM_CORES (not the new macro name). Nothing currently links that component in this repo, so no active bug, but the comment could mislead a future cleanup into deleting the macro once that assumption changes.

🤖 Generated with Claude Code

Addresses AmeNote-Michael's review on PR midi2-dev#20
(midi2-dev#20, comment 5404943130):

1. ProtoZOA_Main/CMakeLists.txt's target_include_directories lists
   '.' before '../Common', and references '../Common' (not
   '../Common/include'), so #include "FreeRTOSConfig.h" in that
   target resolves to ProtoZOA_Main's own copy, not Common's --
   confirmed directly. That copy still only defined configNUM_CORES,
   so the flagship ProtoZOA_Main firmware kept running single-core
   even after the original fix, despite main.c having the identical
   dead portSUPPORT_SMP/multicore_launch_core1 pattern described as
   this bug's symptom. Patched the same way as Common's copy:
   configNUMBER_OF_CORES, configUSE_PASSIVE_IDLE_HOOK, and
   configUSE_CORE_AFFINITY added (the latter two confirmed mandatory
   once configNUMBER_OF_CORES > 1, per FreeRTOS.h's own #error
   checks. Verified ProtoZOA_Main builds clean with this change
   (cmake --build . --target ProtoZOA_Main) -- not yet flashed to
   real hardware.
2. Fixed configUSE_PASSIVE_IDLE_HOOK's column alignment in Common's
   copy (was 2 columns off from every other #define in the file).
3. Expanded the configNUM_CORES retention comment (both copies) to
   also note pico-sdk's pico_async_context_freertos component gates
   SMP on configNUM_CORES specifically -- not currently linked into
   this repo, so not an active bug, but a future cleanup shouldn't
   delete the macro on the assumption that main.c's dead #if branch
   is the only consumer.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
EOF
)
@insolace

Copy link
Copy Markdown
Author

Thanks for the thorough review -- confirmed the include-path claim directly (ProtoZOA_Main/CMakeLists.txt does list . before ../Common, and it's ../Common not ../Common/include, so ProtoZOA_Main/FreeRTOSConfig.h genuinely shadows the Common copy). Pushed a follow-up commit addressing all three points:

  1. Scope gap fixed -- ProtoZOA_Main/FreeRTOSConfig.h now gets the same treatment as Common/include/FreeRTOSConfig.h: configNUMBER_OF_CORES, configUSE_PASSIVE_IDLE_HOOK, and configUSE_CORE_AFFINITY added, matching the comment explaining why configNUM_CORES stays. Confirmed ProtoZOA_Main/main.c has the identical dead portSUPPORT_SMP/multicore_launch_core1 pattern this PR's writeup describes, so this activates real dual-core there too. Verified it builds clean (cmake --build . --target ProtoZOA_Main) against this repo's existing configured build dir -- not yet flashed/verified on real hardware, same caveat as the original PR description for the other 7 targets.
  2. Alignment nit fixed -- configUSE_PASSIVE_IDLE_HOOK now lines up with the rest of the file's #define column.
  3. Comment updated -- now also notes pico_async_context_freertos gates SMP on configNUM_CORES specifically, so a future cleanup shouldn't delete that macro just because main.c's dead branch looks like the only consumer.

ProtoZOA_PicoProbe's two configs are untouched, per your note that they're intentionally single-core already.

@AmeNote-Michael

Copy link
Copy Markdown
Contributor

Tested this on real ProtoZOA hardware (UUT board via the picoprobe on the Main Pico), flashing UUT_FreeRTOS/USB_MIDI_Echo. Wanted to share what I found — the config fix itself is correct, but on this board dual-core execution doesn't actually stick at full speed.

Build: compiles clean, all 8 firmware targets link (ProtoZOA_Main, ProtoZOA_PicoProbe, all UUT/* variants, and both FreeRTOS targets).

Hardware behavior: flashed UUT_USB_MIDI_ECHO and inspected both cores directly over SWD (not just serial logs). Core0 runs fine — serial shows the expected boot sequence and OpenOCD/GDB confirm it's actively ticking through the scheduler. But core1 never actually stays running. Halting both cores and reading registers (via raw OpenOCD register reads across multiple reset cycles, and independently via GDB's -rtos hwthread thread view) repeatedly shows core1 parked at PC=0x00000184, SP=0x20041f00, LR=0x0000015d — the RP2040 bootROM's wait_for_vector loop, i.e. its pre-launch state. This was reproducible across several power/reset cycles.

Traced the launch path under a debugger (breakpoints at multicore_launch_core1_raw, core1_trampoline, prvDisableInterruptsAndPortStartSchedulerOnCore, xPortStartSchedulerOnCore, vPortStartFirstTask): the entire handoff chain succeeds step-by-step — core1 gets the FIFO handshake, jumps through the trampoline, and reaches vPortStartFirstTask correctly every time when execution is slowed by breakpoints.

That contrast (works when single-stepped/breakpointed, fails at full speed, always landing at the exact same bootROM address) points to a race condition in the SMP bring-up rather than a config problem — most likely FIFO/IRQ contention during the handoff window, since both cores install a prvFIFOInterruptHandler on their own SIO IRQ inside xPortStartSchedulerOnCore() right after the handshake completes, and core0 may race ahead into its own setup while core1 is still mid-handoff.

I wasn't able to pin down the exact faulting instruction — further fine-grained single-stepping on the SMP target wedged my OpenOCD/picoprobe session (needed a physical power-cycle to recover), so I stopped rather than keep stressing the debug link. Given this is a genuinely hardware-reproducible issue with dual-core execution (the whole point of this PR), I'd want this run down further — or at minimum stress-tested more on a few different boards — before merging.

Happy to share the exact GDB scripts I used if useful for reproducing.

@insolace

Copy link
Copy Markdown
Author

Reproduced this independently on our own ProtoZOA UUT (UUT_USB_MIDI_ECHO via picoprobe on the Main Pico -- same target/setup described above), and dug further into it. Summary: confirmed your repro precisely, tried two candidate fixes (both ineffective), and localized the failure a lot more precisely than "somewhere in SMP bring-up" -- but don't have a working fix.

Repro: 100% reproducible with a debugger fully detached during reset (separate fresh SWD connection afterward purely to inspect), which avoids the masking effect you noted. Consistently lands at PC=0x00000178, SP=0x20041f00, LR=0x0000014f -- essentially identical to your PC=0x00000184 (few-byte offset consistent with a slightly different build), same SP exactly.

Fix #1 tried -- ineffective. FreeRTOS-Kernel upstream commit f0d79459d (PR #1174, "Fix SMP debugging issue on rp2040") adds multicore_reset_core1() immediately before multicore_launch_core1() in xPortStartScheduler(). Applied it, confirmed via disassembly it compiled in at the correct call site and ordering, reflashed -- 4/4 standalone trials still hit the identical failure signature.

Fix #2 tried (own hypothesis) -- also ineffective, but worth flagging separately. exception_set_exclusive_handler() in pico-sdk 2.3.0 (used for PENDSV_EXCEPTION/SVCALL_EXCEPTION) takes only a local interrupt mask (save_and_disable_interrupts()), not the cross-core spinlock its sibling irq_set_exclusive_handler() correctly uses (spin_lock_instance(PICO_SPINLOCK_ID_IRQ)) -- yet both cores call it concurrently against the same shared RAM vector table during xPortStartSchedulerOnCore(). That's a real, unguarded race in the SDK, independent of this PR. Serializing it (install once from core0, strictly before multicore_launch_core1()) compiled and ran fine but didn't change the failure -- so it's a latent bug worth a separate report to raspberrypi/pico-sdk, not the cause here.

Localization -- this is not a bring-up race. Added a sentinel ladder (writes to a global counter array at fixed milestones, verified via disassembly to never clobber r3/lr/EXC_RETURN) through prvDisableInterruptsAndPortStartSchedulerOnCore -> xPortStartSchedulerOnCore -> the cpsie i / bx r3 handoff in vPortStartFirstTask, plus entry/exit counters in xPortPendSVHandler. Result: core1 completes the entire launch sequence successfully -- handshake, trampoline, the interrupt-enable/branch handoff, and starts running its first task. It then runs successfully for a short bounded window (context-switch counters climbing normally) before permanently stopping.

Ruled out an observer/probe artifact. Since the failure signature is only ever read by selecting rp2040.core1 over the multidrop SWD link -- which throws a recurring Failed to select multidrop rp2040.dap1 warning on every reset in our setup -- we couldn't initially rule out that selecting core1 itself was inducing the failure state, rather than observing a real one. Tested by reading a heartbeat counter via core0 only (never selecting core1) at 0.3s/1.5s/4.0s post-reset delays, with the debug probe fully disconnected during each wait. If core1 were alive the whole time, the counter should scale ~13x across that delay range. It didn't -- flat at roughly the same value regardless of wait, including across windows with zero SWD connection at all. The death is real and happens early, not a probe-selection side effect.

Ruled out USB/TinyUSB correlation. Disabled the TinyUSB device task entirely (commented out tusb_freertos_tud_create()), rebuilt, reran the same delay-scaling test -- core1 still dies in an early bounded window (fewer total context-switch cycles before dying, consistent with less scheduling pressure overall, but still a hard plateau independent of wait time). SOF/enumeration timing is not the trigger.

Also ruled out: watchdog (confirmed disabled via direct WATCHDOG_CTRL/WATCHDOG_REASON register read), flash_safe_execute/multicore_lockout interaction (this target never touches flash at runtime, so it can't be pico-sdk's lockout-victim-handler-displacement issue), and any second software caller of multicore_reset_core1() or direct PSM_FRCE_OFF_PROC1 manipulation anywhere in the app or FreeRTOS-Kernel source -- confirmed via a whole-binary disassembly grep, single call site, at boot, correctly ordered.

One honest caveat: the "core1 is back in the bootROM wait_for_vector loop" read is only ever obtained by selecting core1 over the same multidrop link that throws the warning above, so I can't fully separate "core1 genuinely took a PSM hardware reset" from "core1 stalled/faulted in some other way and dap1 then reports that specific state." What's proven is the behavior (permanent, unsupervised, early halt of scheduler progress on core1); the exact reset mechanism is inferred, not directly observed.

Net: this looks like either something inside FreeRTOS-Kernel's RP2040 SMP scheduler internals below what we can localize further with sentinel instrumentation, or a board/silicon-level power-integrity margin issue outside firmware's reach -- not something either of our known fixes (or the ones we tried) resolves. Happy to share the instrumented port.c/tasks.c diff and exact test scripts if useful for continuing this on your end or on other boards.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants