V80 SLASH shell silently drops host s_axi_control writes to offsets 0x40–0xBF

V80 SLASH shell silently drops host `s_axi_control` writes to offsets `0x40–0xBF` (i.e. when addr[7]≠addr[6])

1. Environment

Git dev branch commit 7a52870
Tools: Vivado/Vitis 2025.1 · SLASH VRT

2. Bug description

Host control-register accesses (VRT kernel.read()/write() over QDMA/PCIe) to a kernel's
s_axi_control interface are not delivered to the kernel for register offsets in the first 256
bytes of the control aperture whose address bit [7] ≠ bit [6] — i.e. 0x40–0x7F and 0x80–0xBF.
Observed behavior on these offsets: a host write does not update the kernel register (confirmed by
JTAG accesses and ILA probes on s_axi_control), yet a host read of the same offset returns the value
the host last wrote there — so the write-then-readback self-validates and the failure is invisible/masked.
Kernel argument pointers written to affected offsets stay 0, producing wrong results with no error
reported.

Addresses checked (V80 compute shell, 4 KB control aperture at 0x20200000000):

aperture offset	`A[7:6]`	result
`0x00–0x3F`	`00`	reaches kernel ✅
`0x40–0x7F`	`01`	does not reach kernel ❌
`0x80–0xBF`	`10`	does not reach kernel ❌
`0xC0–0xFF`	`11`	reaches kernel ✅
`0x100–0xFFF`	any	reaches kernel ✅

3. Temporary Workaround (kernel-side) which works for TAPA design

The failing 0x40–0xBF registers were re-mapped into 0x100–0xFFF (a region that does reach the kernel), and C_S_AXI_ADDR_WIDTH was widened 8 → 12 to make those offsets addressable. (As the good 0x00–0x3F and 0xC0–0xFF sub-bands were not large enough).

@dkumaramd

Thanks for the detailed writeup, @dkumaramd — I've been through the control path on dev and the reproducer in 20260911_tapa_bm25_ila.zip. Three findings, and then two captures I'd like to ask you for, because I think the current model of the bug may not hold.

1. The failing set looks sparse, not a 0x40–0xBF band

In the zip, ip_repo/indexer_top_1_0/src/indexer_top_control_s_axi.v has a hand-edited localparam ADDR_* block that diverges from the HLS-generated map in the same file's header comment. Only three arguments were relocated:

arg	HLS offset	relocated to
`inst_mem`	`0x50`	`0xd4`
`doc_mem_1`	`0x68`	`0xe0`
`doc_mem_5`	`0x98`	`0xec`

doc_mem_0 (0x5c), doc_mem_2 (0x74), doc_mem_3 (0x80), doc_mem_4 (0x8c), doc_mem_6 (0xa4) and doc_mem_7 (0xb0) all sit inside the reported dead band and were left in place — so at least at that point the failure was three specific registers, not every offset with A[7] != A[6]. Was the whole-band result measured later, by sweeping every dword? If it is really three registers, no address decoder produces that pattern and the A[7:6] framing is a coincidence of where 64-byte boundaries fall.

2. Write-combining on the control BAR is ruled out

This was my first suspicion, since the runtime does register access through an mmap of the BAR with no fences anywhere in the path, and the symptom — a write that never lands but reads back the value you wrote — is what a pending CPU-side write buffer looks like.

It isn't that. The driver picks the page protection from the prefetchable bit (driver/slash_dmabuf.c:187), and on a V80 the control aperture is PF2 BAR0 (128 MB → 0x202_0000_0000), which is not prefetchable: top.tcl sets PREFETCHABLE {1} only for PF2 BAR4, and on a live board Linux places BAR0 and BAR2 in the 32-bit MMIO window (0xa8000000, 0xb0000000) while BAR4 goes above 4 GB (0x380070000000) — the allocator's usual split for prefetchable vs not. So wc = 0, the mapping is pgprot_device, and every register write leaves the CPU as its own uncached dword. There is no host-side buffer that could absorb a write and satisfy the readback.

3. So the readback is probably telling the truth

With an uncached mapping, a readback that returns your value means something really is storing it. The simplest remaining explanation is that the write does reach s_axi_control and the HLS register file does hold it — the readback is correct — but those arguments never propagate into the kernel's datapath, which would leave the pointer at 0 inside the kernel exactly as you describe.

What makes me suspect that rather than the shell: the same zip carries hdl_patches.patch and hdl_patches_2.patch, which add power-up initial values to FIFO state and extend the kernel reset by 16 cycles (ap_rst_shift), with a comment about the region's synchronous reset being missed while the user clock is already running. Something in this design was already coming out of reset wrong. An argument register sampled by a sub-module before it is properly reset would read back correctly over AXI and still be 0 in the datapath.

What would settle it

Two captures from the unpatched register map, please:

JTAG read of 0x50, 0x68, 0x98 after the host has written them and before ap_start. Does the register file hold the written value, or 0?
ILA capture on s_axi_control showing AWADDR/WDATA/WVALID/BVALID/BRESP for those three writes, with the kernel-internal inst_mem, doc_mem_1, doc_mem_5 signals in the same window.

If the register file holds the value and the internal signal is 0, this is kernel/TAPA-side and the shell is exonerated. If the register file reads 0, the write genuinely is not landing and the search moves to the NoC → SmartConnect leg — with write-combining off the table.

Unrelated, but worth checking in your workaround

In the zip, component.xml declares reg0 range = 4096 while indexer_top_control_s_axi.v still has C_S_AXI_ADDR_WIDTH = 8 and ADDR_BITS = 8. The shell trusts component.xml (linker/slashkit/emit/hw/user_region/addr_ctx.py) and opens a 4 KB window onto a slave that decodes 256 bytes, so offsets 0x100+ alias back down into 0x00–0xFF. That is probably why 0x100–0xFFF appeared to reach the kernel. If the final fix relies on registers living above 0x100, please confirm the widening to 12 bits went into the RTL and the indexer_top.v instantiation as well as component.xml — an 8-bit slave behind a 4 KB window will alias silently.

@hpc-aulmamei

@hpc-aulmamei i've been working since yesterday on a theory that this is caused by MSI-X registers for the respective BAR aliasing over the kernel aperture. See the associated development branch. Disabling MSI-X does not work by itself, i've now moved the MSI-X register base up to the end of the BAR and doing another build to check if that fixes it.

@hpc-aulmamei

@hpc-aulmamei - not sure which Zip file you are referring to ... please syn-up with @quetric as I had provided the passing & failing builds to him.

@quetric

@quetric - yes MSI-X registers is the cause
The following is what I have done so far with my investigation ...

1. Root cause

The kernel control aperture 0x20200000000 is PF2 (10ee:50c2, slash_ctl) BAR0. The shell uses
CPM_PCIE1_MSI_X_OPTIONS {MSI-X_Internal} but sets no PF2 MSI-X parameters, so the CPM5 defaults apply:

parameter	default
`CPM_PCIE1_PF2_MSIX_ENABLED`	`1`
`CPM_PCIE1_PF2_MSIX_CAP_TABLE_BIR`	`BAR_1:0` (BAR0)
`CPM_PCIE1_PF2_MSIX_CAP_TABLE_OFFSET` / `_SIZE`	`0x40` / `007` (8 vectors)

8 × 16 B places the MSI-X table at PF2 BAR0 0x40–0xBF (PBA at 0x50). The CPM serves those TLPs
from its internal table RAM and never forwards them to the NoC. Writes are absorbed, reads return the stored
value, and the value persists until a PCIe reset. JTAG/AXI accesses bypass PCIe and work, and only the IP at
BAR0 offset 0 (the first linked kernel) is affected.

CPM_PCIE1_PF2_MSIX_ENABLED {0} alone is not sufficient (tested on hardware): the disabled table
collapses to 1 entry at 0x40, and 0x40–0x5F is still captured. The table must be moved.

2. Fix

Move the PF2 MSI-X table/PBA to the unused PF2 BAR2 in both linker/slashkit/resources/base/{compute,service}/scripts/top.tcl
(offsets must be ≤ 0x1FFFFFF; MSIX_ENABLED must stay 1 or the BIR/offset settings are ignored):

       CPM_PCIE1_PF2_EXPANSION_ROM_QDMA_ENABLED {0} \
+      CPM_PCIE1_PF2_MSIX_CAP_PBA_BIR {BAR_3:2} \
+      CPM_PCIE1_PF2_MSIX_CAP_PBA_OFFSET {1FFF000} \
+      CPM_PCIE1_PF2_MSIX_CAP_TABLE_BIR {BAR_3:2} \
+      CPM_PCIE1_PF2_MSIX_CAP_TABLE_OFFSET {1FFE000} \
+      CPM_PCIE1_PF2_MSIX_CAP_TABLE_SIZE {1} \
+      CPM_PCIE1_PF2_MSIX_ENABLED {1} \
       CPM_PCIE1_PF2_PCIEBAR2AXIBAR_QDMA_0 {0x0000020200000000} \

slash_ctl never allocates IRQs on PF2 (poll mode; QDMA interrupts are on PF1), so one vector is enough.

3. Validated so far

fix applied to compute shell linker/slashkit/resources/base/compute/scripts/top.tcl works
now applying to both compute & service and rebuilding

@dkumaramd

thanks @dkumaramd this is great news. Once you have checked service too, please submit a PR with the two TCL changes, targeting dev

@quetric

@quetric
FYI: created PR (#235)

@dkumaramd

Thanks @dkumaramd - fixed with #235

Navigation Menu

Repository navigation

V80 SLASH shell silently drops host `s_axi_control` writes to offsets `0x40–0xBF` #226

V80 SLASH shell silently drops host `s_axi_control` writes to offsets `0x40–0xBF` (i.e. when addr[7]≠addr[6])

1. Environment

2. Bug description

3. Temporary Workaround (kernel-side) which works for TAPA design

1. The failing set looks sparse, not a 0x40–0xBF band

2. Write-combining on the control BAR is ruled out

3. So the readback is probably telling the truth

What would settle it

Unrelated, but worth checking in your workaround

1. Root cause

2. Fix

3. Validated so far

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

V80 SLASH shell silently drops host s_axi_control writes to offsets 0x40–0xBF #226

Description

V80 SLASH shell silently drops host s_axi_control writes to offsets 0x40–0xBF (i.e. when addr[7]≠addr[6])

1. Environment

2. Bug description

3. Temporary Workaround (kernel-side) which works for TAPA design

Activity

hpc-aulmamei commented on Sep 28, 2026

1. The failing set looks sparse, not a 0x40–0xBF band

2. Write-combining on the control BAR is ruled out

3. So the readback is probably telling the truth

What would settle it

Unrelated, but worth checking in your workaround

quetric commented on Sep 28, 2026

dkumaramd commented on Sep 28, 2026

dkumaramd commented on Oct 1, 2026

1. Root cause

2. Fix

3. Validated so far

quetric commented on Oct 2, 2026

dkumaramd commented on Oct 7, 2026

quetric commented on Oct 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

V80 SLASH shell silently drops host `s_axi_control` writes to offsets `0x40–0xBF` #226

V80 SLASH shell silently drops host `s_axi_control` writes to offsets `0x40–0xBF` (i.e. when addr[7]≠addr[6])