From 1d61199f58763793dc3ed9a4504b3fdf0b0e7e91 Mon Sep 17 00:00:00 2001 From: LiangSu8899 Date: Sat, 15 Aug 2026 13:41:42 -0400 Subject: [PATCH 01/32] docs+test(pi05): consolidated Thor guide and the FP16-reference harness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ports the two Pi0.5 artifacts that had not reached this repository. The guide is the canonical usage and performance reference: what runs on the device, how to instantiate and calibrate each precision tier, the latency and accuracy matrix across FP8 / NVFP4 / INT4 / INT4+RHT at one, two and three views, tier selection, the knob reference, the frame-time breakdown, and the approaches that were measured and rejected. The harness measures every tier against a common FP16 reference in one locked-clock batch, reporting latency and accuracy together, which the strict FP8-referenced suite cannot do — it leaves FP8's own error unmeasured and has no unquantized anchor. --- docs/pi05_thor.md | 416 ++++++++++++++++++++++++++ tests/bench_pi05_precision_vs_fp16.py | 240 +++++++++++++++ 2 files changed, 656 insertions(+) create mode 100644 docs/pi05_thor.md create mode 100644 tests/bench_pi05_precision_vs_fp16.py diff --git a/docs/pi05_thor.md b/docs/pi05_thor.md new file mode 100644 index 00000000..54226e22 --- /dev/null +++ b/docs/pi05_thor.md @@ -0,0 +1,416 @@ +# Pi0.5 on Jetson AGX Thor — Usage and Performance + +Canonical guide for running Pi0.5 (PaliGemma vision-language encoder + +Gemma action-expert decoder) on Jetson AGX Thor (SM110). It covers the +four supported precision tiers, how to select them, what each one costs +and delivers, and how to reproduce every number. + +`pi05_thor_decoder_fp4_e2e.md` is the chronological engineering record +behind these results — individual sections there describe intermediate +states and are superseded by this document. + +--- + +## 1. What runs on the device + +| stage | shape | precision (default tier) | +|---|---|---| +| SigLIP vision tower, 27 layers | `num_views × 256` tokens, d=1152 | FFN NVFP4 + AWQ, attention FA4 | +| Encoder (PaliGemma), 18 layers | `num_views × 256 + prompt` tokens, d=2048, H=16384, GQA 8/1, head_dim 256 | 17 live FFNs NVFP4 + AWQ, attention-O NVFP4, QKV FP8, attention FA4 | +| Decoder (Gemma action expert), 18 layers × 10 denoise steps | 10 action tokens, d=1024, H=4096 | all four projections NVFP4, attention cuBLAS FP16 | + +One `infer()` call runs the vision tower, the encoder prefill, and ten +denoise steps of the action expert, and returns a `(10, 7)` action chunk. +The whole path is CUDA-graph captured; there is no Python in the hot loop. + +--- + +## 2. Requirements + +- Jetson AGX Thor, compute capability `(11, 0)`, MAXN power mode +- CUDA 13, PyTorch ≥ 2.10 +- CUTLASS v4.4.2 at `third_party/cutlass` (vendored or symlinked) +- FA4 (`flashrt_fa4.cute`) importable — required for the SigLIP and + encoder attention path used by every number below + +```bash +cmake -B build -S . -DGPU_ARCH=110 +cmake --build build -j8 # .so files land in flash_rt/ +``` + +Locked clocks are required for reproducible timing: + +```bash +sudo nvpmodel -m 0 && sudo jetson_clocks +``` + +--- + +## 3. Quick start + +```python +from flash_rt.frontends.torch.pi05_thor_fp4 import Pi05TorchFrontendThorFP4 + +pipe = Pi05TorchFrontendThorFP4( + checkpoint_dir, + num_views=3, + use_fa4=True, + # ---- production NVFP4 tier ---- + use_fp4_encoder_ffn=True, fp4_layers=tuple(range(17)), + use_awq=True, awq_alpha=0.8, use_p1_split_gu=True, + use_fp4_encoder_attn=True, # attention-O projections + use_fp4_siglip_ffn=True, # all 27 SigLIP FFNs + use_fp4_decoder=True, # all four decoder projections +) + +pipe.set_prompt("pick up the black bowl and place it on the plate") +pipe.calibrate(observations, percentile=99.9) # 8 real observations +actions = pipe.infer(observation)["actions"] # (10, 7) +``` + +`observation` is a dict with `image` (`(224, 224, 3)` uint8/float16), +`state`, and — depending on `num_views` — `wrist_image` and +`wrist_image_right`. + +The FP8 baseline is the same call on the base class: + +```python +from flash_rt.frontends.torch.pi05_thor import Pi05TorchFrontendThor +pipe = Pi05TorchFrontendThor(checkpoint_dir, num_views=3, use_fa4=True) +``` + +**Calibration is not optional.** The FP4 tiers derive activation scales +and AWQ per-channel weight scales from real observations; calibrating on +synthetic data or skipping it degrades accuracy well past the gates below. + +--- + +## 4. The four precision tiers + +All four share the same encoder and SigLIP configuration; they differ in +how the **decoder** stores weights and activations. + +| tier | selection | decoder weights | decoder activations | +|---|---|---|---| +| **FP8** | `Pi05TorchFrontendThor` (no FP4 flags) | E4M3, per-tensor static scales | E4M3 | +| **NVFP4** (default) | `use_fp4_decoder=True` | E2M1 + per-16 UE4M3 block scale, per-block MSE scale search | dynamic E2M1 + per-16 scale | +| **INT4** | `decoder_weight_format="e0m3"`, `decoder_act_format="e0m3"` | uniform INT4 (E0M3, ±0..7), scale = amax/7 | uniform INT4 | +| **INT4+RHT** | above plus `decoder_rht=True` | INT4 after a per-16 orthonormal Hadamard rotation | INT4, rotation fused into the quantize kernel | + +**NVFP4** uses NVIDIA's block-scaled FP4: a 4-bit E2M1 element with a +UE4M3 scale per 16 elements. Weight scales come from a per-block MSE +search; activation scales are computed at run time in the fused +norm/quantize kernels. + +**INT4 (E0M3)** exploits the SM110 tcgen05 block-scaled MMA: the +instruction descriptor's 3-bit element-format field selects +sign-magnitude uniform INT4 at *run time*, so the same tensor-core path +serves both formats with no binary patching. The packed and scale-factor +layouts are identical to NVFP4, so buffers are interchangeable. + +**INT4+RHT** rotates every 16-element block by an orthonormal Hadamard +matrix (H16/4, symmetric) — weights offline, activations fused into the +quantize kernel. The rotation is mathematically inert for the GEMM +(verified at kernel level and end to end) but gaussianizes the +per-block distribution, which suits a uniform grid. It buys accuracy at +roughly +0.35 ms. + +An unquantized FP16 path also exists (`use_fp8=False`). It is far too +slow for deployment but serves as the common accuracy reference in §5.3, +where FP8 itself is measured against it. + +--- + +## 5. Performance and accuracy + +### 5.1 How these numbers were taken + +Thor exhibits **day-scale whole-machine drift** (thermal / EMC state): +the same binary measured hours apart can differ by ~2.5 ms, and the FP8 +reference moves with it. Therefore: + +- absolute milliseconds are only comparable **within one measurement + batch**; +- **speedup against the same-run FP8 reference is the stable metric**; +- every table below comes from one batch (the FP8 references land within + 0.2 ms of each other at each view count). + +Each run is a separate process pair — FP8 child then FP4 child — over +eight real LIBERO observations with matched noise seeds, 20 warmup +iterations and 100 timed iterations, with locked clocks verified from +`/sys` at start. Within-run spread is small (p95 − p50 ≤ 0.09 ms). + +### 5.2 Latency and accuracy by tier + +Cosines are against the FP8 reference, per sample over the eight +observations. `raw` is the pre-unnormalization action tensor; `act` is +the final action chunk. + +**3 views** + +| tier | p50 (ms) | speedup | raw cos | raw min | act cos | act min | gates | +|---|---|---|---|---|---|---|---| +| FP8 (reference) | 46.59 | 1.000 | — | — | — | — | — | +| **NVFP4 (default)** | **31.64** | **1.473** | 0.99904 | 0.99766 | 0.99974 | 0.99944 | PASS | +| INT4 | 32.39 | 1.441 | 0.99838 | 0.99512 | 0.99961 | 0.99939 | PASS | +| INT4+RHT | 32.54 | 1.432 | 0.99918 | 0.99742 | 0.99983 | 0.99970 | PASS | + +**2 views** + +| tier | p50 (ms) | speedup | raw cos | raw min | act cos | act min | gates | +|---|---|---|---|---|---|---|---| +| FP8 (reference) | 38.45 | 1.000 | — | — | — | — | — | +| **NVFP4 (default)** | **27.14** | **1.417** | 0.99921 | 0.99803 | 0.99972 | 0.99916 | PASS | +| INT4 | 27.86 | 1.380 | 0.99879 | 0.99751 | 0.99965 | 0.99928 | PASS | +| INT4+RHT | 27.97 | 1.380 | 0.99941 | 0.99828 | 0.99977 | 0.99922 | PASS | + +**1 view** — the accuracy gates do not pass at any fully-quantized tier, +for reasons that are not implementation defects (§6). + +| tier | p50 (ms) | speedup | raw cos | raw min | act cos | act min | gates | +|---|---|---|---|---|---|---|---| +| FP8 (reference) | 32.65 | 1.000 | — | — | — | — | — | +| NVFP4 (default) | **22.98** | **1.421** | 0.99137 | 0.96502 | 0.99347 | 0.97112 | accuracy FAIL | +| INT4 | 23.65 | 1.381 | 0.99101 | 0.96112 | 0.99332 | 0.96827 | accuracy FAIL | +| INT4+RHT | 23.75 | 1.375 | 0.99222 | 0.96385 | 0.99406 | 0.97021 | accuracy FAIL | +| FP8 encoder + INT4+RHT decoder | 29.06 | 1.122 | 0.99974 | 0.99954 | 0.99990 | 0.99973 | accuracy PASS | + +Gates: `raw cos ≥ 0.995`, worst-sample `raw cos ≥ 0.995`, +`action cos ≥ 0.999`, worst-sample `action cos ≥ 0.995`, plus a latency +gate (3-view p50 ≤ 40 ms, 2-view p95 ≤ 40 ms). + +### 5.3 Cosine against a common FP16 reference + +The tables above measure each quantized tier against FP8, which leaves +FP8's own error unmeasured. Running the same protocol against the FP16 +path (`use_fp8=False`) puts every tier on one yardstick: + +| views | tier | raw cos | raw min | act cos | act min | +|---|---|---|---|---|---| +| 3 | FP8 | 0.99994 | 0.99992 | 0.99997 | 0.99995 | +| 3 | NVFP4 | 0.99913 | 0.99812 | 0.99976 | 0.99948 | +| 3 | INT4 | 0.99848 | 0.99518 | 0.99963 | 0.99936 | +| 3 | INT4+RHT | 0.99928 | 0.99742 | 0.99985 | 0.99972 | +| 2 | FP8 | 0.99995 | 0.99994 | 0.99998 | 0.99997 | +| 2 | NVFP4 | 0.99929 | 0.99830 | 0.99976 | 0.99931 | +| 2 | INT4 | 0.99887 | 0.99775 | 0.99969 | 0.99943 | +| 2 | INT4+RHT | 0.99950 | 0.99854 | 0.99982 | 0.99937 | +| 1 | FP8 | 0.99876 | **0.99421** | 0.99905 | 0.99529 | +| 1 | NVFP4 | 0.99187 | 0.96338 | 0.99380 | 0.96939 | +| 1 | INT4 | 0.99118 | 0.95940 | 0.99342 | 0.96644 | +| 1 | INT4+RHT | 0.99204 | 0.96243 | 0.99381 | 0.96856 | + +Two things follow. + +**FP8 is essentially exact at two and three views** (0.9999+ on every +metric), so using it as the reference in §5.2 costs nothing — those +numbers are within 1e-4 of the same measurement against FP16. + +The same harness also times each tier, which places the unquantized path +on the scale. One locked-clock batch at three views: + +| tier | p50 (ms) | p95 (ms) | vs FP16 | +|---|---|---|---| +| FP16 | 80.202 | 81.028 | 1.000 | +| FP8 | 46.619 | 49.422 | 1.720 | +| NVFP4 | **31.977** | 32.054 | **2.508** | +| INT4 | 32.577 | 32.653 | 2.462 | +| INT4+RHT | 32.646 | 32.712 | 2.457 | + +**At one view even FP8 loses its worst sample**, to 0.99421 — below the +0.995 gate that the quantized tiers also miss. FP8 differs from FP16 by +a very small perturbation, so a sample that moves this much under it is +not being broken by 4-bit quantization; it is sitting somewhere that any +perturbation moves it. See §6. + +Reproduce with `tests/bench_pi05_precision_vs_fp16.py` (one subprocess +per tier, same prompt / observations / seeds as the strict suite). + +### 5.4 Choosing a tier + +- **Default to NVFP4.** It is both the fastest tier and, on 3 views, has + the best worst-sample raw cosine. The decoder FFN fusion (§8) only + applies to NVFP4 weights, which is why the INT4 tiers now sit ~0.7 ms + behind. +- **INT4+RHT when accuracy matters most.** It leads on aggregate cosine + and on worst-sample action cosine at both view counts, for ~0.9 ms. +- **Plain INT4 has no niche today** — it is slower than NVFP4 and less + accurate than INT4+RHT. It exists because it is the base the rotation + is applied to, and because it demonstrates the runtime-descriptor path. +- **FP8** remains the reference for correctness comparisons and for any + deployment that cannot calibrate on real observations. + +--- + +## 6. One-view accuracy + +At one view the per-sample cosine gates fail for every quantized tier. +This is characterized, not open: + +- **FP8 fails the same gate at one view** (worst-sample raw cosine 0.99421 + against FP16, §5.3) while being exact to 0.9999 at two and three views. + FP8 is a far smaller perturbation than 4-bit quantization, so whatever + moves that sample is not a property of the FP4 kernels. +- Ablations flip **different** samples under different quantization + configurations (full FP4 flips sample 0; FP8-encoder + FP4-decoder + flips sample 3 instead). +- The failure mode is a whole-trajectory direction change in the worst + sample's dominant motion component, not a magnitude error. The gripper + dimension is exact (cos = 1.0). +- Decoder accuracy improvements move it monotonically: INT4+RHT lifts the + worst sample from 0.843 to 0.930. + +The reading: with one view the observation underdetermines some samples, +which sit near a decision boundary of the flow-matching velocity field, +and any small perturbation — including FP8's — selects the other branch. +Such a sample produces *a different valid action candidate*, which a +per-sample cosine gate cannot distinguish from an error. Task success +rate is the meaningful judge; that evaluation is out of scope here. + +A configuration that passes every **accuracy** gate at one view — FP8 +encoder with an INT4+RHT decoder — at 29.06 ms (8/8 samples, worst-sample +raw cosine 0.99954, worst-sample action cosine 0.99973): + +```bash +--num-views 1 \ +--encoder-fp4-layer-count 0 --siglip-ffn-fp4 0 --encoder-attn-o-fp4 0 \ +--decoder-weight-format e0m3 --decoder-act-format e0m3 --decoder-rht 1 +``` + +It still trips the suite's published-SOTA latency gate (which wants +≤ 28.5 ms at one view), so `result.json` reports `passed: false` with +every accuracy gate green. Keeping the encoder in FP8 is what buys the +fidelity: it costs 6.1 ms against the fully quantized tier. + +--- + +## 7. Knobs + +Constructor keyword / bench flag pairs. Defaults are the production tier. + +| knob | default | effect | +|---|---|---| +| `decoder_weight_format` / `--decoder-weight-format` | `nvfp4` | `nvfp4` or `e0m3` | +| `decoder_act_format` / `--decoder-act-format` | `nvfp4` | `e0m3` requires `e0m3` weights | +| `decoder_rht` / `--decoder-rht` | `False` | per-16 Hadamard rotation; requires `e0m3` activations | +| `decoder_fused_geglu` / `--decoder-fused-geglu` | `True` | fuse the decoder GeGLU into the gate_up GEMM epilogue (NVFP4 weights only) | +| `encoder_p1_combiner` / `--encoder-p1-combiner` | `epilogue_hw` | `epilogue_hw` (fused, compact store), `epilogue` (fused, full width — parity with the old path), `lut_native` (separate GEMMs + combiner kernel) | +| `use_fp4_encoder_attn_qkv` / `--encoder-attn-qkv-fp4` | `False` | implemented and passing, but that GEMM is not weight-bandwidth-bound, so FP4 only matches FP8 while costing an extra quantize step | +| `decoder_fused_attn` / `--decoder-fused-attn` | `False` | folds the seqused mask into softmax (bit-identical, one fewer launch). Only the fixed-shape state-prompt path takes the seqused kernels, which this suite does not exercise | +| `awq_alpha` / `--awq-alpha` | `0.8` | AWQ per-channel scale exponent | +| `encoder_down_variant`, `decoder_*_variant` | `7`, `10` | GEMM tile selection | + +**Tile selection warning.** Cluster-launch GEMM variants invert between +isolated and in-pipeline benchmarks on Thor: the isolated-best tile for +one projection cost +2.2 ms end to end, and larger clusters +11–14 ms. +Always A/B tiles inside the pipeline. + +--- + +## 8. Where the time goes + +Single-frame kernel trace, 3 views, NVFP4 tier +(`nsys --cuda-graph-trace=node` around one `cudaProfilerStart/Stop` +window). Frame 31.85 ms under the profiler; 99% is GPU kernel time. + +| component | ms/frame | share | +|---|---|---| +| block-scaled FP4 GEMM (incl. fused GeGLU epilogues) | 21.3 | 67% | +| normalization / AdaRMS (incl. 350 decoder AdaRMS calls) | 3.3 | 10% | +| FA4 attention (SigLIP + encoder) | 2.1 | 7% | +| decoder attention cuBLAS chain (QK^T / softmax / AV) | 1.9 | 6% | +| encoder FP8 GEMM (attention QKV / O) | 1.2 | 4% | +| activation quantize | 1.1 | 3% | +| RoPE / QKV split | 0.6 | 2% | +| other | 0.4 | 1% | + +Headroom is thin and mostly hard floors: + +1. The decoder's 9.7 ms of GEMM sits against a ~7.3–7.8 ms weight-bandwidth + floor; the gap is fixed overhead across 720 GEMM launches, which would + need a bespoke persistent mainloop to recover. +2. Decoder AdaRMS (350 × 2.92 µs) and RoPE (180 × 1.58 µs) are at the + kernel-launch floor. +3. SigLIP / encoder attention projections in FP4 (est. −0.3 to −0.5 ms) + and AWQ for the SigLIP up-projection are the remaining candidates, both + gated on accuracy. + +### Approaches measured and rejected + +- **Single-kernel decoder attention.** Implemented and numerically + validated, then measured at 5–7× the existing chain across three + schedule designs. The skinny attention shape leaves the GEMM work + tensor-core-bound (the two cuBLAS calls are ~1 µs of tensor-core math), + and per-(head, row) grids multiply KV re-reads past the L2 budget. + FlashAttention-4 at this shape measures 24.6 µs (head_dim 256 has no + KV-split path). Fuse the glue *between* GEMMs, not the GEMMs. +- **Full-width fused GeGLU epilogue.** The combiner kernel it removes is + exactly cancelled by the doubled weight streaming of the K-expanded + down projection. The half-width compact store is the form that wins, + and is the default. + +--- + +## 9. Reproducing + +```bash +# kernel numerical contracts +pytest tests/test_pi05_fp4_fusion_kernels.py tests/test_pi05_decoder_fp4_kernels.py + +# every tier against a common FP16 reference (§5.3) +python tests/bench_pi05_precision_vs_fp16.py \ + --checkpoint \ + --fixture /libero_obs3v_n8.npz \ + --num-views 3 --output-dir + +# strict end-to-end suite (requires a clean tracked worktree) +python tests/bench_pi05_decoder_fp4_e2e.py \ + --checkpoint \ + --num-views 3 \ + --fixture /libero_obs3v_n8.npz \ + --output-dir + +# tier switches + --decoder-weight-format e0m3 --decoder-act-format e0m3 # INT4 + --decoder-weight-format e0m3 --decoder-act-format e0m3 --decoder-rht 1 # INT4+RHT +``` + +The fixture is an npz of eight real LIBERO observations +(keys `n`, `img_i`, `state_i`, `wrist_i`, `wrist_right_i`). The suite +writes `result.json` with per-iteration timings, the verified clock +state, per-gate verdicts, and `.so` SHA256s, plus the FP4 and FP8 action +tensors. + +Measurement discipline: + +- compare back-to-back within one batch; across batches use speedup; +- warm up at least 20 iterations before timing; +- one process, exclusive GPU, no concurrent load. + +--- + +## 10. Implementation map + +| area | files | +|---|---| +| frontend, tier selection, weight prep | `flash_rt/frontends/torch/pi05_thor_fp4.py` | +| benchmarks | `tests/bench_pi05_decoder_fp4_e2e.py` (strict E2E), `tests/bench_pi05_precision_vs_fp16.py` (common-reference accuracy) | +| decoder pipeline | `flash_rt/models/pi05/pipeline_thor.py` | +| SigLIP / encoder pipeline | `flash_rt/hardware/thor/shared_primitives_fp4.py` | +| attention dispatch (FA4, cuBLAS, seqused) | `flash_rt/hardware/thor/attn_backend.py` | +| NVFP4 / INT4 GEMM runners | `csrc/gemm/fp4/` | +| fused GeGLU store epilogue | `csrc/gemm/fp4/sm100_gelu_mul_blockscale_visitor.hpp` | +| fused norm / quantize / activation kernels | `csrc/fused_fp4/`, `csrc/quantize/` | +| E0M3 quantizer and activation kernels | `csrc/quantize/quantize_e0m3_sfa.cu`, `csrc/fused_fp4/pi05_e0m3_act.cu` | + +--- + +## 11. Known limitations + +- One-view accuracy gates fail at every quantized tier (§6); a passing + configuration exists at reduced speed. +- Task-level (rollout) validation is out of scope for this document. +- The benchmark requires `--checkpoint` and `--fixture` explicitly; there + are no default dataset paths. +- Numbers here are Thor-specific. The SM110 runtime-descriptor INT4 path + in particular has no equivalent on other architectures. diff --git a/tests/bench_pi05_precision_vs_fp16.py b/tests/bench_pi05_precision_vs_fp16.py new file mode 100644 index 00000000..4a2bf2cf --- /dev/null +++ b/tests/bench_pi05_precision_vs_fp16.py @@ -0,0 +1,240 @@ +"""Measure every Pi0.5 precision tier against a common FP16 reference. + +The strict E2E suite (``bench_pi05_decoder_fp4_e2e.py``) reports each +quantized tier's cosine against FP8, which leaves FP8's own deviation +unmeasured. This harness runs FP16, FP8 and the three quantized tiers +through the identical observation / prompt / seed protocol and reports +every tier against FP16, so the tiers sit on one yardstick. + +One model per subprocess: Thor cannot hold two of these pipelines in a +single process. + + python tests/bench_pi05_precision_vs_fp16.py \ + --checkpoint \ + --fixture /libero_obs3v_n8.npz \ + --num-views 3 --output-dir +""" +import argparse +import json +import os +from pathlib import Path +import statistics +import subprocess +import sys +import time + +import numpy as np + +PROMPT_TOKENS = [ + 2, 18075, 908, 573, 3118, 3963, 578, + 2040, 665, 575, 573, 24655, 108, +] + +TIERS = { + # name: extra kwargs for the FP4 frontend (None => FP8/FP16 base class) + "fp16": None, + "fp8": None, + "nvfp4": {}, + "int4": {"decoder_weight_format": "e0m3", "decoder_act_format": "e0m3"}, + "int4rht": {"decoder_weight_format": "e0m3", "decoder_act_format": "e0m3", + "decoder_rht": True}, +} + + +def build_pipe(mode, checkpoint, num_views): + if mode in ("fp16", "fp8"): + from flash_rt.frontends.torch.pi05_thor import Pi05TorchFrontendThor + return Pi05TorchFrontendThor( + checkpoint, num_views=num_views, autotune=3, use_fa4=True, + use_fp8=(mode == "fp8")) + from flash_rt.frontends.torch.pi05_thor_fp4 import Pi05TorchFrontendThorFP4 + return Pi05TorchFrontendThorFP4( + checkpoint, num_views=num_views, autotune=3, + use_fp4_encoder_ffn=True, fp4_layers=tuple(range(17)), + use_awq=True, awq_alpha=0.8, use_p1_split_gu=True, + use_fp4_decoder=True, use_fa4=True, + use_fp4_encoder_attn=True, use_fp4_siglip_ffn=True, + **TIERS[mode]) + + +def load_observations(fixture, num_views): + data = np.load(fixture) + count = int(data["n"]) + observations = [] + for index in range(count): + obs = {"image": data[f"img_{index}"], "state": data[f"state_{index}"]} + if num_views >= 2: + obs["wrist_image"] = data[f"wrist_{index}"] + if num_views == 3: + obs["wrist_image_right"] = data[f"wrist_right_{index}"] + observations.append(obs) + return observations + + +def run_child(args): + import torch # noqa: F401 (import side effects before pipeline import) + + pipe = build_pipe(args.child_mode, args.checkpoint, args.num_views) + observations = load_observations(args.fixture, args.num_views) + + pipe.set_prompt(PROMPT_TOKENS) + pipe.calibrate(observations, percentile=99.9, verbose=False) + for index in range(args.warmup): + pipe.infer(observations[index % len(observations)]) + + raw_outputs, action_outputs = [], [] + for index, observation in enumerate(observations): + np.random.seed(args.seed + index) + output = pipe.infer(observation) + raw_outputs.append(pipe._g_noise.float().cpu().numpy()) + action_outputs.append(output["actions"]) + + if args.cuda_profile: + torch.cuda.synchronize() + torch.cuda.cudart().cudaProfilerStart() + pipe.infer(observations[0]) + torch.cuda.synchronize() + torch.cuda.cudart().cudaProfilerStop() + + latencies = [] + for index in range(args.iters): + start = time.perf_counter() + pipe.infer(observations[index % len(observations)]) + latencies.append((time.perf_counter() - start) * 1000.0) + torch.cuda.synchronize() + + out_dir = Path(args.output_dir) + out_dir.mkdir(parents=True, exist_ok=True) + out = out_dir / f"{args.child_mode}_actions.npz" + np.savez(out, raw=np.stack(raw_outputs), actions=np.stack(action_outputs)) + result = { + "tier": args.child_mode, + "num_views": args.num_views, + "iters": args.iters, + "p50_ms": statistics.median(latencies), + "p95_ms": float(np.percentile(latencies, 95)), + "min_ms": min(latencies), + "max_ms": max(latencies), + } + print("__CHILD_RESULT__ " + json.dumps(result, sort_keys=True), flush=True) + return 0 + + +def cosines(lhs_all, rhs_all): + per_sample = [] + for index in range(lhs_all.shape[0]): + lhs = lhs_all[index].reshape(-1) + rhs = rhs_all[index].reshape(-1) + per_sample.append(float( + lhs @ rhs / (np.linalg.norm(lhs) * np.linalg.norm(rhs) + 1e-12))) + lhs = lhs_all.reshape(-1) + rhs = rhs_all.reshape(-1) + aggregate = float( + lhs @ rhs / (np.linalg.norm(lhs) * np.linalg.norm(rhs) + 1e-12)) + return aggregate, min(per_sample) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--child-mode", choices=sorted(TIERS)) + parser.add_argument("--checkpoint", required=True) + parser.add_argument("--fixture", required=True) + parser.add_argument("--num-views", type=int, choices=(1, 2, 3), default=3) + parser.add_argument("--output-dir", required=True) + parser.add_argument("--warmup", type=int, default=20) + parser.add_argument("--iters", type=int, default=100) + parser.add_argument( + "--cuda-profile", action="store_true", + help="capture one stable-state infer between " + "cudaProfilerStart/Stop") + parser.add_argument("--seed", type=int, default=20260725) + parser.add_argument("--modes", default="fp16,fp8,nvfp4,int4,int4rht") + args = parser.parse_args() + + # Latency is only meaningful under the same locked-clock discipline the + # strict suite enforces; reuse its verifier rather than duplicating it. + sys.path.insert(0, str(Path(__file__).resolve().parent)) + from bench_pi05_decoder_fp4_e2e import machine_state + machine_state() + + if args.child_mode is not None: + return run_child(args) + + output_dir = Path(args.output_dir) + output_dir.mkdir(parents=True, exist_ok=True) + modes = args.modes.split(",") + latency = {} + for mode in modes: + command = [ + sys.executable, str(Path(__file__).resolve()), + "--child-mode", mode, + "--checkpoint", args.checkpoint, + "--fixture", args.fixture, + "--num-views", str(args.num_views), + "--output-dir", str(output_dir), + "--warmup", str(args.warmup), + "--iters", str(args.iters), + "--seed", str(args.seed), + ] + child = subprocess.run(command, check=False, capture_output=True, + text=True, env=os.environ.copy()) + if child.returncode != 0: + raise RuntimeError( + f"{mode} child failed rc={child.returncode}\n" + f"{child.stdout[-2000:]}\n{child.stderr[-4000:]}") + lines = [line for line in child.stdout.splitlines() + if line.startswith("__CHILD_RESULT__")] + if len(lines) != 1: + raise RuntimeError( + f"expected one result line from {mode}, got {len(lines)}") + latency[mode] = json.loads(lines[0].split(" ", 1)[1]) + print(f" {mode} done p50={latency[mode]['p50_ms']:.3f} ms", + flush=True) + + ref = np.load(output_dir / "fp16_actions.npz") + raw_ref = ref["raw"].astype(np.float64) + act_ref = ref["actions"].astype(np.float64) + + rows = [] + for mode in modes: + if mode == "fp16": + continue + cur = np.load(output_dir / f"{mode}_actions.npz") + raw_cos, raw_min = cosines(cur["raw"].astype(np.float64), raw_ref) + act_cos, act_min = cosines(cur["actions"].astype(np.float64), act_ref) + rows.append((mode, raw_cos, raw_min, act_cos, act_min)) + + ref_p50 = latency["fp16"]["p50_ms"] if "fp16" in latency else None + print(f"\n=== precision matrix, {args.num_views} view(s) " + "(latency measured, cosine vs FP16) ===") + header = (f"{'tier':10s} {'p50 ms':>8s} {'p95 ms':>8s} {'vs FP16':>8s} " + f"{'raw cos':>9s} {'raw min':>9s} {'act cos':>9s} " + f"{'act min':>9s}") + print(header) + if "fp16" in latency: + f16 = latency["fp16"] + print(f"{'fp16':10s} {f16['p50_ms']:8.3f} {f16['p95_ms']:8.3f} " + f"{1.0:8.3f} {'-':>9s} {'-':>9s} {'-':>9s} {'-':>9s}") + for mode, raw_cos, raw_min, act_cos, act_min in rows: + lat = latency.get(mode, {}) + p50 = lat.get("p50_ms", float("nan")) + p95 = lat.get("p95_ms", float("nan")) + speedup = (ref_p50 / p50) if (ref_p50 and p50 == p50) else float("nan") + print(f"{mode:10s} {p50:8.3f} {p95:8.3f} {speedup:8.3f} " + f"{raw_cos:9.5f} {raw_min:9.5f} {act_cos:9.5f} {act_min:9.5f}") + + payload = { + "num_views": args.num_views, + "latency": latency, + "accuracy_vs_fp16": [ + dict(zip(("tier", "raw_cosine", "raw_min_sample_cosine", + "action_cosine", "action_min_sample_cosine"), r)) + for r in rows], + } + json.dump(payload, open(output_dir / "precision_matrix.json", "w"), + indent=1) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From 8f1c60c1ed990e1fa4658760ce1ee5e50540a539 Mon Sep 17 00:00:00 2001 From: LiangSu8899 Date: Sat, 15 Aug 2026 14:40:53 -0400 Subject: [PATCH 02/32] perf(pi05): shrink the half-width epilogue's unread store The compact GeGLU store node writes the down-projection input itself, but the collective still performs its ordinary D store into a buffer nothing reads. At encoder shape that is 12.8 MB per call, 21.8% of the kernel's traffic. Aliasing every output row onto row zero keeps the store instructions but collapses their footprint to a single row, so the write-back leaves L2 instead of DRAM. Encoder FFN GEMM 369.9 -> 356.5 us per call at the production shape (781x32768x2048), about 0.23 ms per three-view frame. The decoder shape is unaffected, as expected: at ten tokens the dummy is 40 KB. Fusion kernel contract tests pass unchanged, since no code reads the buffer. Eliminating the store outright is worth roughly 1.2 ms more but needs the epilogue's D descriptor remapped to half width. --- csrc/gemm/fp4/cutlass_fp4_gemm_geglu_il_sm100.cu | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/csrc/gemm/fp4/cutlass_fp4_gemm_geglu_il_sm100.cu b/csrc/gemm/fp4/cutlass_fp4_gemm_geglu_il_sm100.cu index 16f9a332..531e38d8 100644 --- a/csrc/gemm/fp4/cutlass_fp4_gemm_geglu_il_sm100.cu +++ b/csrc/gemm/fp4/cutlass_fp4_gemm_geglu_il_sm100.cu @@ -253,7 +253,12 @@ static int run_geglu_il_hw( auto stride_A = cutlass::make_cute_packed_stride(StrideAT{}, {M, K, 1}); auto stride_B = cutlass::make_cute_packed_stride(StrideBT{}, {N_il, K, 1}); auto stride_C = cutlass::make_cute_packed_stride(StrideCT{}, {M, N_il, 1}); + // The compact store node writes the real output itself; the collective's + // own D store lands in a buffer nothing reads. Aliasing every row onto + // row 0 shrinks that write from M*N_il to a single row, which at encoder + // shape is 12.8 MB of pure waste per call. auto stride_D = cutlass::make_cute_packed_stride(StrideDT{}, {M, N_il, 1}); + cute::get<0>(stride_D) = 0; auto layout_SFA = CfgT::tile_atom_to_shape_SFA(make_shape(M, N_il, K, 1)); auto layout_SFB = CfgT::tile_atom_to_shape_SFB(make_shape(M, N_il, K, 1)); From e17e891f26b305c8cd9169f20042cb81d7baa8c2 Mon Sep 17 00:00:00 2001 From: LiangSu8899 Date: Sat, 15 Aug 2026 15:27:56 -0400 Subject: [PATCH 03/32] perf(pi05): elide the fused GeGLU epilogue's D store (new default) The half-width epilogue's compact store node writes the real output, yet the collective still stages the unread D tile through smem and TMA-stores it. Fork the Sm100TmaWarpSpecialized collective (CollectiveEpilogueNoD) with the D store elided behind the same is_destination_supported pattern Sm90 and the Sm100 ptr-array epilogues already use. Encoder combiner 'epilogue_hw_nod' is the new default: a five-leg alternating sandwich measures 32.513/32.550/32.547 (hw) vs 31.956/31.967 (nod) at 3 views, -0.57 ms with <=0.037 ms drift, outputs bit-identical. Formal 3v/2v agree and pass all gates. Two measurement caveats are recorded in the guide: the isolated kernel benchmark scores the elision as a regression, and under nsys node tracing the variants converge, so the win is only observable unprofiled. The decoder-tile variant is a wash (its dummy store is 0.04 MB) and ships opt-in as --decoder-fused-geglu-nod. --- csrc/fp4_bindings.cpp | 80 + .../fp4/cutlass_fp4_gemm_geglu_il_sm100.cu | 69 + .../fp4/cutlass_fp4_gemm_geglu_il_sm100.cuh | 21 + csrc/gemm/fp4/sm100_epilogue_nod.hpp | 1304 +++++++++++++++++ docs/pi05_thor.md | 54 +- flash_rt/api.py | 9 +- flash_rt/frontends/torch/pi05_thor_fp4.py | 28 +- .../hardware/thor/shared_primitives_fp4.py | 12 +- flash_rt/models/pi05/pipeline_thor.py | 6 +- tests/bench_pi05_decoder_fp4_e2e.py | 16 +- tests/test_pi05_fp4_fusion_kernels.py | 44 + 11 files changed, 1605 insertions(+), 38 deletions(-) create mode 100644 csrc/gemm/fp4/sm100_epilogue_nod.hpp diff --git a/csrc/fp4_bindings.cpp b/csrc/fp4_bindings.cpp index 78dcca56..e0c69f48 100644 --- a/csrc/fp4_bindings.cpp +++ b/csrc/fp4_bindings.cpp @@ -949,6 +949,86 @@ The downstream GEMM keeps its original K = N_il/2 weight. R"pbdoc( Skinny-M half-width fused GeGLU GEMM on the decoder tile (128x64x256); same contract as cutlass_fp4_gemm_geglu_il_hw. +)pbdoc"); + + m.def("cutlass_fp4_gemm_geglu_il_hw_nod", + [](uintptr_t A_packed, uintptr_t SFA, + uintptr_t B_packed, uintptr_t SFB, + uintptr_t D_dummy, uintptr_t compact_packed, uintptr_t compact_sfa, + int M, int N_il, int K, uintptr_t stream) -> int { + const auto shape = fp4_kernel_shape({{"M", M}, {"N_il", N_il}, {"K", K}}); + require_fp4_ptrs("cutlass_fp4_gemm_geglu_il_hw_nod", + {{"A_packed", A_packed}, {"SFA", SFA}, + {"B_packed", B_packed}, {"SFB", SFB}, + {"D_dummy", D_dummy}, + {"compact_packed", compact_packed}, + {"compact_sfa", compact_sfa}}, shape); + require_fp4(M > 0 && N_il > 0 && K > 0 && (N_il % 32) == 0 && + (K % 16) == 0, + "cutlass_fp4_gemm_geglu_il_hw_nod", + "M must be positive, N_il a positive multiple of 32 " + "and K a positive multiple of 16", + shape); + return flash_rt::fp4::cutlass_fp4_gemm_geglu_il_hw_nod( + reinterpret_cast(A_packed), + reinterpret_cast(SFA), + reinterpret_cast(B_packed), + reinterpret_cast(SFB), + reinterpret_cast(D_dummy), + reinterpret_cast(compact_packed), + reinterpret_cast(compact_sfa), + M, N_il, K, + reinterpret_cast(stream)); + }, + py::arg("A_packed"), py::arg("SFA"), + py::arg("B_packed"), py::arg("SFB"), + py::arg("D_dummy"), py::arg("compact_packed"), py::arg("compact_sfa"), + py::arg("M"), py::arg("N_il"), py::arg("K"), + py::arg("stream") = 0, + R"pbdoc( +Half-width fused GeGLU GEMM with the collective's own D store elided: +compact_packed/compact_sfa are the only outputs and D_dummy is never +written (still validated; the host-side TMA descriptor needs a real +pointer). Same contract as cutlass_fp4_gemm_geglu_il_hw otherwise. +)pbdoc"); + + m.def("cutlass_fp4_gemm_geglu_il_hw_nod_v10", + [](uintptr_t A_packed, uintptr_t SFA, + uintptr_t B_packed, uintptr_t SFB, + uintptr_t D_dummy, uintptr_t compact_packed, uintptr_t compact_sfa, + int M, int N_il, int K, uintptr_t stream) -> int { + const auto shape = fp4_kernel_shape({{"M", M}, {"N_il", N_il}, {"K", K}}); + require_fp4_ptrs("cutlass_fp4_gemm_geglu_il_hw_nod_v10", + {{"A_packed", A_packed}, {"SFA", SFA}, + {"B_packed", B_packed}, {"SFB", SFB}, + {"D_dummy", D_dummy}, + {"compact_packed", compact_packed}, + {"compact_sfa", compact_sfa}}, shape); + require_fp4(M > 0 && N_il > 0 && K > 0 && (N_il % 32) == 0 && + (K % 16) == 0, + "cutlass_fp4_gemm_geglu_il_hw_nod_v10", + "M must be positive, N_il a positive multiple of 32 " + "and K a positive multiple of 16", + shape); + return flash_rt::fp4::cutlass_fp4_gemm_geglu_il_hw_nod_v10( + reinterpret_cast(A_packed), + reinterpret_cast(SFA), + reinterpret_cast(B_packed), + reinterpret_cast(SFB), + reinterpret_cast(D_dummy), + reinterpret_cast(compact_packed), + reinterpret_cast(compact_sfa), + M, N_il, K, + reinterpret_cast(stream)); + }, + py::arg("A_packed"), py::arg("SFA"), + py::arg("B_packed"), py::arg("SFB"), + py::arg("D_dummy"), py::arg("compact_packed"), py::arg("compact_sfa"), + py::arg("M"), py::arg("N_il"), py::arg("K"), + py::arg("stream") = 0, + R"pbdoc( +Skinny-M no-D-store fused GeGLU GEMM on the decoder tile (128x64x256); +same contract as cutlass_fp4_gemm_geglu_il_hw_nod. )pbdoc"); #ifdef FLASHRT_HAVE_COSMOS3_EDGE diff --git a/csrc/gemm/fp4/cutlass_fp4_gemm_geglu_il_sm100.cu b/csrc/gemm/fp4/cutlass_fp4_gemm_geglu_il_sm100.cu index 531e38d8..76ea721e 100644 --- a/csrc/gemm/fp4/cutlass_fp4_gemm_geglu_il_sm100.cu +++ b/csrc/gemm/fp4/cutlass_fp4_gemm_geglu_il_sm100.cu @@ -24,6 +24,7 @@ #include "cute/tensor.hpp" #include "gemm/fp4/sm100_gelu_mul_blockscale_visitor.hpp" +#include "gemm/fp4/sm100_epilogue_nod.hpp" namespace flash_rt { namespace fp4 { @@ -173,6 +174,48 @@ using GemmKernelHwV10 = cutlass::gemm::kernel::GemmUniversal< using GemmHwV10 = cutlass::gemm::device::GemmUniversalAdapter; +// ── No-D-store instantiations ── +// The builder's collective still stages the unread D tile through smem and +// TMA-stores it (row-aliased, but the smem->L2 traffic and store instructions +// remain). Rebind the built epilogue onto CollectiveEpilogueNoD +// (sm100_epilogue_nod.hpp), which elides that store entirely; the compact +// store node is the only writer. SharedStorage is unchanged, so the +// mainloop carveout from the builder output stays valid. +template +struct MakeNoD; + +template +struct MakeNoD, + Rest...>> { + using type = cutlass::epilogue::collective::CollectiveEpilogueNoD< + StagesC, StagesD, FragmentSize, ReuseSmemC, DelayTmaStore, Rest...>; +}; + +using CollectiveEpilogueHwNoD = typename MakeNoD::type; +static_assert(sizeof(typename CollectiveEpilogueHwNoD::SharedStorage) == + sizeof(typename CollectiveEpilogueHw::SharedStorage), + "NoD epilogue must keep the builder's smem footprint"); + +using GemmKernelHwNoD = cutlass::gemm::kernel::GemmUniversal< + Shape, + CollectiveMainloopHw, CollectiveEpilogueHwNoD, void>; + +using GemmHwNoD = cutlass::gemm::device::GemmUniversalAdapter; + +using CollectiveEpilogueHwNoDV10 = typename MakeNoD::type; +static_assert(sizeof(typename CollectiveEpilogueHwNoDV10::SharedStorage) == + sizeof(typename CollectiveEpilogueHwV10::SharedStorage), + "NoD epilogue must keep the builder's smem footprint"); + +using GemmKernelHwNoDV10 = cutlass::gemm::kernel::GemmUniversal< + Shape, + CollectiveMainloopHwV10, CollectiveEpilogueHwNoDV10, void>; + +using GemmHwNoDV10 = cutlass::gemm::device::GemmUniversalAdapter; + } // namespace geglu_il int cutlass_fp4_gemm_geglu_il( @@ -324,5 +367,31 @@ int cutlass_fp4_gemm_geglu_il_hw_v10( M, N_il, K, stream); } +int cutlass_fp4_gemm_geglu_il_hw_nod( + void const* A_packed, void const* SFA, + void const* B_packed, void const* SFB, + void* D_dummy, + void* compact_packed, + void* compact_sfa, + int M, int N_il, int K, + cudaStream_t stream) { + return geglu_il::run_geglu_il_hw( + A_packed, SFA, B_packed, SFB, D_dummy, compact_packed, compact_sfa, + M, N_il, K, stream); +} + +int cutlass_fp4_gemm_geglu_il_hw_nod_v10( + void const* A_packed, void const* SFA, + void const* B_packed, void const* SFB, + void* D_dummy, + void* compact_packed, + void* compact_sfa, + int M, int N_il, int K, + cudaStream_t stream) { + return geglu_il::run_geglu_il_hw( + A_packed, SFA, B_packed, SFB, D_dummy, compact_packed, compact_sfa, + M, N_il, K, stream); +} + } // namespace fp4 } // namespace flash_rt diff --git a/csrc/gemm/fp4/cutlass_fp4_gemm_geglu_il_sm100.cuh b/csrc/gemm/fp4/cutlass_fp4_gemm_geglu_il_sm100.cuh index 34e1bd4c..590822bc 100644 --- a/csrc/gemm/fp4/cutlass_fp4_gemm_geglu_il_sm100.cuh +++ b/csrc/gemm/fp4/cutlass_fp4_gemm_geglu_il_sm100.cuh @@ -54,5 +54,26 @@ int cutlass_fp4_gemm_geglu_il_hw_v10( int M, int N_il, int K, cudaStream_t stream); +// No-D-store variants: identical contract, but the collective's own D store +// is elided (D_dummy is never written; the pointer is still required for the +// host-side TMA descriptor and may be the same small shared buffer). +int cutlass_fp4_gemm_geglu_il_hw_nod( + void const* A_packed, void const* SFA, + void const* B_packed, void const* SFB, + void* D_dummy, + void* compact_packed, + void* compact_sfa, + int M, int N_il, int K, + cudaStream_t stream); + +int cutlass_fp4_gemm_geglu_il_hw_nod_v10( + void const* A_packed, void const* SFA, + void const* B_packed, void const* SFB, + void* D_dummy, + void* compact_packed, + void* compact_sfa, + int M, int N_il, int K, + cudaStream_t stream); + } // namespace fp4 } // namespace flash_rt diff --git a/csrc/gemm/fp4/sm100_epilogue_nod.hpp b/csrc/gemm/fp4/sm100_epilogue_nod.hpp new file mode 100644 index 00000000..46f8d47c --- /dev/null +++ b/csrc/gemm/fp4/sm100_epilogue_nod.hpp @@ -0,0 +1,1304 @@ +/*************************************************************************************************** + * Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: BSD-3-Clause + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, this + * list of conditions and the following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above copyright notice, + * this list of conditions and the following disclaimer in the documentation + * and/or other materials provided with the distribution. + * + * 3. Neither the name of the copyright holder nor the names of its + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE + * DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR + * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + **************************************************************************************************/ + +/*! \file + \brief Functor performing elementwise operations used by epilogues. + + FlashRT fork of cutlass/epilogue/collective/sm100_epilogue_tma_warpspecialized.hpp + (CUTLASS v4.4.2): CollectiveEpilogueNoD is the Sm100TmaWarpSpecialized + collective with the D store elided. It exists for fusion operations whose + store node writes the real output itself (e.g. the compact GeGLU store), + leaving the collective's own smem->TMA D path writing a buffer nothing + reads. Sm90 and the Sm100 ptr-array epilogues already support this via + is_destination_supported / ElementD=void; this collective never picked + that up, so the same guards are applied here. Semantic changes are the + three `if constexpr (StoreD)` sites; everything else is verbatim. +*/ + + + +#pragma once + +#include "cutlass/cutlass.h" +#include "cutlass/arch/barrier.h" +#include "cutlass/conv/convnd_problem_shape.hpp" +#include "cutlass/epilogue/dispatch_policy.hpp" +#include "cutlass/epilogue/collective/detail.hpp" +#include "cutlass/epilogue/thread/scale_type.h" +#include "cutlass/epilogue/fusion/callbacks.hpp" +#include "cutlass/epilogue/fusion/sm100_callbacks_tma_warpspecialized.hpp" +#include "cutlass/detail/layout.hpp" +#include "cutlass/detail/helper_macros.hpp" +#include "cutlass/trace.h" + +#include "cutlass/conv/detail.hpp" +#include "cute/tensor.hpp" +#include "cutlass/cuda_host_adapter.hpp" + +///////////////////////////////////////////////////////////////////////////////////////////////// + +namespace cutlass::epilogue::collective { + +///////////////////////////////////////////////////////////////////////////////////////////////// + +template < + int StagesC_, + int StagesD_, + int FragmentSize_, + bool ReuseSmemC_, + bool DelayTmaStore_, + class CtaTileShape_, // (CTA_M,CTA_N,CTA_K, optional: Tile_L) + class EpilogueTile_, // (EPI_TILE_M, EPI_TILE_N) + class ElementC_, + class StrideC_, + class ElementD_, + class StrideD_, + class FusionCallbacks_, + class CopyOpT2R_, + class CopyOpG2S_, + class SmemLayoutAtomC_, + class CopyOpS2R_, + class CopyOpS2G_, + class SmemLayoutAtomD_, + class CopyOpR2S_, + class CopyOpR2R_ +> +class CollectiveEpilogueNoD { +public: + // + // Type Aliases + // + using DispatchPolicy = Sm100TmaWarpSpecialized; + using CtaTileShape = CtaTileShape_; + using EpilogueTile = EpilogueTile_; + using FusionCallbacks = FusionCallbacks_; + using ElementC = ElementC_; + using StrideC = StrideC_; + using ElementD = ElementD_; + using StrideD = StrideD_; + using CopyOpT2R = CopyOpT2R_; + using CopyOpG2S = CopyOpG2S_; + using SmemLayoutAtomC = SmemLayoutAtomC_; + using CopyOpS2R = CopyOpS2R_; + using CopyOpS2G = CopyOpS2G_; + using SmemLayoutAtomD = SmemLayoutAtomD_; + using CopyOpR2S = CopyOpR2S_; + using CopyOpR2R = CopyOpR2R_; + + using ThreadEpilogueOp = typename epilogue::fusion::FusionCallbacksTraits::Operation; + using GmemTiledCopyC = CopyOpG2S; + using GmemTiledCopyD = CopyOpS2G; + + // The whole point of this fork: the fusion op's store node writes the real + // output, so the collective's own D store never issues. ReuseSmemC couples + // the C load pipeline release to D store completion and must stay off. + constexpr static bool StoreD = false; + static_assert(not ReuseSmemC_, "CollectiveEpilogueNoD requires ReuseSmemC == false"); + + constexpr static int ThreadCount = 128; + + static_assert(!is_layout::value && is_tuple::value, "EpilogueTile must be a cute::Tile or cute::Shape"); + static_assert(rank(EpilogueTile{}) == 2, "EpilogueTile must be rank-2: [EPI_TILE_M, EPI_TILE_N]"); + +private: + using GmemElementD = ElementD; + using GmemElementC = cute::conditional_t,ElementD,ElementC>; // prevents void ref breakages + using SmemElementD = typename cutlass::detail::get_unpacked_element_type::type; + using SmemElementC = typename cutlass::detail::get_unpacked_element_type::type; + constexpr static int StagesC = StagesC_; + constexpr static int StagesD = StagesD_; + static_assert(StagesC >= 1, "StagesC must be >= 1"); + static_assert(StagesD >= 1, "StagesD must be >= 1"); + + constexpr static bool ReuseSmemC = ReuseSmemC_; + constexpr static bool is_source_supported = not cute::is_void_v; + + constexpr static bool is_m_major_C = detail::is_m_major(); + constexpr static bool is_m_major_D = detail::is_m_major(); + + constexpr static bool is_im2col_C = cute::is_same_v; + constexpr static bool is_im2col_D = cute::is_same_v; + + using SmemLayoutStageC = decltype(tile_to_shape(SmemLayoutAtomC{}, product_each(shape(EpilogueTile{})), + cute::conditional_t, Step<_1,_2>>{} )); + using SmemLayoutStageD = decltype(tile_to_shape(SmemLayoutAtomD{}, product_each(shape(EpilogueTile{})), + cute::conditional_t, Step<_1,_2>>{} )); + + constexpr static int StageCBits = cosize_v * sizeof_bits_v; + constexpr static int StageDBits = cosize_v * sizeof_bits_v; + constexpr static int MaxStageBits = cute::max(StageCBits, StageDBits); + constexpr static int StrideStageC = (ReuseSmemC ? MaxStageBits : StageCBits) / sizeof_bits_v; + constexpr static int StrideStageD = (ReuseSmemC ? MaxStageBits : StageDBits) / sizeof_bits_v; + + using SmemLayoutC = decltype(cute::append<3>(SmemLayoutStageC{}, Layout, Int>{})); + using SmemLayoutD = decltype(cute::append<3>(SmemLayoutStageD{}, Layout, Int>{})); + + constexpr static bool support_smem_reuse = is_source_supported && StagesD <= StagesC + && MaxStageBits % sizeof_bits_v == 0 + && MaxStageBits % sizeof_bits_v == 0; + static_assert(not (ReuseSmemC && not support_smem_reuse), "Smem reuse requirements not met"); + + constexpr static size_t SmemAlignmentC = cutlass::detail::alignment_for_swizzle(SmemLayoutC{}); + constexpr static size_t SmemAlignmentD = cutlass::detail::alignment_for_swizzle(SmemLayoutD{}); + constexpr static size_t MaxSmemAlignment = cute::max(SmemAlignmentC, SmemAlignmentD); + + // Not unroll epi subtile loop when the activation op is heavy to reduce instruction size and register pressure. + constexpr static bool UnrollEpiLoop = + not cutlass::epilogue::thread::kIsHeavy_member_or_false::value; + // TMA store delay only benefits with loop unrolling + constexpr static bool DelayTmaStore = DelayTmaStore_ and UnrollEpiLoop; + + struct CollectiveStorageWithC { + alignas(SmemAlignmentC) ArrayEngine> smem_C; + alignas(SmemAlignmentD) ArrayEngine> smem_D; + }; + + union CollectiveStorageWithoutC { + cute::array smem_C; + alignas(SmemAlignmentD) ArrayEngine> smem_D; + }; + + union CollectiveStorageReuseC { + alignas(MaxSmemAlignment) ArrayEngine> smem_C; + alignas(MaxSmemAlignment) ArrayEngine> smem_D; + }; + +public: + // TMA pipeline for loading C + using LoadPipeline = cutlass::PipelineTransactionAsync; + using LoadPipelineState = cutlass::PipelineState; + constexpr static uint32_t TmaTransactionBytes = StageCBits / 8; + + // TMA pipeline for storing D + using StorePipeline = cute::conditional_t, + cutlass::PipelineTmaStore>; + using StorePipelineState = cutlass::PipelineState; + + struct SharedStorage { + struct TensorStorage { + using CollectiveStorage = cute::conditional_t>; + CollectiveStorage collective; + + using FusionStorage = typename FusionCallbacks::SharedStorage; + FusionStorage thread; + } tensors; + + using PipelineStorage = typename LoadPipeline::SharedStorage; + PipelineStorage pipeline; + }; + using TensorStorage = typename SharedStorage::TensorStorage; + using PipelineStorage = typename SharedStorage::PipelineStorage; + + // Planar complex kernels have two accumulator copies for the real and imaginary tensors. + constexpr static int NumAccumulatorMtxs = 1; + + // Host side epilogue arguments + struct Arguments { + typename FusionCallbacks::Arguments thread{}; + ElementC const* ptr_C = nullptr; + StrideC dC{}; + ElementD* ptr_D = nullptr; + StrideD dD{}; + }; + +private: + static constexpr auto + get_tma_epi_tile() { + return cute::transform_apply(EpilogueTile{}, seq<0,1>{}, + [] (auto epi_tiler, auto mode) { + auto cta_tiler_shape = get(CtaTileShape{}); + // Use a dynamic stride to prevent mode coalescing + auto cta_tiler_stride = repeat_like(cta_tiler_shape, 0); + auto cta_tiler = make_layout(cta_tiler_shape, cta_tiler_stride); + // This is a multimodal CTA tiler, transform before returning + if constexpr (depth(cta_tiler) > 0) { + // This is an implicit multimodal tiler, match profile and return + if constexpr (tuple_size_v == 1) { + return make_tile(epi_tiler); + } + // This is an explicit multimodal tiler, compose out epi tiler + else { + return shape(composition(cta_tiler, epi_tiler)); + } + } + // This is a flat CTA tiler, no need for transformation + else { + return epi_tiler; + } + }, + [] (auto... epi_tilers) { + return make_tile(epi_tilers...); + } + ); + } + + using TmaEpilogueTile = decltype(get_tma_epi_tile()); + + template + static constexpr auto + get_tma_load_c(ProblemShapeMNL const& problem_shape_mnl, Arguments const& args) { + Tensor tensor_c = make_tensor(make_gmem_ptr(args.ptr_C), + make_layout(problem_shape_mnl, append<3>(args.dC, _0{}))); + return make_tma_copy(CopyOpG2S{}, tensor_c, SmemLayoutStageC{}, TmaEpilogueTile{}, _1{}); + } + + template + static constexpr auto + get_tma_store_d(ProblemShapeMNL const& problem_shape_mnl, Arguments const& args) { + Tensor tensor_d = make_tensor(make_gmem_ptr(args.ptr_D), + make_layout(problem_shape_mnl, append<3>(args.dD, _0{}))); + return make_tma_copy(CopyOpS2G{}, tensor_d, SmemLayoutStageD{}, TmaEpilogueTile{}, _1{}); + } + +public: + // Device side epilogue params + struct Params { + using TMA_C = decltype(get_tma_load_c (repeat_like(append<3>(StrideC{},_1{}), int32_t(0)), Arguments{})); + using TMA_D = decltype(get_tma_store_d(repeat_like(append<3>(StrideD{},_1{}), int32_t(0)), Arguments{})); + + typename FusionCallbacks::Params thread{}; + TMA_C tma_load_c; + TMA_D tma_store_d; + }; + + // + // Gemm Host Functions + // + + template + static constexpr Params + to_underlying_arguments( + ProblemShape const& problem_shape, + Arguments const& args, + [[maybe_unused]] void* workspace) { + // Optionally append 1s until problem shape is rank-4 in case its is only rank-3 (MNK) + auto problem_shape_mnl = select<0,1,3>(append<4>(problem_shape, 1)); + typename Params::TMA_C tma_load_c{}; + if constexpr (is_source_supported) { + tma_load_c = get_tma_load_c(problem_shape_mnl, args); + } + + typename Params::TMA_D tma_store_d = get_tma_store_d(problem_shape_mnl, args); + + return { + FusionCallbacks::to_underlying_arguments(problem_shape, args.thread, workspace), + tma_load_c, + tma_store_d + }; + } + + template + static size_t + get_workspace_size(ProblemShape const& problem_shape, Arguments const& args) { + return FusionCallbacks::get_workspace_size(problem_shape, args.thread); + } + + template + static cutlass::Status + initialize_workspace(ProblemShape const& problem_shape, Arguments const& args, void* workspace, cudaStream_t stream, + CudaHostAdapter* cuda_adapter = nullptr) { + return FusionCallbacks::initialize_workspace(problem_shape, args.thread, workspace, stream, cuda_adapter); + } + + template + static bool + can_implement( + ProblemShape const& problem_shape, + [[maybe_unused]] Arguments const& args) { + constexpr int tma_alignment_bits_d = cutlass::detail::get_output_alignment_bits(); + auto problem_shape_MNKL = append<4>(problem_shape, 1); + auto [M,N,K,L] = problem_shape_MNKL; + auto shape = cute::make_shape(M,N,L); + + bool implementable = true; + constexpr int min_tma_aligned_elements_D = tma_alignment_bits_d / cutlass::sizeof_bits::value; + if constexpr (cute::is_same_v) { // ignore L stride for implicit gemm + implementable = implementable && cutlass::detail::check_alignment(take<0,2>(shape), take<0,2>(StrideD{})); + } + else { + implementable = implementable && cutlass::detail::check_alignment(shape, StrideD{}); + } + + if constexpr (is_source_supported) { + constexpr int tma_alignment_bits_c = cutlass::detail::get_output_alignment_bits(); + constexpr int min_tma_aligned_elements_C = tma_alignment_bits_c / cutlass::sizeof_bits::value; + if constexpr (cute::is_same_v) { // ignore L stride for implicit gemm + implementable = implementable && cutlass::detail::check_alignment(take<0,2>(shape), take<0,2>(StrideC{})); + } + else { + implementable = implementable && cutlass::detail::check_alignment(shape, StrideC{}); + } + } + + if (!implementable) { + CUTLASS_TRACE_HOST(" CAN IMPLEMENT: Problem Size doesn't meet the minimum alignment requirements for TMA.\n"); + } + + bool fusion_implementable = FusionCallbacks::can_implement(problem_shape, args.thread); + + if (!fusion_implementable) { + CUTLASS_TRACE_HOST(" CAN IMPLEMENT: Problem Size doesn't meet the minimum requirements for FusionCallbacks.\n"); + } + + return implementable && fusion_implementable; + } + + // + // Conv Host Functions + // + + template + static constexpr Params + to_underlying_arguments(cutlass::conv::ConvProblemShape const& problem_shape, Arguments const& args, void* workspace) { + return to_underlying_arguments(cutlass::conv::detail::get_transformed_problem_shape_MNKL(problem_shape), args, workspace); + } + + template + static size_t + get_workspace_size(cutlass::conv::ConvProblemShape const& problem_shape, Arguments const& args) { + return get_workspace_size(cutlass::conv::detail::get_transformed_problem_shape_MNKL(problem_shape), args); + } + + template + static cutlass::Status + initialize_workspace(cutlass::conv::ConvProblemShape const& problem_shape, Arguments const& args, + void* workspace, cudaStream_t stream, CudaHostAdapter* cuda_adapter = nullptr) { + return initialize_workspace(cutlass::conv::detail::get_transformed_problem_shape_MNKL(problem_shape), args, workspace, stream, cuda_adapter); + } + + template + static bool + can_implement(cutlass::conv::ConvProblemShape const& problem_shape, Arguments const& args) { + return can_implement(cutlass::conv::detail::get_transformed_problem_shape_MNKL(problem_shape), args); + } + + // + // Static Device Functions + // + + template + CUTLASS_DEVICE + static constexpr int + get_load_pipe_increment(CtaTileMNK const& cta_tile_mnk) { + // Compute number of epilogue subtiles + return size<1>(zipped_divide(make_layout(take<0,2>(cta_tile_mnk)), EpilogueTile{})); + } + + template + CUTLASS_DEVICE + static constexpr int + get_store_pipe_increment(CtaTileMNK const& cta_tile_mnk) { + return get_load_pipe_increment(cta_tile_mnk); + } + + /// Issue Tma Descriptor Prefetch -- ideally from a single thread for best performance + CUTLASS_DEVICE static void + prefetch_tma_descriptors(Params const& epilogue_params) { + cute::prefetch_tma_descriptor(epilogue_params.tma_load_c.get_tma_descriptor()); + cute::prefetch_tma_descriptor(epilogue_params.tma_store_d.get_tma_descriptor()); + } + + // + // Constructor and Data Members + // + CUTLASS_DEVICE + CollectiveEpilogueNoD(Params const& params_, TensorStorage& shared_tensors) + : params(params_), fusion_callbacks(params_.thread, shared_tensors.thread) {} + +private: + Params const& params; + FusionCallbacks fusion_callbacks; + + // + // Non-static Device Functions + // +public: + CUTLASS_DEVICE bool + is_producer_load_needed() const { + return fusion_callbacks.is_producer_load_needed(); + } + + template< + bool ReuseTmem = false, + class ProblemShapeMNKL, + class CtaTileMNK, + class CtaCoordMNKL, + class MmaTileMNK, + class TiledMma + > + CUTLASS_DEVICE auto + load( + LoadPipeline load_pipeline, + LoadPipelineState load_pipe_producer_state, + ProblemShapeMNKL problem_shape_mnkl, + CtaTileMNK cta_tile_mnk, + CtaCoordMNKL cta_coord_mnkl, + MmaTileMNK mma_tile_mnk, + TiledMma tiled_mma, + TensorStorage& shared_tensors, + bool reverse_epi_n = false) { + using namespace cute; + + int lane_idx = canonical_lane_idx(); + auto [M, N, K, L] = problem_shape_mnkl; + auto [m_coord, n_coord, k_coord, l_coord] = cta_coord_mnkl; + + // The tma tensor C under im2col mode only has two modes (M, N) which + // should be local tiled with only (m_coord, n_coord). + auto coord_shape = + conditional_return(make_coord(m_coord, n_coord), make_coord(m_coord, n_coord, l_coord)); + + // Represent the full source tensor, slice to get the tile this CTA is currently responsible for + Tensor mC_mn = params.tma_load_c.get_tma_tensor(make_shape(M,N,L)); // (M,N,L) + Tensor mC = coalesce(mC_mn, take<0,2>(cta_tile_mnk)); + Tensor gC = local_tile(mC, take<0,2>(cta_tile_mnk), coord_shape); // (CTA_M,CTA_N) + + // Apply epilogue subtile, get matching smem tensor + auto ptr_sC = shared_tensors.collective.smem_C.begin(); + Tensor gC_epi = flat_divide(gC, EpilogueTile{}); // (EPI_TILE_M,EPI_TILE_N,EPI_M,EPI_N) + Tensor sC_epi = make_tensor(make_smem_ptr(ptr_sC), SmemLayoutC{}); // (EPI_TILE_M,EPI_TILE_N,PIPE_C) + + // Prepare the thread(b)lock's (G)mem to (S)mem TMA tiled copy (bGS_) + ThrCopy thrblk_g2s = params.tma_load_c.get_slice(Int<0>{}); + Tensor bGS_gC = thrblk_g2s.partition_S(gC_epi); // (TMA,TMA_M,TMA_N,EPI_M,EPI_N) + Tensor bGS_sC = thrblk_g2s.partition_D(sC_epi); // (TMA,TMA_M,TMA_N,PIPE_C) + + // Get the fusion callbacks for the producer load warp + auto pld_args = cutlass::epilogue::fusion::detail::ProducerLoadArgs{ + problem_shape_mnkl, + cta_tile_mnk, + cta_coord_mnkl, + tiled_mma, + EpilogueTile{}, + lane_idx + }; + auto pld_callbacks = fusion_callbacks.get_producer_load_callbacks(pld_args); + bool is_C_load_needed = is_source_supported && fusion_callbacks.is_C_load_needed(); + + // Predication for TMA load (one thread issues TMA load) + bool issue_tma_load = cute::elect_one_sync(); + + // Pre-loop fusion callback entry point + pld_callbacks.begin(); + + CUTLASS_PRAGMA_UNROLL + for (int iter_n = 0; iter_n < size<3>(gC_epi); ++iter_n) { + CUTLASS_PRAGMA_UNROLL + for (int iter_m = 0; iter_m < size<2>(gC_epi); ++iter_m) { + int epi_m = iter_m, epi_n = iter_n; + if constexpr (ReuseTmem) { + if (reverse_epi_n) { + epi_n = size<3>(gC_epi) - 1 - iter_n; + } + } + // Acquire the lock for this stage + constexpr uint16_t mcast_mask = 0; + uint64_t* tma_barrier = load_pipeline.producer_get_barrier(load_pipe_producer_state); + load_pipeline.producer_acquire(load_pipe_producer_state); + + // Execute the TMA load for C if needed + if (issue_tma_load && is_C_load_needed) { + copy(params.tma_load_c.with(*tma_barrier, mcast_mask), + bGS_gC(_,_,_,epi_m,epi_n), bGS_sC(_,_,_,load_pipe_producer_state.index())); + load_pipeline.producer_expect_transaction(load_pipe_producer_state); + } + + // Loop fusion callback entry point + pld_callbacks.step(tma_barrier, epi_m, epi_n, load_pipe_producer_state.count(), issue_tma_load); + + // Commit TMA loads for this stage and release the lock + load_pipeline.producer_commit(load_pipe_producer_state); + ++load_pipe_producer_state; + } + } + + // Post-loop fusion callback entry point + pld_callbacks.end(); + + return load_pipe_producer_state; + } + + CUTLASS_DEVICE void + load_tail( + LoadPipeline load_pipeline, + LoadPipelineState load_pipe_producer_state, + [[maybe_unused]] StorePipeline store_pipeline, + [[maybe_unused]] StorePipelineState store_pipe_producer_state) { + load_pipeline.producer_tail(load_pipe_producer_state); + } + + template< + bool ReuseTmem = false, + class AccumulatorPipeline, + class AccumulatorPipelineState, + class ProblemShapeMNKL, + class CtaTileMNK, + class CtaCoordMNKL, + class MmaTileMNK, + class TiledMma, + class AccEngine, + class AccLayout + > + CUTLASS_DEVICE auto + store( + LoadPipeline load_pipeline, + LoadPipelineState load_pipe_consumer_state, + StorePipeline store_pipeline, + StorePipelineState store_pipe_producer_state, + AccumulatorPipeline acc_pipeline, + AccumulatorPipelineState acc_pipe_consumer_state, + ProblemShapeMNKL problem_shape_mnkl, + CtaTileMNK cta_tile_mnk, + CtaCoordMNKL cta_coord_mnkl, + MmaTileMNK mma_tile_mnk, + TiledMma tiled_mma, + cute::Tensor accumulators, + TensorStorage& shared_tensors + ) { + using namespace cute; + using ElementAccumulator = typename AccEngine::value_type; + using ElementCompute_ = typename epilogue::fusion::FusionCallbacksTraits::ElementCompute; + using ElementCompute = cute::conditional_t,ElementAccumulator,ElementCompute_>; + + static_assert(is_tmem::value, "Accumulator must be TMEM resident."); + static_assert(rank(accumulators) == 3, "Accumulators must be MMA-partitioned: [MMA, MMA_M, MMA_N]"); + static_assert(size<1>(accumulators) == 1 && size<2>(accumulators) == 1, "TiledMMA must match partitioned ShapeMN"); + static_assert(rank(ProblemShapeMNKL{}) == 4, "ProblemShapeMNKL must be rank 4"); + static_assert(rank(CtaCoordMNKL{}) == 4, "CoordMNKL must be rank 4"); + + // Indexing variables + auto [M, N, K, L] = problem_shape_mnkl; + auto [m_coord, n_coord, k_coord, l_coord] = cta_coord_mnkl; + int thread_idx = threadIdx.x % ThreadCount; + int warp_idx = thread_idx / NumThreadsPerWarp; + [[maybe_unused]] int lane_idx = thread_idx % NumThreadsPerWarp; + + // The tma tensor D under im2col mode only has two modes (M, N) which + // should be local tiled with only (m_coord, n_coord). + auto coord_shape = + conditional_return(make_coord(m_coord, n_coord), make_coord(m_coord, n_coord, l_coord)); + + // Represent the full output tensor, slice to get the tile this CTA is responsible for + Tensor mD_mn = params.tma_store_d.get_tma_tensor(make_shape(M,N,L)); // (M,N,L) + Tensor mD = coalesce(mD_mn, take<0,2>(cta_tile_mnk)); + Tensor gD = local_tile(mD, take<0,2>(cta_tile_mnk), coord_shape); // (CTA_M,CTA_N) + + Tensor tAcc = accumulators(make_coord(_,_),_0{},_0{}); // (CTA_M,CTA_N) + + // Apply epilogue subtiling + Tensor tAcc_epi = flat_divide(tAcc, EpilogueTile{}); // (EPI_TILE_M,EPI_TILE_N,EPI_M,EPI_N) + Tensor gD_epi = flat_divide( gD, EpilogueTile{}); // (EPI_TILE_M,EPI_TILE_N,EPI_M,EPI_N) + + // Construct the corresponding pipelined smem tensors + auto ptr_sC = shared_tensors.collective.smem_C.begin(); + auto ptr_sD = shared_tensors.collective.smem_D.begin(); + Tensor sC_epi = cute::as_position_independent_swizzle_tensor( + make_tensor(make_smem_ptr(ptr_sC), SmemLayoutC{})); // (EPI_TILE_M,EPI_TILE_N,PIPE_C) + Tensor sD_epi = cute::as_position_independent_swizzle_tensor( + make_tensor(make_smem_ptr(ptr_sD), SmemLayoutD{})); // (EPI_TILE_M,EPI_TILE_N,PIPE_D) + + // (t)hread-partition for (t)mem to (r)egister copy (tTR_) + TiledCopy tiled_t2r = make_tmem_copy(CopyOpT2R{}, tAcc_epi(_,_,_0{},_0{})); + ThrCopy thread_t2r = tiled_t2r.get_slice(thread_idx); + Tensor tTR_tAcc = thread_t2r.partition_S(tAcc_epi); // (T2R,T2R_M,T2R_N,EPI_M,EPI_N) + Tensor tTR_sD = thread_t2r.partition_D(sD_epi(_,_,_0{})); // (T2R,T2R_M,T2R_N) + + // Allocate D and accumulator registers + // Does directly store the visitor into smem. + constexpr bool IsDirectR2S = cute::is_same_v>; + using RegisterElementD = cute::conditional_t; + Tensor tTR_rAcc = make_tensor(shape(tTR_sD)); // (T2R,T2R_M,T2R_N) + Tensor tTR_rD = make_tensor(shape(tTR_sD)); // (T2R,T2R_M,T2R_N) + + // Vectorized fragment view + constexpr int FragmentSize = DispatchPolicy::FragmentSize; + Tensor tTR_rAcc_frg = recast>(coalesce(tTR_rAcc)); // (EPI_V) + Tensor tTR_rD_frg = recast>(coalesce(tTR_rD)); // (EPI_V) + CUTE_STATIC_ASSERT(size(tTR_rAcc) % DispatchPolicy::FragmentSize == 0, "Fragment size does not vectorize properly"); + + // (t)hread-partition for (s)mem to (r)egister copy (tSR_) + TiledCopy tiled_s2r = make_tiled_copy_D(Copy_Atom{}, tiled_t2r); + ThrCopy thread_s2r = tiled_s2r.get_slice(thread_idx); + Tensor tSR_sC = thread_s2r.partition_S(sC_epi); // (S2R,S2R_M,S2R_N,PIPE_C) + Layout tSR_rC_layout = thread_s2r.retile_D(tTR_rD).layout(); // (S2R,S2R_M,S2R_N) + + // Allocate C registers + // If C smem load is a non-vectorized dst(i) = src(i) then we can allocate C registers directly in the compute type + // to eliminate some redundant pack+unpack instruction sequences for sub-word types + constexpr bool IsDirectS2R = cute::is_same_v> + && decltype(max_common_vector(tSR_rC_layout, tSR_sC.layout()))::value <= 1; + using RegisterElementC = cute::conditional_t; + Tensor tTR_rC = make_tensor(shape(tTR_sD)); // (T2R,T2R_M,T2R_N) + Tensor tSR_rC = thread_s2r.retile_D(tTR_rC); // (S2R,S2R_M,S2R_N) + + // (t)hread-partition for (r)egister to (r)egister copy (tRR_) + TiledCopy tiled_r2r = make_tiled_copy_D(Copy_Atom{}, tiled_t2r); + ThrCopy thread_r2r = tiled_r2r.get_slice(thread_idx); + Tensor tRR_rD_src = thread_r2r.retile_S(tTR_rD); // (R2R,R2R_M,R2R_N,EPI_M,EPI_N) + Tensor tRR_rD_dst = thread_r2r.retile_D(tTR_rD); // (R2R,R2R_M,R2R_N,EPI_M,EPI_N) + + // (t)hread-partition for (r)egister to (s)mem copy (tRS_) + TiledCopy tiled_r2s = make_tiled_copy_D(Copy_Atom{}, tiled_r2r); + ThrCopy thread_r2s = tiled_r2s.get_slice(thread_idx); + Tensor tRS_sD = thread_r2s.partition_D(sD_epi); // (R2S,R2S_M,R2S_N,PIPE_D) + Tensor tRS_rD = [&]() CUTLASS_LAMBDA_FUNC_INLINE { + if constexpr (!IsDirectR2S) { + return make_tensor(shape(tRS_sD(_,_,_,_0{}))); + } + else{ + return thread_r2s.retile_S(tTR_rD); // (R2S,R2S_M,R2S_N) + } + }(); + + Tensor tRR_rD_dst_frg = recast>(coalesce(tRR_rD_dst)); + Tensor tRS_rD_frg = recast>(coalesce(tRS_rD)); + + // thread(b)lock-partition for (s)mem to (g)mem copy (bSG_) + ThrCopy thrblk_s2g = params.tma_store_d.get_slice(Int<0>{}); + Tensor bSG_sD = thrblk_s2g.partition_S(sD_epi); // (S2G,S2G_M,S2G_N,PIPE_D) + Tensor bSG_gD = thrblk_s2g.partition_D(gD_epi); // (S2G,S2G_M,S2G_N,EPI_M,EPI_N) + + // OOB predication for tile quantization "residue" + // Absolute coordinate tensors (dynamic) + Tensor mD_crd = make_identity_tensor(make_shape(M,N)); // (M,N) + Tensor cD_mn = local_tile(mD_crd, take<0,2>(cta_tile_mnk), make_coord(m_coord, n_coord)); // (CTA_M,CTA_N) + Tensor tTR_cD_mn = thread_t2r.partition_D(flat_divide(cD_mn, EpilogueTile{})); // (T2R,T2R_M,T2R_N,EPI_M,EPI_N) + // Relative coordinate tensors (static) + Tensor cD = make_coord_tensor(cD_mn.layout()); // (CTA_M,CTA_N) + Tensor tTR_cD = make_coord_tensor(tTR_cD_mn.layout()); // (T2R,T2R_M,T2R_N,EPI_M,EPI_N) + // Subtract the global "bottom right" corner from the local "top left" corner to get the max relative coordinate + auto residue_cD = make_coord(M,N) - cD_mn(_0{}); // (m,n) + auto residue_tTR_cD = make_coord(M,N) - tTR_cD_mn(_0{}); // (m,n) + + // Arguments for the fusion callbacks for the consumer store warps + constexpr bool RefSrc = false; // Register tensors reference T2R copy dst layout + auto cst_args = cutlass::epilogue::fusion::detail::ConsumerStoreArgs{ + problem_shape_mnkl, + cta_tile_mnk, + cta_coord_mnkl, + tiled_mma, + EpilogueTile{}, + tiled_t2r, + cD, + residue_cD, + tTR_cD, + residue_tTR_cD, + tTR_rC, + thread_idx + }; + + // Thread synchronizer for previously issued waits or fences + // to ensure visibility of smem reads/writes to threads or TMA unit + auto synchronize = [] () { cutlass::arch::NamedBarrier::sync(ThreadCount, cutlass::arch::ReservedNamedBarriers::EpilogueBarrier); }; + + // Predication for sub-128 thread T2R tiled copy + Layout tmem_warp_layout = typename decltype(make_tmem_warp_partitioner(tAcc_epi(_,_,0,0)))::TiledLayout_TV{}; + constexpr bool predicate_tmem_load = size(tmem_warp_layout) != cosize(tmem_warp_layout); + bool issue_tmem_load = true; + + // If tmem doesn't have enough capacity to support double buffering, a portion of tmem (a column of epilogue tiles) + // is overlapped between 2 pseudo-buffers. The shared tmem portion corresponds to the last epilogue tile column of + // tmem accumulator buffer 0, and the first epilogue tile column of tmem accumulator 1. + // Thus, whenever we are processing tmem accumulator buffer 0, we process the epilogue tiles with reversed column order. + // Once the last epilogue tile column is loaded from tmem, the acc_pipeline is released. + // Then, the next accumulation stage for buffer 1 can start. + [[maybe_unused]] bool reverse_epi_n = ReuseTmem && acc_pipe_consumer_state.phase() == 0; + static_assert(not (ReuseTmem && AccumulatorPipeline::Stages != 1), "Tmem reuse requires 1 accumulator stage"); + + // Predication for TMA store (one warp issues TMA store) + bool issue_tma_store = warp_idx == 0; + + // In the reuse smem configuration we have StagesC smem buffers and at most StagesD committed TMA stores in flight. + // The TMA store pipeline producer acquire returns when at most StagesD-1 committed stores are in-flight, so we can + // only guarantee store completion after StagesD iterations, then we can begin issuing releases on the smem buffer locks. + // store_pipe_producer_state tracks the acquire and load_pipe_consumer_state tracks the release, in circular buffer fashion. + // If TMA store supported async transaction mbarriers we would not need this synchronous release behavior. + LoadPipelineState load_wait_state = load_pipe_consumer_state; + if constexpr (ReuseSmemC) { + load_wait_state = store_pipe_producer_state; + load_wait_state.phase_ ^= 1; + } + + // We can delay issue of TMA store by one iteration to achieve better interleaving of non-TMA instructions + // Sync requirements of smem reuse may preclude this optimization + // Delayed stores cause delayed stage releases which causes deadlock when StagesC == StagesD + [[maybe_unused]] int epi_m_prev = 0; + [[maybe_unused]] int epi_n_prev = 0; + static_assert(not (DelayTmaStore and ReuseSmemC and StagesC <= StagesD), "This TMA epilogue configuration will deadlock"); + + // The Epilogue Loop + auto epi_loop_fn = [&] (auto& cst_callbacks) CUTLASS_LAMBDA_FUNC_INLINE { + bool is_producer_load_needed = fusion_callbacks.is_producer_load_needed(); + bool is_C_load_needed = is_source_supported && fusion_callbacks.is_C_load_needed(); + + // The TMA store sequence for one epilogue loop iteration + auto tma_store_fn = [&] (int epi_m, int epi_n) CUTLASS_LAMBDA_FUNC_INLINE { + // Write the tile from smem to gmem with TMA + cutlass::arch::fence_view_async_shared(); // ensure smem writes are visible to TMA + synchronize(); // ensure all threads have issued their async fence + if (issue_tma_store) { + if constexpr (StoreD) { + copy(params.tma_store_d, bSG_sD(_,_,_,store_pipe_producer_state.index()), bSG_gD(_,_,_,epi_m,epi_n)); + } + } + + // Post async fence, pre TMA commit callback entry point + cst_callbacks.tma_store(epi_m, epi_n, store_pipe_producer_state.count(), issue_tma_store); + + // Commit the TMA stores for this stage + if (issue_tma_store) { + store_pipeline.producer_commit(store_pipe_producer_state); + } + ++store_pipe_producer_state; + + // Wait for the next smem buffer to be available + if (issue_tma_store) { + store_pipeline.producer_acquire(store_pipe_producer_state); + } + synchronize(); + + if constexpr (ReuseSmemC) { + // producer_acquire returns when at most StagesD-1 committed stores are pending + bool store_finished = store_pipe_producer_state.count() > StorePipeline::UnacquiredStages; + // Let dma warp know earliest smem buffer is consumed and empty after StagesD producer commits + if (store_finished) { + if (is_producer_load_needed) { + load_pipeline.consumer_release(load_pipe_consumer_state); + } + ++load_pipe_consumer_state; + } + } + }; // tma_store_fn + + cst_callbacks.begin(); + if (cst_callbacks.begin_sync_needed()) { + synchronize(); + } + + // Begin the wait for the producer load results + ConsumerToken load_wait_token{BarrierStatus::WaitDone}; + if (is_producer_load_needed) { + load_wait_token = load_pipeline.consumer_try_wait(load_wait_state); + } + // Begin the wait for the accumulator results + ConsumerToken acc_wait_token = acc_pipeline.consumer_try_wait(acc_pipe_consumer_state); + + // For each epilogue subtile within the CTA tile + constexpr int NumEpiSubtilesN = CUTE_STATIC_V(size<3>(gD_epi)); + constexpr int NumEpiSubtilesM = CUTE_STATIC_V(size<2>(gD_epi)); + #pragma unroll(UnrollEpiLoop ? NumEpiSubtilesN : 1) + for (int iter_n = 0; iter_n < NumEpiSubtilesN; ++iter_n) { + #pragma unroll(UnrollEpiLoop ? NumEpiSubtilesM : 1) + for (int iter_m = 0; iter_m < NumEpiSubtilesM; ++iter_m) { + int epi_m = iter_m, epi_n = iter_n; + bool is_first_iteration = iter_m == 0 && iter_n == 0; + bool is_last_iteration = iter_m == size<2>(gD_epi)-1 && iter_n == size<3>(gD_epi)-1; + bool do_acc_release = is_last_iteration; + + // Reverse subtile order for tmem reuse if necessary + if constexpr (ReuseTmem) { + if (reverse_epi_n) { + epi_n = size<3>(gD_epi) - 1 - iter_n; + } + do_acc_release = iter_m == size<2>(gD_epi)-1 && iter_n == 0; + } + + cst_callbacks.begin_loop(epi_m, epi_n); + + if (is_producer_load_needed) { + // Wait for the producer load to fill smem + load_pipeline.consumer_wait(load_wait_state, load_wait_token); + + if (is_C_load_needed) { + // Copy source tile from smem to register + copy(tiled_s2r, tSR_sC(_,_,_,load_wait_state.index()), tSR_rC); + // Ensure smem loads are complete before reusing smem for mixed types/layouts + if constexpr (ReuseSmemC && not (SmemLayoutC{} == SmemLayoutD{})) { + synchronize(); + } + } + } + + // First loop fusion callback entry point + cst_callbacks.previsit(epi_m, epi_n, load_wait_state.count(), is_producer_load_needed); + + if (is_producer_load_needed) { + // Let producer load warp know smem buffers are consumed and empty + if constexpr (not ReuseSmemC) { + cutlass::arch::fence_view_async_shared(); + load_pipeline.consumer_release(load_pipe_consumer_state); + ++load_pipe_consumer_state; + } + ++load_wait_state; + } + + if (is_first_iteration) { + // Wait for mma warp to fill tmem buffer with accumulator results + acc_pipeline.consumer_wait(acc_pipe_consumer_state, acc_wait_token); + } + + // The current tile in tmem + Tensor tTR_tAcc_mn = tTR_tAcc(_,_,_,epi_m,epi_n); + + // Compute tmem load predication if necessary + if constexpr (predicate_tmem_load) { + // Issue tmem load if this tile's tmem subpartition is accessible by this warp + int subpart_idx = (tTR_tAcc_mn.data().dp_ / 32) % 4; + issue_tmem_load = warp_idx == subpart_idx; + } + bool issue_smem_store = issue_tmem_load; + + // Copy accumulator tile from tmem to register + if (issue_tmem_load) { + copy(tiled_t2r, tTR_tAcc_mn, tTR_rAcc); + } + + // After the last tmem load, signal that tmem buffer is consumed and empty + if (do_acc_release) { + cutlass::arch::fence_view_async_tmem_load(); + acc_pipeline.consumer_release(acc_pipe_consumer_state); + ++acc_pipe_consumer_state; + } + + // Vectorized fragment loop with visitor callback entry point + CUTLASS_PRAGMA_UNROLL + for (int epi_v = 0; epi_v < size(tTR_rD_frg); ++epi_v) { + tTR_rD_frg(epi_v) = cst_callbacks.visit(tTR_rAcc_frg(epi_v), epi_v, epi_m, epi_n); + } + + // The latest we can delay the TMA store is right before the smem store of the next iteration + // since the current TMA store needs to be committed before we can acquire the next smem buffer + if constexpr (DelayTmaStore) { + // Issue TMA stores for the previous subtile + if (not is_first_iteration) { + tma_store_fn(epi_m_prev, epi_n_prev); + } + epi_m_prev = epi_m; + epi_n_prev = epi_n; + } + + if constexpr (StoreD) { + if constexpr (!IsDirectR2S) { + // At present, only FP4 col output with scalefactor generation fusion would go into these branch + copy(tiled_r2r, tRR_rD_src, tRR_rD_dst); + } + tRS_rD_frg(_0{}) = cutlass::NumericArrayConverter{}(tRR_rD_dst_frg(_0{})); + } + + // Smem reduction callback entry point using current store buffer for workspace + Tensor reduction_buffer = make_tensor(raw_pointer_cast(sD_epi(_,_,store_pipe_producer_state.index()).data()), + make_layout(stride<2>(get_nonswizzle_portion(SmemLayoutD{})), _1{})); + cst_callbacks.reduce(reduction_buffer, synchronize, epi_m, epi_n, is_last_iteration, tRS_rD_frg); + + // Copy output tile from register to smem + if (issue_smem_store) { + if constexpr (StoreD) { + copy(tiled_r2s, tRS_rD, tRS_sD(_,_,_,store_pipe_producer_state.index())); + } + } + + // Post reduction, pre TMA store callback entry point + cst_callbacks.postreduce(epi_m, epi_n, store_pipe_producer_state.count(), issue_smem_store); + + if constexpr (not DelayTmaStore) { + // Issue TMA stores for this subtile + tma_store_fn(epi_m, epi_n); + } + + cst_callbacks.end_loop(epi_m, epi_n); + + if (is_producer_load_needed) { + // Begin the wait for the next subtile producer load + load_wait_token = load_pipeline.consumer_try_wait(load_wait_state, is_last_iteration); + } + } // for epi_m + } // for epi_n + + if constexpr (DelayTmaStore) { + // Issue TMA stores for the last subtile + tma_store_fn(epi_m_prev, epi_n_prev); + } + + cst_callbacks.end(); + }; // epi_loop_fn + + // + // BEGIN EPILOGUE + // + auto cst_callbacks = fusion_callbacks.template get_consumer_store_callbacks(cst_args); + epi_loop_fn(cst_callbacks); + return cute::make_tuple(load_pipe_consumer_state, store_pipe_producer_state, acc_pipe_consumer_state); + } + + // API with Global Accumulator in registers for FastFP32 (emulated MMA) kernels. + // The accumulator in TMEM periodically loaded into the registers so that the MMA can clear out the TMEM accumulator + // values for better accuracy. This epilogue accepts the accumulator in registers and take TiledCopy for the + // TMEM->Reg as a parameter to be used in partitioning GMEM tensors C and D. + template< + class ProblemShapeMNKL, + class CtaTileMNK, + class CtaCoordMNKL, + class MmaTileMNK, + class TiledMma, + class AccEngine, + class AccLayout, + class TiledCopyT2R + > + CUTLASS_DEVICE auto + store( + LoadPipeline load_pipeline, + LoadPipelineState load_pipe_consumer_state, + StorePipeline store_pipeline, + StorePipelineState store_pipe_producer_state, + ProblemShapeMNKL problem_shape_mnkl, + CtaTileMNK cta_tile_mnk, + CtaCoordMNKL cta_coord_mnkl, + MmaTileMNK mma_tile_mnk, + TiledMma tiled_mma, + cute::Tensor& tTR_rAcc, // (T2R,T2R_M,T2R_N,EPI_M,EPI_N) + TensorStorage& shared_tensors, + TiledCopyT2R tiled_t2r + ) { + using namespace cute; + using ElementAccumulator = typename AccEngine::value_type; + using ElementCompute_ = typename epilogue::fusion::FusionCallbacksTraits::ElementCompute; + using ElementCompute = cute::conditional_t,ElementAccumulator,ElementCompute_>; + + static_assert(is_rmem::value, "Accumulator must be Register resident."); + static_assert(rank(AccLayout{}) == 5, "Accumulators must be copy-partitioned: (T2R,T2R_M,T2R_N,EPI_M,EPI_N)"); + static_assert(rank(ProblemShapeMNKL{}) == 4, "ProblemShapeMNKL must be rank 4"); + static_assert(rank(CtaCoordMNKL{}) == 4, "CoordMNKL must be rank 4"); + + // Indexing variables + auto [M, N, K, L] = problem_shape_mnkl; + auto [m_coord, n_coord, k_coord, l_coord] = cta_coord_mnkl; + int thread_idx = threadIdx.x % ThreadCount; + int warp_idx = thread_idx / NumThreadsPerWarp; + [[maybe_unused]] int lane_idx = thread_idx % NumThreadsPerWarp; + + // The tma tensor D under im2col mode only has two modes (M, N) which + // should be local tiled with only (m_coord, n_coord). + auto coord_shape = + conditional_return(make_coord(m_coord, n_coord), make_coord(m_coord, n_coord, l_coord)); + + // Represent the full output tensor, slice to get the tile this CTA is responsible for + Tensor mD_mn = params.tma_store_d.get_tma_tensor(make_shape(M,N,L)); // (M,N,L) + Tensor mD = coalesce(mD_mn, take<0,2>(cta_tile_mnk)); + Tensor gD = local_tile(mD, take<0,2>(cta_tile_mnk), coord_shape); // (CTA_M,CTA_N) + + // Apply epilogue subtiling + Tensor gD_epi = flat_divide( gD, EpilogueTile{}); // (EPI_TILE_M,EPI_TILE_N,EPI_M,EPI_N) + + // Construct the corresponding pipelined smem tensors + auto ptr_sC = shared_tensors.collective.smem_C.begin(); + auto ptr_sD = shared_tensors.collective.smem_D.begin(); + Tensor sC_epi = cute::as_position_independent_swizzle_tensor( + make_tensor(make_smem_ptr(ptr_sC), SmemLayoutC{})); // (EPI_TILE_M,EPI_TILE_N,PIPE_C) + Tensor sD_epi = cute::as_position_independent_swizzle_tensor( + make_tensor(make_smem_ptr(ptr_sD), SmemLayoutD{})); // (EPI_TILE_M,EPI_TILE_N,PIPE_D) + + // (t)hread-partition for (t)mem to (r)egister copy (tTR_) + ThrCopy thread_t2r = tiled_t2r.get_slice(thread_idx); + Tensor tTR_sD = thread_t2r.partition_D(sD_epi(_,_,_0{})); // (T2R,T2R_M,T2R_N) + + // Allocate D and accumulator registers + Tensor tTR_rD = make_tensor(shape(tTR_sD)); // (T2R,T2R_M,T2R_N) + + // Vectorized fragment view + constexpr int FragmentSize = DispatchPolicy::FragmentSize; + Tensor tTR_rD_frg = recast>(coalesce(tTR_rD)); // (EPI_V) + + // (t)hread-partition for (s)mem to (r)egister copy (tSR_) + TiledCopy tiled_s2r = make_tiled_copy_D(Copy_Atom{}, tiled_t2r); + ThrCopy thread_s2r = tiled_s2r.get_slice(thread_idx); + Tensor tSR_sC = thread_s2r.partition_S(sC_epi); // (S2R,S2R_M,S2R_N,PIPE_C) + Layout tSR_rC_layout = thread_s2r.retile_D(tTR_rD).layout(); // (S2R,S2R_M,S2R_N) + + // Allocate C registers + // If C smem load is a non-vectorized dst(i) = src(i) then we can allocate C registers directly in the compute type + // to eliminate some redundant pack+unpack instruction sequences for sub-word types + constexpr bool IsDirectS2R = cute::is_same_v> + && decltype(max_common_vector(tSR_rC_layout, tSR_sC.layout()))::value <= 1; + using RegisterElementC = cute::conditional_t; + Tensor tTR_rC = make_tensor(shape(tTR_sD)); // (T2R,T2R_M,T2R_N) + Tensor tSR_rC = thread_s2r.retile_D(tTR_rC); // (S2R,S2R_M,S2R_N) + + // (t)hread-partition for (r)egister to (s)mem copy (tRS_) + TiledCopy tiled_r2s = make_tiled_copy_D(Copy_Atom{}, tiled_t2r); + ThrCopy thread_r2s = tiled_r2s.get_slice(thread_idx); + Tensor tRS_rD = thread_r2s.retile_S(tTR_rD); // (R2S,R2S_M,R2S_N) + Tensor tRS_sD = thread_r2s.partition_D(sD_epi); // (R2S,R2S_M,R2S_N,PIPE_D) + + // thread(b)lock-partition for (s)mem to (g)mem copy (bSG_) + ThrCopy thrblk_s2g = params.tma_store_d.get_slice(Int<0>{}); + Tensor bSG_sD = thrblk_s2g.partition_S(sD_epi); // (S2G,S2G_M,S2G_N,PIPE_D) + Tensor bSG_gD = thrblk_s2g.partition_D(gD_epi); // (S2G,S2G_M,S2G_N,EPI_M,EPI_N) + + // OOB predication for tile quantization "residue" + // Absolute coordinate tensors (dynamic) + Tensor mD_crd = make_identity_tensor(make_shape(M,N)); // (M,N) + Tensor cD_mn = local_tile(mD_crd, take<0,2>(cta_tile_mnk), make_coord(m_coord, n_coord)); // (CTA_M,CTA_N) + Tensor tTR_cD_mn = thread_t2r.partition_D(flat_divide(cD_mn, EpilogueTile{})); // (T2R,T2R_M,T2R_N,EPI_M,EPI_N) + // Relative coordinate tensors (static) + Tensor cD = make_coord_tensor(cD_mn.layout()); // (CTA_M,CTA_N) + Tensor tTR_cD = make_coord_tensor(tTR_cD_mn.layout()); // (T2R,T2R_M,T2R_N,EPI_M,EPI_N) + // Subtract the global "bottom right" corner from the local "top left" corner to get the max relative coordinate + auto residue_cD = make_coord(M,N) - cD_mn(_0{}); // (m,n) + auto residue_tTR_cD = make_coord(M,N) - tTR_cD_mn(_0{}); // (m,n) + + // Get the fusion callbacks for the consumer store warps + constexpr bool RefSrc = false; // Register tensors reference T2R copy dst layout + auto cst_args = cutlass::epilogue::fusion::detail::ConsumerStoreArgs{ + problem_shape_mnkl, + cta_tile_mnk, + cta_coord_mnkl, + tiled_mma, + EpilogueTile{}, + tiled_t2r, + cD, + residue_cD, + tTR_cD, + residue_tTR_cD, + tTR_rC, + thread_idx + }; + + auto cst_callbacks = fusion_callbacks.template get_consumer_store_callbacks(cst_args); + bool is_producer_load_needed = fusion_callbacks.is_producer_load_needed(); + bool is_C_load_needed = is_source_supported && fusion_callbacks.is_C_load_needed(); + + // Thread synchronizer for previously issued waits or fences + // to ensure visibility of smem reads/writes to threads or TMA unit + auto synchronize = [] () { cutlass::arch::NamedBarrier::sync(ThreadCount, cutlass::arch::ReservedNamedBarriers::EpilogueBarrier); }; + + // Predication for TMA store (one warp issues TMA store) + bool issue_tma_store = warp_idx == 0; + + // In the reuse smem configuration we have StagesC smem buffers and at most StagesD committed TMA stores in flight. + // The TMA store pipeline producer acquire returns when at most StagesD-1 committed stores are in-flight, so we can + // only guarantee store completion after StagesD iterations, then we can begin issuing releases on the smem buffer locks. + // store_pipe_producer_state tracks the acquire and load_pipe_consumer_state tracks the release, in circular buffer fashion. + // If TMA store supported async transaction mbarriers we would not need this synchronous release behavior. + LoadPipelineState load_wait_state = load_pipe_consumer_state; + if constexpr (ReuseSmemC) { + load_wait_state = store_pipe_producer_state; + load_wait_state.phase_ ^= 1; + } + + // We can delay issue of TMA store by one iteration to achieve better interleaving of non-TMA instructions + // Sync requirements of smem reuse may preclude this optimization + // Delayed stores cause delayed stage releases which causes deadlock when StagesC == StagesD + int epi_m_prev = 0, epi_n_prev = 0; + static_assert(not (DelayTmaStore and ReuseSmemC and StagesC <= StagesD), "This TMA epilogue configuration will deadlock"); + + // The TMA store sequence for one subtile iteration + auto tma_store_fn = [&] (int epi_m, int epi_n) CUTLASS_LAMBDA_FUNC_INLINE { + // Write the tile from smem to gmem with TMA + cutlass::arch::fence_view_async_shared(); // ensure smem writes are visible to TMA + synchronize(); // ensure all threads have issued their async fence + if (issue_tma_store) { + copy(params.tma_store_d, bSG_sD(_,_,_,store_pipe_producer_state.index()), bSG_gD(_,_,_,epi_m,epi_n)); + } + + // Post async fence, pre TMA commit callback entry point + cst_callbacks.tma_store(epi_m, epi_n, store_pipe_producer_state.count(), issue_tma_store); + + // Commit the TMA stores for this stage + if (issue_tma_store) { + store_pipeline.producer_commit(store_pipe_producer_state); + } + ++store_pipe_producer_state; + + // Wait for the next smem buffer to be available + if (issue_tma_store) { + store_pipeline.producer_acquire(store_pipe_producer_state); + } + synchronize(); + + if constexpr (ReuseSmemC) { + // producer_acquire returns when at most StagesD-1 committed stores are pending + bool store_finished = store_pipe_producer_state.count() > StorePipeline::UnacquiredStages; + // Let dma warp know earliest smem buffer is consumed and empty after StagesD producer commits + if (store_finished) { + if (is_producer_load_needed) { + load_pipeline.consumer_release(load_pipe_consumer_state); + } + ++load_pipe_consumer_state; + } + } + }; + + // + // BEGIN EPILOGUE + // + + cst_callbacks.begin(); + if (cst_callbacks.begin_sync_needed()) { + synchronize(); + } + + // Begin the wait for the producer load results + ConsumerToken load_wait_token{BarrierStatus::WaitDone}; + if (is_producer_load_needed) { + load_wait_token = load_pipeline.consumer_try_wait(load_wait_state); + } + + // For each epilogue subtile within the CTA tile + constexpr int NumEpiSubtilesN = CUTE_STATIC_V(size<3>(gD_epi)); + constexpr int NumEpiSubtilesM = CUTE_STATIC_V(size<2>(gD_epi)); + #pragma unroll(UnrollEpiLoop ? NumEpiSubtilesN : 1) + for (int iter_n = 0; iter_n < NumEpiSubtilesN; ++iter_n) { + #pragma unroll(UnrollEpiLoop ? NumEpiSubtilesM : 1) + for (int iter_m = 0; iter_m < NumEpiSubtilesM; ++iter_m) { + int epi_m = iter_m, epi_n = iter_n; + bool is_first_iteration = iter_m == 0 && iter_n == 0; + bool is_last_iteration = iter_m == size<2>(gD_epi)-1 && iter_n == size<3>(gD_epi)-1; + + cst_callbacks.begin_loop(epi_m, epi_n); + + if (is_producer_load_needed) { + // Wait for the producer load to fill smem + load_pipeline.consumer_wait(load_wait_state, load_wait_token); + + if (is_C_load_needed) { + // Copy source tile from smem to register + copy(tiled_s2r, tSR_sC(_,_,_,load_wait_state.index()), tSR_rC); + // Ensure smem loads are complete before reusing smem for mixed types/layouts + if constexpr (ReuseSmemC && not (SmemLayoutC{} == SmemLayoutD{})) { + synchronize(); + } + } + } + + // First loop fusion callback entry point + cst_callbacks.previsit(epi_m, epi_n, load_wait_state.count(), is_producer_load_needed); + + if (is_producer_load_needed) { + // Let producer load warp know smem buffers are consumed and empty + if constexpr (not ReuseSmemC) { + cutlass::arch::fence_view_async_shared(); + load_pipeline.consumer_release(load_pipe_consumer_state); + ++load_pipe_consumer_state; + } + ++load_wait_state; + } + + Tensor tTR_rAcc_epi_tile = tTR_rAcc(_,_,_,epi_m,epi_n); + Tensor tTR_rAcc_frg = recast>(coalesce(tTR_rAcc_epi_tile)); // (EPI_V) + + // Vectorized fragment loop with visitor callback entry point + CUTLASS_PRAGMA_UNROLL + for (int epi_v = 0; epi_v < size(tTR_rD_frg); ++epi_v) { + tTR_rD_frg(epi_v) = cst_callbacks.visit(tTR_rAcc_frg(epi_v), epi_v, epi_m, epi_n); + } + + // The latest we can delay the TMA store is right before the smem store of the next iteration + // since the current TMA store needs to be committed before we can acquire the next smem buffer + if constexpr (DelayTmaStore) { + // Issue TMA stores for the previous subtile + if (not is_first_iteration) { + tma_store_fn(epi_m_prev, epi_n_prev); + } + epi_m_prev = epi_m; + epi_n_prev = epi_n; + } + + // Smem reduction callback entry point using current store buffer for workspace + Tensor reduction_buffer = make_tensor(raw_pointer_cast(sD_epi(_,_,store_pipe_producer_state.index()).data()), + make_layout(stride<2>(get_nonswizzle_portion(SmemLayoutD{})), _1{})); + cst_callbacks.reduce(reduction_buffer, synchronize, epi_m, epi_n, is_last_iteration, tTR_rD_frg); + + // Copy output tile from register to smem + bool issue_smem_store = true; + if (issue_smem_store) { + copy(tiled_r2s, tRS_rD, tRS_sD(_,_,_,store_pipe_producer_state.index())); + } + + // Post reduction, pre TMA store callback entry point + cst_callbacks.postreduce(epi_m, epi_n, store_pipe_producer_state.count(), issue_smem_store); + + if constexpr (not DelayTmaStore) { + // Issue TMA stores for this subtile + tma_store_fn(epi_m, epi_n); + } + + cst_callbacks.end_loop(epi_m, epi_n); + + if (is_producer_load_needed) { + // Begin the wait for the next subtile producer load + load_wait_token = load_pipeline.consumer_try_wait(load_wait_state, is_last_iteration); + } + } // for epi_m + } // for epi_n + + if constexpr (DelayTmaStore) { + // Issue TMA stores for the last subtile + tma_store_fn(epi_m_prev, epi_n_prev); + } + + cst_callbacks.end(); + + return cute::make_tuple(load_pipe_consumer_state, store_pipe_producer_state); + } + + template + CUTLASS_DEVICE void + store_tail( + LoadPipeline load_pipeline, + LoadPipelineState load_pipe_consumer_state, + StorePipeline store_pipeline, + StorePipelineState store_pipe_producer_state, + CtaTileMNK cta_tile_mnk) { + if constexpr (ReuseSmemC) { + if (fusion_callbacks.is_producer_load_needed()) { + // wait for all TMA stores to complete + store_pipeline.producer_tail(store_pipe_producer_state); + + // Issue releases on up to StagesD-1 previously issued TMA stores + constexpr int release_stages = cute::min(StorePipeline::UnacquiredStages, get_load_pipe_increment(cta_tile_mnk)); + CUTLASS_PRAGMA_UNROLL + for (int stage = 0; stage < release_stages; ++stage) { + load_pipeline.consumer_release(load_pipe_consumer_state); + ++load_pipe_consumer_state; + } + } + } + } +}; + + +///////////////////////////////////////////////////////////////////////////////////////////////// + +} // namespace cutlass::epilogue::collective + +///////////////////////////////////////////////////////////////////////////////////////////////// diff --git a/docs/pi05_thor.md b/docs/pi05_thor.md index 54226e22..c9de95cf 100644 --- a/docs/pi05_thor.md +++ b/docs/pi05_thor.md @@ -150,19 +150,19 @@ the final action chunk. | tier | p50 (ms) | speedup | raw cos | raw min | act cos | act min | gates | |---|---|---|---|---|---|---|---| -| FP8 (reference) | 46.59 | 1.000 | — | — | — | — | — | -| **NVFP4 (default)** | **31.64** | **1.473** | 0.99904 | 0.99766 | 0.99974 | 0.99944 | PASS | -| INT4 | 32.39 | 1.441 | 0.99838 | 0.99512 | 0.99961 | 0.99939 | PASS | -| INT4+RHT | 32.54 | 1.432 | 0.99918 | 0.99742 | 0.99983 | 0.99970 | PASS | +| FP8 (reference) | 46.62 | 1.000 | — | — | — | — | — | +| **NVFP4 (default)** | **31.98** | **1.458** | 0.99904 | 0.99766 | 0.99974 | 0.99944 | PASS | +| INT4 | 32.54 | 1.461 | 0.99838 | 0.99512 | 0.99961 | 0.99939 | PASS | +| INT4+RHT | 32.60 | 1.452 | 0.99918 | 0.99742 | 0.99983 | 0.99970 | PASS | **2 views** | tier | p50 (ms) | speedup | raw cos | raw min | act cos | act min | gates | |---|---|---|---|---|---|---|---| -| FP8 (reference) | 38.45 | 1.000 | — | — | — | — | — | -| **NVFP4 (default)** | **27.14** | **1.417** | 0.99921 | 0.99803 | 0.99972 | 0.99916 | PASS | -| INT4 | 27.86 | 1.380 | 0.99879 | 0.99751 | 0.99965 | 0.99928 | PASS | -| INT4+RHT | 27.97 | 1.380 | 0.99941 | 0.99828 | 0.99977 | 0.99922 | PASS | +| FP8 (reference) | 38.50 | 1.000 | — | — | — | — | — | +| **NVFP4 (default)** | **27.25** | **1.413** | 0.99921 | 0.99803 | 0.99972 | 0.99916 | PASS | +| INT4 | 27.72 | 1.387 | 0.99879 | 0.99751 | 0.99965 | 0.99928 | PASS | +| INT4+RHT | 27.88 | 1.381 | 0.99941 | 0.99828 | 0.99977 | 0.99922 | PASS | **1 view** — the accuracy gates do not pass at any fully-quantized tier, for reasons that are not implementation defects (§6). @@ -211,11 +211,11 @@ on the scale. One locked-clock batch at three views: | tier | p50 (ms) | p95 (ms) | vs FP16 | |---|---|---|---| -| FP16 | 80.202 | 81.028 | 1.000 | -| FP8 | 46.619 | 49.422 | 1.720 | -| NVFP4 | **31.977** | 32.054 | **2.508** | -| INT4 | 32.577 | 32.653 | 2.462 | -| INT4+RHT | 32.646 | 32.712 | 2.457 | +| FP16 | 80.230 | 80.605 | 1.000 | +| FP8 | 47.325 | 49.418 | 1.695 | +| NVFP4 | **31.842** | 31.893 | **2.520** | +| INT4 | 32.646 | 32.718 | 2.458 | +| INT4+RHT | 32.545 | 32.596 | 2.465 | **At one view even FP8 loses its worst sample**, to 0.99421 — below the 0.995 gate that the quantized tiers also miss. FP8 differs from FP16 by @@ -294,7 +294,7 @@ Constructor keyword / bench flag pairs. Defaults are the production tier. | `decoder_act_format` / `--decoder-act-format` | `nvfp4` | `e0m3` requires `e0m3` weights | | `decoder_rht` / `--decoder-rht` | `False` | per-16 Hadamard rotation; requires `e0m3` activations | | `decoder_fused_geglu` / `--decoder-fused-geglu` | `True` | fuse the decoder GeGLU into the gate_up GEMM epilogue (NVFP4 weights only) | -| `encoder_p1_combiner` / `--encoder-p1-combiner` | `epilogue_hw` | `epilogue_hw` (fused, compact store), `epilogue` (fused, full width — parity with the old path), `lut_native` (separate GEMMs + combiner kernel) | +| `encoder_p1_combiner` / `--encoder-p1-combiner` | `epilogue_hw_nod` | `epilogue_hw_nod` (fused, compact store, collective D store elided), `epilogue_hw` (fused, compact store), `epilogue` (fused, full width — parity with the old path), `lut_native` (separate GEMMs + combiner kernel) | | `use_fp4_encoder_attn_qkv` / `--encoder-attn-qkv-fp4` | `False` | implemented and passing, but that GEMM is not weight-bandwidth-bound, so FP4 only matches FP8 while costing an extra quantize step | | `decoder_fused_attn` / `--decoder-fused-attn` | `False` | folds the seqused mask into softmax (bit-identical, one fewer launch). Only the fixed-shape state-prompt path takes the seqused kernels, which this suite does not exercise | | `awq_alpha` / `--awq-alpha` | `0.8` | AWQ per-channel scale exponent | @@ -331,9 +331,13 @@ Headroom is thin and mostly hard floors: need a bespoke persistent mainloop to recover. 2. Decoder AdaRMS (350 × 2.92 µs) and RoPE (180 × 1.58 µs) are at the kernel-launch floor. -3. SigLIP / encoder attention projections in FP4 (est. −0.3 to −0.5 ms) - and AWQ for the SigLIP up-projection are the remaining candidates, both - gated on accuracy. +3. The SigLIP attention projections were characterized with L2-sector + counters and ruled out: at M=768 they are L2-bandwidth/compute bound + (32.4 MB of L2 reads per qkv call at ~1.16 TB/s, with the measured + time bracketed between the weight-only and all-DRAM rooflines), so + FP4 conversion has no bandwidth dividend and the extra quantize step + makes it a net loss. AWQ for the SigLIP up-projection remains the one + accuracy-gated candidate. ### Approaches measured and rejected @@ -346,8 +350,20 @@ Headroom is thin and mostly hard floors: KV-split path). Fuse the glue *between* GEMMs, not the GEMMs. - **Full-width fused GeGLU epilogue.** The combiner kernel it removes is exactly cancelled by the doubled weight streaming of the K-expanded - down projection. The half-width compact store is the form that wins, - and is the default. + down projection. The half-width compact store is the form that wins; + with its unread D store elided outright (`epilogue_hw_nod`, a fork of + the SM100 epilogue collective with the same `is_destination_supported` + guards the SM90 and SM100 ptr-array collectives already carry) it is + the default: a five-leg alternating sandwich measures −0.57 ms + (32.513/32.550/32.547 vs 31.956/31.967, drift ≤ 0.037 ms). Two + measurement caveats worth keeping: the isolated kernel benchmark scored + the elision as a regression (one more entry for the tile-selection + warning above), and under `nsys --cuda-graph-trace=node` the two + variants converge entirely — the win only exists unprofiled, so + per-kernel traces cannot attribute it. +- **No-D-store decoder GeGLU.** The same elision applied to the decoder + tile is a wash (the dummy store there is 0.04 MB), so it ships opt-in + (`--decoder-fused-geglu-nod`) and stays off by default. --- diff --git a/flash_rt/api.py b/flash_rt/api.py index 285e5216..41982026 100644 --- a/flash_rt/api.py +++ b/flash_rt/api.py @@ -426,9 +426,10 @@ def load_model(checkpoint, framework="torch", num_views=2, autotune=3, awq_alpha: AWQ activation scaling exponent. ``None`` selects 0.8 for the encoder-FP4 + decoder-FP4 preset and 0.5 otherwise. encoder_p1_combiner: FP4 encoder split-GU combiner. ``None`` resolves - to the preset: ``"epilogue_hw"`` (fused GeGLU epilogue) with - ``use_fp4_decoder=True``, ``"lut_native"`` otherwise. - ``"direct"``, ``"lut"`` and ``"epilogue"`` remain available for + to the preset: ``"epilogue_hw_nod"`` (fused GeGLU epilogue with + the collective's D store elided) with ``use_fp4_decoder=True``, + ``"lut_native"`` otherwise. ``"direct"``, ``"lut"``, + ``"epilogue"`` and ``"epilogue_hw"`` remain available for explicit A/B runs. encoder_down_variant: Cutlass NVFP4 encoder Down GEMM variant (production default ``7``). @@ -609,7 +610,7 @@ def load_model(checkpoint, framework="torch", num_views=2, autotune=3, if use_fp4_siglip_ffn is None: use_fp4_siglip_ffn = bool(use_fp4_decoder) if encoder_p1_combiner is None: - encoder_p1_combiner = ("epilogue_hw" if use_fp4_decoder + encoder_p1_combiner = ("epilogue_hw_nod" if use_fp4_decoder else "lut_native") else: if fp4_layers is None: diff --git a/flash_rt/frontends/torch/pi05_thor_fp4.py b/flash_rt/frontends/torch/pi05_thor_fp4.py index 930dae9d..28958849 100644 --- a/flash_rt/frontends/torch/pi05_thor_fp4.py +++ b/flash_rt/frontends/torch/pi05_thor_fp4.py @@ -84,7 +84,7 @@ def __init__(self, checkpoint_dir, num_views: int = 2, awq_alpha: float = 0.5, awq_calib_iters: int = 8, use_p1_split_gu: bool = False, - encoder_p1_combiner: str = "epilogue_hw", + encoder_p1_combiner: str = "epilogue_hw_nod", encoder_down_variant: int = 7, encoder_down_x_variant: int = 6, decoder_qkv_variant: int = 10, @@ -95,6 +95,7 @@ def __init__(self, checkpoint_dir, num_views: int = 2, decoder_act_format: str = "nvfp4", decoder_fused_attn: bool = False, decoder_fused_geglu: bool = True, + decoder_fused_geglu_nod: bool = False, decoder_rht: bool = False, use_fp8: bool = True, state_prompt_mode: str = "exact", @@ -147,11 +148,12 @@ def __init__(self, checkpoint_dir, num_views: int = 2, "fp4_layers contains non-live encoder FFN layers " f"{invalid_layers}; valid layers are [0, {self.Le - 2}]") if encoder_p1_combiner not in ("direct", "lut", "lut_native", - "epilogue", "epilogue_hw"): + "epilogue", "epilogue_hw", + "epilogue_hw_nod"): raise ValueError( "encoder_p1_combiner must be 'direct', 'lut', " - "'lut_native', 'epilogue', or 'epilogue_hw', got " - f"{encoder_p1_combiner!r}") + "'lut_native', 'epilogue', 'epilogue_hw', or " + f"'epilogue_hw_nod', got {encoder_p1_combiner!r}") self.encoder_p1_combiner = encoder_p1_combiner self.encoder_down_variant = int(encoder_down_variant) # Tile variant for the K-expanded Down GEMM of the 'epilogue' P1 @@ -189,6 +191,10 @@ def __init__(self, checkpoint_dir, num_views: int = 2, # the gate_up GEMM + GeGLU-quantize kernel (nvfp4 weights only). self.decoder_fused_geglu = (bool(decoder_fused_geglu) and decoder_weight_format == 'nvfp4') + # No-D-store variant of the fused decoder GeGLU (same contract, + # the collective's own D store elided). + self.decoder_fused_geglu_nod = (bool(decoder_fused_geglu_nod) + and self.decoder_fused_geglu) if self._fp4_layers: if not _HAS_FP4: @@ -639,7 +645,8 @@ def get(k): g_fp16 = gu_fp16[:He, :].contiguous() u_fp16 = gu_fp16[He:, :].contiguous() if self.encoder_p1_combiner in ('epilogue', - 'epilogue_hw'): + 'epilogue_hw', + 'epilogue_hw_nod'): # Fused-epilogue paths: single interleaved gate/up # weight (down AWQ inv_s folded into the up rows). # Full-width 'epilogue' additionally needs the @@ -1119,9 +1126,12 @@ def _alloc_fp4_scratch_for_Se(self, Se: int): # Fused-epilogue path: one full-width [Se, 2H] FP4 buffer # between the interleaved GEMM and the K-expanded Down. self._fp4_p1_il = FP4Buffer(Se, 2 * He, device='cuda') - elif self.encoder_p1_combiner == 'epilogue_hw': + elif self.encoder_p1_combiner in ('epilogue_hw', + 'epilogue_hw_nod'): # Half-width path writes Down's stock input scratch; the - # collective's D output lands in this reusable dummy. + # collective's D output lands in this reusable dummy + # ('epilogue_hw_nod' never writes it but the host-side TMA + # descriptor still needs a real pointer). self._fp4_p1_dummy = torch.zeros( Se, He, dtype=torch.uint8, device='cuda') else: @@ -1150,7 +1160,8 @@ def _alloc_fp4_scratch_for_Se(self, Se: int): self._fp4_scratch_dict['p1_il_p4'] = self._fp4_p1_il.packed.data_ptr() self._fp4_scratch_dict['p1_il_sfa'] = self._fp4_p1_il.sfa.data_ptr() self._fp4_scratch_dict['variant_dn_x'] = self.encoder_down_x_variant - elif self.encoder_p1_combiner == 'epilogue_hw': + elif self.encoder_p1_combiner in ('epilogue_hw', + 'epilogue_hw_nod'): self._fp4_scratch_dict['p1_dummy'] = self._fp4_p1_dummy.data_ptr() else: self._fp4_scratch_dict['p1_gate_p4'] = self._fp4_p1_gate.packed.data_ptr() @@ -1402,6 +1413,7 @@ def _capture_enc_ae_graph(self): ae_dims['act_format'] = self.decoder_act_format ae_dims['rht'] = self.decoder_rht ae_dims['fused_geglu'] = self.decoder_fused_geglu + ae_dims['fused_geglu_nod'] = self.decoder_fused_geglu_nod if self._attn is not None: # Fold the decoder seqused mask into the softmax kernel. self._attn.use_fused_softmax = self.decoder_fused_attn diff --git a/flash_rt/hardware/thor/shared_primitives_fp4.py b/flash_rt/hardware/thor/shared_primitives_fp4.py index b7caf4ce..bec72e67 100644 --- a/flash_rt/hardware/thor/shared_primitives_fp4.py +++ b/flash_rt/hardware/thor/shared_primitives_fp4.py @@ -119,7 +119,7 @@ def encoder_forward_with_fp4_subset(gemm, fvk, fvk_fp4, bufs, weights, dims, p1_il_p4 = fp4_scratch['p1_il_p4'] p1_il_sfa = fp4_scratch['p1_il_sfa'] variant_dn_x = fp4_scratch['variant_dn_x'] - elif p1_combiner == 'epilogue_hw': + elif p1_combiner in ('epilogue_hw', 'epilogue_hw_nod'): p1_dummy = fp4_scratch['p1_dummy'] else: p1_gate_p4 = fp4_scratch['p1_gate_p4'] @@ -251,8 +251,14 @@ def encoder_forward_with_fp4_subset(gemm, fvk, fvk_fp4, bufs, weights, dims, # 'epilogue_hw' quantizes at compact granularity in the # epilogue and writes Down's stock input buffer directly. w_il = fp4_weights[l]['gu_il'] - if p1_combiner == 'epilogue_hw': - _check(fvk_fp4.cutlass_fp4_gemm_geglu_il_hw( + if p1_combiner in ('epilogue_hw', 'epilogue_hw_nod'): + # 'epilogue_hw_nod' elides the collective's own D + # store (p1_dummy is never written); otherwise the + # two are identical. + hw_gemm = (fvk_fp4.cutlass_fp4_gemm_geglu_il_hw_nod + if p1_combiner == 'epilogue_hw_nod' + else fvk_fp4.cutlass_fp4_gemm_geglu_il_hw) + _check(hw_gemm( sc_gu.packed.data_ptr(), sc_gu.sfa.data_ptr(), w_il['packed'].data_ptr(), w_il['sfb'].data_ptr(), p1_dummy, diff --git a/flash_rt/models/pi05/pipeline_thor.py b/flash_rt/models/pi05/pipeline_thor.py index 181815c8..3e0bc5a0 100644 --- a/flash_rt/models/pi05/pipeline_thor.py +++ b/flash_rt/models/pi05/pipeline_thor.py @@ -287,6 +287,7 @@ def decoder_forward_fp4(ctx, fvk, fvk_fp4, bufs, weights, dims, stream=0, *, act_format = str(dims.get('act_format', 'nvfp4')) rht = 1 if dims.get('rht') else 0 fused_geglu = bool(dims.get('fused_geglu')) and weight_format == 'nvfp4' + fused_geglu_nod = bool(dims.get('fused_geglu_nod')) and fused_geglu act_e0m3 = act_format == 'e0m3' if act_format not in ('nvfp4', 'e0m3'): raise ValueError( @@ -429,7 +430,10 @@ def dec_gemm(variant, *args): # gelu(gate)*up per column pair and writes the Down input # (hid_fp4/hid_sfa) directly; the gate_up fp16 output and # the GeGLU-quantize kernel disappear. - rc = fvk_fp4.cutlass_fp4_gemm_geglu_il_hw_v10( + geglu_v10 = (fvk_fp4.cutlass_fp4_gemm_geglu_il_hw_nod_v10 + if fused_geglu_nod + else fvk_fp4.cutlass_fp4_gemm_geglu_il_hw_v10) + rc = geglu_v10( xn_fp4, xn_sfa, weights['gwil_fp4'][l], weights['gwil_sfb'][l], weights['gu_dummy'], hid_fp4, hid_sfa, diff --git a/tests/bench_pi05_decoder_fp4_e2e.py b/tests/bench_pi05_decoder_fp4_e2e.py index 8abee476..75eac6e7 100644 --- a/tests/bench_pi05_decoder_fp4_e2e.py +++ b/tests/bench_pi05_decoder_fp4_e2e.py @@ -43,7 +43,7 @@ # configuration the public API reproduces. PUBLIC_API_PRESET = { "encoder_gu_mode": "p1", - "encoder_p1_combiner": "epilogue_hw", + "encoder_p1_combiner": "epilogue_hw_nod", "encoder_down_variant": 7, "encoder_down_x_variant": 6, "decoder_gate_up_variant": 10, @@ -52,6 +52,7 @@ "decoder_rht": 0, "decoder_fused_attn": 0, "decoder_fused_geglu": 1, + "decoder_fused_geglu_nod": 0, "awq_alpha": 0.8, "encoder_attn_o_fp4": 1, "encoder_attn_qkv_fp4": 0, @@ -128,8 +129,9 @@ def main() -> int: "--encoder-gu-mode", choices=("p1", "merged"), default="p1") parser.add_argument( "--encoder-p1-combiner", - choices=("direct", "lut", "lut_native", "epilogue", "epilogue_hw"), - default="epilogue_hw") + choices=("direct", "lut", "lut_native", "epilogue", "epilogue_hw", + "epilogue_hw_nod"), + default="epilogue_hw_nod") parser.add_argument("--encoder-down-variant", type=int, default=7) parser.add_argument("--encoder-down-x-variant", type=int, default=6) parser.add_argument("--decoder-gate-up-variant", type=int, default=10) @@ -157,6 +159,10 @@ def main() -> int: "--decoder-fused-geglu", type=int, choices=(0, 1), default=1, help="Fused-epilogue decoder FFN: one interleaved GeGLU GEMM " "replaces the gate_up GEMM + GeGLU kernel (nvfp4 weights only)") + parser.add_argument( + "--decoder-fused-geglu-nod", type=int, choices=(0, 1), default=0, + help="No-D-store variant of the fused decoder GeGLU (requires " + "--decoder-fused-geglu 1)") parser.add_argument("--awq-alpha", type=float, default=0.8) parser.add_argument( "--encoder-attn-o-fp4", type=int, choices=(0, 1), default=1, @@ -273,6 +279,7 @@ def main() -> int: decoder_rht=bool(args.decoder_rht), decoder_fused_attn=bool(args.decoder_fused_attn), decoder_fused_geglu=bool(args.decoder_fused_geglu), + decoder_fused_geglu_nod=bool(args.decoder_fused_geglu_nod), use_fp4_decoder=True, use_fa4=True, use_fp4_encoder_attn=bool(args.encoder_attn_o_fp4), @@ -379,6 +386,8 @@ def main() -> int: "decoder_rht": bool(args.decoder_rht), "decoder_fused_attn": bool(args.decoder_fused_attn), "decoder_fused_geglu": bool(args.decoder_fused_geglu), + "decoder_fused_geglu_nod": bool( + args.decoder_fused_geglu_nod), "decoder_gate_up_variant": args.decoder_gate_up_variant, "attention": "fa4_siglip_encoder", } @@ -444,6 +453,7 @@ def main() -> int: "--decoder-rht", str(args.decoder_rht), "--decoder-fused-attn", str(args.decoder_fused_attn), "--decoder-fused-geglu", str(args.decoder_fused_geglu), + "--decoder-fused-geglu-nod", str(args.decoder_fused_geglu_nod), "--awq-alpha", str(args.awq_alpha), "--encoder-fp4-layer-count", str(args.encoder_fp4_layer_count), "--encoder-attn-o-fp4", str(args.encoder_attn_o_fp4), diff --git a/tests/test_pi05_fp4_fusion_kernels.py b/tests/test_pi05_fp4_fusion_kernels.py index ad7579f2..6ad5083f 100644 --- a/tests/test_pi05_fp4_fusion_kernels.py +++ b/tests/test_pi05_fp4_fusion_kernels.py @@ -190,6 +190,50 @@ def test_fused_geglu_epilogue_matches_split_chain(): f"fused epilogue lost accuracy: {cos_fused:.6f} < {cos_chain:.6f}") +@pytest.mark.parametrize("shape,pair", [ + ((968, 2048, 8192), ("cutlass_fp4_gemm_geglu_il_hw", + "cutlass_fp4_gemm_geglu_il_hw_nod")), + ((10, 1024, 4096), ("cutlass_fp4_gemm_geglu_il_hw_v10", + "cutlass_fp4_gemm_geglu_il_hw_nod_v10")), +]) +def test_geglu_nod_matches_hw_and_skips_dummy(shape, pair): + """The no-D-store GeGLU GEMM must produce bit-identical compact output + to the store-carrying variant, and must never touch the dummy buffer.""" + _require_thor() + torch.manual_seed(20260815) + M, D, H = shape + N_il = 2 * H + + x = torch.randn(M, D, dtype=torch.float16, device='cuda') + w_il = torch.randn(N_il, D, dtype=torch.float16, device='cuda') * 0.02 + q_il = quant_weight_nvfp4(w_il.contiguous()) + act = FP4ActScratch(M, D, device='cuda') + quant_act_nvfp4(x, act, M) + + outputs = {} + for name, fn in (("hw", getattr(fvk_fp4, pair[0])), + ("nod", getattr(fvk_fp4, pair[1]))): + fused = FP4ActScratch(M, H, device='cuda') + fused.packed.fill_(0) + fused.sfa.fill_(0) + dummy = torch.full((M, H), 0xAB, dtype=torch.uint8, device='cuda') + assert fn( + act.packed.data_ptr(), act.sfa.data_ptr(), + q_il['packed'].data_ptr(), q_il['sfb'].data_ptr(), + dummy.data_ptr(), fused.packed.data_ptr(), fused.sfa.data_ptr(), + M, N_il, D, 0) == 0 + torch.cuda.synchronize() + outputs[name] = (fused, dummy) + + hw, nod = outputs["hw"], outputs["nod"] + assert torch.equal(nod[0].packed, hw[0].packed), \ + "nod compact packed differs from hw" + assert torch.equal(nod[0].sfa, hw[0].sfa), "nod compact SFA differs from hw" + assert bool((nod[1] == 0xAB).all()), "nod wrote the dummy buffer" + assert not torch.equal(hw[1], nod[1]), \ + "hw variant did not write the dummy (test would prove nothing)" + + def test_vectorized_siglip_layernorms_match_reference(): """The single-pass LayerNorms must agree with the reference norm + quantize pair (reduction order differs at ulp level).""" From a3d351e216f493a84fd5e122bf2015d906993c6f Mon Sep 17 00:00:00 2001 From: LiangSu8899 Date: Mon, 24 Aug 2026 11:28:03 -0400 Subject: [PATCH 04/32] feat(fp4): add SigLIP FFN GEMM pair variant with fp32 bias/residual boundaries Up fuses bias + tanh-GELU with FP4 packed + SFD output; Down fuses bias + fp32 residual add with fp32 output (AlignmentCD=4), plus a beta=0 wrapper for plain bias GEMM. The CUTLASS workspace is cached per shape so steady-state calls are graph-capture safe. New symbols only; the existing fp16-boundary siglip_ffn kernels are untouched. --- ...utlass_fp4_gemm_siglip_ffn_f32out_sm100.cu | 282 ++++++++++++++++++ ...tlass_fp4_gemm_siglip_ffn_f32out_sm100.cuh | 53 ++++ 2 files changed, 335 insertions(+) create mode 100644 csrc/gemm/fp4/cutlass_fp4_gemm_siglip_ffn_f32out_sm100.cu create mode 100644 csrc/gemm/fp4/cutlass_fp4_gemm_siglip_ffn_f32out_sm100.cuh diff --git a/csrc/gemm/fp4/cutlass_fp4_gemm_siglip_ffn_f32out_sm100.cu b/csrc/gemm/fp4/cutlass_fp4_gemm_siglip_ffn_f32out_sm100.cu new file mode 100644 index 00000000..323c5606 --- /dev/null +++ b/csrc/gemm/fp4/cutlass_fp4_gemm_siglip_ffn_f32out_sm100.cu @@ -0,0 +1,282 @@ +// ============================================================================ +// FlashRT — NVFP4 GEMM pair for a SigLIP-style vision-tower FFN with fp32 +// bias/residual boundaries (SM100/SM110). See header for the contract. +// +// Up: D_fp4[M, N] = blockscale( gelu_tanh(A @ B^T + bias[N]) ) +// Down: D_f32[M, N] = A @ B^T + bias[N] + beta * C_f32[M, N] +// ============================================================================ + +#include "gemm/fp4/cutlass_fp4_gemm_siglip_ffn_f32out_sm100.cuh" + +#include "cutlass/cutlass.h" +#include "cutlass/epilogue/thread/activation.h" +#include "cutlass/epilogue/dispatch_policy.hpp" +#include "cutlass/epilogue/fusion/operations.hpp" +#include "cutlass/gemm/dispatch_policy.hpp" +#include "cutlass/gemm/collective/collective_builder.hpp" +#include "cutlass/epilogue/collective/collective_builder.hpp" +#include "cutlass/gemm/device/gemm_universal_adapter.h" +#include "cutlass/gemm/kernel/gemm_universal.hpp" +#include "cutlass/util/packed_stride.hpp" +#include "cutlass/detail/sm100_blockscaled_layout.hpp" +#include "cute/tensor.hpp" + +#include +#include + +namespace flash_rt { +namespace fp4 { + +namespace siglip_ffn { + +using namespace cute; + +using ElementA = cutlass::nv_float4_t; +using LayoutATag = cutlass::layout::RowMajor; +constexpr int AlignmentA = 32; + +using ElementB = cutlass::nv_float4_t; +using LayoutBTag = cutlass::layout::ColumnMajor; +constexpr int AlignmentB = 32; + +using ElementAccumulator = float; +using ElementCompute = float; +using ArchTag = cutlass::arch::Sm100; +using OperatorClass = cutlass::arch::OpClassBlockScaledTensorOp; +constexpr int SFVecSize = 16; + +using UpTileShape = Shape<_128, _256, _256>; +using DownTileShape = Shape<_128, _128, _256>; +using ClusterShape = Shape<_1, _1, _1>; + +// per-shape CUTLASS workspace cache (capture-safe: growth happens during +// the uncaptured warmup evaluation) +struct ws_key { + int which, M, N, K; + bool operator==(const ws_key & o) const { return which == o.which && M == o.M && N == o.N && K == o.K; } +}; +struct ws_key_hash { + size_t operator()(const ws_key & k) const noexcept { + return (size_t) k.which * 40503u ^ (size_t) k.M * 1315423911u ^ (size_t) k.N * 2654435761u ^ (size_t) k.K; + } +}; +inline void * get_ws(int which, int M, int N, int K, size_t needed) { + static std::unordered_map, ws_key_hash> cache; + static std::mutex mu; + std::lock_guard lk(mu); + auto & e = cache[ws_key{which, M, N, K}]; + if (e.second < needed) { + if (e.first) { cudaFree(e.first); } + cudaMalloc(&e.first, needed); + e.second = needed; + } + return e.first; +} + +// ── Up: bias + tanh-GELU + fp4/SFA output ────────────────────────────────── +namespace up { + +using ElementD = cutlass::float_e2m1_t; +using ElementC = ElementD; +using ElementSFD = cutlass::float_ue4m3_t; +constexpr int AlignmentD = 32; + +using MmaTileShape = UpTileShape; + +using FusionOperation = + cutlass::epilogue::fusion::LinCombPerColBiasEltActBlockScaleFactor< + cutlass::epilogue::thread::GELU_taylor, SFVecSize, + ElementD, ElementCompute, ElementSFD, cutlass::layout::RowMajor, + float, ElementC, ElementCompute>; + +using CollectiveEpilogue = + typename cutlass::epilogue::collective::CollectiveBuilder< + ArchTag, OperatorClass, MmaTileShape, ClusterShape, + cutlass::epilogue::collective::EpilogueTileAuto, + ElementAccumulator, ElementAccumulator, + ElementC, cutlass::layout::RowMajor, AlignmentD, + ElementD, cutlass::layout::RowMajor, AlignmentD, + cutlass::epilogue::collective::EpilogueScheduleAuto, + FusionOperation>::CollectiveOp; + +using CollectiveMainloop = + typename cutlass::gemm::collective::CollectiveBuilder< + ArchTag, OperatorClass, + ElementA, LayoutATag, AlignmentA, + ElementB, LayoutBTag, AlignmentB, + ElementAccumulator, MmaTileShape, ClusterShape, + cutlass::gemm::collective::StageCountAutoCarveout( + sizeof(typename CollectiveEpilogue::SharedStorage))>, + cutlass::gemm::collective::KernelScheduleAuto>::CollectiveOp; + +using GemmKernel = cutlass::gemm::kernel::GemmUniversal< + Shape, CollectiveMainloop, CollectiveEpilogue, void>; +using Gemm = cutlass::gemm::device::GemmUniversalAdapter; + +} // namespace up + +// ── Down: bias + residual source, fp32 output ────────────────────────────── +namespace down { + +using ElementD = float; +using ElementC = float; +constexpr int AlignmentCD = 4; + +using MmaTileShape = DownTileShape; + +using FusionOperation = cutlass::epilogue::fusion::LinCombPerColBias< + ElementD, ElementCompute, float, ElementC, ElementCompute>; + +using CollectiveEpilogue = + typename cutlass::epilogue::collective::CollectiveBuilder< + ArchTag, OperatorClass, MmaTileShape, ClusterShape, + cutlass::epilogue::collective::EpilogueTileAuto, + ElementAccumulator, ElementAccumulator, + ElementC, cutlass::layout::RowMajor, AlignmentCD, + ElementD, cutlass::layout::RowMajor, AlignmentCD, + cutlass::epilogue::collective::EpilogueScheduleAuto, + FusionOperation>::CollectiveOp; + +using CollectiveMainloop = + typename cutlass::gemm::collective::CollectiveBuilder< + ArchTag, OperatorClass, + ElementA, LayoutATag, AlignmentA, + ElementB, LayoutBTag, AlignmentB, + ElementAccumulator, MmaTileShape, ClusterShape, + cutlass::gemm::collective::StageCountAutoCarveout( + sizeof(typename CollectiveEpilogue::SharedStorage))>, + cutlass::gemm::collective::KernelScheduleAuto>::CollectiveOp; + +using GemmKernel = cutlass::gemm::kernel::GemmUniversal< + Shape, CollectiveMainloop, CollectiveEpilogue, void>; +using Gemm = cutlass::gemm::device::GemmUniversalAdapter; + +} // namespace down + +} // namespace siglip_ffn + +int siglip_ffn_up_gelu_fp4out( + const void * A_packed, const void * SFA, + const void * B_packed, const void * SFB, + const void * bias_f32, + void * D_packed, void * D_SFD, + int M, int N, int K, + cudaStream_t stream) { + using namespace siglip_ffn; + using Gemm = up::Gemm; + + auto stride_A = cutlass::make_cute_packed_stride( + typename Gemm::GemmKernel::StrideA{}, {M, K, 1}); + auto stride_B = cutlass::make_cute_packed_stride( + typename Gemm::GemmKernel::StrideB{}, {N, K, 1}); + auto stride_C = cutlass::make_cute_packed_stride( + typename Gemm::GemmKernel::StrideC{}, {M, N, 1}); + auto stride_D = cutlass::make_cute_packed_stride( + typename Gemm::GemmKernel::StrideD{}, {M, N, 1}); + using Cfg = + typename Gemm::GemmKernel::CollectiveMainloop::Sm1xxBlkScaledConfig; + auto layout_SFA = Cfg::tile_atom_to_shape_SFA(make_shape(M, N, K, 1)); + auto layout_SFB = Cfg::tile_atom_to_shape_SFB(make_shape(M, N, K, 1)); + + using EA = typename ElementA::DataType; + using SA = typename ElementA::ScaleFactorType; + + typename Gemm::Arguments args{ + cutlass::gemm::GemmUniversalMode::kGemm, {M, N, K, 1}, + {reinterpret_cast(A_packed), stride_A, + reinterpret_cast(B_packed), stride_B, + reinterpret_cast(SFA), layout_SFA, + reinterpret_cast(SFB), layout_SFB}, + {{}, + reinterpret_cast(D_packed), stride_C, + reinterpret_cast(D_packed), stride_D}}; + args.epilogue.thread.alpha = 1.0f; + args.epilogue.thread.beta = 0.0f; + args.epilogue.thread.bias_ptr = reinterpret_cast(bias_f32); + static float* d_norm = nullptr; + if (!d_norm) { + if (cudaMalloc(&d_norm, sizeof(float)) != cudaSuccess) return -1; + float h = 1.0f; + cudaMemcpyAsync(d_norm, &h, sizeof(float), cudaMemcpyHostToDevice, + stream); + } + args.epilogue.thread.block_scale_factor_ptr = + reinterpret_cast(D_SFD); + args.epilogue.thread.norm_constant_ptr = d_norm; + + Gemm gemm; + auto st = gemm.can_implement(args); + if (st != cutlass::Status::kSuccess) return static_cast(st) | 0x10000; + size_t ws_sz = Gemm::get_workspace_size(args); + void* ws = ws_sz > 0 ? get_ws(0, M, N, K, ws_sz) : nullptr; + st = gemm.initialize(args, ws, stream); + if (st != cutlass::Status::kSuccess) return static_cast(st) | 0x20000; + st = gemm.run(stream); + return (st == cutlass::Status::kSuccess) ? 0 + : (static_cast(st) | 0x30000); +} + +int siglip_ffn_down_bias_res_f32( + const void * A_packed, const void * SFA, + const void * B_packed, const void * SFB, + const void * bias_f32, + const void * C_f32, void * D_f32, + int M, int N, int K, + cudaStream_t stream, float beta) { + using namespace siglip_ffn; + using Gemm = down::Gemm; + + auto stride_A = cutlass::make_cute_packed_stride( + typename Gemm::GemmKernel::StrideA{}, {M, K, 1}); + auto stride_B = cutlass::make_cute_packed_stride( + typename Gemm::GemmKernel::StrideB{}, {N, K, 1}); + auto stride_C = cutlass::make_cute_packed_stride( + typename Gemm::GemmKernel::StrideC{}, {M, N, 1}); + auto stride_D = cutlass::make_cute_packed_stride( + typename Gemm::GemmKernel::StrideD{}, {M, N, 1}); + using Cfg = + typename Gemm::GemmKernel::CollectiveMainloop::Sm1xxBlkScaledConfig; + auto layout_SFA = Cfg::tile_atom_to_shape_SFA(make_shape(M, N, K, 1)); + auto layout_SFB = Cfg::tile_atom_to_shape_SFB(make_shape(M, N, K, 1)); + + using EA = typename ElementA::DataType; + using SA = typename ElementA::ScaleFactorType; + + typename Gemm::Arguments args{ + cutlass::gemm::GemmUniversalMode::kGemm, {M, N, K, 1}, + {reinterpret_cast(A_packed), stride_A, + reinterpret_cast(B_packed), stride_B, + reinterpret_cast(SFA), layout_SFA, + reinterpret_cast(SFB), layout_SFB}, + {{}, + reinterpret_cast(C_f32), stride_C, + reinterpret_cast(D_f32), stride_D}}; + args.epilogue.thread.alpha = 1.0f; + args.epilogue.thread.beta = beta; + args.epilogue.thread.bias_ptr = reinterpret_cast(bias_f32); + + Gemm gemm; + auto st = gemm.can_implement(args); + if (st != cutlass::Status::kSuccess) return static_cast(st) | 0x10000; + size_t ws_sz = Gemm::get_workspace_size(args); + void* ws = ws_sz > 0 ? get_ws(1, M, N, K, ws_sz) : nullptr; + st = gemm.initialize(args, ws, stream); + if (st != cutlass::Status::kSuccess) return static_cast(st) | 0x20000; + st = gemm.run(stream); + return (st == cutlass::Status::kSuccess) ? 0 + : (static_cast(st) | 0x30000); +} + +int gemm_bias_f32out( + const void * A_packed, const void * SFA, + const void * B_packed, const void * SFB, + const void * bias_f32, void * D_f32, + int M, int N, int K, + cudaStream_t stream) { + // the Down configuration with beta = 0: D = A@B + bias + return siglip_ffn_down_bias_res_f32(A_packed, SFA, B_packed, SFB, bias_f32, + /*C=*/D_f32, D_f32, M, N, K, stream, /*beta=*/0.0f); +} + +} // namespace fp4 +} // namespace flash_rt diff --git a/csrc/gemm/fp4/cutlass_fp4_gemm_siglip_ffn_f32out_sm100.cuh b/csrc/gemm/fp4/cutlass_fp4_gemm_siglip_ffn_f32out_sm100.cuh new file mode 100644 index 00000000..98da67e2 --- /dev/null +++ b/csrc/gemm/fp4/cutlass_fp4_gemm_siglip_ffn_f32out_sm100.cuh @@ -0,0 +1,53 @@ +// ============================================================================ +// FlashRT — NVFP4 GEMM pair for a SigLIP-style vision-tower FFN with fp32 +// bias/residual boundaries (SM100/SM110). +// +// Variant of cutlass_fp4_gemm_siglip_ffn_sm100 for hosts that keep the FFN +// bias and residual tensors in fp32 (rather than fp16): the Up projection +// fuses bias + tanh-GELU and emits FP4 (e2m1) packed output + SFD, and the +// Down projection fuses bias + fp32 residual add with fp32 output. The +// CUTLASS workspace is cached per shape instead of allocated per call, so +// steady-state calls are graph-capture safe. +// +// Up: D_fp4[M, N] = blockscale( gelu_tanh(A @ B^T + bias[N]) ) +// Down: D_f32[M, N] = A @ B^T + bias[N] + beta * C_f32[M, N] +// ============================================================================ +#pragma once + +#include + +namespace flash_rt { +namespace fp4 { + +// A: [M, K] NVFP4 packed row-major + SFA (tile-interleaved). +// B: [N, K] NVFP4 packed column-major + SFB. +// bias_f32: [N] fp32, broadcast over rows. +// D_packed: [M, N] NVFP4 packed row-major; D_SFD: SFD tile-interleaved. +// Returns 0 on success; CUTLASS status | stage flag otherwise. +int siglip_ffn_up_gelu_fp4out( + const void * A_packed, const void * SFA, + const void * B_packed, const void * SFB, + const void * bias_f32, + void * D_packed, void * D_SFD, + int M, int N, int K, + cudaStream_t stream); + +// C_f32/D_f32: [M, N] fp32 row-major (may alias). beta scales C. +int siglip_ffn_down_bias_res_f32( + const void * A_packed, const void * SFA, + const void * B_packed, const void * SFB, + const void * bias_f32, + const void * C_f32, void * D_f32, + int M, int N, int K, + cudaStream_t stream, float beta); + +// Down configuration with beta = 0: D = A@B + bias (no residual read). +int gemm_bias_f32out( + const void * A_packed, const void * SFA, + const void * B_packed, const void * SFB, + const void * bias_f32, void * D_f32, + int M, int N, int K, + cudaStream_t stream); + +} // namespace fp4 +} // namespace flash_rt From 357dba3938f48f93c68fb4294188737f5af751aa Mon Sep 17 00:00:00 2001 From: LiangSu8899 Date: Mon, 24 Aug 2026 11:41:03 -0400 Subject: [PATCH 05/32] feat(structures): add native ggml host adapter (Thor SM110 NVFP4 path) Relocates the ggml-cuda FlashRT layer here as the third host adapter: pure-CUDA kernels (repack, fused norm/modulation/rope, activation quantize, NVFP4 GEMM wrapper) behind a C header, plus the ggml-facing dispatch half (subgraph window matchers and caches) compiled inside the host's build tree. Fused-epilogue GEMMs are consumed from csrc/gemm/fp4. --- flash_rt/structures/adapters/ggml/README.md | 23 + flash_rt/structures/adapters/ggml/fr_ada.cu | 371 +++++ .../structures/adapters/ggml/fr_dispatch.cu | 1314 +++++++++++++++++ .../adapters/ggml/fr_gemm_f32out.cu | 207 +++ flash_rt/structures/adapters/ggml/fr_ggml.cuh | 105 ++ .../structures/adapters/ggml/fr_kernels.h | 126 ++ .../structures/adapters/ggml/fr_qkv_post.cu | 125 ++ .../structures/adapters/ggml/fr_quant_act.cu | 111 ++ .../structures/adapters/ggml/fr_repack.cu | 416 ++++++ 9 files changed, 2798 insertions(+) create mode 100644 flash_rt/structures/adapters/ggml/README.md create mode 100644 flash_rt/structures/adapters/ggml/fr_ada.cu create mode 100644 flash_rt/structures/adapters/ggml/fr_dispatch.cu create mode 100644 flash_rt/structures/adapters/ggml/fr_gemm_f32out.cu create mode 100644 flash_rt/structures/adapters/ggml/fr_ggml.cuh create mode 100644 flash_rt/structures/adapters/ggml/fr_kernels.h create mode 100644 flash_rt/structures/adapters/ggml/fr_qkv_post.cu create mode 100644 flash_rt/structures/adapters/ggml/fr_quant_act.cu create mode 100644 flash_rt/structures/adapters/ggml/fr_repack.cu diff --git a/flash_rt/structures/adapters/ggml/README.md b/flash_rt/structures/adapters/ggml/README.md new file mode 100644 index 00000000..13b46cb1 --- /dev/null +++ b/flash_rt/structures/adapters/ggml/README.md @@ -0,0 +1,23 @@ +# ggml host adapter (native) + +Native C++/CUDA host adapter that maps FlashRT structures onto ggml's CUDA +backend (llama.cpp family), targeting Jetson AGX Thor (SM110). Unlike the +Python runtime adapters (`vllm_engine.py`, `sglang_engine.py`), this adapter +is consumed at build time: the host's CMake compiles these translation units +inside its own build tree. + +Layout: + +- `fr_kernels.h` — pure C entry points (no ggml, no CUTLASS in the header). +- `fr_gemm_f32out.cu`, `fr_ada.cu`, `fr_qkv_post.cu`, `fr_quant_act.cu`, + `fr_repack.cu` — framework-free CUDA kernels (NVFP4 wire format, repack, + fused norm/modulation/rope, activation quantize). +- `fr_dispatch.cu`, `fr_ggml.cuh` — the ggml-facing half: subgraph window + matchers over `ggml_tensor` chains, weight/activation caches, and kernel + dispatch. Requires ggml-cuda's internal headers on the include path. +- GEMMs with fused epilogues are consumed from `csrc/gemm/fp4/` in this + repository (GeGLU interleaved, SigLIP FFN f32-boundary pair); nothing is + vendored. + +Host-side integration (fuse-hook call sites, graph construction changes, +build wiring) lives in the host tree and points its build at this directory. diff --git a/flash_rt/structures/adapters/ggml/fr_ada.cu b/flash_rt/structures/adapters/ggml/fr_ada.cu new file mode 100644 index 00000000..60871fdf --- /dev/null +++ b/flash_rt/structures/adapters/ggml/fr_ada.cu @@ -0,0 +1,371 @@ +// Fused adaLN kernels for the pi0.5 action expert (Thor). +// +// The ggml graph expresses each adaLN application as rms_norm + two +// broadcast repeats + mul + two adds (and the gated residual as repeat + +// mul + add), all on [M, C] tensors with M ~ 10. These kernels collapse +// each chain into one launch. The scale/shift/gate vectors are passed as +// direct pointers (the ggml view tensors' data pointers, which already +// include their byte offsets into the modulation vector). + +#include "fr_kernels.h" + +#include + +#include "cutlass/cutlass.h" +#include "cutlass/detail/sm100_blockscaled_layout.hpp" +#include "cute/tensor.hpp" + +namespace ggml_cuda_flashrt { + +namespace { + +// out[m, c] = norm(x[m])[c] * (1 + scale[c]) + shift[c] +// where norm = rms-normalize when with_rms, identity otherwise. +template +__global__ void kernel_ada_rms(const float * __restrict__ x, + const float * __restrict__ scale, + const float * __restrict__ shift, + float * __restrict__ out, + int C, float eps) { + const int m = blockIdx.x; + const float * xr = x + (int64_t) m * C; + float * orow = out + (int64_t) m * C; + + float inv_rms = 1.0f; + if (with_rms) { + float sumsq = 0.0f; + for (int c = threadIdx.x; c < C; c += blockDim.x) { + const float v = xr[c]; + sumsq += v * v; + } + __shared__ float red[32]; + #pragma unroll + for (int off = 16; off > 0; off >>= 1) { + sumsq += __shfl_xor_sync(0xffffffff, sumsq, off); + } + const int warp = threadIdx.x / 32; + if (threadIdx.x % 32 == 0) { + red[warp] = sumsq; + } + __syncthreads(); + if (warp == 0) { + float v = (threadIdx.x < blockDim.x / 32) ? red[threadIdx.x] : 0.0f; + #pragma unroll + for (int off = 16; off > 0; off >>= 1) { + v += __shfl_xor_sync(0xffffffff, v, off); + } + if (threadIdx.x == 0) { + red[0] = v; + } + } + __syncthreads(); + inv_rms = rsqrtf(red[0] / C + eps); + } + + for (int c = threadIdx.x; c < C; c += blockDim.x) { + const float n = with_rms ? xr[c] * inv_rms : xr[c]; + orow[c] = n * (1.0f + scale[c]) + shift[c]; + } +} + +using AdaCfg = cutlass::detail::Sm1xxBlockScaledConfig<16>; + +__device__ __forceinline__ uint8_t ada_f32_to_e2m1(float x) { + uint8_t sign = (x < 0.f) ? 0x8u : 0x0u; + float ax = fabsf(x); + uint8_t m; + if (ax <= 0.25f) m = 0u; + else if (ax <= 0.75f) m = 1u; + else if (ax <= 1.25f) m = 2u; + else if (ax <= 1.75f) m = 3u; + else if (ax <= 2.5f) m = 4u; + else if (ax <= 3.5f) m = 5u; + else if (ax <= 5.0f) m = 6u; + else m = 7u; + return sign | m; +} + +// Quantize one 16-element block held in registers by 16 consecutive threads? +// Simpler: each thread quantizes one 16-element block it re-reads from the +// just-written f32 output row (L2-hot), writing packed bytes + one SF byte. +template +__device__ __forceinline__ void ada_quant_row(const float * __restrict__ orow, + uint8_t * __restrict__ dst_packed, + uint8_t * __restrict__ dst_sfa, + LayoutSF layout, + int m, int C) { + const int n_blocks = C / 16; + for (int blk = threadIdx.x; blk < n_blocks; blk += blockDim.x) { + const float * v = orow + blk * 16; + float amax = 0.f; + #pragma unroll + for (int i = 0; i < 16; ++i) { + amax = fmaxf(amax, fabsf(v[i])); + } + float desired = amax / 6.f; + if (desired < 1e-12f) desired = 1e-12f; + __nv_fp8_e4m3 q(desired); + dst_sfa[layout(m, blk * 16, 0)] = *reinterpret_cast(&q); + const float inv = 1.f / static_cast(q); + uint2 out; + uint8_t * ob = reinterpret_cast(&out); + #pragma unroll + for (int pIdx = 0; pIdx < 8; ++pIdx) { + const uint8_t lo = ada_f32_to_e2m1(v[2 * pIdx] * inv); + const uint8_t hi = ada_f32_to_e2m1(v[2 * pIdx + 1] * inv); + ob[pIdx] = static_cast(lo | (hi << 4)); + } + reinterpret_cast(dst_packed)[(int64_t) m * n_blocks + blk] = out; + } +} + +// Fused adaLN modulate + NVFP4 quantize of the result. +template +__global__ void kernel_ada_rms_q(const float * __restrict__ x, + const float * __restrict__ scale, + const float * __restrict__ shift, + float * __restrict__ out, + uint8_t * __restrict__ dst_packed, + uint8_t * __restrict__ dst_sfa, + LayoutSF layout, + int C, float eps) { + const int m = blockIdx.x; + const float * xr = x + (int64_t) m * C; + float * orow = out + (int64_t) m * C; + + float inv_rms = 1.0f; + if (with_rms) { + float sumsq = 0.0f; + for (int c = threadIdx.x; c < C; c += blockDim.x) { + const float v = xr[c]; + sumsq += v * v; + } + __shared__ float red[32]; + #pragma unroll + for (int off = 16; off > 0; off >>= 1) { + sumsq += __shfl_xor_sync(0xffffffff, sumsq, off); + } + const int warp = threadIdx.x / 32; + if (threadIdx.x % 32 == 0) { + red[warp] = sumsq; + } + __syncthreads(); + if (warp == 0) { + float v = (threadIdx.x < blockDim.x / 32) ? red[threadIdx.x] : 0.0f; + #pragma unroll + for (int off = 16; off > 0; off >>= 1) { + v += __shfl_xor_sync(0xffffffff, v, off); + } + if (threadIdx.x == 0) { + red[0] = v; + } + } + __syncthreads(); + inv_rms = rsqrtf(red[0] / C + eps); + } + + for (int c = threadIdx.x; c < C; c += blockDim.x) { + const float n = with_rms ? xr[c] * inv_rms : xr[c]; + orow[c] = n * (1.0f + scale[c]) + shift[c]; + } + __syncthreads(); + ada_quant_row(orow, dst_packed, dst_sfa, layout, m, C); +} + +// Fused LayerNorm + affine + NVFP4 quantize of the result. +template +__global__ void kernel_layer_norm_affine_q(const float * __restrict__ x, + const float * __restrict__ w, + const float * __restrict__ b, + float * __restrict__ out, + uint8_t * __restrict__ dst_packed, + uint8_t * __restrict__ dst_sfa, + LayoutSF layout, + int C, float eps) { + const int m = blockIdx.x; + const float * xr = x + (int64_t) m * C; + float * orow = out + (int64_t) m * C; + + float sum = 0.0f, sumsq = 0.0f; + for (int c = threadIdx.x; c < C; c += blockDim.x) { + const float v = xr[c]; + sum += v; + sumsq += v * v; + } + __shared__ float red[2][32]; + #pragma unroll + for (int off = 16; off > 0; off >>= 1) { + sum += __shfl_xor_sync(0xffffffff, sum, off); + sumsq += __shfl_xor_sync(0xffffffff, sumsq, off); + } + const int warp = threadIdx.x / 32; + if (threadIdx.x % 32 == 0) { + red[0][warp] = sum; + red[1][warp] = sumsq; + } + __syncthreads(); + if (warp == 0) { + float s = (threadIdx.x < blockDim.x / 32) ? red[0][threadIdx.x] : 0.0f; + float s2 = (threadIdx.x < blockDim.x / 32) ? red[1][threadIdx.x] : 0.0f; + #pragma unroll + for (int off = 16; off > 0; off >>= 1) { + s += __shfl_xor_sync(0xffffffff, s, off); + s2 += __shfl_xor_sync(0xffffffff, s2, off); + } + if (threadIdx.x == 0) { + red[0][0] = s; + red[1][0] = s2; + } + } + __syncthreads(); + const float mean = red[0][0] / C; + const float var = red[1][0] / C - mean * mean; + const float rstd = rsqrtf(var + eps); + + for (int c = threadIdx.x; c < C; c += blockDim.x) { + orow[c] = (xr[c] - mean) * rstd * w[c] + b[c]; + } + __syncthreads(); + ada_quant_row(orow, dst_packed, dst_sfa, layout, m, C); +} + +// out[m, c] = residual[m, c] + branch[m, c] * gate[c] +__global__ void kernel_gated_residual(const float * __restrict__ residual, + const float * __restrict__ branch, + const float * __restrict__ gate, + float * __restrict__ out, + int C) { + const int m = blockIdx.x; + const int64_t off = (int64_t) m * C; + for (int c = threadIdx.x; c < C; c += blockDim.x) { + out[off + c] = residual[off + c] + branch[off + c] * gate[c]; + } +} + +// out[m, c] = (x[m, c] - mean(x[m])) * rstd(x[m]) * w[c] + b[c] +__global__ void kernel_layer_norm_affine(const float * __restrict__ x, + const float * __restrict__ w, + const float * __restrict__ b, + float * __restrict__ out, + int C, float eps) { + const int m = blockIdx.x; + const float * xr = x + (int64_t) m * C; + float * orow = out + (int64_t) m * C; + + float sum = 0.0f, sumsq = 0.0f; + for (int c = threadIdx.x; c < C; c += blockDim.x) { + const float v = xr[c]; + sum += v; + sumsq += v * v; + } + __shared__ float red[2][32]; + #pragma unroll + for (int off = 16; off > 0; off >>= 1) { + sum += __shfl_xor_sync(0xffffffff, sum, off); + sumsq += __shfl_xor_sync(0xffffffff, sumsq, off); + } + const int warp = threadIdx.x / 32; + if (threadIdx.x % 32 == 0) { + red[0][warp] = sum; + red[1][warp] = sumsq; + } + __syncthreads(); + if (warp == 0) { + float s = (threadIdx.x < blockDim.x / 32) ? red[0][threadIdx.x] : 0.0f; + float s2 = (threadIdx.x < blockDim.x / 32) ? red[1][threadIdx.x] : 0.0f; + #pragma unroll + for (int off = 16; off > 0; off >>= 1) { + s += __shfl_xor_sync(0xffffffff, s, off); + s2 += __shfl_xor_sync(0xffffffff, s2, off); + } + if (threadIdx.x == 0) { + red[0][0] = s; + red[1][0] = s2; + } + } + __syncthreads(); + const float mean = red[0][0] / C; + const float var = red[1][0] / C - mean * mean; + const float rstd = rsqrtf(var + eps); + + for (int c = threadIdx.x; c < C; c += blockDim.x) { + orow[c] = (xr[c] - mean) * rstd * w[c] + b[c]; + } +} + +// out[c] = a[c] + b[c] +__global__ void kernel_vec_add(const float * __restrict__ a, + const float * __restrict__ b, + float * __restrict__ out, + int n) { + const int i = blockIdx.x * blockDim.x + threadIdx.x; + if (i < n) { + out[i] = a[i] + b[i]; + } +} + +} // namespace + +int ada_rms_mod(const float * x, const float * scale, const float * shift, + float * out, int M, int C, float eps, bool with_rms, + cudaStream_t stream) { + const int threads = 256; + if (with_rms) { + kernel_ada_rms<<>>(x, scale, shift, out, C, eps); + } else { + kernel_ada_rms<<>>(x, scale, shift, out, C, eps); + } + const cudaError_t e = cudaGetLastError(); + return (e == cudaSuccess) ? 0 : -static_cast(e); +} + +int gated_residual(const float * residual, const float * branch, const float * gate, + float * out, int M, int C, cudaStream_t stream) { + kernel_gated_residual<<>>(residual, branch, gate, out, C); + const cudaError_t e = cudaGetLastError(); + return (e == cudaSuccess) ? 0 : -static_cast(e); +} + +int layer_norm_affine(const float * x, const float * w, const float * b, + float * out, int M, int C, float eps, cudaStream_t stream) { + kernel_layer_norm_affine<<>>(x, w, b, out, C, eps); + const cudaError_t e = cudaGetLastError(); + return (e == cudaSuccess) ? 0 : -static_cast(e); +} + +int ada_rms_mod_quant(const float * x, const float * scale, const float * shift, + float * out, void * dst_packed, void * dst_sfa, + int M, int C, float eps, bool with_rms, cudaStream_t stream) { + if (C % 16 != 0) return -1; + auto shape = cute::make_shape(M, 1, C, 1); + auto layout = AdaCfg::tile_atom_to_shape_SFA(shape); + if (with_rms) { + kernel_ada_rms_q<<>>(x, scale, shift, out, + (uint8_t *) dst_packed, (uint8_t *) dst_sfa, layout, C, eps); + } else { + kernel_ada_rms_q<<>>(x, scale, shift, out, + (uint8_t *) dst_packed, (uint8_t *) dst_sfa, layout, C, eps); + } + const cudaError_t e = cudaGetLastError(); + return (e == cudaSuccess) ? 0 : -static_cast(e); +} + +int layer_norm_affine_quant(const float * x, const float * w, const float * b, + float * out, void * dst_packed, void * dst_sfa, + int M, int C, float eps, cudaStream_t stream) { + if (C % 16 != 0) return -1; + auto shape = cute::make_shape(M, 1, C, 1); + auto layout = AdaCfg::tile_atom_to_shape_SFA(shape); + kernel_layer_norm_affine_q<<>>(x, w, b, out, + (uint8_t *) dst_packed, (uint8_t *) dst_sfa, layout, C, eps); + const cudaError_t e = cudaGetLastError(); + return (e == cudaSuccess) ? 0 : -static_cast(e); +} + +int vec_add_f32(const float * a, const float * b, float * out, int n, cudaStream_t stream) { + kernel_vec_add<<<(n + 255) / 256, 256, 0, stream>>>(a, b, out, n); + const cudaError_t e = cudaGetLastError(); + return (e == cudaSuccess) ? 0 : -static_cast(e); +} + +} // namespace ggml_cuda_flashrt diff --git a/flash_rt/structures/adapters/ggml/fr_dispatch.cu b/flash_rt/structures/adapters/ggml/fr_dispatch.cu new file mode 100644 index 00000000..59f5f3f8 --- /dev/null +++ b/flash_rt/structures/adapters/ggml/fr_dispatch.cu @@ -0,0 +1,1314 @@ +// Dispatch glue between ggml-cuda's mul_mat and the FlashRT NVFP4 kernels. +// Host-only logic; the device kernels live in the sibling fr_*.cu files. + +#include "fr_ggml.cuh" +#include "fr_kernels.h" + +// FlashRT kernels consumed directly from the flashrt-public csrc tree +// (GGML_CUDA_FLASHRT_PUBLIC_DIR); no vendored copies. +#include "gemm/fp4/cutlass_fp4_gemm_geglu_il_sm100.cuh" +#include "gemm/fp4/cutlass_fp4_gemm_siglip_ffn_f32out_sm100.cuh" + +#include +#include + +namespace { + +// Weights repacked into the CUTLASS wire format, keyed by the ggml tensor's +// device pointer. Weight tensors are immutable and live for the process +// lifetime, so entries are never evicted. +struct repacked_weight { + void * packed = nullptr; + void * sf = nullptr; +}; + +std::unordered_map g_repack_cache; +std::mutex g_repack_mu; + +// The repack allocates with cudaMalloc, which is illegal during CUDA graph +// capture. All weights are repacked during the first (uncaptured) warmup +// evaluation of each graph, so a cache miss while capturing indicates a bug. +const repacked_weight * get_repacked(const ggml_tensor * src0, cudaStream_t stream) { + std::lock_guard lk(g_repack_mu); + + auto it = g_repack_cache.find(src0->data); + if (it != g_repack_cache.end()) { + return &it->second; + } + + const int64_t K = src0->ne[0]; + const int64_t N = src0->ne[1]; + + cudaStreamCaptureStatus cap = cudaStreamCaptureStatusNone; + cudaStreamIsCapturing(stream, &cap); + if (cap != cudaStreamCaptureStatusNone) { + GGML_ABORT("flashrt: weight repack for %s requested during CUDA graph capture", src0->name); + } + + repacked_weight w; + CUDA_CHECK(cudaMalloc(&w.packed, ggml_cuda_flashrt::packed_bytes(N, K))); + CUDA_CHECK(cudaMalloc(&w.sf, ggml_cuda_flashrt::sf_bytes(N, K))); + + const int rc = ggml_cuda_flashrt::repack_weight(src0->data, w.packed, w.sf, (int) N, (int) K, stream); + if (rc != 0) { + GGML_ABORT("flashrt: weight repack failed for %s (N=%lld K=%lld rc=%d)", src0->name, (long long) N, (long long) K, rc); + } + + auto res = g_repack_cache.emplace(src0->data, w); + return &res.first->second; +} + +// Interleaved gate/up weight pairs for the fused GeGLU GEMM, keyed by the +// two tensors' device pointers (same immortality caveat as above). +struct pair_key { + const void * gate; + const void * up; + bool operator==(const pair_key & o) const { return gate == o.gate && up == o.up; } +}; +struct pair_key_hash { + size_t operator()(const pair_key & k) const noexcept { + return std::hash()(k.gate) ^ (std::hash()(k.up) << 1); + } +}; + +std::unordered_map g_pair_cache; + +const repacked_weight * get_repacked_pair(const ggml_tensor * gate_w, const ggml_tensor * up_w, cudaStream_t stream) { + std::lock_guard lk(g_repack_mu); + + pair_key key{gate_w->data, up_w->data}; + auto it = g_pair_cache.find(key); + if (it != g_pair_cache.end()) { + return &it->second; + } + + const int64_t K = gate_w->ne[0]; + const int64_t n_ff = gate_w->ne[1]; + const int64_t N_il = 2 * n_ff; + + cudaStreamCaptureStatus cap = cudaStreamCaptureStatusNone; + cudaStreamIsCapturing(stream, &cap); + if (cap != cudaStreamCaptureStatusNone) { + GGML_ABORT("flashrt: geglu pair repack for %s requested during CUDA graph capture", gate_w->name); + } + + repacked_weight w; + CUDA_CHECK(cudaMalloc(&w.packed, ggml_cuda_flashrt::packed_bytes(N_il, K))); + CUDA_CHECK(cudaMalloc(&w.sf, ggml_cuda_flashrt::sf_bytes(N_il, K))); + + const int rc = ggml_cuda_flashrt::repack_weight_pair_interleaved( + gate_w->data, up_w->data, w.packed, w.sf, (int) n_ff, (int) K, stream); + if (rc != 0) { + GGML_ABORT("flashrt: geglu pair repack failed for %s (n_ff=%lld K=%lld rc=%d)", + gate_w->name, (long long) n_ff, (long long) K, rc); + } + + auto res = g_pair_cache.emplace(key, w); + return &res.first->second; +} + +// Per-evaluation quantized-activation cache. Several ops consume the same +// fp32 activation tensor (q/k/v projections, the adaLN conditioning vector +// across all layers); quantizing it once per graph evaluation removes the +// duplicate quantize launches. Keys use the ggml tensor pointer (unique +// within one evaluation) plus an evaluation counter, so recycled device +// addresses across graphs can never alias. Slot buffers are grow-only and +// never freed, which keeps addresses stable for captured CUDA graphs; a +// replayed graph rewrites any slot before its baked consumers read it. +struct act_slot { + const ggml_tensor * key = nullptr; + uint64_t eval_id = 0; + void * packed = nullptr; + size_t packed_cap = 0; + void * sf = nullptr; + size_t sf_cap = 0; +}; + +act_slot g_act_slots[4]; +int g_act_slot_rr = 0; +uint64_t g_eval_id = 1; + +// Returns cached (packed, sf) for src1 quantized as [M, K], quantizing on a +// miss. Returns false when the cache cannot be used (slot growth needed +// while capturing a CUDA graph); the caller must quantize into pool memory. +bool get_quantized_act(const ggml_tensor * src1, int M, int K, + const void ** out_packed, const void ** out_sf, + cudaStream_t stream) { + for (auto & s : g_act_slots) { + if (s.key == src1 && s.eval_id == g_eval_id) { + *out_packed = s.packed; + *out_sf = s.sf; + return true; + } + } + + act_slot & s = g_act_slots[g_act_slot_rr]; + const size_t need_packed = (size_t) ggml_cuda_flashrt::packed_bytes(M, K); + const size_t need_sf = (size_t) ggml_cuda_flashrt::sf_bytes(M, K); + + if (need_packed > s.packed_cap || need_sf > s.sf_cap) { + cudaStreamCaptureStatus cap = cudaStreamCaptureStatusNone; + cudaStreamIsCapturing(stream, &cap); + if (cap != cudaStreamCaptureStatusNone) { + return false; + } + if (need_packed > s.packed_cap) { + if (s.packed != nullptr) { cudaFree(s.packed); } + CUDA_CHECK(cudaMalloc(&s.packed, need_packed)); + s.packed_cap = need_packed; + } + if (need_sf > s.sf_cap) { + if (s.sf != nullptr) { cudaFree(s.sf); } + CUDA_CHECK(cudaMalloc(&s.sf, need_sf)); + s.sf_cap = need_sf; + } + } + g_act_slot_rr = (g_act_slot_rr + 1) % 4; + + const int rc = ggml_cuda_flashrt::quantize_act_f32( + (const float *) src1->data, s.packed, s.sf, M, K, stream); + if (rc != 0) { + GGML_ABORT("flashrt: activation quantize failed (M=%d K=%d rc=%d)", M, K, rc); + } + s.key = src1; + s.eval_id = g_eval_id; + *out_packed = s.packed; + *out_sf = s.sf; + return true; +} + +// Reserve a cache slot for an activation that a producer kernel will fill +// with already-quantized data (fused quantize). Returns false when slot +// growth would be needed during CUDA graph capture. +bool reserve_quantized_act(const ggml_tensor * out_tensor, int M, int K, + void ** out_packed, void ** out_sf, + cudaStream_t stream) { + act_slot & s = g_act_slots[g_act_slot_rr]; + const size_t need_packed = (size_t) ggml_cuda_flashrt::packed_bytes(M, K); + const size_t need_sf = (size_t) ggml_cuda_flashrt::sf_bytes(M, K); + + if (need_packed > s.packed_cap || need_sf > s.sf_cap) { + cudaStreamCaptureStatus cap = cudaStreamCaptureStatusNone; + cudaStreamIsCapturing(stream, &cap); + if (cap != cudaStreamCaptureStatusNone) { + return false; + } + if (need_packed > s.packed_cap) { + if (s.packed != nullptr) { cudaFree(s.packed); } + CUDA_CHECK(cudaMalloc(&s.packed, need_packed)); + s.packed_cap = need_packed; + } + if (need_sf > s.sf_cap) { + if (s.sf != nullptr) { cudaFree(s.sf); } + CUDA_CHECK(cudaMalloc(&s.sf, need_sf)); + s.sf_cap = need_sf; + } + } + g_act_slot_rr = (g_act_slot_rr + 1) % 4; + + s.key = out_tensor; + s.eval_id = g_eval_id; + *out_packed = s.packed; + *out_sf = s.sf; + return true; +} + +// Grow-only device buffer for the never-written D of the no-D-store GeGLU +// variants (the host-side TMA descriptor still needs a valid allocation). +void * get_dummy_d(size_t bytes) { + static void * buf = nullptr; + static size_t cap = 0; + static std::mutex mu; + std::lock_guard lk(mu); + if (bytes > cap) { + if (buf != nullptr) { + cudaFree(buf); + } + CUDA_CHECK(cudaMalloc(&buf, bytes)); + cap = bytes; + } + return buf; +} + +} // namespace + +bool ggml_cuda_flashrt_should_use(const ggml_tensor * src0, const ggml_tensor * src1, const ggml_tensor * dst) { + static const bool disabled = getenv("GGML_CUDA_FLASHRT_DISABLE") != nullptr; + if (disabled) { + return false; + } + if (src0->type != GGML_TYPE_NVFP4 || src1->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) { + return false; + } + if (!ggml_is_contiguous(src0) || !ggml_is_contiguous(src1) || !ggml_is_contiguous(dst)) { + return false; + } + // Batched src1 with unbatched (broadcast) weights folds into a single + // GEMM over all rows because src1/dst are fully contiguous. + if (src0->ne[2] != 1 || src0->ne[3] != 1) { + return false; + } + const int64_t K = src0->ne[0]; + const int64_t N = src0->ne[1]; + if (K % 64 != 0 || N % 16 != 0) { + return false; + } + return true; +} + +void ggml_cuda_flashrt_mul_mat(ggml_backend_cuda_context & ctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) { + const int K = (int) src0->ne[0]; + const int N = (int) src0->ne[1]; + const int M = (int) ggml_nrows(src1); // batch dims fold into rows (contiguous, broadcast weights) + + cudaStream_t stream = ctx.stream(); + + // The persistent repack cache is keyed by the weight tensor's device + // pointer, which is only sound when weight tensors are immortal (model + // inference). Tools that create and free tensors at recycled addresses + // (e.g. test-backend-ops) must set GGML_CUDA_FLASHRT_NO_CACHE=1 to + // repack into scratch memory on every call instead. + static const bool no_cache = getenv("GGML_CUDA_FLASHRT_NO_CACHE") != nullptr; + + ggml_cuda_pool_alloc b_packed_scratch(ctx.pool()); + ggml_cuda_pool_alloc b_sf_scratch (ctx.pool()); + + const void * b_packed = nullptr; + const void * b_sf = nullptr; + if (no_cache) { + b_packed_scratch.alloc(ggml_cuda_flashrt::packed_bytes(N, K)); + b_sf_scratch.alloc(ggml_cuda_flashrt::sf_bytes(N, K)); + const int rrc = ggml_cuda_flashrt::repack_weight(src0->data, b_packed_scratch.get(), b_sf_scratch.get(), N, K, stream); + if (rrc != 0) { + GGML_ABORT("flashrt: weight repack failed (N=%d K=%d rc=%d)", N, K, rrc); + } + b_packed = b_packed_scratch.get(); + b_sf = b_sf_scratch.get(); + } else { + const repacked_weight * w = get_repacked(src0, stream); + b_packed = w->packed; + b_sf = w->sf; + } + + ggml_cuda_pool_alloc a_packed(ctx.pool()); + ggml_cuda_pool_alloc a_sf (ctx.pool()); + + const void * q_packed = nullptr; + const void * q_sf = nullptr; + if (!get_quantized_act(src1, M, K, &q_packed, &q_sf, stream)) { + a_packed.alloc(ggml_cuda_flashrt::packed_bytes(M, K)); + a_sf.alloc(ggml_cuda_flashrt::sf_bytes(M, K)); + const int qrc = ggml_cuda_flashrt::quantize_act_f32( + (const float *) src1->data, a_packed.get(), a_sf.get(), M, K, stream); + if (qrc != 0) { + GGML_ABORT("flashrt: activation quantize failed (M=%d K=%d rc=%d)", M, K, qrc); + } + q_packed = a_packed.get(); + q_sf = a_sf.get(); + } + + // ggml's block_nvfp4 scale bytes carry standard e4m3 semantics: its + // dequant table doubles the e2m1 values but its ue4m3 decode halves the + // scale, so the two cancel and no alpha compensation is needed. + const float alpha = 1.0f; + // The 128x128x256 tile beats the wide-N 128x256x128 tile on Thor for + // every production shape measured (including N=16384 prefill FFN). + const bool widen = false; + + const int rc = ggml_cuda_flashrt::gemm_f32out( + q_packed, q_sf, b_packed, b_sf, + (float *) dst->data, M, N, K, alpha, widen, stream); + if (rc != 0) { + GGML_ABORT("flashrt: gemm failed (M=%d N=%d K=%d rc=%d)", M, N, K, rc); + } +} + +bool ggml_cuda_flashrt_should_fuse_ada(const ggml_tensor * rms, const ggml_tensor * mm, const ggml_tensor * bias_add, + const ggml_tensor * view_scale, const ggml_tensor * repeat_scale, + const ggml_tensor * mul, const ggml_tensor * add1, + const ggml_tensor * view_shift, const ggml_tensor * repeat_shift, + const ggml_tensor * add2) { + const ggml_tensor * x = rms != nullptr ? rms->src[0] : mul->src[0]; + const ggml_tensor * normed = rms != nullptr ? rms : x; + const int64_t C = x->ne[0]; + const int64_t M = x->ne[1]; + + if (x->type != GGML_TYPE_F32 || !ggml_is_contiguous(x) || x->ne[2] != 1 || x->ne[3] != 1) { + return false; + } + // modulation projection: [3C, 1] from the shared conditioning vector + if (!ggml_cuda_flashrt_should_use(mm->src[0], mm->src[1], mm) || + mm->ne[0] != 3 * C || mm->ne[1] != 1) { + return false; + } + const ggml_tensor * bias = bias_add->src[1]; + if (bias_add->src[0] != mm || bias->type != GGML_TYPE_F32 || + !ggml_is_contiguous(bias) || bias->ne[0] != 3 * C || !ggml_is_contiguous(bias_add)) { + return false; + } + // scale = mod[0:C], shift = mod[C:2C] + if (view_scale->src[0] != bias_add || view_scale->ne[0] != C || view_scale->ne[1] != 1 || + view_scale->view_offs != 0) { + return false; + } + if (view_shift->src[0] != bias_add || view_shift->ne[0] != C || view_shift->ne[1] != 1 || + view_shift->view_offs != (size_t) C * sizeof(float)) { + return false; + } + if (repeat_scale->src[0] != view_scale || repeat_shift->src[0] != view_shift || + repeat_scale->ne[0] != C || repeat_scale->ne[1] != M) { + return false; + } + // t = normed * scale; t2 = normed + t; out = t2 + shift + if (mul->src[0] != normed || mul->src[1] != repeat_scale || + add1->src[0] != normed || add1->src[1] != mul || + add2->src[0] != add1 || add2->src[1] != repeat_shift || + !ggml_is_contiguous(add2) || add2->ne[0] != C || add2->ne[1] != M) { + return false; + } + return true; +} + +void ggml_cuda_flashrt_ada_norm(ggml_backend_cuda_context & ctx, const ggml_tensor * rms, const ggml_tensor * mm, + ggml_tensor * bias_add, const ggml_tensor * view_scale, const ggml_tensor * mul, + const ggml_tensor * view_shift, ggml_tensor * add2) { + const ggml_tensor * x = rms != nullptr ? rms->src[0] : mul->src[0]; + const ggml_tensor * cond = mm->src[1]; + + const int C = (int) x->ne[0]; + const int M = (int) x->ne[1]; + const int K = (int) mm->src[0]->ne[0]; + const int N = (int) mm->src[0]->ne[1]; // 3C + + cudaStream_t stream = ctx.stream(); + + const repacked_weight * w = get_repacked(mm->src[0], stream); + + ggml_cuda_pool_alloc c_packed(ctx.pool()); + ggml_cuda_pool_alloc c_sf (ctx.pool()); + + const void * q_packed = nullptr; + const void * q_sf = nullptr; + int rc = 0; + if (!get_quantized_act(cond, 1, K, &q_packed, &q_sf, stream)) { + c_packed.alloc(ggml_cuda_flashrt::packed_bytes(1, K)); + c_sf.alloc(ggml_cuda_flashrt::sf_bytes(1, K)); + rc = ggml_cuda_flashrt::quantize_act_f32((const float *) cond->data, c_packed.get(), c_sf.get(), 1, K, stream); + q_packed = c_packed.get(); + q_sf = c_sf.get(); + } + if (rc == 0) { + // bias applied in the GEMM epilogue, writing the biased modulation + // vector (still read later through the gate view) directly + rc = flash_rt::fp4::gemm_bias_f32out(q_packed, q_sf, w->packed, w->sf, + bias_add->src[1]->data, + (float *) bias_add->data, 1, N, K, stream); + } + if (rc == 0) { + const float eps = rms != nullptr ? ggml_get_op_params_f32(rms, 0) : 0.0f; + // emit the quantized form alongside f32 so downstream GEMMs skip + // their activation quantize (registered in the per-eval cache) + void * q_out_packed = nullptr; + void * q_out_sf = nullptr; + if (C % 16 == 0 && reserve_quantized_act(add2, M, C, &q_out_packed, &q_out_sf, stream)) { + rc = ggml_cuda_flashrt::ada_rms_mod_quant((const float *) x->data, + (const float *) view_scale->data, + (const float *) view_shift->data, + (float *) add2->data, q_out_packed, q_out_sf, + M, C, eps, rms != nullptr, stream); + } else { + rc = ggml_cuda_flashrt::ada_rms_mod((const float *) x->data, + (const float *) view_scale->data, + (const float *) view_shift->data, + (float *) add2->data, M, C, eps, rms != nullptr, stream); + } + } + if (rc != 0) { + GGML_ABORT("flashrt: fused adaLN failed (M=%d C=%d rc=%d)", M, C, rc); + } +} + +// Cached-modulation adaLN window: the modulation vector comes from a graph +// input (precomputed per step, see pi0_ae.cpp) instead of a GEMM. Sequence: +// {RMS_NORM, VIEW mod-col, VIEW scale, REPEAT, MUL, ADD, VIEW shift, REPEAT, +// ADD} -> one ada kernel reading scale/shift straight from the input views. +static int g_ada_cached_fail = 0; +#define FR_ADA_FAIL(code) do { g_ada_cached_fail = (code); return false; } while (0) + +bool ggml_cuda_flashrt_should_fuse_ada_cached( + const ggml_tensor * rms, const ggml_tensor * view_col, + const ggml_tensor * view_scale, const ggml_tensor * repeat_scale, + const ggml_tensor * mul, const ggml_tensor * add1, + const ggml_tensor * view_shift, const ggml_tensor * repeat_shift, + const ggml_tensor * add2) { + static const bool disabled = getenv("GGML_CUDA_FLASHRT_DISABLE") != nullptr; + if (disabled) { + FR_ADA_FAIL(1); + } + const ggml_tensor * x = rms->src[0]; + if (x == nullptr || x->type != GGML_TYPE_F32 || !ggml_is_contiguous(x)) { + FR_ADA_FAIL(2); + } + const int64_t C = x->ne[0]; + const int64_t M = x->ne[1]; + if (x->ne[2] != 1 || x->ne[3] != 1 || C % 4 != 0) { + FR_ADA_FAIL(3); + } + if (view_col->type != GGML_TYPE_F32 || view_col->ne[0] != 3 * C || view_col->ne[1] != 1 || + view_col->src[0] == nullptr) { + FR_ADA_FAIL(4); + } + // view_offs accumulates through view-of-view chains down to the ultimate + // source, so compare offsets relative to the enclosing column view + if (view_scale->src[0] != view_col || view_scale->ne[0] != C || view_scale->ne[1] != 1 || + view_scale->view_offs != view_col->view_offs) { + if (getenv("GGML_FLASHRT_DEBUG") != nullptr) { + static int dbg5 = 0; + if (dbg5++ < 4) { + fprintf(stderr, "[fr-ada-c5] src_ok=%d ne0=%lld (C=%lld) ne1=%lld offs=%zu col_offs=%zu\n", + (int) (view_scale->src[0] == view_col), (long long) view_scale->ne[0], + (long long) C, (long long) view_scale->ne[1], + view_scale->view_offs, view_col->view_offs); + } + } + FR_ADA_FAIL(5); + } + if (view_shift->src[0] != view_col || view_shift->ne[0] != C || view_shift->ne[1] != 1 || + view_shift->view_offs != view_col->view_offs + (size_t) C * sizeof(float)) { + FR_ADA_FAIL(6); + } + if (repeat_scale->src[0] != view_scale || repeat_scale->ne[0] != C || repeat_scale->ne[1] != M || + repeat_shift->src[0] != view_shift || repeat_shift->ne[0] != C || repeat_shift->ne[1] != M) { + FR_ADA_FAIL(7); + } + if (mul->src[0] != rms || mul->src[1] != repeat_scale || + add1->src[0] != rms || add1->src[1] != mul || + add2->src[0] != add1 || add2->src[1] != repeat_shift || + !ggml_is_contiguous(add2) || add2->ne[0] != C || add2->ne[1] != M) { + FR_ADA_FAIL(8); + } + g_ada_cached_fail = 0; + return true; +} + +int ggml_cuda_flashrt_ada_cached_fail_code() { return g_ada_cached_fail; } + +void ggml_cuda_flashrt_ada_norm_cached(ggml_backend_cuda_context & ctx, const ggml_tensor * rms, + const ggml_tensor * view_scale, const ggml_tensor * view_shift, + ggml_tensor * add2) { + const ggml_tensor * x = rms->src[0]; + const int C = (int) x->ne[0]; + const int M = (int) x->ne[1]; + const float eps = ggml_get_op_params_f32(rms, 0); + cudaStream_t stream = ctx.stream(); + + int rc; + void * q_out_packed = nullptr; + void * q_out_sf = nullptr; + if (C % 16 == 0 && reserve_quantized_act(add2, M, C, &q_out_packed, &q_out_sf, stream)) { + rc = ggml_cuda_flashrt::ada_rms_mod_quant((const float *) x->data, + (const float *) view_scale->data, + (const float *) view_shift->data, + (float *) add2->data, q_out_packed, q_out_sf, + M, C, eps, true, stream); + } else { + rc = ggml_cuda_flashrt::ada_rms_mod((const float *) x->data, + (const float *) view_scale->data, + (const float *) view_shift->data, + (float *) add2->data, M, C, eps, true, stream); + } + if (rc != 0) { + GGML_ABORT("flashrt: cached adaLN failed (M=%d C=%d rc=%d)", M, C, rc); + } +} + +bool ggml_cuda_flashrt_should_fuse_gated_res(const ggml_tensor * view, const ggml_tensor * repeat, + const ggml_tensor * mul, const ggml_tensor * add) { + const int64_t C = view->ne[0]; + const int64_t M = mul->ne[1]; + if (view->type != GGML_TYPE_F32 || view->ne[1] != 1 || + view->src[0] == nullptr || view->src[0]->type != GGML_TYPE_F32) { + return false; + } + if (repeat->src[0] != view || repeat->ne[0] != C || repeat->ne[1] != M) { + return false; + } + const ggml_tensor * branch = mul->src[0]; + if (mul->src[1] != repeat || branch->type != GGML_TYPE_F32 || !ggml_is_contiguous(branch) || + branch->ne[0] != C || branch->ne[1] != M) { + return false; + } + const ggml_tensor * residual = add->src[0] == mul ? add->src[1] : add->src[0]; + if ((add->src[0] != mul && add->src[1] != mul) || residual == mul || + residual->type != GGML_TYPE_F32 || !ggml_is_contiguous(residual) || + residual->ne[0] != C || residual->ne[1] != M || + !ggml_is_contiguous(add) || add->ne[0] != C || add->ne[1] != M) { + return false; + } + return true; +} + +void ggml_cuda_flashrt_gated_residual(ggml_backend_cuda_context & ctx, const ggml_tensor * view, + const ggml_tensor * mul, ggml_tensor * add) { + const ggml_tensor * branch = mul->src[0]; + const ggml_tensor * residual = add->src[0] == mul ? add->src[1] : add->src[0]; + const int C = (int) view->ne[0]; + const int M = (int) mul->ne[1]; + + const int rc = ggml_cuda_flashrt::gated_residual( + (const float *) residual->data, (const float *) branch->data, + (const float *) view->data, (float *) add->data, M, C, ctx.stream()); + if (rc != 0) { + GGML_ABORT("flashrt: fused gated residual failed (M=%d C=%d rc=%d)", M, C, rc); + } +} + +// SigLIP FFN weights prepared for the fused FP4 pair, keyed by the two +// weight pointers. Up rows are padded to a 32 multiple for the FP4 output; +// the f16 Down weight is quantized with its K padded to the same value. +struct siglip_ffn_weights { + void * up_packed = nullptr; + void * up_sf = nullptr; + void * dn_packed = nullptr; + void * dn_sf = nullptr; + int h_pad = 0; +}; + +std::unordered_map g_sig_ffn_cache; + +const siglip_ffn_weights * get_siglip_ffn_weights(const ggml_tensor * up_w, const ggml_tensor * dn_w, cudaStream_t stream) { + std::lock_guard lk(g_repack_mu); + + pair_key key{up_w->data, dn_w->data}; + auto it = g_sig_ffn_cache.find(key); + if (it != g_sig_ffn_cache.end()) { + return &it->second; + } + + const int64_t K_in = up_w->ne[0]; // model dim (NVFP4 up weight) + const int64_t H = up_w->ne[1]; // hidden dim + const int64_t H_pad = GGML_PAD(H, 64); + const int64_t D_out = dn_w->ne[1]; // model dim (f16 down weight, K = H) + + cudaStreamCaptureStatus cap = cudaStreamCaptureStatusNone; + cudaStreamIsCapturing(stream, &cap); + if (cap != cudaStreamCaptureStatusNone) { + GGML_ABORT("flashrt: siglip ffn weight prep for %s requested during CUDA graph capture", up_w->name); + } + + siglip_ffn_weights w; + w.h_pad = (int) H_pad; + CUDA_CHECK(cudaMalloc(&w.up_packed, ggml_cuda_flashrt::packed_bytes(H_pad, K_in))); + CUDA_CHECK(cudaMalloc(&w.up_sf, ggml_cuda_flashrt::sf_bytes(H_pad, K_in))); + CUDA_CHECK(cudaMalloc(&w.dn_packed, ggml_cuda_flashrt::packed_bytes(D_out, H_pad))); + CUDA_CHECK(cudaMalloc(&w.dn_sf, ggml_cuda_flashrt::sf_bytes(D_out, H_pad))); + + int rc = ggml_cuda_flashrt::repack_weight_rows_padded( + up_w->data, w.up_packed, w.up_sf, (int) H, (int) H_pad, (int) K_in, stream); + if (rc == 0) { + rc = ggml_cuda_flashrt::quantize_weight_f16_padded( + dn_w->data, w.dn_packed, w.dn_sf, (int) D_out, (int) H, (int) H_pad, stream); + } + if (rc != 0) { + GGML_ABORT("flashrt: siglip ffn weight prep failed for %s (H=%lld rc=%d)", up_w->name, (long long) H, rc); + } + + auto res = g_sig_ffn_cache.emplace(key, w); + return &res.first->second; +} + +bool ggml_cuda_flashrt_should_fuse_ln(const ggml_tensor * norm, const ggml_tensor * mul, const ggml_tensor * add) { + const ggml_tensor * x = norm->src[0]; + const int64_t C = norm->ne[0]; + if (x->type != GGML_TYPE_F32 || !ggml_is_contiguous(x) || norm->ne[3] != 1) { + return false; + } + const ggml_tensor * w = mul->src[0] == norm ? mul->src[1] : mul->src[0]; + if ((mul->src[0] != norm && mul->src[1] != norm) || w == norm || + w->type != GGML_TYPE_F32 || !ggml_is_contiguous(w) || + w->ne[0] != C || ggml_nelements(w) != C) { + return false; + } + const ggml_tensor * b = add->src[0] == mul ? add->src[1] : add->src[0]; + if ((add->src[0] != mul && add->src[1] != mul) || b == mul || + b->type != GGML_TYPE_F32 || !ggml_is_contiguous(b) || + b->ne[0] != C || ggml_nelements(b) != C) { + return false; + } + if (!ggml_is_contiguous(add) || add->ne[0] != C || ggml_nrows(add) != ggml_nrows(x)) { + return false; + } + return true; +} + +void ggml_cuda_flashrt_ln_affine(ggml_backend_cuda_context & ctx, const ggml_tensor * norm, const ggml_tensor * mul, ggml_tensor * add) { + const ggml_tensor * x = norm->src[0]; + const ggml_tensor * w = mul->src[0] == norm ? mul->src[1] : mul->src[0]; + const ggml_tensor * b = add->src[0] == mul ? add->src[1] : add->src[0]; + + const int C = (int) norm->ne[0]; + const int M = (int) ggml_nrows(x); + const float eps = ggml_get_op_params_f32(norm, 0); + + int rc; + void * q_out_packed = nullptr; + void * q_out_sf = nullptr; + if (C % 16 == 0 && reserve_quantized_act(add, M, C, &q_out_packed, &q_out_sf, ctx.stream())) { + rc = ggml_cuda_flashrt::layer_norm_affine_quant( + (const float *) x->data, (const float *) w->data, (const float *) b->data, + (float *) add->data, q_out_packed, q_out_sf, M, C, eps, ctx.stream()); + } else { + rc = ggml_cuda_flashrt::layer_norm_affine( + (const float *) x->data, (const float *) w->data, (const float *) b->data, + (float *) add->data, M, C, eps, ctx.stream()); + } + if (rc != 0) { + GGML_ABORT("flashrt: fused layer norm failed (M=%d C=%d rc=%d)", M, C, rc); + } +} + +bool ggml_cuda_flashrt_should_fuse_geglu(const ggml_tensor * gate_mm, const ggml_tensor * up_mm, const ggml_tensor * glu, const ggml_tensor * down_mm) { + if (ggml_get_glu_op(glu) != GGML_GLU_OP_GEGLU) { + return false; + } + // both projections share the same input and shape + if (gate_mm->src[1] != up_mm->src[1]) { + return false; + } + const ggml_tensor * gate_w = gate_mm->src[0]; + const ggml_tensor * up_w = up_mm->src[0]; + const ggml_tensor * down_w = down_mm->src[0]; + if (gate_w->ne[0] != up_w->ne[0] || gate_w->ne[1] != up_w->ne[1]) { + return false; + } + if (!ggml_cuda_flashrt_should_use(gate_w, gate_mm->src[1], gate_mm) || + !ggml_cuda_flashrt_should_use(up_w, up_mm->src[1], up_mm) || + !ggml_cuda_flashrt_should_use(down_w, glu, down_mm)) { + return false; + } + // the down projection must consume the GLU output with K = n_ff + if (down_mm->src[1] != glu || down_w->ne[0] != gate_w->ne[1]) { + return false; + } + return true; +} + +void ggml_cuda_flashrt_geglu_ffn(ggml_backend_cuda_context & ctx, const ggml_tensor * gate_mm, const ggml_tensor * up_mm, const ggml_tensor * glu, ggml_tensor * down_mm) { + GGML_UNUSED(glu); + + const ggml_tensor * src1 = gate_mm->src[1]; + const ggml_tensor * gate_w = gate_mm->src[0]; + const ggml_tensor * down_w = down_mm->src[0]; + + const int K = (int) gate_w->ne[0]; + const int n_ff = (int) gate_w->ne[1]; + const int N_il = 2 * n_ff; + const int M = (int) ggml_nrows(src1); + const int N_out = (int) down_w->ne[1]; + + cudaStream_t stream = ctx.stream(); + + const repacked_weight * w_il = get_repacked_pair(gate_w, up_mm->src[0], stream); + const repacked_weight * w_down = get_repacked(down_w, stream); + + ggml_cuda_pool_alloc a_packed(ctx.pool()); + ggml_cuda_pool_alloc a_sf (ctx.pool()); + + const void * q_packed = nullptr; + const void * q_sf = nullptr; + int rc = 0; + if (!get_quantized_act(src1, M, K, &q_packed, &q_sf, stream)) { + a_packed.alloc(ggml_cuda_flashrt::packed_bytes(M, K)); + a_sf.alloc(ggml_cuda_flashrt::sf_bytes(M, K)); + rc = ggml_cuda_flashrt::quantize_act_f32( + (const float *) src1->data, a_packed.get(), a_sf.get(), M, K, stream); + if (rc != 0) { + GGML_ABORT("flashrt: geglu activation quantize failed (M=%d K=%d rc=%d)", M, K, rc); + } + q_packed = a_packed.get(); + q_sf = a_sf.get(); + } + + ggml_cuda_pool_alloc compact_packed(ctx.pool(), ggml_cuda_flashrt::packed_bytes(M, n_ff)); + ggml_cuda_pool_alloc compact_sfa (ctx.pool(), ggml_cuda_flashrt::sf_bytes(M, n_ff)); + void * dummy_d = get_dummy_d(ggml_cuda_flashrt::packed_bytes(M, N_il)); + + // skinny-M tile for decode-sized batches, default tile otherwise + if (M < 128) { + rc = flash_rt::fp4::cutlass_fp4_gemm_geglu_il_hw_nod_v10( + q_packed, q_sf, w_il->packed, w_il->sf, + dummy_d, compact_packed.get(), compact_sfa.get(), M, N_il, K, stream); + } else { + rc = flash_rt::fp4::cutlass_fp4_gemm_geglu_il_hw_nod( + q_packed, q_sf, w_il->packed, w_il->sf, + dummy_d, compact_packed.get(), compact_sfa.get(), M, N_il, K, stream); + } + if (rc != 0) { + GGML_ABORT("flashrt: geglu gemm failed (M=%d N_il=%d K=%d rc=%d)", M, N_il, K, rc); + } + + rc = ggml_cuda_flashrt::gemm_f32out( + compact_packed.get(), compact_sfa.get(), w_down->packed, w_down->sf, + (float *) down_mm->data, M, N_out, n_ff, /*alpha=*/1.0f, /*widen=*/false, stream); + if (rc != 0) { + GGML_ABORT("flashrt: geglu down gemm failed (M=%d N=%d K=%d rc=%d)", M, N_out, n_ff, rc); + } +} + +// Fused QKV weights (row-concat [k | v | q]) keyed by the three pointers. +struct triple_key { + const void * a; const void * b; const void * c; + bool operator==(const triple_key & o) const { return a == o.a && b == o.b && c == o.c; } +}; +struct triple_key_hash { + size_t operator()(const triple_key & k) const noexcept { + return std::hash()(k.a) ^ (std::hash()(k.b) << 1) ^ (std::hash()(k.c) << 2); + } +}; + +std::unordered_map g_qkv_cache; + +const repacked_weight * get_repacked_qkv(const ggml_tensor * wk, const ggml_tensor * wv, const ggml_tensor * wq, cudaStream_t stream) { + std::lock_guard lk(g_repack_mu); + + triple_key key{wk->data, wv->data, wq->data}; + auto it = g_qkv_cache.find(key); + if (it != g_qkv_cache.end()) { + return &it->second; + } + + const int64_t K = wk->ne[0]; + const int64_t N_tot = wk->ne[1] + wv->ne[1] + wq->ne[1]; + + cudaStreamCaptureStatus cap = cudaStreamCaptureStatusNone; + cudaStreamIsCapturing(stream, &cap); + if (cap != cudaStreamCaptureStatusNone) { + GGML_ABORT("flashrt: qkv repack for %s requested during CUDA graph capture", wq->name); + } + + repacked_weight w; + CUDA_CHECK(cudaMalloc(&w.packed, ggml_cuda_flashrt::packed_bytes(N_tot, K))); + CUDA_CHECK(cudaMalloc(&w.sf, ggml_cuda_flashrt::sf_bytes(N_tot, K))); + + const int rc = ggml_cuda_flashrt::repack_weight_concat3( + wk->data, (int) wk->ne[1], wv->data, (int) wv->ne[1], wq->data, (int) wq->ne[1], + w.packed, w.sf, (int) K, stream); + if (rc != 0) { + GGML_ABORT("flashrt: qkv repack failed for %s (rc=%d)", wq->name, rc); + } + + auto res = g_qkv_cache.emplace(key, w); + return &res.first->second; +} + +bool ggml_cuda_flashrt_should_fuse_siglip_ffn(const ggml_tensor * up_mm, const ggml_tensor * bias1, const ggml_tensor * gelu, + const ggml_tensor * cont1, const ggml_tensor * dn_mm, const ggml_tensor * bias2, + const ggml_tensor * cont2, const ggml_tensor * res_add) { + const ggml_tensor * up_w = up_mm->src[0]; + const ggml_tensor * src1 = up_mm->src[1]; + const ggml_tensor * dn_w = dn_mm->src[0]; + + if (!ggml_cuda_flashrt_should_use(up_w, src1, up_mm)) { + return false; + } + if (dn_w->type != GGML_TYPE_F16 || !ggml_is_contiguous(dn_w) || + dn_w->ne[0] != up_w->ne[1] || dn_w->ne[0] % 16 != 0 || + dn_w->ne[1] % 16 != 0 || dn_w->ne[2] != 1) { + return false; + } + if (ggml_get_unary_op(gelu) != GGML_UNARY_OP_GELU || gelu->src[0] != bias1) { + return false; + } + const ggml_tensor * b1 = bias1->src[0] == up_mm ? bias1->src[1] : bias1->src[0]; + if ((bias1->src[0] != up_mm && bias1->src[1] != up_mm) || b1 == up_mm || + b1->type != GGML_TYPE_F32 || !ggml_is_contiguous(b1) || ggml_nelements(b1) != up_w->ne[1]) { + return false; + } + if (cont1->src[0] != gelu || dn_mm->src[1] != cont1) { + return false; + } + const ggml_tensor * b2 = bias2->src[0] == dn_mm ? bias2->src[1] : bias2->src[0]; + if ((bias2->src[0] != dn_mm && bias2->src[1] != dn_mm) || b2 == dn_mm || + b2->type != GGML_TYPE_F32 || !ggml_is_contiguous(b2) || ggml_nelements(b2) != dn_w->ne[1]) { + return false; + } + if (cont2->src[0] != bias2) { + return false; + } + const ggml_tensor * residual = res_add->src[0] == cont2 ? res_add->src[1] : res_add->src[0]; + if ((res_add->src[0] != cont2 && res_add->src[1] != cont2) || residual == cont2 || + residual->type != GGML_TYPE_F32 || !ggml_is_contiguous(residual) || + !ggml_is_contiguous(res_add) || res_add->ne[0] != dn_w->ne[1] || + ggml_nrows(res_add) != ggml_nrows(src1) || ggml_nrows(residual) != ggml_nrows(src1)) { + return false; + } + return true; +} + +void ggml_cuda_flashrt_siglip_ffn(ggml_backend_cuda_context & ctx, const ggml_tensor * up_mm, const ggml_tensor * bias1, + const ggml_tensor * dn_mm, const ggml_tensor * bias2, + const ggml_tensor * cont2, ggml_tensor * res_add) { + const ggml_tensor * up_w = up_mm->src[0]; + const ggml_tensor * src1 = up_mm->src[1]; + const ggml_tensor * dn_w = dn_mm->src[0]; + const ggml_tensor * b1 = bias1->src[0] == up_mm ? bias1->src[1] : bias1->src[0]; + const ggml_tensor * b2 = bias2->src[0] == dn_mm ? bias2->src[1] : bias2->src[0]; + const ggml_tensor * residual = res_add->src[0] == cont2 ? res_add->src[1] : res_add->src[0]; + + const int K_in = (int) up_w->ne[0]; + const int D_out = (int) dn_w->ne[1]; + const int M = (int) ggml_nrows(src1); + + cudaStream_t stream = ctx.stream(); + + const siglip_ffn_weights * w = get_siglip_ffn_weights(up_w, dn_w, stream); + const int H_pad = w->h_pad; + + const void * q_packed = nullptr; + const void * q_sf = nullptr; + ggml_cuda_pool_alloc a_packed(ctx.pool()); + ggml_cuda_pool_alloc a_sf (ctx.pool()); + int rc = 0; + if (!get_quantized_act(src1, M, K_in, &q_packed, &q_sf, stream)) { + a_packed.alloc(ggml_cuda_flashrt::packed_bytes(M, K_in)); + a_sf.alloc(ggml_cuda_flashrt::sf_bytes(M, K_in)); + rc = ggml_cuda_flashrt::quantize_act_f32((const float *) src1->data, a_packed.get(), a_sf.get(), M, K_in, stream); + q_packed = a_packed.get(); + q_sf = a_sf.get(); + } + + ggml_cuda_pool_alloc hid_packed(ctx.pool(), ggml_cuda_flashrt::packed_bytes(M, H_pad)); + ggml_cuda_pool_alloc hid_sf (ctx.pool(), ggml_cuda_flashrt::sf_bytes(M, H_pad)); + + if (rc == 0) { + rc = flash_rt::fp4::siglip_ffn_up_gelu_fp4out( + q_packed, q_sf, w->up_packed, w->up_sf, b1->data, + hid_packed.get(), hid_sf.get(), M, H_pad, K_in, stream); + } + if (rc == 0) { + rc = flash_rt::fp4::siglip_ffn_down_bias_res_f32( + hid_packed.get(), hid_sf.get(), w->dn_packed, w->dn_sf, b2->data, + residual->data, res_add->data, M, D_out, H_pad, stream, 1.0f); + } + if (rc != 0) { + GGML_ABORT("flashrt: siglip ffn failed (M=%d H_pad=%d rc=%d)", M, H_pad, rc); + } +} + +bool ggml_cuda_flashrt_should_fuse_qkv(const ggml_tensor * k_mm, const ggml_tensor * k_rope, const ggml_tensor * k_cpy, + const ggml_tensor * v_mm, const ggml_tensor * v_cpy, + const ggml_tensor * q_mm, const ggml_tensor * q_rope, const ggml_tensor * q_scale) { + const ggml_tensor * src1 = k_mm->src[1]; + if (v_mm->src[1] != src1 || q_mm->src[1] != src1) { + return false; + } + const ggml_tensor * wk = k_mm->src[0]; + const ggml_tensor * wv = v_mm->src[0]; + const ggml_tensor * wq = q_mm->src[0]; + if (!ggml_cuda_flashrt_should_use(wk, src1, k_mm) || + !ggml_cuda_flashrt_should_use(wv, src1, v_mm) || + !ggml_cuda_flashrt_should_use(wq, src1, q_mm) || + wk->ne[0] != wv->ne[0] || wk->ne[0] != wq->ne[0]) { + return false; + } + // K path: mm -> reshape -> rope -> cpy into an f16 view of the + // persistent KV buffer; single KV head (Nk == head_dim) + const int64_t head_dim = k_rope->src[0]->ne[0]; + if (k_rope->src[0]->op != GGML_OP_RESHAPE || k_rope->src[0]->src[0] != k_mm || + wk->ne[1] != head_dim || k_rope->src[0]->ne[1] != 1) { + return false; + } + if (k_cpy->src[0] != k_rope || k_cpy->src[1] == nullptr || + k_cpy->src[1]->type != GGML_TYPE_F16 || k_cpy->src[1]->op != GGML_OP_VIEW) { + return false; + } + // V path: mm -> reshape -> cpy into f16 view + if (v_cpy->src[0] == nullptr || v_cpy->src[0]->op != GGML_OP_RESHAPE || + v_cpy->src[0]->src[0] != v_mm || wv->ne[1] != head_dim || + v_cpy->src[1] == nullptr || v_cpy->src[1]->type != GGML_TYPE_F16 || v_cpy->src[1]->op != GGML_OP_VIEW) { + return false; + } + // Q path: mm -> reshape -> rope -> scale, head_dim x n_head + if (q_rope->src[0]->op != GGML_OP_RESHAPE || q_rope->src[0]->src[0] != q_mm || + q_rope->src[0]->ne[0] != head_dim || wq->ne[1] % head_dim != 0 || + q_scale->src[0] != q_rope || !ggml_is_contiguous(q_scale)) { + return false; + } + // both ropes must share positions, freq factors and parameters + if (k_rope->src[1] != q_rope->src[1] || k_rope->src[2] != q_rope->src[2] || + memcmp(k_rope->op_params, q_rope->op_params, sizeof(k_rope->op_params)) != 0) { + return false; + } + // NEOX rope only (matches ggml's rope_neox math replicated in qkv_post) + const int mode = ((const int32_t *) k_rope->op_params)[2]; + if (mode != GGML_ROPE_TYPE_NEOX) { + return false; + } + // f16 KV views must be row-contiguous (head_dim elements per token row) + const ggml_tensor * kv = k_cpy->src[1]; + if (kv->nb[0] != sizeof(uint16_t) || kv->nb[1] != head_dim * sizeof(uint16_t)) { + return false; + } + return true; +} + +void ggml_cuda_flashrt_qkv(ggml_backend_cuda_context & ctx, + const ggml_tensor * k_mm, const ggml_tensor * k_rope, const ggml_tensor * k_cpy, + const ggml_tensor * v_mm, const ggml_tensor * v_cpy, + const ggml_tensor * q_mm, const ggml_tensor * q_rope, ggml_tensor * q_scale) { + const ggml_tensor * src1 = k_mm->src[1]; + const ggml_tensor * wk = k_mm->src[0]; + const ggml_tensor * wv = v_mm->src[0]; + const ggml_tensor * wq = q_mm->src[0]; + + const int K = (int) wk->ne[0]; + const int Nk = (int) wk->ne[1]; + const int Nv = (int) wv->ne[1]; + const int Nq = (int) wq->ne[1]; + const int M = (int) ggml_nrows(src1); + const int head_dim = Nk; + + cudaStream_t stream = ctx.stream(); + + const repacked_weight * w = get_repacked_qkv(wk, wv, wq, stream); + + const void * q_packed = nullptr; + const void * q_sf = nullptr; + ggml_cuda_pool_alloc a_packed(ctx.pool()); + ggml_cuda_pool_alloc a_sf (ctx.pool()); + int rc = 0; + if (!get_quantized_act(src1, M, K, &q_packed, &q_sf, stream)) { + a_packed.alloc(ggml_cuda_flashrt::packed_bytes(M, K)); + a_sf.alloc(ggml_cuda_flashrt::sf_bytes(M, K)); + rc = ggml_cuda_flashrt::quantize_act_f32((const float *) src1->data, a_packed.get(), a_sf.get(), M, K, stream); + q_packed = a_packed.get(); + q_sf = a_sf.get(); + } + + const int N_tot = Nk + Nv + Nq; + ggml_cuda_pool_alloc qkv_cat(ctx.pool(), (int64_t) M * N_tot); + + if (rc == 0) { + rc = ggml_cuda_flashrt::gemm_f32out(q_packed, q_sf, w->packed, w->sf, + qkv_cat.get(), M, N_tot, K, 1.0f, false, stream); + } + if (rc == 0) { + // rope parameters, mirrored from ggml-cuda's rope host setup + const int32_t * op = (const int32_t *) k_rope->op_params; + const int n_dims = op[1]; + const int n_ctx_orig = op[4]; + float freq_base, freq_scale, ext_factor, attn_factor, beta_fast, beta_slow; + memcpy(&freq_base, op + 5, sizeof(float)); + memcpy(&freq_scale, op + 6, sizeof(float)); + memcpy(&ext_factor, op + 7, sizeof(float)); + memcpy(&attn_factor, op + 8, sizeof(float)); + memcpy(&beta_fast, op + 9, sizeof(float)); + memcpy(&beta_slow, op + 10, sizeof(float)); + float corr_dims[2]; + ggml_rope_yarn_corr_dims(n_dims, n_ctx_orig, freq_base, beta_fast, beta_slow, corr_dims); + const float theta_scale = powf(freq_base, -2.0f / n_dims); + const float scale_f = ggml_get_op_params_f32(q_scale, 0); + + const ggml_tensor * ff = k_rope->src[2]; + rc = ggml_cuda_flashrt::qkv_post( + qkv_cat.get(), (float *) q_scale->data, + k_cpy->src[1]->data, v_cpy->src[1]->data, + (const int32_t *) k_rope->src[1]->data, + ff != nullptr ? (const float *) ff->data : nullptr, + M, Nk, Nv, Nq, head_dim, n_dims, + freq_scale, ext_factor, attn_factor, + corr_dims[0], corr_dims[1], theta_scale, scale_f, stream); + } + if (rc != 0) { + GGML_ABORT("flashrt: fused qkv failed (M=%d N=%d K=%d rc=%d)", M, N_tot, K, rc); + } +} + +// --------------------------------------------------------------------------- +// Vision QKV pad window: {mul_mat, add bias, reshape}x3 + {pad}x3 -> three +// padded-weight GEMMs with the bias in the epilogue, writing the pad buffers +// directly. Widens per-head projections (SigLIP head_dim 72 -> 80) inside the +// repacked weights, so the runtime pad kernels and separate bias adds vanish. + +namespace { + +struct grouppad_weight { + void * packed = nullptr; + void * sf = nullptr; + void * bias = nullptr; // f32 [group_out * n_groups], zero-interleaved +}; + +std::unordered_map g_grouppad_cache; + +const grouppad_weight * get_repacked_grouppad( + const ggml_tensor * w, const ggml_tensor * bias, + int group_in, int group_out, int n_groups, cudaStream_t stream) { + std::lock_guard lk(g_repack_mu); + + auto it = g_grouppad_cache.find(w->data); + if (it != g_grouppad_cache.end()) { + return &it->second; + } + + cudaStreamCaptureStatus cap = cudaStreamCaptureStatusNone; + cudaStreamIsCapturing(stream, &cap); + if (cap != cudaStreamCaptureStatusNone) { + GGML_ABORT("flashrt: grouppad repack for %s requested during CUDA graph capture", w->name); + } + + const int K = (int) w->ne[0]; + const int N_pad = group_out * n_groups; + + grouppad_weight g; + CUDA_CHECK(cudaMalloc(&g.packed, ggml_cuda_flashrt::packed_bytes(N_pad, K))); + CUDA_CHECK(cudaMalloc(&g.sf, ggml_cuda_flashrt::sf_bytes(N_pad, K))); + CUDA_CHECK(cudaMalloc(&g.bias, (size_t) N_pad * sizeof(float))); + + const int rc = ggml_cuda_flashrt::repack_weight_rows_grouppad( + w->data, g.packed, g.sf, group_in, group_out, n_groups, K, stream); + if (rc != 0) { + GGML_ABORT("flashrt: grouppad repack failed for %s (gi=%d go=%d ng=%d K=%d rc=%d)", + w->name, group_in, group_out, n_groups, K, rc); + } + CUDA_CHECK(cudaMemsetAsync(g.bias, 0, (size_t) N_pad * sizeof(float), stream)); + CUDA_CHECK(cudaMemcpy2DAsync(g.bias, (size_t) group_out * sizeof(float), + bias->data, (size_t) group_in * sizeof(float), + (size_t) group_in * sizeof(float), (size_t) n_groups, + cudaMemcpyDeviceToDevice, stream)); + + auto res = g_grouppad_cache.emplace(w->data, g); + return &res.first->second; +} + +// One projection triple of the window: mul_mat -> add(bias) -> reshape -> pad. +bool vis_qkv_pad_leg_ok(const ggml_tensor * mm, const ggml_tensor * add, + const ggml_tensor * resh, const ggml_tensor * pad, + const ggml_tensor * shared_src1) { + if (mm->op != GGML_OP_MUL_MAT || add->op != GGML_OP_ADD || + resh->op != GGML_OP_RESHAPE || pad->op != GGML_OP_PAD) { + return false; + } + const ggml_tensor * w = mm->src[0]; + const ggml_tensor * b = add->src[1]; + if (w == nullptr || w->type != GGML_TYPE_NVFP4 || mm->src[1] != shared_src1) { + return false; + } + if (add->src[0] != mm || resh->src[0] != add || pad->src[0] != resh) { + return false; + } + if (b == nullptr || b->type != GGML_TYPE_F32 || !ggml_is_contiguous(b) || + b->ne[0] != mm->ne[0] || ggml_nrows(b) != 1) { + return false; + } + const int64_t group_in = resh->ne[0]; + const int64_t n_groups = resh->ne[1]; + const int64_t group_out = pad->ne[0]; + if (group_in * n_groups != mm->ne[0] || group_out <= group_in) { + return false; + } + // pad only widens dim0 + if (pad->ne[1] != resh->ne[1] || pad->ne[2] != resh->ne[2] || pad->ne[3] != resh->ne[3]) { + return false; + } + if (pad->type != GGML_TYPE_F32 || !ggml_is_contiguous(pad)) { + return false; + } + const int64_t K = w->ne[0]; + const int64_t N_pad = group_out * n_groups; + if (K % 64 != 0 || N_pad % 16 != 0) { + return false; + } + return true; +} + +} // namespace + +bool ggml_cuda_flashrt_should_fuse_vis_qkv_pad( + const ggml_tensor * mm_q, const ggml_tensor * add_q, const ggml_tensor * resh_q, + const ggml_tensor * mm_k, const ggml_tensor * add_k, const ggml_tensor * resh_k, + const ggml_tensor * mm_v, const ggml_tensor * add_v, const ggml_tensor * resh_v, + const ggml_tensor * pad_q, const ggml_tensor * pad_k, const ggml_tensor * pad_v) { + static const bool disabled = getenv("GGML_CUDA_FLASHRT_DISABLE") != nullptr; + if (disabled) { + return false; + } + const ggml_tensor * src1 = mm_q->src[1]; + if (src1 == nullptr || src1->type != GGML_TYPE_F32 || !ggml_is_contiguous(src1)) { + return false; + } + if (!vis_qkv_pad_leg_ok(mm_q, add_q, resh_q, pad_q, src1) || + !vis_qkv_pad_leg_ok(mm_k, add_k, resh_k, pad_k, src1) || + !vis_qkv_pad_leg_ok(mm_v, add_v, resh_v, pad_v, src1)) { + return false; + } + // identical geometry across the three legs + if (mm_k->ne[0] != mm_q->ne[0] || mm_v->ne[0] != mm_q->ne[0] || + pad_k->ne[0] != pad_q->ne[0] || pad_v->ne[0] != pad_q->ne[0] || + resh_k->ne[0] != resh_q->ne[0] || resh_v->ne[0] != resh_q->ne[0]) { + return false; + } + const int64_t M = ggml_nrows(src1); + if (M <= 0 || M > INT32_MAX) { + return false; + } + return true; +} + +void ggml_cuda_flashrt_vis_qkv_pad(ggml_backend_cuda_context & ctx, + const ggml_tensor * mm_q, const ggml_tensor * add_q, ggml_tensor * pad_q, + const ggml_tensor * mm_k, const ggml_tensor * add_k, ggml_tensor * pad_k, + const ggml_tensor * mm_v, const ggml_tensor * add_v, ggml_tensor * pad_v) { + cudaStream_t stream = ctx.stream(); + + const ggml_tensor * src1 = mm_q->src[1]; + const int K = (int) mm_q->src[0]->ne[0]; + const int M = (int) ggml_nrows(src1); + + const int group_in = (int) (pad_q->src[0]->ne[0]); + const int group_out = (int) pad_q->ne[0]; + const int n_groups = (int) pad_q->ne[1]; + const int N_pad = group_out * n_groups; + + ggml_cuda_pool_alloc a_packed(ctx.pool()); + ggml_cuda_pool_alloc a_sf (ctx.pool()); + const void * q_packed = nullptr; + const void * q_sf = nullptr; + if (!get_quantized_act(src1, M, K, &q_packed, &q_sf, stream)) { + a_packed.alloc(ggml_cuda_flashrt::packed_bytes(M, K)); + a_sf.alloc(ggml_cuda_flashrt::sf_bytes(M, K)); + const int qrc = ggml_cuda_flashrt::quantize_act_f32( + (const float *) src1->data, a_packed.get(), a_sf.get(), M, K, stream); + if (qrc != 0) { + GGML_ABORT("flashrt: vis qkv activation quantize failed (M=%d K=%d rc=%d)", M, K, qrc); + } + q_packed = a_packed.get(); + q_sf = a_sf.get(); + } + + const ggml_tensor * legs[3][3] = { + { mm_q, add_q, pad_q }, + { mm_k, add_k, pad_k }, + { mm_v, add_v, pad_v }, + }; + for (auto & leg : legs) { + const grouppad_weight * w = get_repacked_grouppad( + leg[0]->src[0], leg[1]->src[1], group_in, group_out, n_groups, stream); + const int rc = flash_rt::fp4::gemm_bias_f32out( + q_packed, q_sf, w->packed, w->sf, w->bias, + (float *) leg[2]->data, M, N_pad, K, stream); + if (rc != 0) { + GGML_ABORT("flashrt: vis qkv padded gemm failed (M=%d N=%d K=%d rc=%d)", M, N_pad, K, rc); + } + } +} + +// --------------------------------------------------------------------------- +// GEMM + (bias) + residual window: {mul_mat, add bias?, add residual} -> one +// GEMM with the bias and the residual in the epilogue (C may alias D; the +// epilogue reads C before writing D). Covers the encoder/vision o- and +// down-projections whose residual adds were separate bandwidth passes. + +namespace { + +std::unordered_map g_zero_bias_cache; // keyed by N + +const void * get_zero_bias(int N, cudaStream_t stream) { + std::lock_guard lk(g_repack_mu); + auto it = g_zero_bias_cache.find(N); + if (it != g_zero_bias_cache.end()) { + return it->second; + } + cudaStreamCaptureStatus cap = cudaStreamCaptureStatusNone; + cudaStreamIsCapturing(stream, &cap); + if (cap != cudaStreamCaptureStatusNone) { + return nullptr; // caller falls back; allocation is capture-illegal + } + void * p = nullptr; + CUDA_CHECK(cudaMalloc(&p, (size_t) N * sizeof(float))); + CUDA_CHECK(cudaMemsetAsync(p, 0, (size_t) N * sizeof(float), stream)); + g_zero_bias_cache.emplace(N, p); + return p; +} + +const ggml_tensor * mm_res_residual(const ggml_tensor * chain, const ggml_tensor * res_add) { + const ggml_tensor * r = res_add->src[0] == chain ? res_add->src[1] + : res_add->src[1] == chain ? res_add->src[0] : nullptr; + if (r == nullptr || r == chain || r->type != GGML_TYPE_F32 || !ggml_is_contiguous(r) || + !ggml_are_same_shape(r, res_add)) { + return nullptr; + } + return r; +} + +} // namespace + +bool ggml_cuda_flashrt_should_fuse_mm_res(const ggml_tensor * mm, const ggml_tensor * bias_add, + const ggml_tensor * res_add) { + if (!ggml_cuda_flashrt_should_use(mm->src[0], mm->src[1], mm)) { + return false; + } + const int64_t N = mm->ne[0]; + const int64_t K = mm->src[0]->ne[0]; + if (N % 16 != 0 || K % 64 != 0) { + return false; + } + const ggml_tensor * chain = mm; + if (bias_add != nullptr) { + const ggml_tensor * b = bias_add->src[1]; + if (bias_add->src[0] != mm || b == nullptr || b->type != GGML_TYPE_F32 || + !ggml_is_contiguous(b) || b->ne[0] != N || ggml_nrows(b) != 1) { + return false; + } + chain = bias_add; + } + if (mm_res_residual(chain, res_add) == nullptr || + !ggml_is_contiguous(res_add) || !ggml_are_same_shape(res_add, mm)) { + return false; + } + return true; +} + +bool ggml_cuda_flashrt_mm_res(ggml_backend_cuda_context & ctx, const ggml_tensor * mm, + const ggml_tensor * bias_add, ggml_tensor * res_add) { + const ggml_tensor * src1 = mm->src[1]; + const int M = (int) ggml_nrows(src1); + const int N = (int) mm->ne[0]; + const int K = (int) mm->src[0]->ne[0]; + cudaStream_t stream = ctx.stream(); + + const void * bias = bias_add != nullptr ? bias_add->src[1]->data : get_zero_bias(N, stream); + if (bias == nullptr) { + return false; // zero-bias alloc during capture: run unfused + } + const ggml_tensor * residual = mm_res_residual(bias_add != nullptr ? bias_add : mm, res_add); + + const repacked_weight * w = get_repacked(mm->src[0], stream); + + ggml_cuda_pool_alloc a_packed(ctx.pool()); + ggml_cuda_pool_alloc a_sf (ctx.pool()); + const void * q_packed = nullptr; + const void * q_sf = nullptr; + if (!get_quantized_act(src1, M, K, &q_packed, &q_sf, stream)) { + a_packed.alloc(ggml_cuda_flashrt::packed_bytes(M, K)); + a_sf.alloc(ggml_cuda_flashrt::sf_bytes(M, K)); + const int qrc = ggml_cuda_flashrt::quantize_act_f32( + (const float *) src1->data, a_packed.get(), a_sf.get(), M, K, stream); + if (qrc != 0) { + GGML_ABORT("flashrt: mm+res activation quantize failed (M=%d K=%d rc=%d)", M, K, qrc); + } + q_packed = a_packed.get(); + q_sf = a_sf.get(); + } + + const int rc = flash_rt::fp4::siglip_ffn_down_bias_res_f32( + q_packed, q_sf, w->packed, w->sf, bias, + residual->data, (float *) res_add->data, M, N, K, stream, 1.0f); + if (rc != 0) { + GGML_ABORT("flashrt: mm+res fused gemm failed (M=%d N=%d K=%d rc=%d)", M, N, K, rc); + } + return true; +} + +void ggml_cuda_flashrt_begin_eval() { + g_eval_id++; +} diff --git a/flash_rt/structures/adapters/ggml/fr_gemm_f32out.cu b/flash_rt/structures/adapters/ggml/fr_gemm_f32out.cu new file mode 100644 index 00000000..0312fbed --- /dev/null +++ b/flash_rt/structures/adapters/ggml/fr_gemm_f32out.cu @@ -0,0 +1,207 @@ +// Block-scaled NVFP4 GEMM for Thor SM110, fp32 output. +// +// Vendored from FlashRT's cutlass_nvfp4_w4a16_gemm_sm100.cu (Apache-2.0) +// with ElementD changed from bf16 to fp32 so the result lands directly in +// ggml's fp32 dst tensor. The Sm100 CollectiveBuilder dispatch under +// KernelScheduleAuto produces the block-scaled tcgen05 mainloop when built +// for sm_110a. + +#include "fr_kernels.h" + +#include "cute/tensor.hpp" + +#include "cutlass/cutlass.h" +#include "cutlass/numeric_types.h" +#include "cutlass/detail/sm100_blockscaled_layout.hpp" + +#include "cutlass/epilogue/collective/collective_builder.hpp" +#include "cutlass/gemm/collective/collective_builder.hpp" +#include "cutlass/gemm/device/gemm_universal_adapter.h" +#include "cutlass/gemm/kernel/gemm_universal.hpp" + +#include "cutlass/util/packed_stride.hpp" + +#include +#include +#include + +namespace ggml_cuda_flashrt { + +namespace { + +using namespace cute; + +template > +struct FrGemmConfig { + using ElementA = cutlass::float_e2m1_t; + using ElementB = cutlass::float_e2m1_t; + using ElementC = float; + using ElementD = float; + using ElementAccumulator = float; + using ElementCompute = float; + using ElementSF = cutlass::float_ue4m3_t; + + using LayoutA = cutlass::layout::RowMajor; + using LayoutB = cutlass::layout::ColumnMajor; + using LayoutC = cutlass::layout::RowMajor; + using LayoutD = cutlass::layout::RowMajor; + + using ElementPairA = cutlass::nv_float4_t; + using ElementPairB = cutlass::nv_float4_t; + + static constexpr int AlignmentA = 32; + static constexpr int AlignmentB = 32; + static constexpr int AlignmentC = 128 / cutlass::sizeof_bits::value; // 4 + static constexpr int AlignmentD = 128 / cutlass::sizeof_bits::value; // 4 + + using ClusterShape = ClusterShapeT; + + using CollectiveEpilogue = + typename cutlass::epilogue::collective::CollectiveBuilder< + cutlass::arch::Sm100, cutlass::arch::OpClassBlockScaledTensorOp, + TileShape, ClusterShape, + cutlass::epilogue::collective::EpilogueTileAuto, + ElementAccumulator, ElementCompute, + ElementC, LayoutC, AlignmentC, + ElementD, LayoutD, AlignmentD, + cutlass::epilogue::collective::EpilogueScheduleAuto + >::CollectiveOp; + + using CollectiveMainloop = + typename cutlass::gemm::collective::CollectiveBuilder< + cutlass::arch::Sm100, cutlass::arch::OpClassBlockScaledTensorOp, + ElementPairA, LayoutA, AlignmentA, + ElementPairB, LayoutB, AlignmentB, + ElementAccumulator, + TileShape, ClusterShape, + cutlass::gemm::collective::StageCountAutoCarveout< + static_cast(sizeof(typename CollectiveEpilogue::SharedStorage))>, + cutlass::gemm::collective::KernelScheduleAuto + >::CollectiveOp; + + using GemmKernel = cutlass::gemm::kernel::GemmUniversal< + Shape, + CollectiveMainloop, + CollectiveEpilogue>; + + using Gemm = cutlass::gemm::device::GemmUniversalAdapter; +}; + +using Sm1xxBlkScaledConfig = cutlass::detail::Sm1xxBlockScaledConfig<16>; + +// Per-shape CUTLASS workspace cache; entries live for the process lifetime +// (weight shapes are fixed per model). +struct ShapeKey { + int M, N, K; + bool operator==(const ShapeKey & o) const { return M == o.M && N == o.N && K == o.K; } +}; +struct ShapeKeyHash { + size_t operator()(const ShapeKey & k) const noexcept { + return (static_cast(k.M) * 1315423911u) + ^ (static_cast(k.N) * 2654435761u) + ^ static_cast(k.K); + } +}; +struct CachedWorkspace { void * ptr = nullptr; size_t size = 0; }; + +std::unordered_map g_ws_cache; +std::mutex g_ws_mu; + +void * get_workspace(int M, int N, int K, size_t needed) { + std::lock_guard lk(g_ws_mu); + ShapeKey key{M, N, K}; + auto it = g_ws_cache.find(key); + if (it != g_ws_cache.end() && it->second.size >= needed) return it->second.ptr; + if (it != g_ws_cache.end()) { cudaFree(it->second.ptr); g_ws_cache.erase(it); } + CachedWorkspace w; w.size = needed; + if (needed > 0) cudaMalloc(&w.ptr, needed); + g_ws_cache[key] = w; + return w.ptr; +} + +template +int run_gemm(const void * A_packed, const void * SFA, + const void * B_packed, const void * SFB, + float * D, int M, int N, int K, + float alpha, cudaStream_t stream) { + using Gemm = typename Config::Gemm; + using ElementSF = typename Config::ElementSF; + using ElementD = typename Config::ElementD; + + using StrideA = typename Gemm::GemmKernel::StrideA; + using StrideB = typename Gemm::GemmKernel::StrideB; + using StrideC = typename Gemm::GemmKernel::StrideC; + using StrideD = typename Gemm::GemmKernel::StrideD; + + StrideA stride_A = cutlass::make_cute_packed_stride(StrideA{}, cute::make_shape(M, K, 1)); + StrideB stride_B = cutlass::make_cute_packed_stride(StrideB{}, cute::make_shape(N, K, 1)); + StrideC stride_C = cutlass::make_cute_packed_stride(StrideC{}, cute::make_shape(M, N, 1)); + StrideD stride_D = cutlass::make_cute_packed_stride(StrideD{}, cute::make_shape(M, N, 1)); + + auto problem_shape_MNKL = cute::make_shape(M, N, K, 1); + auto layout_SFA = Sm1xxBlkScaledConfig::tile_atom_to_shape_SFA(problem_shape_MNKL); + auto layout_SFB = Sm1xxBlkScaledConfig::tile_atom_to_shape_SFB(problem_shape_MNKL); + + using ArrayElementA = typename Gemm::GemmKernel::CollectiveMainloop::ArrayElementA; + using ArrayElementB = typename Gemm::GemmKernel::CollectiveMainloop::ArrayElementB; + + typename Gemm::Arguments args{ + cutlass::gemm::GemmUniversalMode::kGemm, + {M, N, K, 1}, + { + reinterpret_cast(A_packed), stride_A, + reinterpret_cast(B_packed), stride_B, + reinterpret_cast(SFA), layout_SFA, + reinterpret_cast(SFB), layout_SFB + }, + { + {alpha, 0.0f}, + nullptr, stride_C, + reinterpret_cast(D), stride_D + } + }; + + Gemm gemm; + size_t ws_size = Gemm::get_workspace_size(args); + void * ws_ptr = get_workspace(M, N, K, ws_size); + + auto status = gemm.can_implement(args); + if (status != cutlass::Status::kSuccess) { + std::fprintf(stderr, "[fr_gemm_f32out] can_implement FAIL M=%d N=%d K=%d (status=%d)\n", + M, N, K, static_cast(status)); + return static_cast(status); + } + status = gemm.initialize(args, ws_ptr, stream); + if (status != cutlass::Status::kSuccess) { + std::fprintf(stderr, "[fr_gemm_f32out] initialize FAIL M=%d N=%d K=%d (status=%d)\n", + M, N, K, static_cast(status)); + return static_cast(status); + } + status = gemm.run(stream); + if (status != cutlass::Status::kSuccess) { + std::fprintf(stderr, "[fr_gemm_f32out] run FAIL M=%d N=%d K=%d (status=%d)\n", + M, N, K, static_cast(status)); + return static_cast(status); + } + return 0; +} + +using ConfigDefault = FrGemmConfig>; +// 2-SM tcgen05 tile: 11-15% faster than the 1-SM tile on Thor for every +// prefill shape measured (M >= ~256); slower at decode-sized M. +using ConfigLargeM = FrGemmConfig, Shape<_2, _1, _1>>; + +} // namespace + +int gemm_f32out(const void * A_packed, const void * SFA, + const void * B_packed, const void * SFB, + float * D, int M, int N, int K, + float alpha, bool widen, cudaStream_t stream) { + (void) widen; // superseded by the M-based tile choice + if (M >= 256) { + return run_gemm(A_packed, SFA, B_packed, SFB, D, M, N, K, alpha, stream); + } + return run_gemm(A_packed, SFA, B_packed, SFB, D, M, N, K, alpha, stream); +} + +} // namespace ggml_cuda_flashrt diff --git a/flash_rt/structures/adapters/ggml/fr_ggml.cuh b/flash_rt/structures/adapters/ggml/fr_ggml.cuh new file mode 100644 index 00000000..e821e086 --- /dev/null +++ b/flash_rt/structures/adapters/ggml/fr_ggml.cuh @@ -0,0 +1,105 @@ +// ggml-facing interface of the FlashRT NVFP4 path (Thor SM110). +// Included from ggml-cuda.cu under #ifdef GGML_CUDA_FLASHRT. +#pragma once + +// Host dependency: ggml-cuda's internal common header. The consuming build +// must put ggml/src/ggml-cuda on the include path (the ggml adapter is +// compiled inside the host's build tree, like the vllm/sglang adapters run +// inside their host's runtime). +#include "common.cuh" + +// Called at the start of every backend graph evaluation; invalidates the +// per-evaluation quantized-activation cache. +void ggml_cuda_flashrt_begin_eval(); + +// True when this mul_mat should be routed to the FlashRT block-scaled NVFP4 +// GEMM: NVFP4 weights, fp32 contiguous activations/dst, no batch dims, +// shapes within kernel alignment. cc must already be checked by the caller. +bool ggml_cuda_flashrt_should_use(const ggml_tensor * src0, const ggml_tensor * src1, const ggml_tensor * dst); + +// dst = src1 @ src0 via activation quantize + NVFP4 x NVFP4 tcgen05 GEMM. +// Weights are repacked into the CUTLASS wire format on first use and cached +// for the process lifetime. +void ggml_cuda_flashrt_mul_mat(ggml_backend_cuda_context & ctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst); + +// True when the 4-node FFN subgraph {mul_mat gate, mul_mat up, GEGLU, +// mul_mat down} can run as one fused GeGLU GEMM (interleaved gate/up +// weights, FP4 intermediate) followed by the down GEMM. +bool ggml_cuda_flashrt_should_fuse_geglu(const ggml_tensor * gate_mm, const ggml_tensor * up_mm, const ggml_tensor * glu, const ggml_tensor * down_mm); + +// Execute that fused FFN; writes the down mul_mat's dst. +void ggml_cuda_flashrt_geglu_ffn(ggml_backend_cuda_context & ctx, const ggml_tensor * gate_mm, const ggml_tensor * up_mm, const ggml_tensor * glu, ggml_tensor * down_mm); + +// pi0.5 adaLN modulate window: {rms_norm?, mul_mat mod, add bias, view, +// repeat, mul, add, view, repeat, add}. rms is null for the variant whose +// normalized input arrives from a previous graph split. Outputs written by +// the fused execution: the bias add (consumed by the gate view later) and +// the final add. +bool ggml_cuda_flashrt_should_fuse_ada(const ggml_tensor * rms, const ggml_tensor * mm, const ggml_tensor * bias_add, + const ggml_tensor * view_scale, const ggml_tensor * repeat_scale, + const ggml_tensor * mul, const ggml_tensor * add1, + const ggml_tensor * view_shift, const ggml_tensor * repeat_shift, + const ggml_tensor * add2); +void ggml_cuda_flashrt_ada_norm(ggml_backend_cuda_context & ctx, const ggml_tensor * rms, const ggml_tensor * mm, + ggml_tensor * bias_add, const ggml_tensor * view_scale, const ggml_tensor * mul, + const ggml_tensor * view_shift, ggml_tensor * add2); + +// LayerNorm + affine window: {NORM, MUL weight, ADD bias} -> one kernel. +bool ggml_cuda_flashrt_should_fuse_ada_cached( + const ggml_tensor * rms, const ggml_tensor * view_col, + const ggml_tensor * view_scale, const ggml_tensor * repeat_scale, + const ggml_tensor * mul, const ggml_tensor * add1, + const ggml_tensor * view_shift, const ggml_tensor * repeat_shift, + const ggml_tensor * add2); +void ggml_cuda_flashrt_ada_norm_cached(ggml_backend_cuda_context & ctx, const ggml_tensor * rms, + const ggml_tensor * view_scale, const ggml_tensor * view_shift, + ggml_tensor * add2); + +bool ggml_cuda_flashrt_should_fuse_ln(const ggml_tensor * norm, const ggml_tensor * mul, const ggml_tensor * add); +void ggml_cuda_flashrt_ln_affine(ggml_backend_cuda_context & ctx, const ggml_tensor * norm, const ggml_tensor * mul, ggml_tensor * add); + +// SigLIP FFN window: {mul_mat up (NVFP4), add bias, GELU, cont, mul_mat +// down (f16), add bias, cont, add residual} -> fused FP4 Up GEMM (gelu +// epilogue, FP4 hidden) + Down GEMM (bias + residual epilogue). +bool ggml_cuda_flashrt_should_fuse_siglip_ffn(const ggml_tensor * up_mm, const ggml_tensor * bias1, const ggml_tensor * gelu, + const ggml_tensor * cont1, const ggml_tensor * dn_mm, const ggml_tensor * bias2, + const ggml_tensor * cont2, const ggml_tensor * res_add); +void ggml_cuda_flashrt_siglip_ffn(ggml_backend_cuda_context & ctx, const ggml_tensor * up_mm, const ggml_tensor * bias1, + const ggml_tensor * dn_mm, const ggml_tensor * bias2, + const ggml_tensor * cont2, ggml_tensor * res_add); + +// pi0.5 AE fused QKV window: {mm k, reshape, rope, view, cpy, mm v, +// reshape, view, cpy, mm q, reshape, rope, scale} -> one fused GEMM over +// row-concatenated [k|v|q] weights + one post kernel (rope/scale/f16 KV +// suffix stores). +bool ggml_cuda_flashrt_should_fuse_qkv(const ggml_tensor * k_mm, const ggml_tensor * k_rope, const ggml_tensor * k_cpy, + const ggml_tensor * v_mm, const ggml_tensor * v_cpy, + const ggml_tensor * q_mm, const ggml_tensor * q_rope, const ggml_tensor * q_scale); +void ggml_cuda_flashrt_qkv(ggml_backend_cuda_context & ctx, + const ggml_tensor * k_mm, const ggml_tensor * k_rope, const ggml_tensor * k_cpy, + const ggml_tensor * v_mm, const ggml_tensor * v_cpy, + const ggml_tensor * q_mm, const ggml_tensor * q_rope, ggml_tensor * q_scale); + +// pi0.5 gated residual window: {view gate, repeat, mul, add}. +// Vision QKV pad window: {mul_mat, add bias, reshape}x3 + {pad}x3 -> three +// padded-weight GEMMs (bias in epilogue) writing the pad buffers directly. +bool ggml_cuda_flashrt_should_fuse_vis_qkv_pad( + const ggml_tensor * mm_q, const ggml_tensor * add_q, const ggml_tensor * resh_q, + const ggml_tensor * mm_k, const ggml_tensor * add_k, const ggml_tensor * resh_k, + const ggml_tensor * mm_v, const ggml_tensor * add_v, const ggml_tensor * resh_v, + const ggml_tensor * pad_q, const ggml_tensor * pad_k, const ggml_tensor * pad_v); +void ggml_cuda_flashrt_vis_qkv_pad(ggml_backend_cuda_context & ctx, + const ggml_tensor * mm_q, const ggml_tensor * add_q, ggml_tensor * pad_q, + const ggml_tensor * mm_k, const ggml_tensor * add_k, ggml_tensor * pad_k, + const ggml_tensor * mm_v, const ggml_tensor * add_v, ggml_tensor * pad_v); + +// GEMM + optional bias + residual add fused into one epilogue. +bool ggml_cuda_flashrt_should_fuse_mm_res(const ggml_tensor * mm, const ggml_tensor * bias_add, + const ggml_tensor * res_add); +bool ggml_cuda_flashrt_mm_res(ggml_backend_cuda_context & ctx, const ggml_tensor * mm, + const ggml_tensor * bias_add, ggml_tensor * res_add); + +bool ggml_cuda_flashrt_should_fuse_gated_res(const ggml_tensor * view, const ggml_tensor * repeat, + const ggml_tensor * mul, const ggml_tensor * add); +void ggml_cuda_flashrt_gated_residual(ggml_backend_cuda_context & ctx, const ggml_tensor * view, + const ggml_tensor * mul, ggml_tensor * add); diff --git a/flash_rt/structures/adapters/ggml/fr_kernels.h b/flash_rt/structures/adapters/ggml/fr_kernels.h new file mode 100644 index 00000000..cc8c8423 --- /dev/null +++ b/flash_rt/structures/adapters/ggml/fr_kernels.h @@ -0,0 +1,126 @@ +// FlashRT NVFP4 kernels for Jetson AGX Thor (SM110). +// +// C-style entry points implemented in the fr_*.cu translation units, which +// are compiled separately for sm_110a with CUTLASS. This header must stay +// free of CUTLASS and ggml includes so it can be consumed from the regular +// ggml-cuda translation units. +// +// Wire format (NVFP4): +// packed: uint8 [rows, K/2], adjacent-pair nibbles (elem 2i low, 2i+1 high) +// scales: e4m3 (positive/ue4m3), one per 16 elements along K, stored in +// the CUTLASS Sm1xx block-scaled atom layout for the GEMM shape +// +// ggml's block_nvfp4 scale bytes carry standard e4m3 semantics (its dequant +// table doubles the e2m1 values and its ue4m3 decode halves the scale, which +// cancel), so they pass through to the GEMM unmodified with alpha = 1.0. +#pragma once + +#include +#include + +namespace ggml_cuda_flashrt { + +static inline int64_t round_up_i64(int64_t x, int64_t m) { return (x + m - 1) / m * m; } + +// Scale-factor buffer size in bytes for a [rows, K] operand: the Sm1xx atom +// layout tiles rows in chunks of 128 and K/16 scale columns in chunks of 4. +static inline int64_t sf_bytes(int64_t rows, int64_t K) { + return round_up_i64(rows, 128) * round_up_i64(K / 16, 4); +} + +static inline int64_t packed_bytes(int64_t rows, int64_t K) { + return rows * (K / 2); +} + +// Block-scaled NVFP4 x NVFP4 GEMM, D = alpha * (A x B), fp32 output. +// A_packed [M, K/2] row-major, B_packed [N, K/2] row-major (used as +// column-major [K, N]), D fp32 [M, N] row-major. +// Requires K % 64 == 0, N % 16 == 0, D 16-byte aligned. +// widen selects a wide-N tile (use for N >= 8192). +// Returns 0 on success. +int gemm_f32out(const void * A_packed, const void * SFA, + const void * B_packed, const void * SFB, + float * D, int M, int N, int K, + float alpha, bool widen, cudaStream_t stream); + +// Quantize fp32 activations [M, K] row-major (contiguous) to NVFP4 packed +// [M, K/2] plus SFA scales in the atom layout for problem (M, x, K). +// Requires K % 16 == 0. Returns 0 on success. +int quantize_act_f32(const float * src, void * dst_packed, void * dst_sfa, + int M, int K, cudaStream_t stream); + +// Repack a ggml GGML_TYPE_NVFP4 weight tensor [N rows, K] into the GEMM's +// B-side wire format: packed [N, K/2] with adjacent-pair nibbles plus SFB +// scales (ggml half-scale bytes, unmodified) in the atom layout. +// Requires K % 64 == 0. Returns 0 on success. +int repack_weight(const void * ggml_blocks, void * dst_packed, void * dst_sf, + int N, int K, cudaStream_t stream); + +// Pairwise-interleave two ggml NVFP4 weight tensors (gate, up; each +// [n_ff rows, K]) into one B operand for the fused GeGLU GEMM: output row 2j +// is gate row j, row 2j+1 is up row j. dst_packed holds 2*n_ff rows; dst_sf +// is sized sf_bytes(2*n_ff, K). +int repack_weight_pair_interleaved(const void * gate_blocks, const void * up_blocks, + void * dst_packed, void * dst_sf, + int n_ff, int K, cudaStream_t stream); + +// Rows-padded repack for the SigLIP FFN Up weight: rows >= N_src are zeros. +int repack_weight_rows_padded(const void * ggml_blocks, void * dst_packed, void * dst_sf, + int N_src, int N_pad, int K, cudaStream_t stream); + +// Quantize an fp16 weight [N, K_src] to NVFP4 wire format, K zero-padded to K_pad. +int quantize_weight_f16_padded(const void * w_f16, void * dst_packed, void * dst_sf, + int N, int K_src, int K_pad, cudaStream_t stream); + +// The SigLIP FFN GEMM pair and gemm_bias_f32out are declared in +// flashrt-public's cutlass_fp4_gemm_siglip_ffn_f32out_sm100.cuh +// (namespace flash_rt::fp4). + +// Group-padded rows repack: n_groups groups widened group_in -> group_out +// rows, pad rows zero (used to widen per-head projections for FA head sizes). +int repack_weight_rows_grouppad(const void * ggml_blocks, void * dst_packed, void * dst_sf, + int group_in, int group_out, int n_groups, int K, cudaStream_t stream); + +// Row-concat repack of three NVFP4 weights (shared K) for the fused QKV GEMM. +int repack_weight_concat3(const void * b0, int N0, const void * b1, int N1, + const void * b2, int N2, + void * dst_packed, void * dst_sf, + int K, cudaStream_t stream); + +// Fused QKV post: RoPE+f16-store K, f16-store V (into the persistent KV +// suffix), RoPE+scale Q (f32 out) from the fused GEMM's [M, Nk+Nv+Nq] rows. +int qkv_post(const float * qkv_cat, float * q_out, void * k_out_f16, void * v_out_f16, + const int32_t * pos, const float * freq_factors, + int M, int Nk, int Nv, int Nq, int head_dim, int n_dims, + float freq_scale, float ext_factor, float attn_factor, + float corr_low, float corr_high, float theta_scale, float q_scale, + cudaStream_t stream); + +// Fused adaLN modulate: out[m,c] = norm(x[m])[c] * (1 + scale[c]) + shift[c], +// norm = rms-normalize when with_rms else identity. x/out are [M, C] +// contiguous fp32; scale/shift are [C] vectors. +int ada_rms_mod(const float * x, const float * scale, const float * shift, + float * out, int M, int C, float eps, bool with_rms, + cudaStream_t stream); + +// Fused gated residual: out[m,c] = residual[m,c] + branch[m,c] * gate[c]. +int gated_residual(const float * residual, const float * branch, const float * gate, + float * out, int M, int C, cudaStream_t stream); + +// Quant-emitting variants: additionally write the result quantized to +// NVFP4 packed + SFA (atom layout for an [M, C] activation operand). +int ada_rms_mod_quant(const float * x, const float * scale, const float * shift, + float * out, void * dst_packed, void * dst_sfa, + int M, int C, float eps, bool with_rms, cudaStream_t stream); +int layer_norm_affine_quant(const float * x, const float * w, const float * b, + float * out, void * dst_packed, void * dst_sfa, + int M, int C, float eps, cudaStream_t stream); + +// Fused LayerNorm + affine: out[m,c] = normalize(x[m])[c] * w[c] + b[c]. +int layer_norm_affine(const float * x, const float * w, const float * b, + float * out, int M, int C, float eps, cudaStream_t stream); + +// out[i] = a[i] + b[i] for n fp32 elements. +int vec_add_f32(const float * a, const float * b, float * out, int n, cudaStream_t stream); + +} // namespace ggml_cuda_flashrt diff --git a/flash_rt/structures/adapters/ggml/fr_qkv_post.cu b/flash_rt/structures/adapters/ggml/fr_qkv_post.cu new file mode 100644 index 00000000..2195d27e --- /dev/null +++ b/flash_rt/structures/adapters/ggml/fr_qkv_post.cu @@ -0,0 +1,125 @@ +// Fused QKV post-processing for the pi0.5 action expert (Thor). +// +// Consumes the fused QKV GEMM's f32 output [M, Nk + Nv + Nq] (sections +// [k | v | q]) and in one launch: +// - RoPEs K and writes it as f16 into the persistent KV buffer's suffix +// - writes V as f16 into the persistent KV buffer's suffix +// - RoPEs and scales Q, writing the f32 tensor flash attention consumes +// +// The RoPE math mirrors ggml-cuda's rope_neox (yarn corrections included) +// so the fused path is numerically identical to the unfused graph. + +#include "fr_kernels.h" + +#include + +namespace ggml_cuda_flashrt { + +namespace { + +__device__ float qkv_rope_ramp(const float low, const float high, const int i0) { + const float y = (i0 / 2 - low) / max(0.001f, high - low); + return 1.0f - min(1.0f, max(0.0f, y)); +} + +// mirrors ggml-cuda rope_yarn (forward) +__device__ void qkv_rope_yarn(const float theta_extrap, const float freq_scale, + const float corr_low, const float corr_high, + const int i0, const float ext_factor, float mscale, + float & cos_theta, float & sin_theta) { + float theta = freq_scale * theta_extrap; + if (ext_factor != 0.0f) { + const float ramp_mix = qkv_rope_ramp(corr_low, corr_high, i0) * ext_factor; + theta = theta * (1 - ramp_mix) + theta_extrap * ramp_mix; + mscale *= 1.0f + 0.1f * logf(1.0f / freq_scale); + } + cos_theta = cosf(theta) * mscale; + sin_theta = sinf(theta) * mscale; +} + +// one block per token; threads cover the K/Q rope pairs and the V copy +__global__ void kernel_qkv_post(const float * __restrict__ qkv, // [M, Nk+Nv+Nq] + float * __restrict__ q_out, // [M, Nq] f32 (head-major rows) + __half * __restrict__ k_out, // suffix rows, head_dim per token + __half * __restrict__ v_out, + const int32_t * __restrict__ pos, + const float * __restrict__ freq_factors, // nullable + int Nk, int Nv, int Nq, + int head_dim, int n_dims, + float freq_scale, float ext_factor, float attn_factor, + float corr_low, float corr_high, + float theta_scale, float q_scale) { + const int t = blockIdx.x; + const float * row = qkv + (int64_t) t * (Nk + Nv + Nq); + const float * krow = row; + const float * vrow = row + Nk; + const float * qrow = row + Nk + Nv; + + const int p = pos[t]; + + // V: plain f16 copy + for (int d = threadIdx.x; d < Nv; d += blockDim.x) { + v_out[(int64_t) t * Nv + d] = __float2half(vrow[d]); + } + + // K: rope one head (Nk == head_dim) + for (int i = threadIdx.x; i < Nk / 2; i += blockDim.x) { + const int i0 = 2 * i; // pair index within the head + if (i0 >= n_dims) { + k_out[(int64_t) t * Nk + n_dims + (i0 - n_dims)] = __float2half(krow[n_dims + (i0 - n_dims)]); + k_out[(int64_t) t * Nk + n_dims + (i0 - n_dims) + 1] = __float2half(krow[n_dims + (i0 - n_dims) + 1]); + continue; + } + const float theta_base = p * powf(theta_scale, i0 / 2.0f); + const float freq_factor = freq_factors ? freq_factors[i0 / 2] : 1.0f; + float cos_t, sin_t; + qkv_rope_yarn(theta_base / freq_factor, freq_scale, corr_low, corr_high, + i0, ext_factor, attn_factor, cos_t, sin_t); + const float x0 = krow[i0 / 2]; + const float x1 = krow[i0 / 2 + n_dims / 2]; + k_out[(int64_t) t * Nk + i0 / 2] = __float2half(x0 * cos_t - x1 * sin_t); + k_out[(int64_t) t * Nk + i0 / 2 + n_dims / 2] = __float2half(x0 * sin_t + x1 * cos_t); + } + + // Q: rope + scale per head + const int n_head = Nq / head_dim; + for (int hp = threadIdx.x; hp < n_head * head_dim / 2; hp += blockDim.x) { + const int h = hp / (head_dim / 2); + const int i0 = 2 * (hp % (head_dim / 2)); + const float * qh = qrow + (int64_t) h * head_dim; + float * oh = q_out + (int64_t) t * Nq + (int64_t) h * head_dim; + if (i0 >= n_dims) { + oh[n_dims + (i0 - n_dims)] = qh[n_dims + (i0 - n_dims)] * q_scale; + oh[n_dims + (i0 - n_dims) + 1] = qh[n_dims + (i0 - n_dims) + 1] * q_scale; + continue; + } + const float theta_base = p * powf(theta_scale, i0 / 2.0f); + const float freq_factor = freq_factors ? freq_factors[i0 / 2] : 1.0f; + float cos_t, sin_t; + qkv_rope_yarn(theta_base / freq_factor, freq_scale, corr_low, corr_high, + i0, ext_factor, attn_factor, cos_t, sin_t); + const float x0 = qh[i0 / 2]; + const float x1 = qh[i0 / 2 + n_dims / 2]; + oh[i0 / 2] = (x0 * cos_t - x1 * sin_t) * q_scale; + oh[i0 / 2 + n_dims / 2] = (x0 * sin_t + x1 * cos_t) * q_scale; + } +} + +} // namespace + +int qkv_post(const float * qkv_cat, float * q_out, void * k_out_f16, void * v_out_f16, + const int32_t * pos, const float * freq_factors, + int M, int Nk, int Nv, int Nq, int head_dim, int n_dims, + float freq_scale, float ext_factor, float attn_factor, + float corr_low, float corr_high, float theta_scale, float q_scale, + cudaStream_t stream) { + if (n_dims % 2 != 0 || Nk != head_dim || Nq % head_dim != 0) return -1; + kernel_qkv_post<<>>( + qkv_cat, q_out, (__half *) k_out_f16, (__half *) v_out_f16, + pos, freq_factors, Nk, Nv, Nq, head_dim, n_dims, + freq_scale, ext_factor, attn_factor, corr_low, corr_high, theta_scale, q_scale); + const cudaError_t e = cudaGetLastError(); + return (e == cudaSuccess) ? 0 : -static_cast(e); +} + +} // namespace ggml_cuda_flashrt diff --git a/flash_rt/structures/adapters/ggml/fr_quant_act.cu b/flash_rt/structures/adapters/ggml/fr_quant_act.cu new file mode 100644 index 00000000..5fd82300 --- /dev/null +++ b/flash_rt/structures/adapters/ggml/fr_quant_act.cu @@ -0,0 +1,111 @@ +// fp32 activation quantizer: [M, K] fp32 row-major -> NVFP4 packed + SFA +// scales at the CUTLASS Sm1xx block-scaled atom-layout offsets. +// +// Vendored from FlashRT's quantize_fp4_sfa_bf16.cu (Apache-2.0) with the +// source element type changed from bf16 to fp32. One thread quantizes one +// 16-element block: four 16-byte loads, one 8-byte packed store, one SFA +// byte at the tile-interleaved offset. + +#include "fr_kernels.h" + +#include + +#include "cutlass/cutlass.h" +#include "cutlass/detail/sm100_blockscaled_layout.hpp" +#include "cute/tensor.hpp" + +namespace ggml_cuda_flashrt { + +namespace { + +using CfgVec = cutlass::detail::Sm1xxBlockScaledConfig<16>; + +__device__ __forceinline__ uint8_t fp32_to_e2m1(float x) { + uint8_t sign = (x < 0.f) ? 0x8u : 0x0u; + float ax = fabsf(x); + uint8_t mant; + if (ax <= 0.25f) mant = 0u; + else if (ax <= 0.75f) mant = 1u; + else if (ax <= 1.25f) mant = 2u; + else if (ax <= 1.75f) mant = 3u; + else if (ax <= 2.5f) mant = 4u; + else if (ax <= 3.5f) mant = 5u; + else if (ax <= 5.0f) mant = 6u; + else mant = 7u; + return sign | mant; +} + +template +__global__ void kernel_quantize_f32( + const float4 * __restrict__ src, // fp32 [M, K] as float4 (4 elements) + uint2 * __restrict__ dst_packed, // [M, K/2] bytes as uint2 (1 block) + uint8_t * __restrict__ dst_sfa, + LayoutSF layout, + int M, int K4) { // K4 = K / 4 float4 chunks per row + const int block_idx = blockIdx.x * blockDim.x + threadIdx.x; + const int row = blockIdx.y; + const int n_blocks = K4 >> 2; // 16 elements per block + if (row >= M || block_idx >= n_blocks) return; + + float vals[16]; + #pragma unroll + for (int c = 0; c < 4; ++c) { + const float4 raw = src[row * K4 + 4 * block_idx + c]; + vals[4 * c + 0] = raw.x; + vals[4 * c + 1] = raw.y; + vals[4 * c + 2] = raw.z; + vals[4 * c + 3] = raw.w; + } + + float amax = 0.f; + #pragma unroll + for (int i = 0; i < 16; ++i) { + const float a = fabsf(vals[i]); + if (a > amax) amax = a; + } + + float desired = amax / 6.f; + if (desired < 1e-12f) desired = 1e-12f; + __nv_fp8_e4m3 bs_q = __nv_fp8_e4m3(desired); + const float bs_dq = static_cast(bs_q); + + dst_sfa[layout(row, block_idx * 16, 0)] = *reinterpret_cast(&bs_q); + + const float inv_bs = 1.f / bs_dq; + uint2 out; + uint8_t * ob = reinterpret_cast(&out); + #pragma unroll + for (int p = 0; p < 8; ++p) { + const uint8_t lo = fp32_to_e2m1(vals[2 * p] * inv_bs); + const uint8_t hi = fp32_to_e2m1(vals[2 * p + 1] * inv_bs); + ob[p] = static_cast(lo | (hi << 4)); + } + dst_packed[row * n_blocks + block_idx] = out; +} + +} // namespace + +int quantize_act_f32(const float * src, void * dst_packed, void * dst_sfa, + int M, int K, cudaStream_t stream) { + if (K % 16 != 0) return -1; + if ((reinterpret_cast(src) & 15) || + (reinterpret_cast(dst_packed) & 7)) return -1; + + const int n_blocks = K / 16; + const int threads = 128; + dim3 grid((n_blocks + threads - 1) / threads, M); + + auto shape = cute::make_shape(M, 1, K, 1); + auto layout = CfgVec::tile_atom_to_shape_SFA(shape); + + kernel_quantize_f32<<>>( + reinterpret_cast(src), + reinterpret_cast(dst_packed), + reinterpret_cast(dst_sfa), + layout, M, K >> 2); + + const cudaError_t e = cudaGetLastError(); + return (e == cudaSuccess) ? 0 : -static_cast(e); +} + +} // namespace ggml_cuda_flashrt diff --git a/flash_rt/structures/adapters/ggml/fr_repack.cu b/flash_rt/structures/adapters/ggml/fr_repack.cu new file mode 100644 index 00000000..9d94dfda --- /dev/null +++ b/flash_rt/structures/adapters/ggml/fr_repack.cu @@ -0,0 +1,416 @@ +// One-time repack of a ggml GGML_TYPE_NVFP4 weight tensor into the CUTLASS +// block-scaled GEMM's B-side wire format. +// +// ggml block_nvfp4 (36 bytes / 64 elements): +// uint8 d[4] - one e4m3 scale per 16-element sub-block; standard e4m3 +// semantics (ggml's doubled dequant table and halved ue4m3 +// decode cancel), passed through unmodified +// uint8 qs[32] - e2m1 codes, sub-block s at qs[s*8..s*8+7], byte j holding +// elem[j] in the low nibble and elem[8+j] in the high nibble +// +// Output: packed uint8 [N, K/2] with adjacent-pair nibbles (elem 2i low, +// elem 2i+1 high) and the scale bytes at the Sm1xx atom-layout offsets. +// One thread handles one 16-element sub-block. + +#include "fr_kernels.h" + +#include +#include + +#include "cutlass/cutlass.h" +#include "cutlass/detail/sm100_blockscaled_layout.hpp" +#include "cute/tensor.hpp" + +namespace ggml_cuda_flashrt { + +namespace { + +using CfgVec = cutlass::detail::Sm1xxBlockScaledConfig<16>; + +constexpr int GGML_NVFP4_BLOCK_BYTES = 36; // 4 scale bytes + 32 data bytes + +template +__global__ void kernel_repack( + const uint8_t * __restrict__ src, // ggml block_nvfp4 stream + uint2 * __restrict__ dst_packed, // [N, K/2] bytes as uint2 per sub-block + uint8_t * __restrict__ dst_sf, + LayoutSF layout, + int N, int K16) { // K16 = K / 16 sub-blocks per row + const int t = blockIdx.x * blockDim.x + threadIdx.x; + const int row = blockIdx.y; + if (row >= N || t >= K16) return; + + const int blk = t >> 2; // 64-element ggml block + const int sub = t & 3; // 16-element sub-block within it + + const uint8_t * b = src + (static_cast(row) * (K16 >> 2) + blk) * GGML_NVFP4_BLOCK_BYTES; + const uint8_t scale = b[sub]; + const uint8_t * qs = b + 4 + sub * 8; + + uint2 out; + uint8_t * ob = reinterpret_cast(&out); + #pragma unroll + for (int p = 0; p < 4; ++p) { + // elements 2p, 2p+1 live in the low nibbles of qs[2p], qs[2p+1] + ob[p] = static_cast((qs[2 * p] & 0x0F) | ((qs[2 * p + 1] & 0x0F) << 4)); + // elements 8+2p, 8+2p+1 live in the high nibbles of the same bytes + ob[p + 4] = static_cast((qs[2 * p] >> 4) | ((qs[2 * p + 1] & 0xF0))); + } + + dst_packed[static_cast(row) * K16 + t] = out; + dst_sf[layout(row, t * 16, 0)] = scale; +} + +} // namespace + +namespace { + +// Pairwise-interleaved variant for the fused GeGLU GEMM: output row 2j is +// gate row j, row 2j+1 is up row j (N_il = 2 * n_ff rows total). +template +__global__ void kernel_repack_pair( + const uint8_t * __restrict__ gate, + const uint8_t * __restrict__ up, + uint2 * __restrict__ dst_packed, + uint8_t * __restrict__ dst_sf, + LayoutSF layout, + int n_ff, int K16) { + const int t = blockIdx.x * blockDim.x + threadIdx.x; + const int row = blockIdx.y; // row within gate/up + const int which = blockIdx.z; // 0 = gate, 1 = up + if (row >= n_ff || t >= K16) return; + + const int blk = t >> 2; + const int sub = t & 3; + const uint8_t * src = which ? up : gate; + const uint8_t * b = src + (static_cast(row) * (K16 >> 2) + blk) * GGML_NVFP4_BLOCK_BYTES; + const uint8_t scale = b[sub]; + const uint8_t * qs = b + 4 + sub * 8; + + uint2 out; + uint8_t * ob = reinterpret_cast(&out); + #pragma unroll + for (int p = 0; p < 4; ++p) { + ob[p] = static_cast((qs[2 * p] & 0x0F) | ((qs[2 * p + 1] & 0x0F) << 4)); + ob[p + 4] = static_cast((qs[2 * p] >> 4) | ((qs[2 * p + 1] & 0xF0))); + } + + const int out_row = 2 * row + which; + dst_packed[static_cast(out_row) * K16 + t] = out; + dst_sf[layout(out_row, t * 16, 0)] = scale; +} + +} // namespace + +int repack_weight_pair_interleaved(const void * gate_blocks, const void * up_blocks, + void * dst_packed, void * dst_sf, + int n_ff, int K, cudaStream_t stream) { + if (K % 64 != 0) return -1; + if (reinterpret_cast(dst_packed) & 7) return -1; + + const int K16 = K / 16; + const int N_il = 2 * n_ff; + const int threads = 128; + dim3 grid((K16 + threads - 1) / threads, n_ff, 2); + + auto shape = cute::make_shape(1, N_il, K, 1); + auto layout = CfgVec::tile_atom_to_shape_SFB(shape); + if (static_cast(cute::cosize(layout)) > sf_bytes(N_il, K)) { + return -3; + } + + kernel_repack_pair<<>>( + reinterpret_cast(gate_blocks), + reinterpret_cast(up_blocks), + reinterpret_cast(dst_packed), + reinterpret_cast(dst_sf), + layout, n_ff, K16); + + const cudaError_t e = cudaGetLastError(); + return (e == cudaSuccess) ? 0 : -static_cast(e); +} + +namespace { + +// Rows-padded variant: rows >= N_src emit zero data and zero scales +// (mathematically inert; the extra rows exist only for output alignment). +template +__global__ void kernel_repack_rows_padded( + const uint8_t * __restrict__ src, + uint2 * __restrict__ dst_packed, + uint8_t * __restrict__ dst_sf, + LayoutSF layout, + int N_src, int N_pad, int K16) { + const int t = blockIdx.x * blockDim.x + threadIdx.x; + const int row = blockIdx.y; + if (row >= N_pad || t >= K16) return; + + uint2 out = make_uint2(0, 0); + uint8_t scale = 0; + if (row < N_src) { + const int blk = t >> 2; + const int sub = t & 3; + const uint8_t * b = src + (static_cast(row) * (K16 >> 2) + blk) * GGML_NVFP4_BLOCK_BYTES; + scale = b[sub]; + const uint8_t * qs = b + 4 + sub * 8; + uint8_t * ob = reinterpret_cast(&out); + #pragma unroll + for (int p = 0; p < 4; ++p) { + ob[p] = static_cast((qs[2 * p] & 0x0F) | ((qs[2 * p + 1] & 0x0F) << 4)); + ob[p + 4] = static_cast((qs[2 * p] >> 4) | ((qs[2 * p + 1] & 0xF0))); + } + } + + dst_packed[static_cast(row) * K16 + t] = out; + dst_sf[layout(row, t * 16, 0)] = scale; +} + +// Group-padded variant: output rows are n_groups groups of group_out rows, +// the first group_in of each group copied from the source (group g, lane l -> +// source row g*group_in + l) and the rest zero. Used to widen per-head +// projections (e.g. SigLIP head_dim 72 -> 80) entirely inside the weights so +// no runtime pad kernel is needed. +template +__global__ void kernel_repack_rows_grouppad( + const uint8_t * __restrict__ src, + uint2 * __restrict__ dst_packed, + uint8_t * __restrict__ dst_sf, + LayoutSF layout, + int group_in, int group_out, int n_groups, int K16) { + const int t = blockIdx.x * blockDim.x + threadIdx.x; + const int row = blockIdx.y; + if (row >= group_out * n_groups || t >= K16) return; + + const int lane = row % group_out; + uint2 out = make_uint2(0, 0); + uint8_t scale = 0; + if (lane < group_in) { + const int src_row = (row / group_out) * group_in + lane; + const int blk = t >> 2; + const int sub = t & 3; + const uint8_t * b = src + (static_cast(src_row) * (K16 >> 2) + blk) * GGML_NVFP4_BLOCK_BYTES; + scale = b[sub]; + const uint8_t * qs = b + 4 + sub * 8; + uint8_t * ob = reinterpret_cast(&out); + #pragma unroll + for (int p = 0; p < 4; ++p) { + ob[p] = static_cast((qs[2 * p] & 0x0F) | ((qs[2 * p + 1] & 0x0F) << 4)); + ob[p + 4] = static_cast((qs[2 * p] >> 4) | ((qs[2 * p + 1] & 0xF0))); + } + } + + dst_packed[static_cast(row) * K16 + t] = out; + dst_sf[layout(row, t * 16, 0)] = scale; +} + +__device__ __forceinline__ uint8_t fr_f32_to_e2m1(float x) { + uint8_t sign = (x < 0.f) ? 0x8u : 0x0u; + float ax = fabsf(x); + uint8_t m; + if (ax <= 0.25f) m = 0u; + else if (ax <= 0.75f) m = 1u; + else if (ax <= 1.25f) m = 2u; + else if (ax <= 1.75f) m = 3u; + else if (ax <= 2.5f) m = 4u; + else if (ax <= 3.5f) m = 5u; + else if (ax <= 5.0f) m = 6u; + else m = 7u; + return sign | m; +} + +// Quantize an fp16 weight [N rows, K_src] to the NVFP4 wire format with the +// K dim zero-padded to K_pad (standard e4m3 amax/6 scales, e2m1 nearest). +template +__global__ void kernel_quantize_weight_f16_padded( + const __half * __restrict__ src, + uint2 * __restrict__ dst_packed, + uint8_t * __restrict__ dst_sf, + LayoutSF layout, + int N, int K_src16, int K_pad16) { + const int t = blockIdx.x * blockDim.x + threadIdx.x; + const int row = blockIdx.y; + if (row >= N || t >= K_pad16) return; + + uint2 out = make_uint2(0, 0); + uint8_t sf = 0; + if (t < K_src16) { + const __half * xr = src + (int64_t) row * (K_src16 * 16) + t * 16; + float vals[16]; + float amax = 0.f; + #pragma unroll + for (int i = 0; i < 16; ++i) { + vals[i] = __half2float(xr[i]); + amax = fmaxf(amax, fabsf(vals[i])); + } + float desired = amax / 6.f; + if (desired < 1e-12f) desired = 1e-12f; + __nv_fp8_e4m3 q(desired); + sf = *reinterpret_cast(&q); + const float inv = 1.f / static_cast(q); + uint8_t * ob = reinterpret_cast(&out); + #pragma unroll + for (int p = 0; p < 8; ++p) { + const uint8_t lo = fr_f32_to_e2m1(vals[2 * p] * inv); + const uint8_t hi = fr_f32_to_e2m1(vals[2 * p + 1] * inv); + ob[p] = static_cast(lo | (hi << 4)); + } + } + + dst_packed[static_cast(row) * K_pad16 + t] = out; + dst_sf[layout(row, t * 16, 0)] = sf; +} + +} // namespace + +int repack_weight_rows_padded(const void * ggml_blocks, void * dst_packed, void * dst_sf, + int N_src, int N_pad, int K, cudaStream_t stream) { + if (K % 64 != 0 || N_pad < N_src) return -1; + const int K16 = K / 16; + const int threads = 128; + dim3 grid((K16 + threads - 1) / threads, N_pad); + auto shape = cute::make_shape(1, N_pad, K, 1); + auto layout = CfgVec::tile_atom_to_shape_SFB(shape); + if (static_cast(cute::cosize(layout)) > sf_bytes(N_pad, K)) return -3; + kernel_repack_rows_padded<<>>( + reinterpret_cast(ggml_blocks), + reinterpret_cast(dst_packed), + reinterpret_cast(dst_sf), + layout, N_src, N_pad, K16); + const cudaError_t e = cudaGetLastError(); + return (e == cudaSuccess) ? 0 : -static_cast(e); +} + +int repack_weight_rows_grouppad(const void * ggml_blocks, void * dst_packed, void * dst_sf, + int group_in, int group_out, int n_groups, int K, cudaStream_t stream) { + if (K % 64 != 0 || group_out < group_in || n_groups <= 0) return -1; + const int N_pad = group_out * n_groups; + const int K16 = K / 16; + const int threads = 128; + dim3 grid((K16 + threads - 1) / threads, N_pad); + auto shape = cute::make_shape(1, N_pad, K, 1); + auto layout = CfgVec::tile_atom_to_shape_SFB(shape); + if (static_cast(cute::cosize(layout)) > sf_bytes(N_pad, K)) return -3; + kernel_repack_rows_grouppad<<>>( + reinterpret_cast(ggml_blocks), + reinterpret_cast(dst_packed), + reinterpret_cast(dst_sf), + layout, group_in, group_out, n_groups, K16); + const cudaError_t e = cudaGetLastError(); + return (e == cudaSuccess) ? 0 : -static_cast(e); +} + +int quantize_weight_f16_padded(const void * w_f16, void * dst_packed, void * dst_sf, + int N, int K_src, int K_pad, cudaStream_t stream) { + if (K_src % 16 != 0 || K_pad % 64 != 0 || K_pad < K_src) return -1; + const int K16 = K_pad / 16; + const int threads = 128; + dim3 grid((K16 + threads - 1) / threads, N); + auto shape = cute::make_shape(1, N, K_pad, 1); + auto layout = CfgVec::tile_atom_to_shape_SFB(shape); + if (static_cast(cute::cosize(layout)) > sf_bytes(N, K_pad)) return -3; + kernel_quantize_weight_f16_padded<<>>( + reinterpret_cast(w_f16), + reinterpret_cast(dst_packed), + reinterpret_cast(dst_sf), + layout, N, K_src / 16, K16); + const cudaError_t e = cudaGetLastError(); + return (e == cudaSuccess) ? 0 : -static_cast(e); +} + +namespace { + +// Three-tensor row-concat repack for the fused QKV GEMM: output rows are +// [src0 | src1 | src2] stacked (N0 + N1 + N2 rows, same K). +template +__global__ void kernel_repack_concat3( + const uint8_t * __restrict__ s0, int N0, + const uint8_t * __restrict__ s1, int N1, + const uint8_t * __restrict__ s2, int N2, + uint2 * __restrict__ dst_packed, + uint8_t * __restrict__ dst_sf, + LayoutSF layout, + int K16) { + const int t = blockIdx.x * blockDim.x + threadIdx.x; + const int row = blockIdx.y; + const int N_tot = N0 + N1 + N2; + if (row >= N_tot || t >= K16) return; + + const uint8_t * src; + int src_row; + if (row < N0) { src = s0; src_row = row; } + else if (row < N0 + N1) { src = s1; src_row = row - N0; } + else { src = s2; src_row = row - N0 - N1; } + + const int blk = t >> 2; + const int sub = t & 3; + const uint8_t * b = src + (static_cast(src_row) * (K16 >> 2) + blk) * GGML_NVFP4_BLOCK_BYTES; + const uint8_t scale = b[sub]; + const uint8_t * qs = b + 4 + sub * 8; + + uint2 out; + uint8_t * ob = reinterpret_cast(&out); + #pragma unroll + for (int p = 0; p < 4; ++p) { + ob[p] = static_cast((qs[2 * p] & 0x0F) | ((qs[2 * p + 1] & 0x0F) << 4)); + ob[p + 4] = static_cast((qs[2 * p] >> 4) | ((qs[2 * p + 1] & 0xF0))); + } + + dst_packed[static_cast(row) * K16 + t] = out; + dst_sf[layout(row, t * 16, 0)] = scale; +} + +} // namespace + +int repack_weight_concat3(const void * b0, int N0, const void * b1, int N1, + const void * b2, int N2, + void * dst_packed, void * dst_sf, + int K, cudaStream_t stream) { + if (K % 64 != 0) return -1; + const int K16 = K / 16; + const int N_tot = N0 + N1 + N2; + const int threads = 128; + dim3 grid((K16 + threads - 1) / threads, N_tot); + auto shape = cute::make_shape(1, N_tot, K, 1); + auto layout = CfgVec::tile_atom_to_shape_SFB(shape); + if (static_cast(cute::cosize(layout)) > sf_bytes(N_tot, K)) return -3; + kernel_repack_concat3<<>>( + reinterpret_cast(b0), N0, + reinterpret_cast(b1), N1, + reinterpret_cast(b2), N2, + reinterpret_cast(dst_packed), + reinterpret_cast(dst_sf), + layout, K16); + const cudaError_t e = cudaGetLastError(); + return (e == cudaSuccess) ? 0 : -static_cast(e); +} + +int repack_weight(const void * ggml_blocks, void * dst_packed, void * dst_sf, + int N, int K, cudaStream_t stream) { + if (K % 64 != 0) return -1; + if (reinterpret_cast(dst_packed) & 7) return -1; + + const int K16 = K / 16; + const int threads = 128; + dim3 grid((K16 + threads - 1) / threads, N); + + // SFB layout for a [*, N, K] problem; independent of M. + auto shape = cute::make_shape(1, N, K, 1); + auto layout = CfgVec::tile_atom_to_shape_SFB(shape); + + // The atom layout may address up to the padded (row, k) extents; the + // caller allocates sf_bytes(N, K) which must cover the layout codomain. + if (static_cast(cute::cosize(layout)) > sf_bytes(N, K)) { + return -3; + } + + kernel_repack<<>>( + reinterpret_cast(ggml_blocks), + reinterpret_cast(dst_packed), + reinterpret_cast(dst_sf), + layout, N, K16); + + const cudaError_t e = cudaGetLastError(); + return (e == cudaSuccess) ? 0 : -static_cast(e); +} + +} // namespace ggml_cuda_flashrt From 7b7e8fc3985c0ddea3b6319ab27c4c176df539da Mon Sep 17 00:00:00 2001 From: LiangSu8899 Date: Mon, 24 Aug 2026 11:58:29 -0400 Subject: [PATCH 06/32] feat(structures): pipeline binding and qualification gates for the ggml pi0.5 host Adds the jetson_pi_edge_pi05 pipeline binding (vla_tick_pipeline, complete hot-path coverage, 11 segments mapping the host's fused windows onto catalog structures) and a qualification runner: manifest validation and structure-version pins offline, plus an opt-in on-device gate that holds the steady-state action chunk to a stored golden (exact match; the adapter is bitwise deterministic after warmup). Verified all gates green on the live host and red under a deliberate pin bump and a kernel-set change. --- flash_rt/structures/adapters/ggml/README.md | 9 + .../ggml/qualification/goldens/base.png | Bin 0 -> 150969 bytes .../goldens/pi05_thor_action.json | 1 + .../ggml/qualification/goldens/wrist.png | Bin 0 -> 150966 bytes .../adapters/ggml/qualification/pins.yaml | 16 ++ .../ggml/qualification/run_qualification.py | 177 ++++++++++++++++++ .../bindings/jetson_pi_edge_pi05.yaml | 97 ++++++++++ 7 files changed, 300 insertions(+) create mode 100644 flash_rt/structures/adapters/ggml/qualification/goldens/base.png create mode 100644 flash_rt/structures/adapters/ggml/qualification/goldens/pi05_thor_action.json create mode 100644 flash_rt/structures/adapters/ggml/qualification/goldens/wrist.png create mode 100644 flash_rt/structures/adapters/ggml/qualification/pins.yaml create mode 100644 flash_rt/structures/adapters/ggml/qualification/run_qualification.py create mode 100644 flash_rt/structures/bindings/jetson_pi_edge_pi05.yaml diff --git a/flash_rt/structures/adapters/ggml/README.md b/flash_rt/structures/adapters/ggml/README.md index 13b46cb1..84aff171 100644 --- a/flash_rt/structures/adapters/ggml/README.md +++ b/flash_rt/structures/adapters/ggml/README.md @@ -21,3 +21,12 @@ Layout: Host-side integration (fuse-hook call sites, graph construction changes, build wiring) lives in the host tree and points its build at this directory. + +Qualification (`qualification/`): the pipeline binding +`bindings/jetson_pi_edge_pi05.yaml` maps the host's hot path onto catalog +structures under the complete-hot-path contract, and +`qualification/run_qualification.py` gates it — manifest validation and +structure-version pins offline, plus an opt-in on-device gate comparing the +steady-state action chunk against a stored golden (exact by default; the +adapter is bitwise deterministic across processes after warmup). A catalog +version bump or any numeric change in the fused windows turns a gate red. diff --git a/flash_rt/structures/adapters/ggml/qualification/goldens/base.png b/flash_rt/structures/adapters/ggml/qualification/goldens/base.png new file mode 100644 index 0000000000000000000000000000000000000000..97a77cdf94e938abf160a58a8a98579d1570d4b0 GIT binary patch literal 150969 zcmV(uKQ;b+e#tIyX4IE`VCa{wTyj2FWx(i_Qm9H0XH;2Ono1lTt%nk z2S^p%bEjRT0yrMr%viaDkOai5tTBJ{c7^PxZzdQE)0|nm@u(QQJ7EPlM`!rliiQg- zq-E&HgVNnSm701#K4cEcu8k;wI|Z&5>7ofcK-H=vtHdIut7@C%9eIncCu9z6P_Qga z6;AnO4Mt`_Azp7|uL*OuBv*%9nrbi)M$BB*O{9MbQa>AFTBm;o#oB=EPHjwC-)nlK z3@kFCUAq$@>ajEh3hT(0q{UDaNha+Qu*_<*1O2j$j9RlwtSv7b7}+>Hs_&}Gk@&~< zJQbqO0^m!h+jdTz_d@{?stG))T39jve?h4ElWNLl2+LchzYMMn53*LP)c7#lMO4|G$q z+^UED4!-d0U7veQ}96lv? zS_O?FF>1hW%EI;!Z{f4py51yPR7X0)g zugeC=h5+&vH=PQghBApJk7~D@!9dHG90tP-I#$-*vcO)5kPITF=K885lv0Rsn7wK#DD98KT%y>1nKR-W1yyxdb(CTQ|WQR1x}%V6D0&FjyI~XCZ(e z09kX(6W#M}u`d!qJMtmFbP3lLG&Rfi0eS}xDUW_aLe4iojUZ zz8vK%J<-ux`ByY{qI&Vghf#TVlPC~xp9iFeK0Zq{rDZ@HRe+uR*)NKc_&!Wh3S(xh zyTL2lK{s@-Y}kFW<->8}ytBcR8&Og0Uhl4nuC6du5r#fDixGJeaXp@Ao#L>w&dpEm zgaOjDruIdGvC9m7qzdiImjU`Wjgm%|4bKSx+2!#_#WoAPXSOmrRGW~1Fv5>pg^s<# zidta)x>zk76yMD`LF`a(TBaW%$k$*a{E6~fG4tsDY?HbCiD#KtgYPR#r)lr;r&qA$ z{@wJi=E4?7A&?nVRBw%ADh0AT4LliDM`L~Qc`gb<$)lZeT!v%A4l?;a>gTVEmY{oM zFdDhujl;B}rbMds>yOTfAa9m|(u*J+Ed}5`Wg^&;?V^Yj8fe3P;t3eU2)OWz^(rIZ zq8S6|`&RV;kBn}Xh)>I`LyI+ioVfr3e`CLovmfzSw2%Zz#mJ@u8D1i|o+0RqQoZKN z$POm2%bL}Wny;1TMnW-0$Mt^_FFjh+y?raP&8p!B^c4>cm$D7+)gGyaH9b6D@F@qr zD0$!T!uC79_@ic^G4g`j>Y_d#GnscW5~wW*Z%1d@-t8=1*4E`}!>nrxmj}8CjrI|1XlQ% zy*ESU`O*?dg*J&Sub2&tVmIE`XVM)iy1eHL@bdhv2=H-JW-A^$G(++PUD~ruOuo@k zQ`qIXhJxV;xz6}7&L4?x9SsJG&36oV$5Pq!4Cv1;>|x?k7@6IFjif-IjM7l8+8-KC z!sm;Tb30BQx+CTDC?0FIt%feG8ZJ1(2TL2)Y8Y~uXS2nqzCA}fYNzv`Sp_Zz7#W-u zb(Lc19j7@us4f6$;MYlXB=9r14%I`9KUx$+i2(6LD1Z3*rqpFJZ%vjgU7kv6aMu9~ zL>IQJq+We}P?3{F2>#h!W>%u4<+akLz~hs}=L7=5IyIy2o0}t#p#4|h*5umWxVD^F zq06njIv{C&Ayj9>n;5!TSq0r4=hSh|%n4OmVjx?(00LTh|4Xz;m>g4r7=De9!G zxr9&gkWavD6|`;MNL~t9V08_(NJHNly-@ksP$PHD!QuhIY?i<*YkRg~i*+^@iF{Tx z*VJ>Acfwx5(|yiZ{yAaqhXV`d;&mI{7;m*@n#N{UPa$MOe=XfP7gY0tEvBtrRDX#N zJr6rI8HyV_vrrhF;IP){u69X>)b&+V;T@8U_+I^SdtWhlUBEmJD7(6GNrnQ@MrpaG z>vxgp1VR+A9tehK>r*}%9lPo95l1Cdj{<3v(-wA>X_zo+u-ybeSwt8##z%ai^2%bN zE8hCtz$j zgme@^>s zzuMGud=EXLH#H%>m{mnlGd?+{KE8oq^#gi~#Mc!9ERaO0%u}C!)XC-w6TpMQI{tNe z-zMTj2vZ7K7e3KH9XXEbVFZwG@Nprf0PKcKbL!a@t0Ui3Llq!7y9M)?Zn+Lv&pZ1@gE!=AlO-Jot@2C|51+g6Br6@jO z)<{YSLV>6#-x}(VoAf_(0<8LiPhMZnpBD7zkJ`LKsAb8J*+I$;#)#zVQb)BXlf1{y ziRfn|PT8;ImxoYH=R)Wj4K@$F)3o-kODVd5Gx)8561;v+4XFPcW(3}YPxJGhC@dWrE?F;Xn2K4Dt zub0z<#u&IYmbZc}nvn_oIT*0*n1@l*{4)R0_>h5X~YLzWDv>Oc!ju3_WiM`(?XDs-bI@ z$Eo=-pOU*xsa4~aT+RL#jYpYNztO@r*+L`tSnAt?)Do8!0_I#K(gnl9eW2K zwVF1J0cQmK)u0s&< z5BPNO1z02h&vzAP8VS%u^wJl*Aas?;#Q6av?q4L(hM9CKEaIa*;Dx~oR7q@k%aBE3 z&XH=T(3q&FePZdg>NaV*8txpFgcbl+)*5|ujROoLA06Xd9Wb#kY~8T6yfv>IOkR>( zs&}CN5@@)%F~TB5<*T|@*T5rsA!p*sZ|Ms7tmB$@%*sa&@4h#6dZ?dcj@=O~R1 z0eMHcUYg3u%tMa4ls>hdYw3szQVxtoDy2{NB@3`S8^u2vvrfKTk*|huIu*=TFHj6I zt()aj2WO$0ai~H;;Z?=lM4}8L5Ln3r{G{2;;}}NR^_N)mAwQBjReCwN4)i2%G_B_R z`MW>k7aWY#(=_yiX4J#UK;*SKY@r~d0nWCu9hWee0#wF{>DI7pSl0F}9;w6ReP5al z8@X!X+19>f<2m5|;=MmYK3pe7qlz-!P$QXz67BR-Sz#y7b0I{3@wt zg4&Rhwe{O%vs02_NuT9!~v2qwGAo|;>_79h_k!c zs5A7pG8JP3aC$;gtTiq347E_pG$iQxXKyy-5iyhk@cV}pqkQPIq#i#sPCJ31BG=um0X@v+4X(U(Em~@!ym_exxEy_GA8ooUT3+wz$5VPa39=sq`i{HO|LfWtK^ zQ2GrpP8m!QXIA4%#Y1voq#b^fAHL2i1gwhIIC8Dd8nRp2fZ`?sgZe!VmT!P+am1Tw zL|QmS{t?$pC z84}s7=Ha2g=9r*6UdmYS7a@>eQ}|&L=zw_uJtMKO8k^Aq7#PDCQ)uoNgr}5K}2J`g$DG+k2?aLf*ql-z`qDOs{Vqc4Sw_RU7nM_}BuLOLZ-o+|I_5mQkNLzx09YN! z5jNPz0-+T`q_xcG(9a3B69Sl(u+D=k(p6IydELqrxe|*#2GLXf8JKunVQA>xDK4sX zxR_gs_>BX12wSCTbr!JTS-qbo0!Q;Y*f~U5m7ufB$HXtX8hW$Z-7QVC(XlBIuD3Q7 zo@$FPu`Z$`k{Q>m+q2~)2x@t4OWkfi z=;n(an`+`bmLduW4@%I%jB<}W>WvZacj~x+g~HA|Gl~JSOMl~vU2h{t&eLXM2Emw% z;o4*skgDJMID1_-g5^DwGNr%rm+EzXKjM;sp@mimpcw1fQQYpVd8stj1JV9;bgC^I zby2(3nM8GdJkCzj67}p_w0^%zW(}Zp&6d7@42=E7&oVz3kiRJT8w*s1d?lf*$xk!w z&tUFZbI#oJw;5`PZE@HlV1;(608{nE=iN`Hd7$5TJTWt1Ft10l3nST}UBsxUAW8G6 z%a%G8Fgxve=UcU7K89-{3YzZa5O)Y908~)uEO;|6Z9`ARq+YKu=KS3FjkvdMoH?WC z5L5uej)V#$F7hjJo;{h=In3=FRqSjfstuf(7)C)IxRVI%-X=f4GSRW=A67arI(2-~ z(12-W!t{xkCjef7WhLwIfigyhyUKdKbpjWKQ?SfU#|tZuG@@$RIlIqUu=-O#{7dTH z-B(ed+*2ZziFXXn;@L;;^Mh$4&G3TS1QIo;IRUZgoi#nh_zY(WSC!a; z-*Bp>;y=z$7036yBKFc~Gqxz(Pv%s@t>q?j3mgyL3h#*%3dL4HQ0fieNo$vIcdA`}-3?{HpEeyt~4go}Q8C?^w~B?a0EX-av~t!v5%HqtsNU zjlRN;VH^t`R9uJ04(1soMvvn&nn_e$?*&5GFA~J3mnggmj9LJrYwGu5zXVc86u<-1 z^DNg9Z04A0=iXAzn-?DaHSgXC|M3(I3}m}H$1}zsV(*hSr4UmhWdlGNA zk5y-rPv-t^L)>f37UnV4ey7P}TK6C75{=pPyK|F60))|i9GO`m1oXG&WB=hEsaND? z@S92U4%+7y6;h=L^wK#Y={RsxN zlamhM5|TcJW&{^JE#s$9X_vLSOZ(i85?GLS*e0r-pSCK^>kZ@TyO)Tms2fVcI}x1A zeRV;cirC%RT&65X;%rx&0!^qpUVWM4UMdD%VU(mm)O}Gcxd3YL-vl{kBlm z+mr+^*UEH@Rl6_I`$%;lIG@(*JIjHGm+xB7oR8w%Zlk3itaRch@?ZjMyHeD$Z z(0%vq@?_;%|L{?9VGY9N-a#qRkPg+i49mP@F&ri~irx2i!EjJpy)DG4SA zS}}n6>F2RMLfmem$oSVt?mpq@{8zO-oypE5dmc)OIox*N8LSb-I3BemQgv{QEcSeA zxhlF4wMthvE*QEaGXM`ot5VF{?b2T8)U-{!5myj+eHb)E-XYoozDIJ(9m&LXYj}id zJJ1&wpF=a(#p$)sK-wpzK+@Q4FIPM+1efP#bM?;Z_dc664%`eRr;%Z3;HSMybqhf; z>Hp_9<-oUn^?hL%K2Nr;FsKbmJSW)!4Ly#S0FIkE7+pTSKpF>klx9;=#{MuKmcAR} z=AKPN<;&kNLQC)C7$?k!%HCn8FGKj40Il=3At}X=bwF)f$uE+}KH-!`uNT@RPWjW> z3lxElScoyDQH_3t9iPO-sXV(a0B$I2910ue_7!+F8Uq*-h~0J_>owlABO`rJ^2o_K z=%^ICyO4OKqjPDE$WLl~R6>@5t|Pe3nY)8YY4@toAJf0!sSk+Wd(c(pK@i0W%P~2+ zfEK4r!Hwk2ow{1xwkiDuK#GPy;=X}cp6-%tbwl0L=zrt{&fZ`yL_Lz$=KvXkyrD|R zOdrsv4+4pT2N@SM6Uo519YKIwkQAG3O_WU*)j9J5SJg)0(XU)YH=<;Uth_zp#;-SI z+VmCJl^Vsk_Bp+nsx;A);fyiNm(OJnCDn6}(XRqCoh^3tr|?%?-W%V2ohPU&=nTt==jS3w=*-b)IIgRs zcCU@}onxz-5vB27=U%6*eawhp_~+gx!~`t>{ZdtLnDu*BIFi0^pG|!z0J`Q!dZ;J| z$kc$D4mH0+X3Vumr_OsILBedzQ7P@ezv?9-k_kNMd^49Op^ebqg`bruUBLAVoz&ed zZAjS;xdBm>FUp$;sBe{|Tu+}de}`2`Egrc^s5o{{CfF1FbI@8^1XY%bM3R z54jB|#w^-r%54}Nt;dv#=Kw2PimrtM`KSW2C>dqI{VZIn7+Ja_0JE#ul7_>yV%*b&r`njgCvH-C1I@pibz(?Mp{ybJtlG z02r92dkKe?x548*A+tNdzL!Z?sz7t0rEF){;^M-tAnJn;!@0hdiGC9Ir>wK6>tn}R zOU`k8X3RNM;d84n77yn%^FDFNiyN9wk;GA%$Sz9AzRLpJ1)2}0zHFnqa4k+`+d|3$KN zfX!tYN;#BdFYRv)_VdQA>xJb&gIQ9bjFd#{t3h}<=1ycaWxTa8w?RCYldsZX&8pHI zx$5)C|F2%^>Lc zRwhtmM`BX=*zX3RMgxMI_^IucYJmW)D_aMU8y}GAi2zg25UbvL^W=tSNXcgI*XH&S zhnu+GQHEsW6>t!R0zT&oi?jfhvzy248h}s*Z2J7ZNhVC~0>`mL6~L z1vaJtH%9)p9H0}1Zh*-U3sq;V*Ti+I2VUUzn{NnuM+gvBpmF%z=6m4ltlnr5{P9LB z^F09~OrFcQzHFTZB4~=d0^Fp4tR@%TTejsTL$FY%F3ZlRzLF4qHVBeJ3hI^4{G`(2 z@>GASa{vzIWoSU9#2`76x+~|%W+sWfYLDTul7-gKjs64aX5PQua5C5F>TDYVU8T?C{zV!NSVbjYhLScwY=fCNFKa-7@6jkGZoGH}Pjox@etB zkQkV32$PpH%&SaV(jH$wCDZzH`>XF)raG1m(v41lCj z!j@~*6|hYuHWh6s;;C)?F89cKjV|8`A7WSYLhro5cLv*l?FxA}XKl2i|nzX;tORk*NRs%B~T4~iW>~{^JC1I9AfFl9U zIvFAyTHy8Q3ViKSY)2|MSY%yGlEYyk$c5Vo-pmwvb@vd#!c||#fOzpEMQWJ8s;vyc zP&5>KaIG;ws=qx6GxG_Y9lZw-K{;CKXNH`)zrl&ae|ZuF{=jY3E(E(U`&<+G2~F-Am1(ql0 zMvaoWK&93MqH1fl2Mdh#7Uu+profNG9(F8)b}ZvFCmk9ze({+GC!9%r+LsWZGS_(C zWDPL&3BQIIt!hA}HiP=^5HX5wZS?8{e@XS1{mx>+&~KeefJbtDI2f5d+j>)=rflUL zSEri$LEX^01;AGo6O^*n-r^)}5^#2D198iL;RQWw3S<0bmGiJ#y$QmI`*tVVIhyK?>XJu5RZ3nx;o;r++`iPZxEVqyrQ(Fn&)L=dGkyc?g17E(X3 zfq2#9G9FI1zng}Pl1U4An3$ccV9u5|ZwM~LOC)7>^KJV3p?f+A-UA1V)k7<^^I*+< ztR+Wl@I7>0Eh%Jyinrl~BtrSnZov7t^ueU}fZhOFb=OeJHgYlv&xS^i0{ZUB5mO?-Ype%x%%^4rml zmOi1BRS8?2F0erKpq2x~A8;SPL~7ECGgZuu9KX(e2sYm(iEwH|05HId`JF5|Qn$JwQ| zGxW;hVIH<1mXpE=E9zbbDspWK%ewcbKS755w2`u;LwaA1iZb9173|XJ6HVEFMT!|w zLOT+9XqkJV3x=Jn_<}K#WP)w5hB^8`#2#&O<6X?PX$V>iEA_<|DyKE!lfLs}UsjW% z%Qa!-vz9{{lgpauR8tDCqO*5RJpGv=C^t+(6~QGOsD^f!XVh=d$C=oQE4>_5Awz)*Qg$k{kDlJmWO>v04!nAi1txAc!2xIbvYn`e!mFN9o@!hzBUOt^+u zXcp634-I4@J$p65dx;n_8Dy4`u`M;9Uz)cp7@doeMriDHo2Aus4%Ln45$w)_K0!{5 zC9aT&G8)S^PC#B8iZ+?xK#3o6RJFXtXQoVRPmwH`mQ{GLq++b&YxY<_7SKL%QEN3l9*;Z%iWV$FS&y+olMYBI<;-^-@wnerS|LN`-5BJq9#F5wcfgi$@XvU0--nG%~e| zEa9CDrJh;|M^MT#m~SD+-02+2Nl}Z;p!)Y_vYiWkMUTiHH)%~l2q%-HnIf~aV&pGk zP)UqDnH|my-ey^TCJq3*mXv#RDCn5$*iz-!g=Lhhc=F1PCddb7JC$IoXE!M(;}r4{ z8Q;pV{}O9rJ{!oNr1?mnI2`;3Gs?Kwr}d$7doz|Qy#1h#%t>OWJzp4ssvHw?UN`0d z%UM(-6{L(_^G+!vy$eGr*5gm`zbeaBB9n0r1-bZ4D%~?nkDHJo!~_W6s>T>DmefV; zP(F-la`n&}J_qlgjQA zVf69ssy>lmIwg_n)j(Nt!d!}GskY1gbrr-4dL^THJXEMFz@aRyjI%?wPm&vx?C5!H z)d%a;JuJS%_QeML;}h?4Um|mhNAT{A0(9JOm$ie5rK`b_xhj(2g4!4VmMb)N8DIMi z?&O^;R>%wrn9jN?aWzu4 z*!(ReJKSluU=UQ59Wr%4t<2DO0XU)NAb)Ymg>GS3**YJw0;$}=JY!^7O3|3ZI#IT* zfv~hXP>QJ!BW(PzLtGmb9#9In`2n6-U-B3*l^XXCNJ7RXB)|P#S>W))HIozRA>B8aj@W8_iVMPZ->|J6nG0SH5`GV?j!_J3%S<${uWDihsJhJBD zN&Vp$*Fw=+?PX<-#-oE5Ns;tyKYuP#Q9M}{8V^`A)VWM5jL$zEHUb2}#HVi&S{RJr zf`F6iz#u*r1-qTA+Ilv8+1iQD@}79P@pw>6LoWH;svXx}NfNW!=X2u|i-&c(T?|#6 zz7w{h=H|q-uvP8gz}uY0**3=Wd6mj=PSL2Ds|&+0-943j>4xhoL@Ut6lhCxyb9I_H z$*tqlP6u*`o_a_t$GuT0Ftmj(*V5h-LQ_l8F6z@{ z`lgs(^F7${#tM}!a#!W|9qZFOlg4g^q32nZV9suD=X&uGZz!>}zy3v1dsHz>P%IOA#CgWz6=*B3flxZm z@KH|u>Fmb_1tpFxt-nJwSPm93Pm#LMbOM_5r%9CKM zM+4S<=LfvgT7oyUZ=4JZ*CWmA-N?beis+&24k+;1$#6V6 zOghH+$%bn|E=n)XFUe1Q*>Rt(tP6DrXZO4~?e1h)tOTuCy6IFyV1-tnyvfRoO8ow4 z!_trhYSe-sgGDd++B`m{3bzppO4f0O{#x~8IG`t3f8Vo)a28a&TRXMY=GxNYtB$ep zYp{y@4)AQ8w!+I_CAEHq#=0U|V@{qFOMpWyDg3b*<)cLq?bs{;N^C!q7h8q8@JCxF zn3)CBaOrBg8wG6q!j=~vR0#dp=R_oGa9m^u zB;pwU85kwo1lR3JaR!Qn+w3S{UO}h8ecfjnZZ}&|&=}^uf>w22lE%H1MP?}Bg>{51 zkBaZAqPzg5UEY10%h(O)Kt;}8Otu&N1`yZ!m9^W5*o!Wa`{RJwF6DgPggLP*1bH|8 zOj2C~N)}CtNA1phc0N0%)CdQ83-N~4=YH;b*7m$)1M-~ns9je%*Ry1YMzq zXzUQp0DMFYC~hIUwsenN6WD1UZ8L2e_g7=UV!2;y5@PDrz|epl1GgAxX;muw1u5o; zQc10ajx)P4u&j-%BhDOZ)oPf3oYk|UC0mK&Ws_gGMNvlGISsA(rijoE+{y~-u%|w= zHOrV4sP%{03-_7pPPlX%7!bpH{3q}Bavk}yI9^+QJ`;3fT87uch@y_b7=zcCsoo3I z0D&;m09j)RED|N-rP3DJdLkay@WZ6Pd_tC~#-B8GfI9aDi$(5~ASM;c~+2DDj1$oec;E^gFbzSZ;FHw}n95fxt#V zZdGuw1J_RD0lsi*MsjiOlHbU=w_+u*S0O@ z(SJ*HW5LE)y9WNuJ@A>4p6Co;JWnFYpWpk@<1li$^sLp zSOei$`<;hXm=DKSVX`v=seSCL803-=;v8)#(3Xk<9)`tQTG|CRdDCq%G#wh^<)bYm zTJFjmPl|Z~y9SF7`EyTJ73M2T4C_=CHwc--o^8VIWG#~wL;#4wm$42?q7B5GUhPA- zUy2VM6>$Pj;S-(|*>hYK!f?}CX|Cvtowvta8oQXW5{o-FLK?s}sveX2(UswiRTrpo zi~!41U}Y=T+$2FM3g^CaP~?AEvJ}eQhptqsOYROR=_EDIhJ4>jcdzerm+6%*rgNh0 zar^wjAJ=Rfb%%K7DZtl8ke?h@>Lc|O)8_4PpY^Jkb>7z|ncK03Mj&SPwlQMWj(>$* zONLO#DtousB79;ff}}#c3UEWtbeBXrxq%6V*?JP6##5ZoPgNhw$+x2Jmvv8_C`%!f(&o#aH+bZzt6$s(0iZ2ZvpRl$J3`;yu z9gzhHA>TMlyPSNjhT+&e!d&H~*n;!2+_jN1N#Pq_T0L1SRB4@iR8r~(3j6Xuuka`C zUt@lZ61lFfCf2b~T!I^1FnZ)SL~~yb115V7rd5r++b>NQTzN;W z>>9+~_PIJTvlm9Krnw<{v!x!#;YZ%y?g%5mi#i<#As*Le^NBOdrU%R_1%ls>g5il| zfIe(oQd{oT-#C`L;uy;v**Fo~*?kTWP%dgbU)&TYC39ga10!YNE+$I$W9m*CHI()q z)*}x*2|@kmzKl&azQVUD^lose+vZWuoxqyt`7^+uocU3-N^AhOL!{Qp*yQW@E)?4w z#%A^m-AE|02Oe57sR{sq(cTt#g*45Fc`TRhXKfBFN z!uBblfR{M5;U{aC)@n{*ix%n|Z(+rKeAX$Eg7a`JT%~s1A=-y8(r#(O+UM>BxN`#K zJ#%11*e<|1o`K?bJ)pW2_U#;#$5qz~4APq2*6H$@&-fc^TonPQYCmySM8jrse$ddV z9NxY=a6=g!6yN$q-3oJ;`w9IXP>W=*%uGBK(%-D`A2eCqj79Ggb85R7q`T_FQXUJ! z!j-Wd_5x?QCO86SERPn!e}zHhVFsI~RGGLBk#}U+rkvI;)qg`K0dP2lQ?bCUw=J>HW`k*hhajN#8saG(-!7E&Z`V%UV20_j? zIaQe0wOMr7yz4q9z!zr3xiKjET66nA0$)cWK(@zuavz?K-qj7MxriqhVWqIew4-bb zaFSvNpo4zp(jwFr_zbs2@Tyx;QrBNJzsUprLT7=^q!;+H4v*K*{ykLDXeGBA^OF)(mgQVC7IG*+QqJVpF>0$HR{I~?yJ4NAp$Sn8cHj!2(r2Aou=+fY?obf(}qhx0~D z7thH~+@qnz_iQ+vZTgEsv4I>~5dLi6uGDeCfuiao%IbU+`5M0kXC5cBGZZaC9MYrz z4G&5$xu#kis3QciC5`|#Ty%puxe|xg;ymHU`sC5VqAb?~4pni;riGIz)4ojh6_xa> zIotrNKwtsx5G~EdZkbCl_&4nZC9~GbvOwg-E5T2rCG>Ukew0M>H*6fEcW|Os{|7}s zSZmn=T?Ji1hU3Qd6nx#D-uYtgN-hf<`0-E&!i{KnUg?iV$Y^yK%`II!vA}O8!$Dzh|jrCs`yHS#vy?4gl@Phb>e!($2Q`RwMEauym>GJ{9r@!oKD`X zLhh9vwcH&xkn==dE&e-VpgUc^Lu35Qv#vfY&`-BB+Gok$Hff~JD3LBbIvaHNR3{(T zjtz{)_GS3adCzt#6CknU(~?E#n(R~`)1`4M)#p3;J%h`VUrM>|O#N6c$R?a5vQ~jL zKVXD+dCl_mw5jPEl9<2Srg5;4n*JO0IkeXVOet|l)`^*SV~dr^bYekDcqwvuO*Zan z`b8x>X*tGh;Y-|!d# zzgCy-f-7Sr>`$|x+wWSIv;#jvz+ zf)giBzrL=i7R&rXP{UjoCR!&l4IE=R+)HjIfO3{h^e9x8xE#QM6q%G70tL}K%A_Z# z1~1_#w1!O%b1QwjL@t@>8oU{^7S$sre|_=*y~QU8%Or>CHdX|;dVE+UUO?enwlMdg zSn$84RhDSHP^y2~Cmg*VSFoe$V_t6X{_JM);l3X7YX`+zLK3@ z!gQ1_g0L8xc^iAkAv*P}Wq&;2R#^P9km6%J$`Q;xhj7~%hOCB!pFPec0#lkFs>86Q zOThd3I@W3{m1HTo?e8n~LM~}Rv;6Ag4 z+YmQ?H=7Mmwfyj_LpN20pYZq>1iqBirur&sDvbd_W?!&fXVfUL zng{V{YK+uxZ%uj4tT)oe3Df-<#mA=dL&X8q76Q!`RmDoz!5(teVI#%xdewdyTn=iC zgRdad-Jb>sYjp^X4LPtbjxXsA75x92bbIEN{dYzKFe;gUdk3?R3jt?7foEI&`nsuj zR7(vF#cEV5MUAf@t_`3cK##l-vM$F7oPW=-8)b-nLsQ@VPA-!mTFGct|MEhH5O=Lql z>lgTGCp`D69(!LSeerv*@#G}8z#DIjH`9f#Y+K+fbJAsxO%_6>c;mb{_sn^AAx7_Z z#y+H8_26B+ZHumJS@5NUqI_$XZ4s$o*TF(v;0NUv9XYW z_{ie5=Mdp*0bk+I?}tcL^SKssU|L`Y?1?x7zm2qR>$mV~;VsLw?aG>Lkq}<5(EpCjQhW(*nnSn{|EN<1 z6?gkSv^EU}vp)&MCmcN+%S{~FvU?N1;EUaj4PYM#sY!-i(uo<;GQhgB!ctArz@zC8 zO7(=adCC*V#xEc_DleXsNvwOrbL!-7=CJ`Fuz;F|9N*8d7l!5X+<`~euJIkBa&n*j zH`N!aILOPFoYF%mhr&*4M;tz}tSf|fb)llfaiNR6lG zi2nuUcFsjMRG0R>fGOF~A%H&&>jYMe9<7JHw<1uBuatJRWjO5CDqSx`u}i_+kHtp! zjBlSjsX3i4U81G_Y0wMycv?`t01h#aT{n2lY*Uo7)jg3`eRHkNT~mXZON=0a__HF1 zp1D!9SkV6~#2gWma)V!HR5s_5<6*hN9#AT_(p$4Y z!RkR3b6ziaEWeb!`rwaP@c14+bJGaV^#wig_FH zEq(E1)m1$;Rb5@F!lzbggGv@R?XY8|b>^8g1&0|`gF-^E7tai}(ybu>)x_u#8WQd< z5Gpi#IO(Ey<&Duf8BE13)(%lP+bYJGOE%ad97a?(M}$CTy<`Uii1eH{qHI^@PW${R@bL}Z7!_GSax-mh0FW^hxoSLPd>|hy)ZLy0VbK92kJ?(js?Fn7-Jg8$xgx+U=^Hc$*XtYAJi7I9%xV^p6hpo z60G89aw-J<%(p>UF^NZRqqFTZPWj1;b!r!0e~q^xVk*gllw@OZ!{z0``!!sk=OFyZ z$Pwz~S)ftL=oihj+WT16bPAMhz<^4^9F_KF8r~1PEMajZ9e___v-~!ynC!}PT(<=pFsw&^j62!J=W&8&k|4*#oJI0Qbm_$n@S%5>NF^!c zl*h}3_(B^NKHMKbG{>)3L_EaY{oUqP<$>K8o*5oe)RpRD}=Ga5D}CsD7OGdF%|`HUO<7WC-|Wd3z1FA5I}11ZMoz3N5Qj)A?^ zjPmDDpl{7iI$Ih?#ecc6F`D9M^l0;?i^jSx;-~C{G_e>71jf~A^;Z!h^}yL#t;3MO zYo{>)p(axvN>sOz-F~@<;VP^JBFj1*1i;B+R~E#<1HrI6t)7{3F_y$HX<$_DOAM6Z zB1`8lgg3Dgv>g7*3m$>QxtBu|;DI9iSK@JsjcxgAk3a>OumBr!?)t(vQKb8c=rK4b zf0j-_D7Y^`7S7*3^rG35`be_4MqVN5XZ3#9X3?zrO z5H&ysD`|U|!Fa$fOdJ{IP<;Mdw|4Rzz}-lnLM-j_{?fwa-5`P-iJ{17wKrZ<+s~6E@=>$HOWH8fC{zho@-N(#8txTGE zt@Hk)D(JH9&^a(cD(F9iJss}%?nr|-H@aEPnzAm_`i{R1kW5(R`$yjKKaF6y2>PH8 ztv-5R-jV*R`lHuNQmOzMPAS8B8tFWov=gUQ4aR7D2t}W@(E&Xc(kS^ z&ctU?ZA`v#XKlxnyI8KwaqLLB8&<@BkqmJ(dDe+aGq;dDE{?&%@Uf0+*|s=m{_hmY zO~BhX(P$!a176M>rC^eoQ6tsFVMcION2N)WX1w|Aq^!6rR>ex&Z@qzV3#k@4fyi^< zlV!6x+?epNL(S1u;AP|r_w{U@;|XaZN{4S{5C!uV{9jNH%^n@=QUF}CqC#Zu6i|Iu z9k^~XBsyQSAD3Y#5))>h;AL33^87iKnrQ})7A4UZHW&yy4f;D=Q9jDq7p!Fuufchx zX}FuOOLU$B?vi+^JhxYC%Ga+FUXwXfv-~DQ4Q`3c5Ba{rivfZ(#L)NVMVckVv+sk! zo>ec>X_q~@go=q0$x}i|-^~M!2p(z7Jj3%3AqY0xS(O6b$~sB%gUqPxX_YM^F1`2A zEQf}A@ZLdNN;-`%*>arsm`I>~8 z0txC0N2v6+=0hS)qwGH`h3*-Gde?asiiy!nDIp>;=sN3I_8GZ@q*55)SfCd-kxxE%ZUh_gN&cYN+_-&* z*phF<5P3R9*KCJS1==9j29Bh76f&${?j*=dYH0l9QkKe_pLq5InxE;|VNb4}yVLNg zsS%pxmf21~33cvKPu+%|13vR#06hZuWgtX&1m@T4FS8)hU`on*s?%JskxU{7-(h$ z#otc{3CX6cX8q|@?#yyvUmC(KmYooI{lyrO^LPt(PG6yuCO^svy9{gR=;0LsQcR5B}Sron0K*eUWR|Rz$ ztGjp^|-NYl+GPePC@0dO_9|wVHtX@S677>yI2`F z6cFgpi%M&0DkAM;p!M~X0cBm=<{8z1L<7{*iIIBeBs?(6+UhV)JYn~y6e;sy@$c}t za$%ESLimLf!SzzkGpf_Tr+PDC{$_{55pQJi*{;vk`xmb{jFvIa&9QmK)eDcs4mU_NSpk7e$Y`E ztTfuNJ{$-dx8jP+8}`ph^!HFp&K#IN{#yr)Sb3U3Hzg>G2d z9hVCd3BJB>{)XuP%O!7$aESk%rja9;;~muxkR5B$d+?%;?)}DcrUn_eKClCY)o(>+ zc-nXZL`{e;mU(kL;f?lwg16#4c9ZY!-GebItUI@dDcrXe=pTz?Ok$rws|vX_QO5EE zq??{kU^QG?chDUljp6G0=|BYn{nI*wq+V`Q9}9pBo%y)Of|mXy6CAPfiR5fez~AX5 zV;QIR00IBg`6SPImXzDU@ihiT(YBNdPAfX$WaEmh*4N6}&MB**7C8Q48%BwS6O^>Q z6tgLR0Y&$m<4KmRq!Q=u3Hh1g$)oSphC*Pgi`RC-x@Xa$xHIk8p@U}!A%dpQn#K5Q z^(65%Wk6&Ufl!p@R9XNm5K}=s!D05}pS8lb2FX zVKHQ5+*A(&g?(o>;YrtkoTrC(KI6O)ITk0SNCNR&A1i#vwwmDPh zDE2j^>pqgF(Tk%ZhyLfd=M+L76*Jw`^yd@B)7!8P_ z3K++kQt^k~969+XZ2m=PjQ8apyZO>1DBl?0LHA>*4QvQ|j?!WGhbXG|r_W};gRFGa-c z=U5oZ@{?f&@RrZ97!Y)55KPWf@)k!cmaf4{UOs}$n9OFS@n$Cq46Q;^%%jUFb3NB< z);%P}Tz#u8Hc)&yP|2MS!QK7qWj?A>-T@+XUwW47VW(TbQ`Jfy|6&`{kGIq{072Lv zgOXS2)PmP@fu zbRVAG`rL9t@*r5s8tiMXCwSXJAvJIlV=_h{661D?`+U<|ush38$b!)pa|=xM(!TLP zpQ)uJo`~C=1mlnPT0@pjP%k<*CcTTx$oDQjUdg)IZdU`l87P)1*EC8j+(Syn` zP096YY|S!;$9&0W&Bx6wy~iqpGuz6=oVV_B_`%|mX2dmOsthERb5`R*$^G_(IW0zu zy|vd;R^x%fero)R**ze>NR>Aru&%mn-UVB zwB1$MM{e8+_C;MRMF@jw07;M&u~fh}B}r~S4G89gCvx5~Fa>!a1>%t#@_Jn$^$&!; z^67TXzWD58GT1QPCgmFB&xVkMS0fR~Tp^TUs6#Q8YD?t>CC^3OD&^ zEH#~GvMf=uRcXoo^Skl%SwT&!-uJsXjBs3k@WeG*X z01=CkNp5Ei(0KAAL%S;8Y+~+I#}}SaH+E@5VAdV&(0C3GRncSSokxz)uq3 zmq0gZN^r9MvNqnN$u#i^DqN45x=cHl|t2vD_Ztp>2ydj)9W_EHkKKR`pmMSth50>e~J7jY?vXP zga1Prv66x7oH80#R6^z=iX*33at;9Ticb8ysP^h}z9+V2a�_g7km0_9r;YKbdQx zLms(msQb$Yoa}WTjO5o8JHO(RT%gFkikj#zZdlU`GPo-s5MN@+@oHg{k^*`^i(=c` zA{wLv_9^0U?a{K=`7Y;g({t}~P(2(w@sS3B84)G(RLv>ikkM~iUw<<|ax318yYh-S zuk?#>(Us=*YAKJ_=&~Hi;xpab*CvHSrUp0`wE(y?2 zevZm`tX1QKg4C>X_0McpZ4Xp!TM3&Y^9O>>7=>T0tPSscZQtJLv{6Br8-iM{#tCh& zlBUVdIG4Uw;&em z>Jo9TT*L6;&Xten%T-+hf*)uTGTOXbMJi25Ti(HkR8(qR3iJTWeep&(A;O??OUJt> z&Z{>&ZU;LC`IvFV)pRDe6ya47GPV2qq1eK#h#S@c=3$PP|ttC9u^qLTzxfog78LD#*QaK;KE~ zp*Pg2R&W}3UO!wI!I~H?d}3VbKbfWL5P|P;LZXS;1bB86tsj;~g%$XMb}HOYpgpni z-=VLD@sG*Cr!%nAF>OXvAg7o#6p+0x$($@Aawqy^i;>}CLMLZrvFJPYhb4*y zD_(bL!v+^1E{{E1_x1Xt99xK4l_aqu^kC1f0-Hnta+wwW9twlpD~wj?{0w|e^JvZl zA4T2@`1damVaepk%0^qWKxWSuuD?e#RmTgH>Rg%ts}zd@EUsH4lvklUZ3uspo~pz| zA6+E2dE#DmtVB5t1WXgE#BZ8t+n$#wAbJ@ z+n~kNVNr6wxJ;-z41sbl&; znvRgej_;D+o6RVhP9Ih)aM~;zK^Od@F>kxHuY^R3I?5SC17GEjuwugT?e^cu<2s3U zlPjsV83U9{7hDk|Iw(ba*D1Y6p2*=%+Xn9^B9?vzQ+irg>_>8oyl@uzqK6yji&O zDGl0+6O8mU*Wo?ImBrt0Y(&a%8s{*Y47onlM|qg~`7DfQ#^0t94|1c7Er#ew8vxgM zt)!-K%`g(qQ=g@!h6u~e>l)q2N77UNyWm0d)HCRXxa*!E7v7!?V*rH~-NFHy+?no# z%v~zC-w#zzsvS^y9}W?eoV+dI!6WSzJeyYKJr|5k7jkVK6ylVQ+`Pa`qB8KjpJdP| zk>r?P{>)#SXoNFBESMg*0?S;K#wRS;jxjfPKw~OFmNPKc!_S1JUHpoqEE?K>JRghN zrg18B-%hS)iW7VCwB6rI`RHWXTCtLNvZ58RtvKwFHy(!d}FV_TTJrcKS`c&>hWZ8jTs+ zT*$180{VzRs9^Wn)*4U6uq-4B@lDL5`;FA1gJpiaW_Gpq!K3;IY) z0^!G=q2TF?cgU@6Q}G~fnEcmuV*;O|1mb!cQyVC^m(EW~xgCOwtHPh7IL=F{{2WTn zIeuM0z6?s@;v+8kZBnPtdC`%NRUc$tUE1Dx^Df7Dei0kPY9!P?9*qluUKlXKC^7$7 z+S-oa0joFE%knUDcD)al5#`}&a7|F}biU3+@z;3^12v8wz(l4nI9PA?g$A*9=&9OCjyrp)TaUr@N_~qcMu2}W6_C=TW zECfQzzYUcX_FQkZA&u)~(o(<(-=L{(A2UXQeOkDvo>#I76n{Zix5Qn`b z?J~WjZQOI+KHI`jF6#H^D|Y##6WD&LfrtLnqS-2uQ!wd_&_8(BY4k@~kO*I2&%3A2 zZS?^=S8M2%1&z`DX#~n>^+5tMvus4_ve3Gua4*2Y0!;HDg0W59$%d!_BwmQN&CrsN z4&=13UDM)K5KsI=wx+76#RQeq$Y@VaA|-?f!jWR=u1*h?$!MxK7Dp117WtQUWA}_B zyD8?Wri3I2FWFv=>Aw2%CHO6TvzdvqP@U2|Vep9n@z8&#wPX#{)GLFjH^PrGEkVpi z^J;n@*zG1T`m_iGIralkyzJYpoLG<3*1T>JN9Ja3UzA8yjNX;p2!Mh&DkH?b`SPxd zwCnJ0x8-ZJLQ@Dy1)|~sLXe&h2EoV0S6UVH;%_{-?YGa~{OZ-ok)O@T5CQSTWmml&)y_-1!07NP_!q2tME_-C2{w?Z5RK8B5;oWWv9E`ns zqGre78rn6c_#S;b90HebrLw5;C^F|&ATXd2h#S>2x8-L$Oe}E~Bv8v83@`@=@Er9}0aoL56*4IYig0l2($DPL*=_hrSE zg~unqs)xd18(mbrWN7z=NUO_ffHF_>F%kyAzOwvk`pyV%G0EBu;ggjt$v~NyF%~6xiYhYr zA+MO$wXB4)#+p#$Y`_%7N)v36NoQ+l^33U;myTTfD&0&N)eTxvmo>NuxfVi zX3xYsylNt^Bqgut3koGjFZl4E-gz?r;wF!8JER)Ni}rd72PG(Od5k35%dL|OZlDI~ ztmy#|bu$H>$Y9Ej(lY0AZzcfeO1r4?b>UkyAw;X4)tE68%3WWUZ;r%GPa`jPJqJ27!j;VkCVCGYS&2|&I>!d1;&6-M|w zQr|p)mZ}}UA>8B3wuu&hu zcEbGqd!7h#@Pvwkccuh}!zXz_d+6a|ibq=RG)68Jw zeUCzj8Jpy@Dp>GfeqPKHr!RO+UV!+^^>EVZxpi6cvWX;z3P&)0e6?e63fxHXhs2xa zJ7_I+$8Y&39d!`qNHHmy$UZzY=H?7iKZ4O5mP6@qrp%DdOyTI}G0+~yAdGV4+v{8i z2C>se%R&2$seb8P3nbX>+o=;MLPfUPF3IVf9)vatDw^Xs0K2t06&xpoLYcl%^emT2 zOJgnya7igi{JYw*12MZxC)yZXpf(xWwP&}i7z1iXr_2nq=CbuQF|h5})lgmvL4eD}n6z(gmvWDcXKY?Ae)Ud5(zPsS{s5?wr8z_=Oriyq=!B!6C|J5Cy1t8cZ9E3wH~iEJ3r6VJutuLKHyB5RpC5 z;7-ZQt!mz&U8mq}e%mJ;A7u;L?PpH8bd~+xEhE3uBIBMe9j zt=hkQ*CC>;HI(ckfJND=87o=&k%6uIw9Wp^?tUDtog9WI1?Q_*l3k0q z0k*VQ6tvP`8+KF$A>3jrvWSx1HFzTf29tdL*@gAAulc>2L}xF)<_RzfY5e*odl)N2 zz(lm2D^;D0SMil<4lNF0kU=`YxifRCttWu(?MubgpcB?X$e*<8syQYg!?WRQBv1RE z8>Y9yip%f{2KU=yzi57)nis)r#Deh}xfJsFVO;IN7y-r$u=G8AR6Xhk)VV2DJ30Z@Z;Xd+$mCsb1!rfxB zKQ48qi{m2=lF=+>bK%RFlQWZ z=%QX~0Sx-x|1(i8p<#iMnAG>cUO~%Gc=g}cHRoJbh`GZ%lbrI_wUkpA%fw1^A5nz2 z!$9kUGF6aQapAE78v~&APa=8h^t+$fK5>;;C7U_uMf}~(7$E{e5qqQ7S&NIogm?1$ zB%u?O$)r$XnA~v?p(BW23AZs7B|RXM6doFTvKoJ&!$QB&_%||Nh}sNv^-N^YhQ{(} z$6oRvalaJj?=M6x3%%6S8QC&FmDRrF&`-q+CGt<4W$gYG9#oi!_U`EaWFMiHpz6u8mF$~4#^@hq3GPXamiZPV&WMThWa<#;7KysFFp{dR)1C{MA& zx(Yz_K@Y%BZ8Z;YcIaNoyQ^K*X?5@wt-qt(f^B>9gu%70fB794QvtEJ@=9$*&fuab zF~rX&OA;f`NoHR=wmS(d6BE6hYQFdR0 z0+%VY{=3Oj5#M$-m>#;xyk5Cu1O(Rr5Ftmpc1C>@q}|#Dzreqe3{q}UVr(uy2cqEL z4uvP%iwFZ1d!;E+w2^#eZ|2+KQSR^v|H zk8W`w@(YtKExto5&=<^9$24kj4XIV8?oVThIrO6X*eFB&y}5X{o1MZNfcVdBjm!6{ zQ&9QY8M-y04N9#pOrSbV2e z`2>ory|9X?B$^-@2p?%ao5;sGl^tD2ySqFOfo{khiCnFwC8)vhoj63bYwdf5@Lu?S zn>9DiTKZ-_&OzRH^8SwUcoF>*DeS98Hznn77YuUV<_Dl5wUAdEn29YkVNLuGXR zR=OlF`q}puP6GTQ@d6`dZ9TOvUa`l9FcAVXnhAi4t>JT81%W~w}i2zJ-e=VZR|$c5Sa|%isdmOJ$BeEbye>D*;m^s zn+JU@5tJ|Lo>?R5a0$s>CG4!4?jGk0^vU7@61Emn=eDDiKJX7XMGu~$r1SEcI4-D-_+}B0MQcG^6nA{B-90@0|lQLtG=+_!^CSc(Rt!`_9cSnV?q2YO2#&& zBWU~C4hW7yE>=tQnCk!%cQqA>LGO0i=s9bQ4{UQ57=hrFq{q-`uHp<^)Z75Q1_cBa z3^ailFFPQdf_FFeC-)Ejd zrVUVB`wKT+Tx1K(%u*e84z}ln(db9sRyDAr09Eyxh4V#25F$SCPnfrZx`)!j+T*f$ z9$@jo%kq6&hY0qs#xigkh)DO$N_tP>kS_Ezi{$qyjeCC~Grk>&37zX(b_1uYgqbgt zB;4BcUDo5C?%q7x5+Gbw4El{*o2TTsw$tH>FDa}F?lvOUrTgfR zLy|5|CiMj9*#*V< zP;Dh%6-zA*Z8=gJk3V+rl2-?XPhPuXmJhXhR6!XdXZ(tbt0=0gL6z3V;j?Llm#F3) z7xJp4?TIFcF4+|1wkP?z6AUAU=7&NA3=CBSrwN_cccBhE1y6_rrL?0MQg@YgZbbt; zjWm?l+Vl$APC&W!UPZ;egf=jzhck1FT5t+J0)r_6gLMX{2NG+*=LR*MEfLw~^gXGn zD*Bf|n}T@9Z!q-D?yz#988ht!CxRpM3PdR8hqS^^K4w7t2BBmZ?Q& ziv<}<>ip$cet3%m{gC!-uZ&FO@m|O-$HC8R)qO1wE zXcUe*h%p(fKwbu-wS*Z@^;z8!^8`fCf!~9q5T^OQ)?r6Y9xG}_U*z^21QiUW&i0y$ zVjo8RCxHZ$#srE49^4XaWbC0bvJA5JfwWTR89a6J&Bd6%R8b+qR1%ZYB`E!N@ zBLWkoT0$YR2LgI8M&2s&-~)bJgcyK5joaa)DP-1tO|yvz?6~O{5v04L9Ck@r`|;Xe zj$#x~C=vpLCF8cR-L=`r~5A`Gu7NdpH6V z^WHc@awyR{%C`pbfrB~Sj*m4$MJGM98;P7WbQV@rf9);TpB`^sp7&~J44IVqrtbPq z0jbik`=~)eFHuKBV}k+o!_7l(34k|^fR^5!Bft_zFnIX!il$EPqdLTa|9tJe!I>CbQuC%jObDOvN%Spp?MKGplq*lPqtG@ZRPHeoVV1(C)BeB3WL z7Cctc7PZFGo^u(AFrM6^y&N;FZ8+!yT0yyFNT%*bQ2k|d?(%-5X7}<8SI~AW57<-F z>aOneq_13QNL)%4j4Ink_F1=}Z#F9RY+NC%^SI7pM@6UEJ`@%b;vL9I&g=H+6c{kA zp&kG}k@P_GB6{{1fV84r8~0+xIAg|ylqtyKu1zZsKe0TOANwk+HvVGR#j5T5M^zQ; zs>l#y0xVKt^$|oH&*Bk=GjsXzH%vo;>b(|buHYJfIUKiYJ755@7|Y0ivG{VOp1& zF~B;~VD^mb9dpaq8G>%VBslvPqdfXOp=<3^=|S&bkQtIIwUuC>f&#wEWe&ICo|sq^B1>f10p>YgkkmIt{Lcd~d%A551?H1V<5 zV{A>vdKJg_C|o(X1eGC3a^5v+V69mcM;014{U#KmIMa4*Co?ujsDXms0RF<4C>+pc z+4YHMeQ1ml1HJJt(o|uTrE~)O`edfHKuY4Jo}%3%0?|XY+Hwr#VPzR$;HRBz;1BdG zHBfxGP!RAe_EW{LxR;ab(0X!YDor_78_SR?^cqF4-UM^koro+;mu|VLTRQWiq;ssC z!NN@#cU&SR?X6${P_!R#nX<9B#+cH~_#*>~CJk9f{tyfMU(8Nf$z4Le-xuJ)yI)3N zp9B7=!#4G&ZG!~L)Y^)dDptjf2lY~nd#>Ul%gaG5j4+rLw;KF<{ zpo7PF*V9|sfj2w1`S`FvM(#G{=Q@8y$V#h215Tbv`hkyX6t}|8y!6|T zt48F%A)mpv9pd+a6#{{GEJ>-A^Oj~^2@$o%GAFA&R~5T1^VXF$A29-n)h*>4LCnwK zlzT-C=k)Lm935(mjUkdm!xetLnf4?=P664+TOab7yf9rCFmg$Mv;pijUDb}Y_N*&4 zZfG)$9rdDA@Ss1zEp(BCk)0D+stPOTD0dNNmg?yU%?;PDiEj7FdJf?I@ojSR;NPf+ zHEv-`9yaBrtkB0El*}^$5c2RrF?2FEDXuC@janZ~D`rlwyV~ z9f{w*4|m*U7#=QcRYc|w>>a)c4lBU+7)Ci%X6iRB!Il=nz-XB=0pmaR)FJulS7QsZ zL2K*Cw<22(uBpq^8@pWmeHkG44eRo5~Rhcjk$8gga6v>QS)jjr9zxYW>pJJ6kdBO;S+Dd0Ce0sucFe0#9A&Zx3NSct( zkni7DUE>hm%2Wk5HfFR}V6XaqshVN%j zVu8|a?uL}xZa%oTCp5HUqFQ^s*&PUC1m&62i>9nIh^dsM;bNz4L>O{kY8v8T`n0&V z71&>PB>UIuo;+~n*jQiO-C6`|AS_op6`IgFqajeb$Fju$WQzuwdh69eOXF~OmnV^` zWgzNH+nue-9q3!FFK<=sO*KdjIcDi4w&-;wr``!;F4~5mpNBCmdQmnk%b)kEOfl;8 znyg^lb@@r_4AeLn;;BtKa>kN{eET$lwu1Q$^HHO=Ku@afb6H{QUF{DR$_qfDG`r<7 zr^rYgUKYD}yT%88P@(G9gpT8TS}jk2mZC4ySS(P}vn47h?6K38+Lw|4hi@*~;bm+H z$NWEMVvV^XErbP;3H-mu2<0uDtV2zA4TFMnGE5NB-YUkz-)s* zoLO}+s*77|^I|q0OJLKyF~-1d;dR~S4??iQePKTFHU6dfj4Qp|NXhGB2Ax9;7~fJ3 zj6|=_%{0u`iz8SziAz1dPwjQq^;DKnvX&?rn`-Vk$H$G?=(cl}TVOSy!8H`I^4VU- z>Z8mT&jged0uVG$h1UAv8exfp=XgO@hqszbos*lrBXogqFS)9PlaCp;0I0+e!Nqxr zOSm)+fy8iSpHJ;Toob`41p0kUOHaQX{RH!EuZtqVf}+F2Du~dqnJ$oq@80Ps;2o$6 zvs{n~tZs0QWj!eCFLkqbl|3{OWnm?QpNHlE2~{x~8#q7v>LV6Pu6{tPS}Q$)B2u1F zHqpabcMi{)lXPNkah?%f<&|O&puVqe^pa{{Di6nkAoTL$bdxxeJ$+pC)LC@LDh_NCuApp%$twM7zZ8)50x#n z758$A5hmXrh^=yGBn(2`b35=s0#TZTEtWlbk6yqGt=!yFWGt|SY(hFm|ILRGs*yrY zh|Knj>8dES37)2hp}69vtH>f32@pAl!}Es{0BG`WMS(2_$l+RPah;6`g!fE)ufl!& zyhH#)pCxl6b_b|g!6Ys-O|S~wjnH`z2uL}ejVch2W+oawsW$euPtlpOfC2wII!kV4 z#Nil&!ckx}*Y2`A5OHjXWVP2vdC?e}cg~Jya1*ew&r-ZF*>Z2D^A2lNhG7u*W9Fmo zJoK~H5ftFWoDJVs7zZH?OGQwG=TU8kkj4-=Q#_}J63-KEf&u;dZhW zuAJ5e<#hZxE)P-jwW9=NnUGXzP2)My7jK7AsSeA{S=tR4oYTr!=}LU^Yoy{r%v8Ou+;{Afhxcw%e!qn^u9U*i zRbFCrKmsk!LOOih7qd8ehQ}o^zasY)$3!+_R{XnbW}cdRACB!c`ExnX%Sv*x{i^c^ zsbd7#K~4OC>Z4^+9v%vDxIF=qrz24l%<=RBotcwKXOx;=a)^oS5NM`>!G_8xeJ8&Y zbYAx}kdn$#J6!04Nz1auCRU{`w#_1>-r;OmqROj7q7rxh=Q9PJ3v*<8{S-F(YTW)P zefknEV2tbBi$u%z7fswg%mOQ&thD%c?p?27RH-LUU%Z$ud?|ZgAX#DrXMh-@|2uwKk`%&Z3?jROg~P8Yxhbkx)P0{snt3*i}Df-RO!D(>heX*M^GN{5=cXYkGR z+=jsG-bGKebt)kv_r8F$f-ABy&H2Q6H5)Gnvbr6;0LsU}D0WnlC2cj1zG6P?PePZL z<|XVj^wFfbiD8HX!Cac8-}K$3(RyAogRi5<{Acmo-)cDU3lF!MQ zrFYdYyo+t-^;7%@kk7QO7{Cdtd$QFU^vdGMRXo>GpFvw2&w_4bi8L@a3oGSM5yw`2 zqTs*T5HK||Fy0rUnqzB@6E#oWr?%uK>V5^&kUkJrF<_Lo>u#F}K>nKKjP<%F4cx){mJ9}{SgOm71-grUB|lgLl0w~vqBax;*~-$|&;UijPwM0|eM{g1qkBU_d|#bqz5{FXE~WtX zyw66Vih>l4Uxl4< z!IJ5mG-=WpFon~84w@P=B|MqtlH`m`aF2^TO#PAz!i#U8OThCQ=#9$ z+#{reAj{yHU8_L^=Opt&!3g-%a{>O9w$#Pt{6|A{)DBGV!6pH(XOG|tcax_Lrz1bw|bb*t(Z67xZCI5ymy z20l%*QN}0^g}W+4OsB_H+f{`lsDxl5%2<>_u8;nS(@P$;8HI@aNW^f%YN^eJ7DI?o z;v(-9xgOU)DdF=d>_Abh%P{VoG3z)gC!oT~8p6jzemwf_T#M9$$mZ zGD-yv8g}bSAAduhuO`9y;kAY~G%@fh z;|YFS4;$due;D8?JQnSlXU{bcd%=CI_H)cmlDzNtHMwD*bIk5*6PLTTiV0^sg)lA^ zQE3ja_9@z&TFTEQB)*U&=ylO_n|Z1s9#(2rMJtlt1j)cenV|tZb+QPGNSx5Oe-&dy z2T^hFk&vLyuFF&(UokLiA^cMvwH6LC_Cot0J=E?O5y4C~ZqX^dKwS6a(Ph|afK*KR zNRRa{8`PjebZvSEZY?-B0Ww=o>^55GffNAW0v_7*oNoXpwG%uDtS(3Xa3q#lIT^IY zk!2%MtjhHw_j2$KCiZBR__fxqHtam*v3L16_Omrnjv^@(;OU1zXvzc z4OSUgUYoR0?hjl|Y3f##q~{w$rvsJ^LYi(|mKd{}UpYo<<3+P4^QQ;JN_}!t=5i{g ziFu`YG7$K2jC>}5bqoRq^{={kpb z;!o@a$)n$d`Du)3>E7FxYuO7eq$>s@nbDjj&(Rc_3D{*QA1S`N!V+P*=auHIiNCos zNjuy)7+Kz(K6UdWJvy;r5jeS_rXj>e=z@6-@pCW>+@=ThfI~Tn8FDjKcFF`jUw}m6V4`Ck;ifhx!S0HF z)aa~p7(J4SzSj3JjDHiW$STRHdWDjer`ed!s@Rge@M^n1T`%H z!mn28pgEOG6hfy;DOaBG*XR&WbL~;2FxQ)`!4dQq0CN^)l6>93lh54|WU`r|0;kvF z#b*t%^#cHzlj!xzL_mRjv#VxM@J)mS+wZ9gMENH>tI-FTdAsjsK>+!BP&<&v1|1j_ z{0UzxjtpL%aBbMscM8Z<&9Ip{RccXZrao4G8K^I);hrh^!ja(^emKASvl0OP&r-hK ze?mD;ok@rstlvRybrqGiVm<{x-|Yq*!3GXb0!gr=7X93JvN#8ZN9OT=%Doyx_eRk` zObfbcD2-x!l}E92CoJ~*}6 z9V&TPe}ShZLj$Ei%Fx=%jS`Kl+bn8}j3BSxd|)Rz!;m{gabCs8+1%Ndjm>xco@Tqn zM=Mz=7c2$=#|IsYC_Gg7VwPgrZ=>5E&W(CRAm+|(U&78IUJWb*ULN=g;l%9N-T~Dx z3idcxtDHCaqLAS0bwDt9*TKYQ+dK)Y`*xI+h+J}m7u(~dYLT>EkyH^m4N#y@Qa@(W3jVPF| z`KToEtb7EIO(Ltw#JrdWo`9%liRCshduy#q_J?5T6}a(}hx4N-SEBCNG{xY=@smOk z@)Zb&%B%@x-(=gE*|h*=$9h}6oJ4Hcf^E=ell}%Z?%ME65qp>ME_WY&x97M~UEK6g z5YL6x9cQbQR_Qw2AI#QqHFFtpdR)cBoHF>P*(b;Ony3^rHMco*k>5a>rk@*!d@+$Z(R1S&@_vd7)@HIOdbh^SbAD6Yw^?F3VV4?IcFz<5Vu zwRM06r>E!>GXusQc@vdy9D5QL0lWEkg+MmX7)xD*;wQ~M6LPtqR$AGpi?^D-_ip6;Fz;f>cA`_HA!Rgh&gMAr0yO5VhJOzGF_%v6L78Q{XySA0dr zBfBXPsJ!}|p5_`)@B$Q=k?fO#+q<-_S0@_Rb>nq;C1hCabLVl+!h$SZIE?~&vc((S z(IP#^a!UIB`xZ%BlCnr4q~)npA`PT|Z{{fJ)~xqZp@Zj$LO&26N)qPNHx2Yw4CvZ@ zk42OjGCMUMZngSfc^CflDrov>xKpqC-06Hm@A7nx#5B=~JmqgW8^@}UXQfK*pUA{Lqk@YvIt5&BwLPeL1H2i!xjY5#lUKvYifqnQ4p zY2z_FVircaCIz%kUZUH)Fv#AOr(zwWQXU*Fzp!mg94vPJHKygYHJHy=CRp z&WWQyrCH7b7vJb*Z_!Df=7}mhS`!oNYzyhyq1!HvKEAg#xUk7XnPaxdK^9Bgscy&s z2dTt0aKboza_$g5s>S65!2>`T00026Y}>YN+qP}nJ=?Zz+qP}n_P5CGbHbvv06{>$ zzbg`Rw<4XbN_dL?ZXKySq){zr8s5;P2c3OFuY2LgPp1jH1b-u7z_M5J^Y&bQzs~M? zWEg%PBb#Q@8RfF|V!WfWWW>FE1hTVrFLh>ZZZ@nZ@y-hs@TH5zkdOFS=rfRGIXIP( zgFOl%9N!bQF|o^~<|)y^^Y=eobH5M>sL>bD?-!8tTD5K(GYh-|R1>d5mgMU)6lg_? z$v(eq3$`<$c4XT=4|I;WQYprV-EP$kI_4Gg0LLxC$vN!PaztU>&A|9cP@>XXWI&$H z$L-m<_U;@0zGDw!SD7?tCd6>OfaUP3dpIO6of^BU)O_weRHFQ>Uoghda z(y>d<(<3bt?59A_JlP&2WZ~!(3~PZX^8h4keBHEE2eO^!qz#bqQpUSMwrnt}Y>f zJiTtLY}VR~M`B>gg)c9Ej1?2zqNcb8|jF(GcdY@)RvZOyJ8_sooXSg z34HGGf43zt9F7e;hX1lpc)6@GJnsLNu>WE;5$VaBO97!j!N2!{`UP$jI)H?se_ z$+DqNe`mrb*L4{Hl#lf~qm$s@n+rB!A1?skP(^P`H#8hQvD%sdE9AljKKIn z*FwA+`Y8$XZOwmnuR@^F^if!6i+R~?ejaH{F-Ic(^y2Rol$R)*9pwF(40oBFfJ~HM zZm6Jx8unqi^=1^!t^gaWs3lB+UvfyZ4QiG0GBsDDhF+eVRC1^#CeIOa;{6_rUt?t= zw6+u8W;6KRsd=S($v>QAX|{30pH2!XP$+M&#{ z!YziSDGx4SBYPU@PfG>bYy#=xrXH^~isw_Zdh$L;?gon0Ir_s0IkE`OnS&oWc0E2= z@G0uWd?ap=RaU}{-d8dqXL?tYa~#G%5~Oo2u$SxvPIYVtQ#_RGpUs(u3@Hin9+C%) zJWTr8HYf2zs%%amp|H#PyNve%8jlE9EL%M0Coue#>VTg&b=cNja8fIgS@7_@{`
MEtd>f@!2ecx{bT*a3!? zacV#CtA3!%(21?M=D@ZdwO#e61OMWiyZdoK)Ek5WmRev=3RXoK-d1;57RG&xJ_s3W zI*h+C)ZP?R-9Y?u>=|J87jEMGFNcPcEzr2w`BpE2-LBWRC*TDYIjh)z>*x=^9}N{L zCa!^u&~oevUDf~l{Njt0`#96iua+(YB}K-6h&?aoOPxvT26Tf-t<)CK(3oC^HNo2} z_XnOvoK3#+XAq#90Shtw@{N$v!)Frz9-IWZ?%ts>TtN!2nkBLEvoasoMU*iWU0Vg?=9p>f>fHHFMJ;4eUE;c=fuSoPNM6qHHe*Xa`ineZ%Ch zCG#f%Qg2Km+~;Tl{)onC>(67ONYSg9mD*Dfbh^JgQ$W)CI8|!J#Ta;;lP(rG16H6x#^u`nzlv?zQ#b^#Y6cBL}#ytE0Cv!{pQQ0&0=6Bb*#uoNu>gBfjOoHfsHS7TzXG4Blsk4?|Y{V0HLOLhpMa?1_cRYaEnDoH!rs_JqK_F zYaeadFXTCkCCDOg9S-EE=e8Vgl1a=_#Fu8+=@s-QjY<{`Dnw^staZgq;n;qz>oDH% z<+hlDOvnC95U;%j;%RTw81Uh7+I#+Zf%S&5s-O987-Pj^EpI|b?ZIO7hwafm2`tez zD@<7x!{R*k4?R!C9#IjwSNgS3MSqcGPR>cRkj8|h)7HU78L*$%ri1U#CdzVv#l0H) z@uKXu1fyie*wq0E2rPm>L@8p~9u72nCFRkhpaR(uX@FjZ4~{1&MpN72RfmC1gii?N-}>UUDyDtDQvGU`w6P4{(CwNw50^dp92nqc;2GSPB)7lgsFgZ<6$h2$&Uj-}KC)5b{sD<1*yH7{(#xRoj zPhvnM$rEpjkp;euyD_Tio#glMbpDyEjJzJYt6hL1LEDptE6eaFdBavNfi=*Fyb7>;!GYGFsExWH!Y(ZkdoOCan8Is`wG3n?j%Jztl2(9yJui{ zG4h=QxD)eS@q&rwqs6($&z{{VlwnrJpkZ=5=zyu9kFQ`e&+k$JbA%c;Z=P*{qNxjW zbL6aM7gg`L`NE^sZ1dDR)cFbw-jQpMzAq0B2;ILmVx*q*=0ov=j9T3Q>=$u5nCS?{_9t=(lps6>kzx&4m<{CyW3Qg|zV2{R zuQQ&R5}c7O&a)*ow~)!mr!2Y#2@oM($u;HorKzgj8?7J!sK5i_Mk!Jg%K|&7`20Bj zumI-R%+lk5dd<&9-DzPHw=}dLXm|oN#j04@M-C%G{VSJWm5Z*29b4eS5<9G6+!UOQ z&#-dCk-z(N9L7PH;(B<03nSlxKe=Gic;u{=CUGSoR6aYTqC7TaY=MjL{J0kmD}GCyS>OEYn0{ z>07wBXZN1;TCT1iEjuOaF`o54wd$5g`vD*cZWXwP0Fk~NNwrM~O=4)P+9f^2WqZ*u zg5BB$_A9Z`12B5!{T!E#rK~^pmfB>pE7pri3|=<+8-hjQYwMQED^1C5jMN zPp^0NB?14?yreucQ)L;Ln`bO|>EkXo49^d|UW6&Zn@btdf~04dIi^6cuQDiUzRON# z%o~0*wu}cwx9T;>w->1prJNgGnqOx4TFa3z~%m+W#$w?h}wr?~p-Y?h^9E3@@L@wOMwqx7Gr;wqnd*e2t6I=cw@ zVcOJ%cun?LX;B;!0T4x`%=61lGj%_w?msy5T!a-<=~BD_+Rb zF&(7DBCRz55uNE*06H+1(#8!Mn^LO$ttb$F?PQI{8%#SUit=EI5Tqd8_=Haicv5fX z(3am_tV8!90yI+FTQV>KbT{01BhC-4nx3EJ2y?Npu`Xw?`~>=P^wcpH7&l~z#hkQ ze&xJ@0Em#o9orhb)X!h1cQcF}ete)|2*5z-+v*V>T0@L9Am)lv)lBu2p{A(`I3U77 zNNcS(;=}izJ2<`}eq{b6->gE1dPrA3`Up zoCR}njDV?bejV3`e20n%e5&d7u(+9w5#FT3nSxBJ|nQk?7^1Zikv?|v-f|Gk5H)ykL9IpJi?B~F4z8&Uq>|Vsg#JzKJ?TM2cJv}aB|~FQ zabkPGXSAT>Q#?0KZSnFWq?y=|wBO8oN+4EbY4L;x8k4P$9-%KYjs+4E`)G-9!A`01 z@o{6{`sP?!-h;zAS2VJK94nFQL7`CFeGZYmOxy=a>B7-Hbc3;{e~e_1H1-a@-q$xR zZ2mD3jc}Wvt!8=l7%JHc@|JSWeH6oiu;tfaQYqSvoalU*n{b~l2=#}n(j&LUx;)Jr z0S4*jPicH`Q2wLs>adh_EAodRiPLOmZ6{J3vSt_n0~+0|Xj1JupMqxy^Xcsg?zpwb z%7U!iPx%b(g|cmAg+Sh2Ty3PQIuyi9AS)qxYws|F98I>k*({JRL&2kgYkz>_-?^%fwjhnR(`Of}#eG=ZBAuZDsZVkiiT%`(p?c zUo8-bx0XA^Wdk6&){T;A!jD{8HLWVCx8Qf17aQ07sEg=&K^o33F!oh8B6MMD+EhYY z(HudiA5lm!zOQ3(8Ca*kMGljD10iBHM1Fwu8fVt}OBB!F&67UCeFeE+wRQE%@+>9& zSvQj-(fxfuOqOVroo&X^Ic7Ixr}F?KX(v$P5K*Kip0{-)b)PmfV~(r4*q=_F59EREm^mPOk4ZR_;=3DEUI3OlytX}9BIxN-%8~lS%{~%5Bvw0WZf9`PR$ZZl*`sQR}wD&&OFHl;%GnLu}y^q z@zMtfXgRN;u)aI0ncEQ8S$AZeXZ;|XJCYkqw*8JVFASU84_9E$h)F@glRt6W^3LR6 zf=Y9T*&BcwP}=v@K7)dW{LMCF3%bx{hnw?ZtKwvz@=OnWf6;arT<@L|{%>`|m3^B? z+tr__Nl+TSRrdN<>MS$Cz7}zeR!R%vXW;6^&lbRwafbGkT4L2D-G$Ovt?J!1^i}|6ZV80P>1fX?t>KqHgY#=3GS(%Xm{FZrsl~i zu>V{R7dME#=_B*qf`ZlhtyM4y-+-Ez*Qk!A8T5M`E@`Neryey(Knj2W{87C)+x-qf zTmYPQGX7eqYNIa_eufz*AaUXs)&ki#yO7zouA-Ye&Q2t5Jb?XehRCDg&b17#@(b<< zQi?HUVgl!z06qS8+x$XYjjVwa`t;x+fjfZ?8wJe<5EPJSCww8c1+BS0ZV1Ub9i%#~ zP$Bc7D;&JPjM`XDQ{SQgUUF*J4AiSh;5(=xbZe|Q1PAotY!v{2{kfjZ3V@$m=0MA ztIMU>j86kfKrv41q+4ktALrC{hIhS>=PYW;0PaoFK6pi>Hc}&m7JA0ARyr8<^-2n$ zOH+B6abI4 zkM|TUBT5m!5^4KKDA2VINDXV9s{asjCue(TispVr&Nc;7>>Aex^O}|!{t27wfcIj8(wuj|=jc^-FZu9eYa?Ou#nbqT_x8Jyrb(|6}tN9p({dR;@d2eNv; zm!a2W(Y8R%RS|;xVzmrRu09E$e&(#Zu0qzZF53HT^UHKp;L!)tCDfR_f0--@3^oE~ zkBw+=lJPVjV+O-Y?IUAD)GnuaL!QR(&i#|kH@CeKucndV;8AWDlXI|3Gt`1e12G>< zJsZh0ws`AxKA?nWt@iKC2#`&FDq=kgCaI!Pp}o3e>xf6yl|#8b}*wkL`#b{ zEo#`xV5ju54`GK!V%3k-D-Ob-4=#!>iVG+~q-0s^T{wW-Q7jzE>8BZ1+1?A|sr@NQ{nzA_xu+Z>cR zZv>(0_1(Gwqj=vE_Vt}tRXEwXp1>~G{Ejf0oMM7t!B*T<#}%()#tH{87|3*>QN*4A zqpr4x9_2{>9P|v)m5o`2ODp-*zx?W&VZ-VCYR9b1yc?qKdst???$|ayXeYE;W7eF0 z$57Vh!plG^{xCo0hYHuRxpg2%6pS%1fRUpUiChl;Bz+a(0rF6Cxj?+Bh-_}+vQmaF4Z7i2MD7{<))(1NSiRfT4)W@pi9yYjLk;nWryHMm zVEev}=G_Q1610f~R2gj}G~9Cq2{KJkzO}KMD}-J%{dDgi+ATclv|(VYiQ=4TX* zL%%e1*tpbWR2lLV7~xM{?qB#Yy`0tC#L)^z9M$tDa?K*O#qt93jC>#mwxi zGS;7jN-*UtrE%j)gciA8^H=BrPr|qmt3DVQbdb0ApcHGFj(n~tW`f;WtjOMy47G4_z-QL&#;@SUp54#vj+aaMaBEQh0CU@S!&{GPyq$-El0rx>P%G7Kw@0&6#3_*hI( zlU)W0^Vl7S+_T-jBejB~O3;-duHp}PAM8`EN{)_!=AzE&>~2AWp@CWkM}2fW7J!0q zE=PyDV(wi6mMuq};oe8S*pfx*36r9Q zu+7*tPwPrr@~752(}wI6pcViePuwqJdx|a%FiCq;!`q>R8|p%--ox@ip}qvfMyT*# zA{j`@q3Cy}m9(ZiTI0NoF75HYjnOD%LO*SQ%Oqi*uugPeg6l-m)j@C!vGf)LL>4B} zG~y3$8|wNEv=C~rDJc=}lH^Fux+c60oon+|){qXhF{jK{<+&U0%My15@!|)3z-wrm z&ebp~BEqh;DZL==#(@@YTSr_kb|`G@%Y+n4K8Pz)td>8xoYnh?vQwyhqFE-VzpU!` z2qZ_ux70jt@7fQP1K%IrL`3H)C7hx5u&r4e@BO%}hM=XW-UicI^8-Ly4qvr8DSNH& z7RWoTC}vBVo9DDWBe>7=aYb#1gC!_yawEnI$L5*HJ(|Fwp!V z1;S1;_8_G?S*JK=O*t!L?2G|=%TglW%nRWP1m#ySufyr*gGboi+e;@zfuemu^afSm zk>so^tB}k7BU)>m^%4Y(qAK`^T#t^O-NK6;>`HHgt<>;Qzdf?j&_~ zkq==pN-z+&nyrUhYF((i1Z~gbfUku=Ax!eyW0#Bd%J>1rm6I9+)*C1FXh~j!fh`o6 zW|;~?-9PP}W20J~DdsyZk6RP~!~l>HVqw7v{CdMtI4(@mOKS?^r0J@(?XTm7cHAPW zDO2pQ)&*-ilOI>7-<|f4hNLUUv|3vx$s((JOyg5rlWt`$`j*MxBVq;m2UDGCc zPO|-P>EXB2F%j?aSZMcszFwG$in4{PD|s=2<^sV4v|(jh6sVEjWocMQ$9R4S27MDA z1wbR97*0#HoBMHZls9~b4!G9r2ucSdMfv_JEX_ZGYqi}hK8oa1^2ZF9FMcJ7{w{_Q z<#dk!9Q!JS*#7-fg;oWEe?<3Z7b3eTsjo`>l#FG?FNmMqX)45Cgk9GW8Mj%ZAwKXK8 zk*fTa#0q&oQY||t1Q%%FK{+dl?DPN{&(5ba0>GQb>?BE!Osp=~*^O5`{Q=H4yEYGQ zF(Hk4Y|T6Gt5i{FyRBi`TmQMZ3O@M$6?g*8w06MfN()&0{4m76*8_ryFa$1fIR4*V z(MH@~&=-VyS&%3~=}%!3oBoE|AUZRuro5fu)NWgCrg1z$W`_MAbz#bNO7Mf#CUldluxDB0-oU00IH0OTx;o5#iL|^`?OxR!RWvx*zhs`43;p+wXxJJ06!K zhIJ|2+OeTub(f2~(_xvSP*5Oxx^zs(I$k`ENcP-+o}5U1txuCRMO?bM2RpEd z3ipVWfGcbsbqj_}s*UZwtpmcm7`jEq>)mxN6tLaV4kRzpLlSH((wJ>fST6w{4}Rg43Y-^frXuzzU{j5dEs9LARM7FdX3e<{AFF1CqFx zriQ$vok(KtW$+Vkgr!on?6k;Sd60J$ITK!XbhGprwm`ol{DruJzjDhyykY?3R9#_e z2zVVD&qo*XxBh@FwO$@9`SRGTACuT6Gnc5Tv&Rq*15danW-7s!vBaGfO_dbA(VdWt zKvtu**7X=yaw=?bQK?+?T-G?Q!^d9%#2ioQd&lU`|1idxBMA$jf~EQ-JGFT1E0O^0 zE^|nvmXLq^HfS827G{b0dH&YB`6n~8m{nj>cia&~Z`w{Vc5oD4yEQ)bwiixjiImXg zC0};8YaDJgu;tKTer%5dx7^#%5M6^BiqE9kSs?Cl5!+$4HX+m#t5D$u-8R5-2lr{^ ztl)X=I<0ihTZ-tPzR|a#jk$`+I?@9DH?NMFWWo6B{RJLNcI3F7ayD>t_;GyKPIw~d z@1&?iPCKiq)`Mu>Pzec92`lG?Opty)ZCT+!J}S}faI6$lx?dMB74lDZjpmW20{Npk zT-4eI*WMR7M}2_`CMb91AbvMCpfFp=JPfFW;rI2VwxyWP%Xmf>xJF=}n?PkSf{iy} zVm&9)n`=l1=rcWFX#I~o@1gAIF8U_VL86){j6Gj?0sY;Ea{Y~7`J2>S<78cr`te8B z<0ZQ-66DqZ=_v8k3@Ee&Gmg!{+jeTjPDr?sk9#1X`CA4I;Weme{jj`-#9^N<&J(?X z>|D5ExskpRnE{w@v}FX3Tv9P?&`zIgO^8n(lt7W^ahdmDDwlgMvKeJZl0hP|$3}>% z8U)1G1})02NN^iVWeOD0u9D6h)I;nb*%I9n#4DYJuFP%R1L$m(C^lSF4qRP3Vh^Vh zm&zLN=`NK;l3^#<)Jcq9f-oOtZT_9TCcpTvGNr{>bxox9OSJrOKp6`WK|91)!7Z`v zN9ZUZ7}QT2g(~L|_|rfbaWjDi!WCq#;cgdHrZ>eK&@$ayZ1Z03uO~@f> zL>>R7BR9I3>YD8sKjfPffTSdiV3I=C& z7;9m0Ex0NB;VYcKy@r(!71Uk>KlA4C#E2QfX`|?gOFG1dmC2T%ESX8xRKWb(CFPb* zReIaWr7Qih?zc7WdPf&V4ZmVRK=I+fj@lvNd5J5sXzQuL+wyMHSU>$C^W{yuD!LQ} zsMUY~kOG(Mt@wc6k-SL5H$6w#f*W;T{j>nEm~R1qw4hl@&Z{R9=1U3duxEc7VjpOkJdbe~io+LL53h;Ti8+hJ0sI;)(l;N0BAFYuZ;$Uw|n|yD^imk(lzd}wx zYjdQMLN`snCAjC|@za7oRW$E(&wX{hg->MEoFfh+_oR zETz8Q*PQ`*;J-B|K+~}@fjZeu)6z2Ez$fUm>$NoDDF6bZk>?TghG;Z@3>Gtqe8OL~ zKy!_Hq@m^u+rH*k2$1@qc>Jc4cM=UAv=@Jfo}1*2%HDO0kOf@AV3M>|te;>FI!sC*5A;;aqN z>{e#`B0uEcuuOQ$U)fwfI?8*-i zlzl0GdVwLS@ne+dR6rMS$r41=c8x-QzAqEjv3&8hW?yzk zSo~4jObdw;m0Qr^d!9>NLY=SJe>rL;I7#74a$J_xZ!Cl6zcZ8YaVOoUnl$zc>L*4k zK9?%9K|L`<@wV5e;l;lE1PeEAzo8X> zqgjgnY-Qg;G?8YT&rG`*K;*1hTB+!nM+wy}_P(OZ8C03xKg)7`IJUUYSI-k9G9$F3 zW#Da$ya~27W=sv(L}mi+=l|++&GiuOx)$)e>{8|bScz>#UqjYakD7#Z=&R(0j2CUt zHj5LCFlCIT2rR@2LuMbro{1HglUx*(qa&a#YeK}eX%0dscb?=+TE{Spu&yX^!i!w{ zQ*Z9Vjw|Sj`{7^=n`lNF`~^){Ek(dCuNL7BkMvRSlSX2m0RSlC)v%UIZfYwr7eFmK1+fz$UJ)fy_$By&S&;$sQKCoRp+nC|Bp0>MmiQ-EG*#SlH z9zhAfEF#UvKdUUS5}njO*YDP1y!=Ae76!IwZ?^kP$ht0s&08P6ztY`Q3?a+X`+ z2Q}u2WC1bg`nvVgv)Rb(Sx^Ja^yc>D=HbyrqFNhHz-f?@ou2UK?k(ii=;mm5kB^e0 zZ_n7^)t-U5)LMLNEa{N^3+`B1l2vR6zTQ@khAqOP!GDF1+fH2UK9G2wGk?Q0Y+TJq zv>=7z9QobB)Hjw}o;%}3PmG7YIh2L=6%2|+3I(;j87;{H54X&u(cUw2YMrkaTO#-0 zjAvB`ipC)yG-b{_tv_Tdl$kch{C{W6~E?0igv67;}&G>^N_jU1MbT@m+HFV67r7w z!>U(}vJH{`+;&$yH!b0o>geGubv8@o-~5^|p2fds_x=oU0Nd#CWGR51y~rwIFi(lS zsl-}3Dlqo5@!YzevT2w;-Uz(>E4#K?_$q)_K+oat3jLq8rSY5N&KkAo$EJiMexvUk zk(6vU&w(LP{08>gYrs8kC(|d}lg{tm9;d^s&=7|ta$MBxcsHF0pk>KojT}PGXVZZI zY>l6zXT!^$=WE_~@3JAe#FU>X6r~e4rMaEIHES#JqVe_})YK0LM z0w3@BY(Z>SMPaGVsWJdq5f4WNY2w5LF4jl)azerKwFP(xTohpEw!c{jk9Jg-eLJa) zHsG}l^0{JU*q_Hgk@FW`j~*>5>p-1ooP#StgrtGAlxWp82e4qa6Z)u9?kgdfxz>xd z=vC2&qP_OI*Sr(D&dNz^U%dC5f22$w^v5`pgw1hR7aC6U_e!km6Z$7?ErsRakrXe# z?vrh6cPvp}^)?@p=fB*kuOF^TW#0t1&K)LvulNX&OMtKa)a}f7zqQD1^vpBhskxm! zEJlffrLMuM5uywQZRZ{!0>+y!nkJ4CGEj3T%eZIao07Z^hfC)ep>i{UvT;3xQzb=l zUxSVY9E399PO5e1yXAB6=(^Oe%6Gm#kBPDelxeQEZI zL0r4&4MZT04*hk$Oc{Igc-nj6YI|t7Ua=!ClDm;eJ_h)e)q59CusWouJ0yMFpo1tA9`DDdLFM7g0nFnW zI4YmXf*Yh1j4TAyWnH6l-=m1#njtsC8KZxco{b?K;_0^iW0qJr6b53q#-XMRjW6VG zLG6rfS~;}PnbS)vjy_uhmU~;d#Mm$>|GYePu(^_Q-CszdNF5&`vhmCVg{ru7O)#wN zJ|lFHyVwJ1=LJ@4XM9-z>=;+M@7x%Z4&fvR>w!VqHwtBA{F>B?MB0-z@3;4j5SQ-4 zqOc4>Z?&W;Y`PF*r?;UKsc*GR6JL;+I0xr!pn+RPPQ7qcx}US6lMJ_aL>kk&G5= z)e65YpYPZ=OCDhK5n}MuYlWlmePVVW0|2{qHDJCAVKg040A&ZxsuCb?g}ae7=~)iq z*+|Nwy)icuMPb6X`XfTFcG9_ia*m@<@nikEziX zn&bGJ#&-U}(&;U|E-;DFEC!5VKN9z}C9-37Nug93LUFg@0)^?kO0-x~fZZO^Y~p7w zS?)~L!ME428Qa(3X#8TdttBydd~Sxh_l`XRr`olK)m8-xc`=%GBSF`BvHMN7bF(BQ zzspI@8VzCu3b?fyLfLLdbw$20Rb$=RgqijqTYyBpt){`xJkQmz-}L_a1C$&_z)h~s zjaACrw({E7>cqel=`M{dUoeXl@DDh8`|<0f8UV5p)Dqo-JBWh;p?GD9<;K*kV#cqjtr26W#q$~OTgCSSATc_t(W zU~!vnsa#po^j*BGGRif#(sa;_d);7;PCf$t4;8($iTSaD)g;~`E@<#vXq0c(1Obx| z&LF0g^+=2_qWwZ)jizDc(SnisRon=`Cq*`)ERRVmu_3L_<@n8M`=Uhfe>)x~iS{OD3^zurfmMQ-umif19|Ajfm7F z#}Yq})1C7bhWs^n0^|6B{eNu--#EAWCvr$bri3%9Y%>r+VYVJqRmng*VrOZk2pm z@$;9*5F7VNzZuFj6mJL66@lu*rxEYe{CNsQIFWwp2V%!HHmif<9J6gk<;eO&k0u@X zWv=fm48V=%Pq1}K=QjMACVcp_4lyA6?@W1dluPDI2}Xx;*p1N26sENxeoO={F*~(x z*`?Mvc^^4%>Ypb&*)Vrw)}A;Hu1IEX(kYah2KL)1{XkgqW^m`@`euIpJLRU=WA^D* ztDvbpFcY(PjJIp;Npd=Ts3X6sheKS3M zI5{p8P$ff;4ZcZI4W(uR@N>tHK5hPfl+jb1;3Zi{zP+WECycfNS?D+6h>g_C^`K}x zWKiV8v$Qnh#d%)XP(M!#*uLK{`!6LAa&AQFNy#-7H{L6QR1g9qJeSmM=1Qe|XBJ| zTy5srhWXqa{6QmoZ{I!-1q;Osw&2g024w4lUWqTSjP+tJ(bQG6TcgTvrrUo5p(&$8;yb@&uJFHRADMLEucYu-vDN{Y5z zk1-AO@hXHd`KU^k8HL|OKaVz=PnAiziQB&c)jW2|yMAqJ- zMTkeW35)`FzkKALgVi;hp~m1RtBtgd-iwKhP$1)+9C z`EI0pC-d59A32br?Er@3ERW)Vb}~)qpx%(e!J4GrhtT<0Suxy?X_Ik2%_|i2 zmKCbKcVi0=D_a#0bj!r2&lY{_Kd!9QnkcpxV_u_X{Ihzczs>Tq8dh{H?cfdzD8VY&a9|rY7 z(4-+wN5cx$aGe5RQS4XkMlL*hUjS}|flX~?GAOOatWE_jH-edbd3uw`n9QrKaNOWH z9!(ff*cQReL06#{5JeNH$7d^Sp#ZaZA-{K2e&qgA1xQv9?O-jU(u_kEx^r0G}NGDG}6`P1>KDnF-xvwX-m4IrCc`zqM zV0^AGfu}3)?SMuoA99KxyP8D0NJsP#4%zFgbB>TdLbqnv&=Uz)XB&O?Kl8Pc20=C( zBA7`c9HGRQ-(|zKp6XtC;mC2hzLz>XISeMyR(5xDFJGqO)5Wq`7P>>qa|o-$nD4JR z;BV=l-#nBnW74YTsgrxvHC1o0j76>_lC>rW^B1xIeX))To%5aiDZ6M=+i2^zW}cim zbsz?O2&8%colHKvqqZmZ61&BuR-6i?Ss%7c%z2`6(IDO5EV?@>HTQp@m-jr=w6WBg zg&QSVo%JFqdehd;?E$p8e^x;6oIJMTVn*!^=JL#Iqr8xO0xBYp7n16@R zG>!z?#tgzctK;5}^g#b=Hgae&9&?2|!_{dw87Pt}9mt=GyPB}Xa-W{W?qZqwqHaIw zmg0u1AK7nFs$sK4E)!g3d<42ewe8I@fbn@Q%)!A2vNfp~FR*wG+r64>X8V_rq1SK1 z%$T=~NR9@TO;Yk)+zMLW29-Xk5rLnc-Gu+%)tIh^!|l79JOFvHfZi~~7b)HWT`r-F z|5yb8(&m1JJCi%d#5(wQ^K1PKXu(k9T4Di?2@_m~8poG}OQc2^+XCdviP_?)hCJFr zQOlIX{R#2q0@_`C$__JCG6_FtbB+pzt%0Bn z=x5DM>=UugnhjE}sZln|*+6fFBbDO_?A@Vp**^xm~Ftf945IkKui8 zmOGBYHF}wAx*$f^nS+QVwjQBJyu8?LYz~oiYZ_cuHbegf3{^5dlgKjDD@7sKP~E+cWWtfV43M zv7*3AE%kb^9A&7B+mQG8Xzn&lM_2@mLc(KM{-`sato%ixCP+`V(-XV?j; zDOIo*m`KI(k`u$UD_O2^LDK+1Re0QCf1mRE2XD%>zlt*HXPGDy{Z6U%wRlH}bQV@P zVI)^>%6xd9MY2;w?lITD-(AX`6sc-Ty9JCf>#ZwT_bWA;-T^7w=U;QUN@)C8f1WW1 zG9e2}IF|l+2?{W%6h%4c;FhsvwoyM^O#su2=WhX|1$J$gTq89p{}Mu+RUfF<^?^jf zDwcA=+ChuI7HQiog=UB@U2mS8lYTLmLYDE6mf6MbWsRw;J=c)ii*tR}`>*>)Ofo`? z-Zi(iZ`2NNN|4i@3$S_&3P{2O_Ar&o7Fq*x0rteq;IIM^LBy5ceF%BM$a%X7nM`$t z8Xq;kp-~9WvlwaU4&P_*>7K!Qvj3>y$uZqJWv%u*@b99187_WT-EfgRO{#W64;DJ7bp}FYfquc#* zYdZn6Um#xW>ZD^T!Cn_$kG=x@japFX1~hAa%7U2)G2TOBx|tj6NfrSM?A#X-uaAKe zjknTEjj>580fPz15Dp5nz=AG@(@4!E>5Wg7g4@dL_HQutGZPLMrTd7xW;Kz|77*C@ zi`_q0d|>PC5F^vG+nzH+S4p!_j5uk9LOvds_2sZaRI#*3N0nKx>dd~LZ=CEF1wV{) z_7Oq6$y=4F^$kHsNNbkJ(7>72nz?F{vMiC(&&)4{$9Tq9gP)XFw>0SvVF?7ubSI!2!HA zljiIMc_TU_Ih??A7YmpB5et-L@giqh13r&7l6OSaUc+jj&LAGXus;~ZaYB$`uiv9L zOE7{5hv)VI8JyYKsr`Alo@2}lrv3;xB2>AgIs+TC64=wKa&^kfoSiyo9B1om-ntQ= zhLuqFx5lst@lU>MhwfbNeob5D5)9M9$@RP5s>VfPUH*y3bIY>_^D7ppKKW04<-c}d z>602Bv>jYIp_dLhbuz;o*?)~K>Ou)$mLzsYNIb}H*HL;^f>m%X8T+$Gx2VRK@Sca zHy>X~dnGJKN$@P1q!Fw$N#w;5L~i-FLlXdqhlfBw?px+ktoYg5c`iXjw7>hsGF_`^ zN+a0bR4OGWq7r~Bs>l4di}XiKX1~%YE}q0iZL&85iA-d@?r_STq&OPE> z41TZent4zCo7Gj^ud_p-2yIzOiS-6Ep_GKCT)?>fygrP%*x+uQvV_4`-Fb_c;;(Xl z#C7s0=j!W0z{P=1?Hjvt21?M5sJB50Jl|N9#3m0TuyM3}&_@0 zFY>k`NJV8su!wmdUtN79NDb3SoeemS ze0UxCpS#MP`qJ+auS27tcx3G8k^odD^whpK0LkKCUaPiEz4(qc)q)0s`|*19Sr4fZ z-sJ;ZnC;9G1pq%khLThFMKJFfRGN1G!ZsrzAq_+>o;QpxZjtojac$Z$Dpp=8;!S|_ zv)b89uM~OR7qlx#zB(VNJ4Z)sMRvM!$}g9E>W6BVoEZGQtXtv?4fYbFfsqhqYV~WO zuMED3`&64cXU+gyIKz$xFm(Uwukhmb)LJLzMXbz=C%&5!Ef-o(ed!nP!>m4x!}63# zfB95Lq8K2^9W&R}LTdg-q?nXjy^p9 z%#!C1;R=A})wck{l=n^qO`Bl)=Rxw=O}*FqU!tY*Pnly-#a2t`G@GB41FE*C)ww45 z4zMeqYe18=A+-T-nLzUt#H(&P5Y@8|DN)Sblor$ZLL9x@dqS z)t()L8aWZEjNa#})g%qINRHJuD=uxbGg#1kp@pRM&2^HrGmmL{4Sb>c2iVPXK87&b zD@fLz7fzQilkYz~Mun}YudjU*8fNQnPC|s0+@0?})cH%HebUn#`uvn|+lK0)CbMdR zw>7QMJ<2nu6E%-rgRy%b3z*PuKGHL5;~WDeWWUn@&1gN@whEU{V&J8kmL3Cw zs8TVX|9hWTW%&MNDpF5>=Qtex znyW4VweRJj@{4Gic(Kw@t|sAooALRL*RHCr{Kz@*Z!ViP8TF5jz+1m^(Q-Bj8a0 z>8t;ebzU_b{5hvlvjp#-0u-g-pOSdU`^Bv2k8Hvi*qYW{iEnyxweg?G!oqs7(#ICt z7qzLOeHXD&)(1POCkb#U=|DDNnqU^hsIW~(*N1&u#K5B{(A}k;2o3UzTOS*J$e?O7 zkl~{f#tE-tFRxLC)AnN}XRL8~g|F5nbG2a-4#4ubNpue*XiBwPnoKtx9}7x5&0=F@ zP=3~B`vHACX4+V=-zXoS<VDhm8u#_zEdBhg7k!yifgTF| zlP^pYrIV-`e5pwJ{>ZgW%4yl|4Y7aMIj0fAf!~M~F3bABKU)&X0r*@0q`WW9!FLkr z$y(+#@o_chw5szl^mv#9wrO8@_wY_+l4g$aw#E-f z|Lfe@^y4$y(7pG)50+$Ev%l#Zu!J4aU{?7$9KvA+1g>W|;tlO{ER$seZBC&z#MZ3n z4$N%rpUht;k8(U;azJ2190e0=Y(O@L%nV#82gGcn&uH3AK_a+%x@`dq%R8xXOPeja;ZhvKP0fY;@$Urqog{K<=OKLm zR1y?L#1BR>&}@7a+R_kY-x5#NLsdX9V0pKh9l2Bg95q>Q00`}MTq9v@j1!&)cduM2 zam!2~@rXYW1m_9SvMk@~X>5y>28J#?zxq>H4+U$Y|vfNeT%+~MiosIj#60v_Pt-% z59}Dpe~StJ=ZXYv`SmZb1XD>rXp$#u(1F)@VtTV1U^uv5!7x|CO$>Ih2nA^;?Jihx ztZ3pqJVo7rTnHK*w}dfFg8X%P_PJzyd5f6n_oEIG??YLCwjd-Z_o1NMXN*c$2i@*& zJIm>*oQSR;;52&6i0}q~ohe|RSmo#)!)QQBvkH(qnCVCQq;ICA=Lp#G-g)*^MAelu zT^AUS@~zs?DLpPk`K)=269=W|pFacy(*ho7gQq)Ix(HwLCD8Il>{Q+D^~hh17L`N4oR$mZp{2=(kBqphvi)N!Qxzo7!+PQ=udUX=$pIiV&}U*DywelIUs|XigU2l@o_s%M{f>ga zl^hxeRUR0oaSNH792M1N62 z!_R|91EjMFWaa&tzLd^IF4m$XB8LW_Z-DGIt@!FyVR5@${3VbxxibQ@wO^G)&qT-m zG|8)CAEXqW>P0!JR_)H$`j7tI^oj+L|#l9a>*( zwRh98>9^8%TCrJxk;~|jxHxFH3nw5MAn$*TZw$y~Xzw1%Mv9$r>M=|s*hTZ<3i4F* zb9!iysk`6_w>Yf$8QhRA{W<>J4^HWjM zcf1%lAFzjI1g~D0oDD&baenL+!g4$TFeCm?=!t}I~HXk|P{YVW?H#+64qjSVbGy6}N&U#_b;h&54w-T-v6je7YOuIsM5 zKN3PLj^K!4V5wtEiiUq5m?y9Fo_*|8-ry;O(IqRDJsw$~>fG$9<%3i>OTIm0qsCAW zz17BMnZXXE{}F?XyGPrxICaQfm<1OBPAP>ChIfiDheiy@u{JenFVZ#?A-_mKj!Bi&& zD1${neyvXBo923_#tuW8{Bv8zk-_6O>dr123?DZaHAfXDt@6>u3$lg^PINXu%s2PY zp*B(B93TI8s5^)r0ec$sw6tdxI1nRisTYEiMO;}mWNXoJ^b+-~>D@Tn9+hvm9Ks>F zlr(ZUyPr2iQ469SEFk}CV5fE!Sg)hSvjFc z#^{XghiR$%t~ejHCD6yKho`rvpQH$Lzb7jK6#nZ!WU`3fHD?4*O@sh zTet^_VOo9_f`}`UklqH7b8VDYPA&_rSdT@^w$ZS{Us{|KeYH$gVa3d{$oen4YwL|q z7>0drdD3H!w_i_Q2T=6kep|tMUGNFjW^|8W`0({w@w&TgnxJocw%r_u1zE?zPzBVs zJzhmg#nq|)(2iip01o)?vga4#OkrAv?#53Etb&7Ro%pG=+2AY|n*f60XYf=Bl8JCcV&G1SrY6=dNTr94V{w&$IlZS8!=aK->Iy1% zII=ESCGsEtSXUofFLrsIMM(q%96F}x{yOq_Om*T*ORlKniJq3VvOC_WrUg>}Po%L1 zhBJCN0LnBF9`V;7d1E#0xCQ%7omfNjmLNdN9-KRyp#RI{*XlU{Dxzy?lt}&C%V9ph zCgt6RxG^Qh%8icy+41*P64HaM)ZOxdZ=T%i9+g&0efM2FYZYdnTE}grxbnfgNR=9% z`-WMw&Pu;a5XmF06Znj_!k_~?8GyK(^cK@Yl|X&~Kpu79hVBNPvJjUUh!9>>giq^k zSC6_u>ksdNm#?~kIoM4`3h9q)8a6$z-vlh8;chMB51eSN$GR*N8WEP@o>`|cj1!JA zdp~YyHAS1|SLZfWfDg#yMMGomaL;{&phi z3F!hdwf*grx^wnKs_25LqUz?mVH7+I%am|CO^|L^TKbB?96@bPP{&~+jk^iImhaKq zasS;-87x81W&MywDGo#=c6Y6AK%YLBcn z0jkxk(>5XuhJCxF^vDkBL8WcUgpVXTQcTFkSgn&3aE~T6Ge~wpq~sen)`YdvXu4F= zogeqv@)8j1gEiy>H`Ep8kED8-TitFd3c; zC?KbhEl{;SB;d7~%ijCfC--z(GtnAw!h zcO2sFEdj-Zk4{>rYLFjBCCe-V?tX!N3|6T(tMnuDAgMHgRr4Ya5d0d>l76xt1Ggc` zhhn?cWAb)C$NnH=`gE6SB3h>?-VB||2w?&n^lC8ww1F8zuUq23eB~#EuJ3Yj&ha}R za}xx0SMNdj*+m>-h3a7LJq$q2{ZXxn!G-emn(+gP9G76MVW~54sHatCCvDAwbwg3` z=5hikh=PxNcAs9jobYte*cW2C64*`QD+}P^g$BHc6Oi_3V@8t>{Vf4tsIrifmtrbh zKK$DnyT;;E{2P8leQXT2>05U=aC2wcX^b+GYA~@?~_g)+=SBRM+S3PXlKbou()Ka zxvZ?(#fKQidsybxHXtzF`Q-U*hl5=04SwNAD#X1Ep*SIje|+g!`{-ALkAm?UG1Wid z$i~@Dmi`NS_Gm&buqJu4EPnDINV9h`sBfFty`YGXn%le6wjgU1PPQ8QJ!MfX)U9{`pnKtd-B_znlKT5V~f* zp6?gQ(+Uy&N!Bodyh3K*wQL*E;h`=<#L0A7{zXUuZGmxPX0on4W2T3+nRuZI7kPut zra3ibbi0u|r7?72zQ5Z1+Vw(vf_3SFBLNl6lujy7#8+M8>h-bvnP`(+^nDM_qfIO! zNnSxYSaR?r&UF+1>Gp-=EhqK#nK^0F|Gwws!ZboESkQNuBLAX7pJ-PkRx|KHI3^vt zzMgx;!;Wa?6&1po(fYmsJ?w}DQi|jWDs?wnEX8p-C2xz78mzOg%Q%#>OfDAVPKCvf zFL|*hmN#UDoD4SuFn6ZXNa}K`5#3{y*4+-p3f}epTTqS*`6Aqf_ZdW~w~ZZ@|%&9ij|jx=`)Tk$Ft{DaTizwf4oX8EKo$Ht}SEPd;J1 z7y+*m&E4>~{5d!3NnMfws=QQZ$ZWCQ){Qkj6_`v9eOnggpj&E8?CCRkUfH+7t&jfKo6>WZwM&|1};l^&HvbjIvUO84Ij-R~4&3lchK=&n*z-e-SPTf|~lR+hIJZ~R&^E708D0jh}thxZ6IYl6dp)z`t6a-o1( zm>W-CazSV?z%|Q_eWdhOzm&UGr6CV1OSFz_+lv~VJLv`xkZDC(aaStuj&6}h@g;D3 zn{Z%%4i#V^3JZtbZ0J)J?wA%EI)@ccqH(~$<@I3yXR);?bw+{3K@c`YB;x3~wMd{; zAA_wi_^*n$h7lPU#mHo3@nh*-BFtYUEsaO4gULnpg@hBvIp%n2W+Z}6=`zrXJC-M< z)$M%I`m>?|kFewZ8*tTcqOQOy++aqkp152cN>Oj6N>~({`7B=yCxrgFFI83m2nf

C#xSsK&Fv)dh6MznBNe(jj>v0fP52R`R^;kz=?Mt-Qo?dLfQD#$kcq43An#jk(ve+PcV+$ zc=ZI#1#5MsLarZqbstE8L)k)nHP$?cCo>0{XxQ^E=cUn@rxWD^pVur1%l!k26glg&~OJBb3p;$ z;@w33g*|M`0Pj*2fe1Jq`d@pwqx{I()2Sc{O~Xg20-NLDvYICfz-c(5<&utIu)?0?0#@EB?L`$V=S=9%i{ZqtqSO?kT6tFFE*m3(sdzgm z^|SN~Z?2-N2ZhAdXIwt#%~Hs56?xI(0?1=FQ7~gffmrHm(4k@sfyKy*JrXlpfQ;qV zr#Kyclc$1C!K_q^(!$4Q5l;Gunog>B3{t^4K^%fQ53ib-OdX)p##xq*G-E)ibhoO! zzBzS2H0M}}FfO&3ROY1Zq zWI;j>DTHjLV*|`gQ+MX*FSe?76~Z5qFN?q(jBrFd+w6H!#|dH>?~<2iZbOYGBM)0M zOs#Y^f6@0~8YM^n1ec@(h53;8d_-Jf41khz@uu73+~HObhE@HZXyeW(DU&eHP>BoV zt_EDehHxYt6*a8CUA#p1bM2sXB-}d6iWQn$dIjf5kjbiE=BV;5y|{Oc@9Lk(PT$xe zy+$*Npmq|f)-d3rAB#I}fqe&OIExw7SZ?HFnM3AZGVkqbGS}25YSj-0mAxTi znJ&X)3Uy2RVev4DF_j_K{Ic~J9*36-pI&9lAG+s2$;kQW9Oa_&doe;ZTJSSGYKfO) zEbp7vxC&)}R(C3M2U=~-+KNr?Xn~mHVkZdm_`KUZWz-_cBNJzagwrFQqn{d4(HQqE z^jVk=x{_sRs4C#kC16H5n|9p+$iMx~MgQg6_^2hWK4Pdi0{G$)!J9}ZwFmIyR7vHm z5)%x-m4t+mG^gwBzpyg}#URX%)*ITAa~axOjj2>-BsU@~;_i@pU|K2WB+3-lyKpf0 z))Mqp%|?Q;F)r+-3N$goMMmrCE7+BT{e>&s1@0YG)usKjzW~bl`D3a_`e1A5KZ$8z z4W+rQi-mDl$R-*5#^(4`b}$ztii=B9XZlO7j$GWbKDSNP8PPwLozA^1ksdM87U?@_ ziV>~DMg}kn;{X*{KpgD5l|IJ7HL2f-167=Px^)#!5I9?l>QuouVr~w5Am{tLyw!In z`(r^XO?DQ&W3t#*J&a$f=O+eWCN`gcj^m;`tTrx=KhX+1*_O>o=%AZ>2iWTy*#U)SUx z=7zVlGn^@AJG|1|G3X1hF;VF1{yS8iBmD&>7PV#;Sjh}v`BRq_&=NwP0RF88Fb<|M zkK*+D=Vy_3g7v6q!p_d)Olj0jHp1TWo%$oP&@`AhnFYTuw+1shL#8*WKYkH` zHL+>FPzkxmapYaYJwKe{x@KdTj$8P{-1>!}6|32IhJp0f6|dlCBlYd zo4Q9g?^v|AsTI^kCO%dxXLj}Na!CTlt|WPoA+`NHaG5 zP6wfatC^cjkC_UXiIug%!ADks5VM#L%sKBL*;3|Ka=ov$k+TI1-J{4R?FS^7=jg(h zpr*x;y*+9du;o{boh(-ah~(u*n9xX2w)H{DpIQAeXNIn**%paMQnyryh%yDUAdDdI zyS9{G9HuL#kU9a0kL#U!9t^!CZH5sQ>z~^pP!brCUMWO|XK>6Ss6qp4Cd5dB*G_-P z<|RlIlt71l30XrPV~zu^9l&=dH2@w74DUp*s~IWYv&Sx&DByI3XL|_dDGYwCrUM^& zAjM>VtAjuOD$@}E2pkjRQwZoz-ZQ6hJUEX|f#O!LphOHLV3_5=Sj^!P|EZS;FV=#c zr&t!u&f0tDMkH01$wkl;Jt0tlY|)C&_Y+f0lkj%BZILeCFKkjsQ}Wrdb7487%w+QT zzL6Xtt0a$3RB7@PmNBJC-c&v^m*C6c6$)PF*x^ZF(4W(R-Zq}R7|ySxwSw~rR=m#~ z7PAX}ZO0S1>*}`HB12h3YyOl=p@GhWCOK=&I9^W>iK`vC&SNP_g?zu|lS*=Yk+#WC7fjbF``SZ~n_ole&am;i@>_K^ft z8%pb$Nzyu3IcT!E0Xv_(T znmJ9yuRt;g;fbOkF}T@#)+RuHLEUd^=;z)hE?*EmW8l~RtjJUhk=+LLN|s6oN+x1azbUs41c<$e@IBbdSj2sv~a9$F`Y=F$5$I?x)jwm?$3 zF(r?Xs6*Pelxf{ZSr)jfC&pX|ot+ss2zWx~(LXW;AN5aoNiQY0fp3DIEEPx!dh!1x zK7H8OxkR~uDu;Oc##R+c%^3=pgZnBDR#}o5=${5Q>r6o+=wP-1xeJBgaIOWD&C~o< z>rXUIpOu28>1BN9?~kMCJ`7E>2pn4{33$s)SYDmjgk7vb(D4aG!)Zf-W-zPPY8bL* zpy!9Uj3gQL7CY6quv1I<`HQ}ZTzs}&mj%gFWlH3xy9S|f4d0Kk0cn36!WKKfJ@Ki@ zpnX(H^ud3UsyH&|GMTK>>EI2UNzh;7>l!+1wkj$nmJ0LDRFvh<(e5?^9nOT_u3x#& z@>Gb7LQow+cMW&aL7Kvp!yO2G8wZ;D5(f@!w@~v*HS-(_U?cpeM%Il4E_UkFD6V+Q zd&{e_A1`Sp}&{~5An zLjew38i$j>Q$*rVo&7_0Jy=1hcR%j^3%oIDZKxyMsNHT#oBDViz$N1ePQhm=vU%jR z9(W%-)Jz*3WLff)1aV>VBie`^yP>idieJ z`m5V)3iGh`#j_oSKfbY)#Q9FGy=9W{gn#J`5>CAux!YdZ4Kg(ry#C4`hPN9vd_=>C z{$ZCKycpeNK}X^BP_grZ_e92^fD=vy1AtymzJK$0PmUI}l7Olz|Am$o8@HQ)W})o- zh~5Sd{|s+UUb5t%mkbr-F0^kmk6hk_Fko(%*vTv?J|Klhopw)EmqQLpQt$)~d z&s+oaK*ugZ+Nx9V*x3o$scC6n;TrE&*4rUCE!nqo`pt-N`)|a83IALVc-T1spLf31A5R$n9Vw*HCHKNNk&$0~_k(ULgvu3A5-zJw zKo1Qc9X}E7)Z#5;!8L3d?xcdXz41ss2|Zns_{WbG|C`KqAjBj6-}}1tp-)E*QNHnU z%U^Enhls5e2+K-Ivl!bW3P&$`U5oAY7Ioltbp7WAR?dt<;&Qlvdm1xOMqrHD~4k zU2lpbmttpz4v_B~dpMK#!FCZKyy&i67~xVjpEvb0XWDYA#O9;n#;Rq)j$LVc?f)Q$ zI_F?u_ka!^iMg|HG6=+n%@3PD0)UYtUdk?%6-<<2XIkPCN=hf9_zsoe-pi)%0a;OOR|;qptU?*~lX$AJ>BgyH!Nw*WG8!=W=Nks*Ef&?h42Q}XLUJjwYo_KNGcqXgN#>X*C?%D^0E7%ZOg(*G@ zkKR7xM%XvYEk|*KE{1v?7k#_)LsV+3p|KM-GXaHT*(Q+uHC@O8xU~oohy_1xbtFI z_5g=?@@eRJgcm9Ga2;@ZQ~ILNm?xPM^*}?Y*83c0@V{&DrKo zXHwF^FMs|O?xPFtFm1ho~8?c0i{JXC#X}=Q3T^i!LSp-kf~rwPKJ7fn_{34 z!7h@fc}7m_>!ZTSEBbAi{@jV@ZdpR*yp&O3RY=tV6OJp=K}_IkdgR-D|X$G4fvZEY;Pg$uq9nL}aXbbpHHCMvqLN zhpX}{={jRs{Jijh0FDb7Yz$uNV8mdr<&YP672)x#@;iNAU^b1}y&Cj=1OQfsJ$Po5 z`>B;i$v#In7`oFB$`s6`Zu4RPb~N&sGqa-oXpj({nXNv9k_UrXgDo>KOjr8#*6XRNd1P}a5H z`tLM%_+nB2K4bwUw4#h5agmJY>%EO$;Z#B{4nTYn!rvqvkPA1_sA>|!*FR-Z=BLAu-enXw7THE4Ye*Yx)-m%23#^JOMQPtSwSe_` zeHY^i@la%wslVmed`K1!Zkds%cbW7loTKpxLKT{%EZ@;K+2#k=|K?D-okBI0_lrKi z+@q89620dxU$ZvJFuquyEmrCF@E>?GbZ<)myWgu<3(yI`qfjWIIB8cti30N#L!$f9 zaLAl4HW95`cf_!|Tzzhl|HHSP+m<+8k#_RxUd8ucsD2O4Rus|}Pl&29e&0@i`8XGI zGjdFFxQuD2SGKVZ!`wKzMrR;?cjuE{q$9=8Xu5t}UGPcNBHiGt<|^SCXX8`+JXZ^m zVHwQ&27j*wf6Q__$7$^diYEx+pYmM}gX$>g#%~W;oWfiFrYXQTuKnerl{K%$#nnQ7 zzW6K(gFCPIZ}u;PUx4UgF3x=vIujjV2Bqx z0$KF|>Vl&6b*CB5oPa6O_4N6aw(wTCF$YV>^yFX>r9+nyaQCXR5sls7UH?)bctfIE zR2aet)DG4_O;lbaf)ZwhBp&jIKK&Q^AS=f`x{|WngE#@ik&~R2MZQd9E;6Zu{Yn>V z^_>#fNudC2JHEhT2%g#S5Buhy=>3_6DQVy{sHi#}7MN3W!o;JSTFS*rV?FrO>X{H_ z3Cp%!T~8ZdZ7M_-u`h|8PTB|2%Y?{jkf6DqnM#qz7wAQ7I1Tb`k5t6pIQ#iqlP@my z+%Sj1j#}0@M)kVdH5he^*M!GQnHn<9UMMcZd9eMdCs^({$czzH46!($Z+D*i#V<&) zG>gN+$=YTN?T(_VOn#LnxD^p*^BMYkl#+@XVo3g7R8TlV@m(sD}SeWYmwrQ84&kP3Y7Jh4D10P!xeVj~aL36M1 z$}!~6M)khnb|SB*PF3W#QV)uBUqRuX`+dq2Q&eGki5zk0iisOu1^=ImA>$VVdHBsV~>BJjqlwkT59asvaF+n>bc4E>PkngoR~u-7W7$2uhW zv<_Qv`&E;&3&e$Gb7AW2Osl^bBpoX-D2vjS%Bc-t`+M>;GT%~6S->k$oL6tc9i|kC z^a`tvR-eOadsQw$=_Q!T)Z7&Ss$l&!f&Gr}N$W0d*~&4S!~%Cbn0Mj`37+e97tV!m zcQE-L!k`lA!EcJ%-sbmEGYM}W*Irh?B0-0ci3S#B3)c?<)1*8!K+C`a7Q5N^ahH1R%km z!C*s{(Da^T{|%uL)B|vkKW>aB=(-MVwqfJm6@}TPd+ASQOy8-CG*UiT8*S4q^%T|( zklZt15!-jTlLgj-(6~^WI;dN0Y2R;3OHwbXhBNpQ$SNG(l8MWr(N+^=B$1?q1+5Bu zY&*dW)Ky*U;TCuyg4Q-Qf9&~Wb2Y!6b!%EDCR=PI5Xd24ig7%&P9xVbKtvxu+$$q9M^=U7ja6jasiJ3l&%lpwN49uDq!>X+PdS0c*eFq-hLR# z22P2$rm<;#oFRZWgFX0XzS(7>$Q-JT-cAuX3Y-91%5QQFmvOwsTvH^uRv-<4B^iDG zR~E&tWT)n?qo7iTgTI^xk(Jq|Krj$+uLd7%PQhp5>pkThJ)PSfkns+{NG+xQC_?Y% zlgerB;*4icHMT&mP}BNEwJRC(ZEM!AS{aRzvb~E?plbMu8|Jbht())>C?_cAfV+gr z)#6Go?FWxyZq5))pl$RpS5iO>8<7YpknGGzyYLw-#9liK^OukUK)GuJv{qP9n$DoM~%(;vCwc^%UzC)4jAdqpuUhnhaH2#bh8R^RHwP_JSsaULd z3f0ALdOo=-=7+i>l{1p5Q99SSn#It({Z4fn3G$RW*k8p|@L+pGM z1BC8Z!d>89;Kj35Hx&RJN5~C3Vtv7hVJn!K$7ij@H*8b_o92pDb96)_%zPz<;=T#p zM|I&#OLjtC<+ZgJ>p;&)b%n>KCdwtYKuEWo8^n%NHxTE0DVE0-x&sYa1p6jxU96Ex(=dBKxH+CJ(1&oUGiwa z#CMQA6V{Y>KAVbK+26Yq$a1081Es18&^1Qr8hYc-FyFtcj_pKnDClu)|KNs`L z6%K(cnp+&0u##jHR*aQUjrbbrvZ~N)c+G^v$^Li=ArR+pl`&Wh)Mb3MzDPwUG9vjE zN;D~Qn+_aI4AcepPR3M;bK!>tTa4`L6k|cLIwQ5OQCcP=f z22J2A8EM)dJuncn#Za~j0laoTQTXdd^{MN8$0g`~^R_22-+k19=TxrxJ~=BeuwaZx zs=r+A^$>0>?jd5dmz6p84x9ill62?UoOhz+wuWj^x2^srKrg4xU zFeXJl7Nyh#0H@R=aDPb8!Mv1QCT7$^yH4~y!o0HFUOsOI-XIB51%fa3(D&OREvFDh z=*&15x_*Cr`;4FJGh;lHo~W?@xIOG>2{tm9{YP>SjdSnLfNFO|@FHqBM9+MQXH(aI zQpcbpSHM)hmV~8JDI73)o#3B#lgW#YljiSXsg^eqbf3!}8QlBlHE$+amU|$-H#37$ z2Y3Q6*2h@7fs}+j$en#%j*-&X{4C!AZ>+t4Wf>T<`$_SGSl1w;o{{vEaboRTtUZlV z@g=}>>;0c@bji(yCKMsCA9sWV!?;TJf(9itabAqkp+&?{H`(IX({{UjmQ39bmHgOj z+ejK4M3(CcC&D#@mp;MX%$^6a^?thT5^85eE0pJnpCndqV(~&6AQ%KM*}J#1ibx9` zrIMQ%5~7FR!f}E)-|{nyzw}I$3={2?Omd~p2A+P0rs8(_wfTES^(&b$w+Qo~#O0`g zX1ojz*NfXdkB45w6~gw`L-VBOtTl%qsE^qjon30^rocO@X1#zr=x&eGpKU`k1ZXx# z0))gXWbA;P(ri-`P4W316;|R*##S@^!S3&v$ih=WPuV8SH>55d8QWpKx_=&02~(F*MouNJ!GYYFg9}Y z$a_*wr(Vg+gdnvkV$2g@JVDqmb%XcO9Lp&ps7}kM%zsu_sx87X_YZ@&PZt`N$1hb9 z_if|C`To{YPv9(=_$&K|f5OtzK|%jroYssFEX%W09yh+3MNz%AlpvZdcW8+X`F^0bxxX~ zg6Eex&C*n2iK@nW?LlGYQ+45I_p1-n*e8EIpTM$dk`m^}Tw8Is-hrHj`#*y2+#?qo zviFp#tPbO=pyp!KX(~#bGjOR*Y+g!fXZmC)>E3FRaEwlpK*Wa2e_g(d2IO`E9$NU9re3d=)|A#{PkCL~h}s=8=zgaFO&` zvZTVOrFtAR8VY+Ra$P~y*?ihz33tfHoFB#^fhxC$`+@d7L3OB#AKdEMDm>YEH%#Id zW7HL5e6^U>>P*#FC79rE{Lp;!*@Pk1k)mSbTyzo+;%RfJH(uQY=asPbHwYF;2s4nu zombOxX;KZM-+$GJySdElzIZ>_Jwq|TI^{HiPUX15m4zHcRdmZj!G*7~r_Qu4bDVLO!9G>t#++VLSo-H&u=m(lGctOIer|D-f3InO z?^Ol~q1c)Jxq$Rk02#jNA<>uW?40qkTCLPDLaZ?|stP!OCE=j^@l~+^I-@omo0}q? z3^|}cD~2c3-b#qH^5Z1b$Ia)dS|=0**lT{@%g*eQp zy%j>IE7iBqCypL^Vbxs|qCZGgtENv^fCDo_UeyrSy*4VV8*Rz!8^tN~W>6#KBDF_) zo{x%Z(RjydnaES)c@H{tip1&~mscW#&8v6$dOWGYL(jl)Osu3Xc$JR%=AO-GAJFRF zAA*vswn2I)z|#>szG78dLA)F~r4y}*>7+y8!<>-y9~;$A%jRLtMO@hiV+!$l$l$aa zu_e2Uz8=-;$?{X&8B>y&2xzEv+5T0Le&izK|6N7Y)pyTDJ1@j4wDTTN`ydnES=7IX z_|bU1DI;6;2O5NaeaowLyD}h1@h5so>W1J6=eg5w708Rsx9+{fxY9wmt;kY-`JMAyVaZ2uPd@Y^QrHUz4g7%wTk zbRM!et*Ybk!L(pC$=`OgRD)Yvtmh_P+R}gJaQ^t`=L>#(n)9lnv)=6fG+D?Z1T46)kSW0T20pAWrQ)yZEaD408Nhl)@X$v4=>CmbFT;0DjWz zL)W)_EjMwqI&2<91E5uYOg12ImE{>-zIj6RHC@GlJGkA_d^iouu*|%ja4UV)N)O{y z8SP<>(?|JB)|PI6*mFd;o}JwBZHBsvd0z)M62U~Pk(%Ij6uS2W<9;R1kTu)E0Z zaF#UvzL)pkUYr|tvgjStgV;2;QcI_;lmZTnrS`9$j27$D5(J?TXMUFB3bdnR{LKS; z+KXHA@5)+V7k4-}(*_Dzq-UM?@yzt=2;>g7M2rZ`n)bSzO?(vB0Xo5n&@92Uu@~@G z8aJp9NLpWerP9l*TA(fo#t2_FB(l2t!j^@4q${ftp1bmEkm^GCjdMJf#btyI3v2VS>fSu|1_c+g&A(uGzh@0P9R(w^FMhUKoMEXRn^a5Y}YT?-ri;%g#fn5Esl20 z8`2ITv-2A&LrZ%rn{;hn(03^em!dh+Kek6Puek(g7z??_EaYsJ>nO%nCFnab0OzutAtVNmWTdc z3wOKwKKAz+ZMXE6LHGSMyqEoxFIFBBNphWvp?W)39e^53e?)d(9$X=Q6XB^=_$z(y zehPzbb1$!4#>aYaozT^BxU+>yvX@(H7IRHMYKLN)ozKhV+|&Tq1!yI2;|Qq9D_^=z!dQO-+}GEi&70`OIKr{lf0 zB+Xu(cOyp#0<^o)lkkC@#fMExI}2J1#67jeB8SSgNFi05-vHxeZcXQrg#Nz62IF>h zZrbe#n~lwTm65IgWKaNHPSd}2!Qk%jFz|u$p>kDL;23;=ls1Js5;sPTlhbxMT@#5m zMw-T_ldNGUuYweJwPCuElBbuXJ^H>+N0bL1?;Mko?9>LHG;1wQLac3dooE#@(KIE z>J(J5DTHP>B)0rU1TiI!+@9pYx&rc+6ohCq4`^RPBNI837GAN9jShKK2(ZHYvmNeeC@cd14 zPDEv84%e*03uVQ$#ZpxOOF*>0hnDz{-X9o7S&J0^$^Tv13yF7rrFAvxX=$`t;LW2|Fd zPJZ))s+>VQ2g;s%YWnB?>fXWmqY+7+H$lF1Qob~zT|C2Z6^8~=u@r|P1Hh?KJLNMV z5H#@zUt6gNY*spb>S9fF2oe!$DNQdyQq84x*%Ot5zJ;7Q8=afKd4kSG73-~d;Ox2F zwawF|aQ8fwj4A^t+KyigEWY#>6Cfh8LO3eQibdjehx|*2ctl|celXeF1~~7Tkhsem zl=%CzJYE4Rxwz*O{P$3?pg~j^&w4+-r2l!U%wELhb?GojHE@?F&6ZuR%dap-%qpW8 z$IZ?_y_|Ppv&S7`ooLVT29QoZApRK3)Uszyl06qbc;&Asn}Gct2I(yo(d3QCchxiN zu|_O<+~+#?{;R?H1&*_}?t<)`xfER!bTE}w>%;N{ggJ;ZmZB+gcA3y2B$aCfVJjc~ zfHPHqMh`D^3Rdi5y_3E47ptY)`v=O}{TU88t7X17m!X6m)dp;1P-BJoM5#UcSZ7sz`wgGJxL<7v*v^gJpBp+PF#!s@#_+2gXf0aOhuvn_|yq*=E|(-STVa>W0=2} zA)yWzLek8LEkXajAH3?obt(<9{0SCl!?HBCq$=pwNGdp=;mL+G>4j&={Se9SvA`f* zQ%zQQ#IJP$A-H#yCO+iT*YxFUJnrn;C#nl5>b`mA2OL2t-qHyyU>1Ona;j=QdGbeq z4tO{djlN~e8PW0mnTnFT5drDJ0@j?+#SmScN1k^B+^RuFz(LmZtoVmBhIYkIz+#B? zZY@A?+Cx;c&b!@Be#w$*xi$2h(10h?8B?bI0O27pQ~=sU{GuGUncC2qCL$;*T7O;j z$A&^UFcynl+b3qOs!dxN7sCyV`FS?%!Kj0qqRY}+dl?(O`1T`{dGh zE)$@F-a>6snusz1Cm6i<{e)&r8ym(xjzpL)?wHT+7|V%Nar==ykv>U+;yKum&Mmqk zA0-VIhB*HoqJ%v&Uba%{1ro5=cePkn)(G-JN4##?*Rbx0@la{7>gn(%)mJio3qt`5 z5G9GLdh05M+B}f~8<${tY;HAa@@@b{Z_?-s9oL0^WL2N_apIhDmYJAz8#d?^4>p_- zp0s?)y9G&nieOq};B_f0aHFe3TQG^{aHaa-=-u&Xg=PVmdAwnsyNQC(Q{`?M#g#t6 z>Nu*{o31Cy%rA+$!AI2z%RyN@|vV(v;*Q7vFe<_$a_<+JP z4BtUjqRbq8S$yJ-0Ga{eVT6KGFdgeaRs942T@tYQS3S)aD*}Eie&BWbX3+-j%gKG= zi0J{;37ZyWo@Y`Rk_BBEDtFhyEv}p4;IF-3?qrJ6&{tcWO}$SrG4Jcay_~{{$xhG@ zur5~*ih8%Dhs@PtiFr^RtKf5kr#6TN*ZT%1FZcpl&|FkJa>TWV)nsN zDWpTc*?ORq0xI<0_y&lY2QZoh2`ZL%l^?%<4p?j)Yn64dkiRCd<~%Xm?qU0k&H*w< zZLi2QWDXbou(Av2z)Xt2M#gZE=mZ`U*s~);)8JJ&aX&8^Vyz7Ed3rrvJVE)yduo_?_Xdpq*=3CNKfITfEI(x+Wt* z+NZQth81X5!v2h?8($nlq}UYh0oD!_w=XIkmsg@gBwS-xup?8oobOQ?_x^b2(dof2 z<0$08I0(Dpy#$Et;;pBkYnD=_LGsojS$T$=pHL(c{v7=re=;c}?}A7iFa7;mSdkB} zx*dBnlS`d972x-JOhRQ$LA)@x8kibiAndOf@}Y30rsa*1-zT;iPL(!U2y=(GRNa2ISwHVR|i zrx6UmzhSa#Mw$k{@tu2be|jxMOCkH3hdoN>RUDiW8+1$_#JOg&203@lN-+xc>ox&= zo)IUS9f-cvCaX|0u-pY zLZ|hKoFS6+A33Z$+-)~2ZN_mN493qA5l9F_XAJS<^>3ltyjyCvd*C(p1EO{))^w#$ zTt)UKHnHcy{u7Lp1dJI=dKVnqzbj;mV&ZpCb5HlsADTwn$3qd;#^%nkE(Db*{Q!U1mumdsFU}49@_C)RlTHXY%g6FC3A8J0I58OiEp_r-IK{ zg~+XUXRqR{wH#Pzf|9h4miu!@KDQ?r@Nn5J2rFr;cP}8YuuD9yIuQs9RC2kbfA?sxsr(Q3?15>^E`+wO@)| zOOPx1rE``=9u9PYIKW^{ZHqf@JIn!sN&2+B`L>RBEAQFiFE70(SR9BTsV6%*1Cqa1 zI{J@8Y$+YGQb1Z?B?LutF^Yu*WJPORIR+B5*kqC0vUm-mE1{1r@SdB+;E8USDaB+i zj(pyKO$h%cOfd%im1MSoGG^dUw$IPqzf1l6wJt@LOt8h*U_AeAlBY4!y$wp{8RBhM z{Gs{@t&$xb)II-F_(Ing7F;hr>*a#Hli(oo2o(J~lX@lOZWIb)uyYO>48M(C)BKV$ z^CVvzd;{AQ(Lpa!L@|W};6Nv1tH?~{w3xfvr9?67Cl+-H+3A6SrmQ>fp8+bW&=Tb` zlK_Z~7*}EDfeGh}2WDZA+y{dLiA&oW6lIngf?3R2X*gVSx`>9<({!*;Y@|^;oEBLS z!eaO6jtTu70|JK6H%!4REvs+_9#fuL2R>%GmU{0GU0u&vp@!lbQc+wMAekXEkNK*< z$-_yZ6Jaq0a~D%>lB}g&A$EfgERAQ$Y2)JWZ3{6ukdLuFDD`232`dMJaU~7+B z-k%GJN2C3?S2xsheXahbhcQ0ie;c3@*fUk$hTG-sVJWwNl73sagWSfCRny@dK;E=W zT|F?r@5$sF&-+4Ui)uR`!EMhJpigywtD=-+?BgYXUQ82O(sg)*4OF$2RbkHxyZV~p z>8W<^q2MlIRH&cbhc^(!Wcxd6u&r(rTXnhbcLY5>*rB6+Kdenv%-1^2JA0N#_a@ zZlIarTCvHw=bAS5P3n_$51WAX7cFC=Yer;42<_5#e$OR~y0_)e#DrF@e!Lxb4DjXh zT!3Y1KqC5gmA#VJ`>^^#Y-3kD*>_)!cJb;w6U>Sq5kKcyYgWT7_z$awJHnI z{5XyqBODQ~aSJ(n&f9>IjCnA1+~I{?iLa|96@aeJa2HAl4QlATh>-*gw$igu>gSxu zD90Po_sg?I{B5Z&9!Sg27d$o8;>`XGuT8s~*}%|0z2X^}SKYmbIuK!8HY8~1{|54m zpyc)jk?Y;+b8xe~DKFv6iRAB&g(;gEOhah5r$!kBm%@#n5uAiv1S|k{CGMF;m)iu0-RY6 zLpR=AX<2)l`ZBPa@o(TUwU%RM`JIn-q(|;f-#9ZHlKmWIwRiPrwjT?Pql9)3f`0eq zs&%dU5^;0vqs=;BKjM47&ADnS>YAA2;F!yqtsOI(jI?;%%-Hl?m0;{CjslGUFOqv{ zmPg+AhUHy3XRb=4e|TK5Hp!}3rOxtDYsUFzi{waO#nXIQ=Z1R5``+z*YuOBiROt|| zlKPs8P$?AptU=0lmu&%%&-qyxUP7p@?j%;6g~N*2M_xl?g1H{=2FBiT$|pLdOpXj> znL?2IrBPq1k(qF4uMc{*ikS2zvVu3$(r@YQHT}HaF%}cx{WF%T(N4lB=Mkr6du6^k zRRk0nX_%COS2$@kU#poa@KE^;=oFhRQ#A7wr$tibw9b+F8N*Doqeh{^lXjbVKkiW( z8q7{b@WhDN*T;zhxXI#gP2EXDsorL32YrG-z8$+itnH6lm$RCeHXX+Fvwm;#T8U_X z00mGX-qkuXaXn`73Jcn^_%=*b0I}%(%^ubiDK;eVL=tI-X@% z_R>hb5axsqcuU6%qjG%ca*HHcbHN}fuC z23d~&a}*~^r@GA9(DKiN+WLGM7yjk@kprhIi3~(9TrGp6VZR-dvk**7FLy4H8C{@f zztysZ#pc)Ws_S;R>NnRB<>xg&PLC*)f41V~DvyeGL9qBu$G4vt^WfdI3DPFo`lI*oemjQP|wU{hDYzX4prEKy6Oj2A4%QMr;X%R`2Oov|K1XLCwxUs8=Ie#htDD zpU3G@Ie@Os1_^;%KJ$Z}m94)y*V`Td3ns@h z^>Oo7`_qLnhs$d3x{A^IFzcrjy^r8JlN}U_m^nv`>zCXBAyoWF;!*v^sBCW!y92S$ zGtVup)yr=92i~<5-UQ>>TS(!i^iDz&P<1sKN9p-k!mop}18a0geBK8!%33b$5}u@; zVJfyQJrEbjWOkhnJszPn^9Ne-Va)at315!p&(b%*(66hQJ&?W0{_?*J#pbhfNuYVO zppqkGd+7SXG+vH66ePF_t+bNjS(%H*j+g@Tyy-qITKt{O-co+$q5v26GR|htePQmJpduIU; z+)^k<9s4QDSGozzLvnra^A<=xmhJ@yt$&nQ!-cUXB*;1NX5beoqH}q)3|r3890#~6 zWp#aUCt_(czK=cS0Y7T8rl^`82c_GLOb(sxwvG0dKE0+x?sqIJJS88FGPlN0UtF|QbcuR1boCy%RU3WrsVxpa1*tT!k?_T6dISik z!F6ulean)r(wHeWmJd9cFPDMeyyDNbzJOifVt^ST7jchIfCEGzx_D4RCt7#5!4lY# zF8G=CGd@GYagRIrhdc{?LHYQo9>0F_Qj`=I>LpSk007U0R}y1ccZ65wQQQ;*Tut~6 znvkC@3Xy6L$mc{Ux`wd=x}S66c%;WI1_bP==4Hy75z{ZyQYZ@uHBpcO^aeri_wwu5 z#16$>LPv8>r#(Rrei97oTHj6PJQR_t&&=2r^tuC8^OTo)L*Qht)tFNmhU%@ihTl?`_Cr z6ciLKa)v&shD=3ZYB?n za)}gQ%0TDVY0?~RvKx!^YG4s)c>^m)Ck+w6xF}H%DuV=9&envA(nY8k;PI9 z2nONb9KcAz_|#y`?&Xk5@Wc?44t8U{uI@QOkyu)&Ng#&cAKpvJ+~2Yul{#34B8N%{ z)7n_~j?cMM^J7oT-$)yGWHKvwO}hE|g#?b45IA=!U@T|Ru=g9gQw?ZmiF{M>wo9?f z=Mt)_YTyUb)}iM94)>IcFSg(IB%b2$x(Mzvzel#oyF=hJX1ue20KGsA;RtCtbcZWB zR+res-4|gcJE)f-%st=0Nk*d?wF@WaGj-Ds@!2$@O35k5R>g+YA~FS`J~OmSe4hZtM5 z{2vE$3$G#+2GMzDnT(d_9UttJm`9gP<3w7ngc$-rh8E~bOI_3hkR^iFwIJ9;T*%?I z%q^7@9&&$@m+Z|yDK9{0=aIP9wbG5caz1tO2>tniW)n4=xU@V@o6wPaPh9biHGeJG z%8iEkZiT$c9tt!37dZ5-SeAQuL6knN+9zm}G)Y(ptp5r$r3UapsVNv&9~TIqsUt<&YxcRd*?fiv%K?q&p3`1y#ytdbSbO^PV^ zD)b>7nSQ4F8jMXA`CR=De0nF1b0#ZzNVl5>_-xU}OPKY=nmSOH*wy)036S1Tnh@3S zOq9{7pU06Fc)Itodb9(Y?lEO%J~#yGdX*T>N<|v9sK4ue=M+)}!@K7}#?JWI!MFat z^f)u`*Y!=K*M#5#^D1nc`a4Qs2@14 ztHWA1jX$>^Ef*p(S$nEOY90)OjJdzLcf0BuV!d_=jcb;WpP{EL-WoC* zqL5ZZaPbDgqB?{8xXbbyF=%21Q80V3O}CvM5(OEIJM29bpPXVHNzcW5G^$SJbo*loY&S6}X@kS=d}$yc;3wsk zns#(!rpoGBa=+z3k%8KBF8b#Iv%C*kWlgVZb-DuvwYq3v+%d4272i#UcIr(q9e)gk zobPz(+88Pb!BH1JJ0r#siE30mqdeltQr6}#uQev^jZ6D~AlIYLh=ZVmR|$n9pfAz% zW14-(sxlh@KTh+(Mfxp|9Cj3_I8Uxw%@eaAFK^3POZZ#bu&;OsI&gZ}DndnVR;Ge= z_aK0QKA0X=BkUvcSS>#NDSK(gSmkW|)HR>#58)@?Q&8ATu=T$toLFOoF2GhPW@IN9CfNH@(Cbo-p0ltKXj+k!RgX0?)D#gqN)1+3#l z2&UaC7q7pK)jEP298QPjVd%B;6T}3$xlHO%d;mA^ GiQ~vD*R=8x+Fo27f<&8!7iz@;f@L56qi|Cs3&`@RyEGHW zh+03}AJx;)^1e?AN1E)$fc}`e4p}c0Er%S5PlLzmf*5}^#*vXOFIHr3Rn{Q6W1=Qg zfen5*u&zIOGQT^O;v_fA0Z+D;O}~lRx9#X9M4Nvw7!b=6KMx-0wCgdTGP$!f&?H2` z{8SGeTf5pn1_yrWd4?AD%nf3l6z@r!2E~f?!_&G3&oz5C$Ziy$eN`@&85jSi8T7`o zM@@fZyKcpW+tsWFY_4tH|7`g>KohU`-3FcxL*KMhj*`a=i*$&`C{*8*oV7n_4h*?v z&_)T&J=3|!(#b3C);$4v#O{CFJZc7lX)yP4JZN)eLg2SA>dNk8H@nyyS3qi@J|5q; z?9eM8U~K<~>@sWzjkGzU3ju5Fa2>9yIu~pO1X~0mc_WtYuY;+4Jte8c%1{M>5DX+n zv9v#kHFy55J{%}7{%M(LBx}8HyrpzH`jc6(o=G0W$4}xa@&^*~?`YH^G6tq`0$lSh z0ky#iXio&b8Hf_FlHX&XqRn9i{yVn8eyEhM>)ex2RquBL#IadJTuSGoAP21Gg-Gjm zHSIV(^km8&KBHnSrqFF9U4vpaL@syLj9D2beRu$9qvZ_S zZ9+m=28ka#3iO)ec=B}Jx$N_zFLJL|I1$~pd75oY6dZ7{Omlh>UINx%@9&o$iMmb< zNsk&62{1W~ly<&G+pkyaO5q#Jmp2D&B}r=2onB%5?pnc%M5yjF2~%dbAqjqx5EWuG z;XB(T21RRt5nACldi`f<>a&#?=RgD@Kkh!|$?j3)z7O2|3JG*}CIQ`=+jhmGdx$4A6c*5ut(DBbE{-D3av#HZ9lr7>!! z1>uaC_M$HX{*O?p-M6I#syem_Y4MoPcG(&q=)t{X5ym!LO@-cv*J~*yc6R9Gg_n0G z)i1lfUJ79?h%J*1)zKXDJO^lCZfwzp{yv9wkkZ#OG^l50NV|#4DwH`3EI7aiM}^#; zXY@=Z0RqM zR%UV&M;42~KAfX}UZy)*C3V|(weAk;^SP?B5@eLt_`oWwFI4)%{d)m`+wUubP*k93 zNom|2t!iG*>C^0xzxViXeCRB1GtgfLy)DhO7;Lw9JJs+9>nlwT$BjQhk?x4C*@vQ+ zxAMUvzW|ss^MF4BkS+7>3Z}P(nbz_98l~jHs7K1e3YrLDT+(w79lH^d53Tg+7(@-f zd%#e_7AWLruViT?hmX$lcI2S2WEmeBv^+%lG|{<%-d@g>>Ry`vUacDJ2%Ak_24Kz1 zyhk2esjMWSy$eD4R?6j9r@}9F{;43V^J!)$h=U`5iK(kYWK;|z|4f4IZH&++1i$E) zNdv8}aS8w4{h7mkH!k0F-ffXuXP2k*EVm4tm+PKE7Htzop3G=oXA!TLAZAv-hHB(g zJ^;*cWDfHxy&((4k;1a`s$;Wnk5}nIKmJi9X6*aqLQ|p^Q)~~Ov4Pe;>i7!=To5e4 zUv)(P)=e7VXRTQcbiWZ0j;J3@t2gMldPlA?ykFVB284~oC6T)C{42OnrE<8QWE}x7 zL)@G~^=%Jn`2B_ikLkQFsOz;X!9sN)7IOUVqamIQNeRB%VqU{r>2p`V8@`b2= znqeCu^a(sW*d#@^K)Fs1k#V#;E9;}<=#z^ZCH5O8(TxUh=@JYItS9uK6#-Wz*l)B-!YZ^V}#Rkc$Z5o%lisVBJHMG#E0z?r~<>=lXK47MM ze^r80^i_he3G4AK+{=!}Wozt}IuM`Qp)xNd5q}K!o9F7BuI{z{WA6>&t|DOS)4Tnc zQI4J2kN5_sH*@f#sx`kSoGrNWkyfb0a12`)gq2Soh$iZFlv~+GrMxzz(+`lI?9Qa) zUi8epM3MbmNc`pkGbB7O&y@Z?KL#-o+Y%7Vb$5Nc4S{Q?qY$Wj!J!;IiKBt^;M5}M zygG@Y@g=4bUIOJF8-7RG499}_4cPp~c$FLt-3J6caci`?u$NoXr$LiSEwIN1zM+2S z_zpe)a-{OW`D=#zLQ4HW>&ZmL19s1h%9NZtGvx6MdOX0D+>Z*<8h;WkiHWH`@9~w* z+}mCbvJ|jz1+FnNL7D#JDz=uA*LFbg{79lIT#!Z+#pPQ71R5#u-G>lG zm5JsCzWVG}UhCo2YbI2;0G{2%Q>I<=YfuGasb*feuO>gyg)aVqx04xleO@}%-!z5L ztECM&!B!L?rS)0|87Jb@6-$FzR!S}c$##o-n5x3(To--$Dch22FKv80ag0%LdYQsv ze%ob}aD}kR34Qr5%din%{W1&(s?ffIicD6_+b7X>fyywy;L>BKs716#a@rxv%9haR zdI}J*1)k1;Iey9*gj|3gb}UgwF1TK5_Q@{KT*pAT?Gp@)LF+*dVj<=CE0;HVPIM{= z*$cJzBcas43XV)KF*?we3`I{jgb|A4JJ>`b%d<+w)n!rd<{b#@`3H5OI5zS6Hn&{q z-XCJv=&`$-f`kGBfTQ*)v_8vmo|2!L6>(S=qmyse-eawsPT}#p4rs^2um?ODa(X_^ z$2VU=JfN7-iFWL|;UrdeEc|C4^ukXu(iLhn^qTjK{T=5lo_RuyX0Q*f2uKYeUWC#L z;J9*FaPJhYLN@_sn6Z3zy#>tFOK#{}d?@rn1*A_-+lLDsvewUn)I4+CPxPB02^t7{ z#-dvhjZ&ghGi{U#fN`~WQ|$pa4(llKhMQ7d|=_xIIa zE}xM`1WflP#3xMoo)v09Y4iSFsUP7Ex;Z^6$P+~h+999t=m?7RR#axGpubAQz;G2{ z)^3KcJvsQN&3`GdM`+QUdsspfaMG4y22;D=rtgrB)Z$>}|oI$1oYa+3rD})|tjnctD)CF`CVy@bcTU z20L#=Sn1V^02)0*_0vn%3^gTp?vM$-+ATY~HqP&XsyVt2OxnWxo|a8L!&snGZ4+?| zA_`vk_2+ZQeC)A)K?^gJ=XoA{P%r%#16Trq2HNk8a=20t%>yFM2QJK+G^01cV_ZU( z<0Ju_R0G8ij;0*}OXR;OTMz@URh_|;X#1p`F(akUjKfBn-&)}}?35Nz>~mBAM&OZz zA>cdVhz{Pyj>ksHqH@O4Ym&ul^mB1xqz{1K!p*qGv4j+arS#o_1nxZ(BF1Y-f%qdS z|8RW7>)Q5^$nwv+x;q5Wyf;KU!j*R5xk*kt~run_VskW^R>B9G5_#`d**Qe0j+7$a9wL;# zSKjeKBdaC}0w$mgd!bW~^8PQB)5bzWk3IBSO}dy-1j`XO4f^vP*6*SB%hzbl{NYXL-b(=z=; zp~^@Lm_{ocdm`VqVtsn1v37@Gv_~;m%&UnE_HnUst!cR^>&$a@XyDZmM=TOYW>|?| zkUcT8&rJz&&%lIRUI^x3pOKMvmD-8}Ur zP6&=S53IZGc(X?{)?f%NKj=8XB?=jR$C97I=35k3x&V7Mn&@ks7nBPj1f@_!Hvli| zs#(nYluV4Hq?xJ%rR`J~w9qNd3hAK;jjUr2ihOeloT!JiFx8za%uf{@LOb0~mgX5= z^h2$mLx0BAfW@x)n_=j4+;w-<1^UB!>*+#KL`q$}8`wmR?D&oE){du=Kzd}m6vOkr zw{4fXr_O?9bJm+jAH$f#vaPdN%KdK;t~GJ-Mh6E8c>#O91!M>DKdl37#0R{0)>9Rc zel99~vjAsJ*z!h|{^FvtdgYiUHeJ&r!LpI`cQ$P8r+0NVT1ANvSAtT~zz1@VB2^qP zPK^l8w;RTY4`zBLqY6Uc4hK;O5~ykP_O?Ll&5AOOJ0l&Xd*Z$>0Z-0H{tvUTkmP^* z;PSpAZo0Kn4Pc{SO8k#;U^}?w8>JnZM@sfWJG-_IKAzJMKLUOUQ}XU~gP@tAYwHIK znPUxj4G&mLLOMM?bCaJ4$Kk~nO7p(#?Wi`y2ebG4^OGJ;F`a!1f3YQXX)h*I-JC4* zz9q^pPIO(&43jwFVxqO(; zHuYd!yAA+m^Ge+W>}sta7?9P8u}^aa@y~!u_;PQIf9r#k>yavNLfo+kJ5blN2bpLF z+NOH0+Y8~%mCJ_-r+4JeyeQ7D3VCRJTo2V_=yQ3wYV<7yy#309{CXrR^7&KAE=C%&x9CIg$}#L8CdP!odmkM=r+yg4Xx6+GiN~aQv5{KWOq0um$kfY?Ix& zEYxI9>Zkf^f_xznlasEBYDEMfP4{B&Lb4Ww(IS{!Z`%16fv# z)??Q0(aX-AWj%-Fiz7?8$)5QSq}9K-%acc9Stz8Gp(J&GmJ*|J4N0e=4Dhws7bg#@ zTc;joFJ;}Oq9WZ#NrBDyNl5noadJ672j>6U;@n|RV?c=G!~$}oo^F2s2vZ9ce1j|j z+TE#y{w9!T-|V8S1NQ#F5L^^k+bH9O%7>f7Qv4^{^-i9>#0@JOBB ziJ$+}7gCuLh_^|1>med-k+ z``hHk=1w>e${=j8pMx*tv*f%cn`pMH`dhtYpy~@dy~Cu7aU9npf`sXUKgJZGb!mbs z(300#F#LW={r9P4uRIp|Yu?mQ4ye>;kWJ>UcgSfYmc%h9284LKMtHi7z}%1c&<#bP!F@;O4= zTi%!mavl*LP=jK}y*$2_Z*`Viv6czDy*Myt^ovgZ^DCwl%&+*3qg&czEhKkBJx^9@iG}t;M8xh{Dv^$ zbE}R{Bw}UR)%S|tfL6#nQ_{~Up^lp^syG)Ele0H$A~$*e)9Zi*Q%9MxlmQIw&G}%| z@8bh9%#&Wuh_cL0h-8z$OdHOVcANotlS^HRgK7q|hRaito5GFgY(v&|&u+Qp0xo^Odn|g?y)+Jk?z}OZpjZ4ZUh2^E=#M51tFo^w=_{T$m`KQQ7& zpHQG1+)49O3?06deBw77H!)07#8f4mkX$`xNx5(Z^OmYtam)FyfCgB;&nXKa)U~g} z{#-bAYCwuldTU&PE#7qitzrzdPV~Osn|AJP7BUC&I<>I{_bQSG^zh^RQhXP!%Dsck z^yt&JkP7D_tYo|$o6!Kqlpwj^uj{Vr8~ieobpV+9=hB2 zm=@I4+2A2}-1)<{z50S|#JX$iB7ne?`!SBKCk>7et0s7^*qh#HF&Yqt&+*Tx$x{Y$ z;$U8f7zhVKxEeX7iAyOY_pVolJxY>K)z?^|ML*`=Q zpl$HFndkm(@#|-U-ab_$KdR2pu`15HQo-stBk&)Og?dRxLEq#TT+=Hb29uA{FNp;S za`9HJm-~C*n-Kg7M`8)U)8PvX16Um+mc(1a+wIBRY{~g5eK^|^mX;iy?KvfAMkaFQ zo$gLFNbUo(Ne2NQ^0ZiyD;p=MVkoJQ$kCy;_nQ6R*gHf zrG7(@{1cO_`6*2EP$4++52P%JsXhT)+E^|WV6DeQ5ahh32jKH91p?fA%~P@)nl8YU z$aC-`FoD$HnWtag&;3F)LO6KI=7ZWR1vmXnPnURX?>x4{TStTE>K=%f6{s}QyWzHX zWe`*l-IfSk9uw2RA!7{*;_DOIn-yeQ(A)`NNSKm25a58IM*Ow_Mf|JtAa|Ak?&87MIHOQRU;69_B|I$1id3 z35q9Gyv4XxhTGB&pJ~Ql(z|k|vN68xXNVrw(j#F3_aIxd5pY78fJjgOWLgT+i&H%S z3@rlfarD6t$x>KR(Y!KKM(TMx9arkuPa343E8|67l?bXwDyZ^i{Cl8=Kqn)iqlOae z10lzeQ^pgHu|_v4Am}9=m-%(VVApiE;c0jB;V~P{VU7|Nfw>Fa+2|60^pG^xMK{gM zqBXhW8Q89>93UxA%+Mb}G!M)VYAUit>-%WD!Oe(Q6F3J|p$h2TONTs;Doqu+!M?v- zR9w+>v2s~bUEd8Z>X)_s7>YjyCfDm>Dt-m0z-`1Q6XGdAw_-BZg20aVbLd4}r=2LZ z)NLi+)@HS-NrnwJuxc}$1qk`?&?-47-_q6J8U}1JaWI!$AQ|kC;EF6FlCBonqW+ixUM?t%ee{PIu$toAU;d@>adZ1oK+leQPXOS}o5ZsVUV$io zn>D=d0N`S)pdm~&Ik|>f>c;tya9SM!NYt%^PvJjr3ufLKBM77#(%0*sI2OV(Tkj*$ z%u;4lnoX?$kGur&k}{Iq5Zdf}83)2Oq^`By=v~4C|2}aM9bqxBpo5A)90j{j+)3o% zyFfi{%;l38LZub*s%9JzoStUu50O8d-8L9b_H(N3!#!dLdk-fvoR z>xqMFQ0F2YZ%5LD<1B>WxDx?y5kc2OLc}Yoc6a&+VEVn)Go_pz5aM2`#71^8M`UJb zPrG0rox_XdR>sd@J9apRr-QW1gk-i`ru@-!-4R!)f1$uhJ&sbwy^(?y*MCVe7}6r( zOz^-wQ|!M%KY&%NJ+!3k^%U$A=(&!B`gox3E$~c$-F{AH(Mjf3Qi3->F(xdun!XBD z9u0yrn0SR3i6Hv)qN!6k?(ED?JT!%_o#fT$eV+oo+m15e6ao_N%&C<_$f%rW(t-_0 znbF8izgu8E#Fl*3*ic=_yElfZeNaj9@*RoqiaW59miOIEUO$zCpLrdH<01)aJ`F$u zvKM475v{47ZcZa%A3IQJpAfYk<9rW@g+7uN3!&LWe8P- zwOuq=VkPBp{cgiVa=91wUop>43+b1irqR@cA)KQh@LL#U3c2)kzrV`D-agS~H-@-B z;&Le8VBDuKI@9iu(g~4W=<#UZyou*G7o`~;0eSOom}uOVTN@tHdUVu~W1JHwGmY;$ z_l{8?W@F4b^UEE*a>FK|7H4S+RQ8N)2q-wX*)V;7bn=RR7KA%%J^q>bX0eoFa zA8RYTC2Bk=y!Lp9@s5k0vWCtleEWUCpcQ~yQ|#{+fqx;Rx8^W=oYLEv)4jW_$^X9i)P(AlIi z^)VDvC8z3LHS3ppbf~{jt&g)&ZG;h7+uI{jM~(v=!hg?Om44q*l4*uA&N~lq@R(uH zdYNtYkvNU?y36Qm!~Kghaj!;u61)yNZYM$3?c zu6m^NV#wgD4caY#gA=M0`5?r8#=>fqFmXW>%sR9wFb7Mr^$Y%VGB~*Y3N-7+2THey z62PW@)z)h~ovEl?1f4xvTuQJ(pJ~F#((T=OqGS0_NfHkA$wm(+O_+ta+C;)MIVw^2 za^B=$?jm~|63fyDDrx--OJQd7^8X;%3J~bT4l;jm8iY$VywOQ}n?753s1XnqFuxbc zN={k?jzrS^7IHLYdomHp)2|G&F=X*&)+erY`dpt33}G7q$KgCEcR? zPV=;ftZw>}xAZGR)zNp2XxNP2WsK92Z>7el0*=Cf6qzVzN*hweZ62yi z6`;5hEqF$R-R-ux}TjT!k>_Rw&w$ z+99mB{yyovx5Xa65X1SCy1H}a?`J`8H9dX_{_WXqg^ubek?SIDr)SJ{Ad1wg8sWOF zuRQ|9`hEd6rQW|RI3Y>&(N-%E3+o_7VO?sOVYERVr$$z4Y=385lI=_+2W#jiXQ1yl z`gdCaSgMzD*r{P{#*@+>>WBlNhAHi25Fs+a%_1ZU@Go{)SP>|f!d>{X%}o*D0Z&RU zrbN2ebt-byzrz@<3=LgAP}v>V$&7g<#+8Z;+#*_*#008>uVH`6BuRPp5#zTs{YQ>p zsj}Vh7@9i*4MZ`kBTwFxb9nL=G8|`WkNx5eAiWLVksW{B{H535&|dCeUhTZRz69$o zTL6=+O68;;hNB z>0Fi&sN{4JudW<4AL!X|j~Ls}Z=d>X@eFyXQP70n`!HB_ow0m_hj#IMVQR_bqA8pN z-`bgu0;m=ycRQK}+9Ri*qVZuS=s|@vK0eZfSr+@+{j^QP&CF9;!$0A8i2c=W<3!le zQ?JOmZgFr$z5s#R&%At##LUU|js{V>Lq@Le^_C>Z^nu6Nx!_gOl68Ifbdf%GUnbI3 zL!3ZGF}whw@-JJRj!+uQZXe1B$A;*js9y6UO|7quQs&S-=dJ70J~(4MsxBq>%0cB@ z*|rIElga#rs_V}oazoz78p`YIbffTvA9}Q^9UUrLRC{>cBAe>d6&<~^!Rd{3JDFK0#6n?ie zu<}B=72_KUI5#zWrCDags7OPW1t4q>2TbSgEGSMBmP>RQD^S;AI`KTGx`4db#RWCb zVv_Z2#CB0f0g(3=kUSU`OU7TQcFcPcU@b)$E3Llw{e3zXZrCp#N;JrXlurXSL9fhC z*ja<-iD`|xf)pQPq>6TAKo5z{WM(v*a+jfODtuF;DUbrw|6YI?XvSCae`~@3K0v|0 z7yrVl8}0z<&G<3#RA$4R4ZbpDWSkMr+q>@o&sH0#vC4d0^>p-Hg;c%7yP@i*?!hAx+S2`V7qp-dlHHi0V`O4^BYy(n)noOxXo zd#2QXiy(>@74;r}+@WajbU0t9QE96=cwS6yOCrSu;aascBp>HeGep!T>6<$C;1vi6 zYlJ^ce&Oew9N2!9rf+JpVLrDodD7*lrn>i`H6h<@f$^^ z>vmy*P=wA^dp%bo(lFy$N)8M}P=IXXAYhlo(Q)&bA7M9>@wNAaCNg04Ck=FkDgJ2k zx-OK%N8bR^N>2gRYM$Q_=RH-cTU{BeD9BWq9KH+n`jr&=)&!Ui821sB=iDn!wYY?i zxf7kRFi$?td=W$>SxNY*a4O!Zk<|HzH0sVCp4-?CNgLqw87T6IdW9Y&j(as z(0CMh*rp?oEPsbO=|T>Enh9i!dtxPellkc92m2sf>;8ObeLahwkv8Mf(48`_?D}_v z+vSAYOH)hor2E}b`O?-FPE-@`+K2CC3%Mr_$i-J z4VZ@2FyJ%t?_A0NcB}TTYLc|6En1&%X!tCrzx9ptpD+HrN1w_0KC~^c)**)!AGnT3x zqP+9FOX>{~ZX4ntvHv;DctJS0&YCN2_?Gn_w=>!UxoC)h(ofy?vOa$siWUcHblHP4 z`|_*^?|wZeFtBl*z{hP6{U!!)H)}v?AG-p^vbNKO`ns4|Qg3o&iM5@Gy3U)#prUV{ zI=O%CFj%evbHp)REn@Mxb;8r#-h!CiDIEFi?K#tvW*DNsAiSOa)mfy(cJ zWPqsIF3}ap$3YKrX=ip@mU=Us@O8HX%XRThzctBrFIoiukR8VBFKo$<0%uMJRNb3p z#nyqn;yNh979o#RXP(4H?6LY(=C?6V8qZy9xBxI?_@VSx%tsFFl4RP05^i_&pkPtA zjO@!*&J@Im3-xs`BHJ}REr{GS<65yXgnje5^@A-c^~b{ZKl6Ile_S|QCkAjm_6N{? z4bj(12g^dQsFXpfveM+t)b*X4R%K*rC;fqQpZk*`7&u%3CM<3! zMj|umJw^^aDNpun$pI1Y`#C*V9QXRQm-oOE{-~l{XsZ zvhzoo{R0?Y|Kid=9+@H$416h4OE2@DpFR9g9Tri;Y2=(x4geHBz5k3VVpx+ zRSoVtLXbheO9FSqWLq$AJR@yze{>`=x1DWYX+68B%oU6Io)1E%QMSoRFj#Ld{VK2o zhLHD9CEZxE<6P&h_|J%*u)tD^&GJAy7S{ztpA>3#{4TRbbDH+0vk=f{NZ&cIYy&3? z$SWF~S|B65aJYL3J6+mA^fX5OYogJf2jv8RyEw19x0Q;HXI|?Z)@&`z&$W2&5Bsr; zOt|zo2#qffAJpqU!hvj5pProg5UazZdndb~S$Dp24e|o@;Og=mu^FrzTq`!^N>R3g zZ*5hD!+yaOtPgPM_*|Hz3;WBH5PnKpsKeDlzW70T7T?p-yt?4oCsr|NwtHgfoT4&g z^(~z~@+SQ_1BL+nB&N!BzQINpjWFm~c&|wkrOGckY3MQ_TW?1bapm&C$MC7F7-vJBF83nf* zGl@^}-RA@a=1v@0*R}3TmvCtikl~zlfP#a=oz0`#(iS#Qsq3iL`Q!>DA4`QZMBt3&k>$nwX*vrvKPpxv1+9)k)mvS}oG@C$f?XQvAsKDX-c7Xv zVO6gQQAZ$Hk*P7w8wI+%Ty(z7u0ye8r|(D7`A5}#)qL$%-7~y=AgnQ`@}g~AJ*MNo zB%1ML7=H0;o2oKp`GHS6icfwKeauVRRudSXy5f4jLA#_W)iAeI{U=73#uxIuGR)A!O zz3FvHir~U2a=p|nd$bvvWy@8sQUBrt^bafES&su6Ou4J+*eGu!dmut z0Y$l*BQ)}Wa*bnMH%s{lgn@OV*@>0tH??^?JKnB5a1Ce7^4-vffXTarr`{P9eN%)* z7&gMDa@*&vUNdj*>v>Gvc0*^bBr`Yo7pTpyh>d&F&K5$KogMev!oaBLzqqsuYF=hs zs)nD?<+d*fTtolqi;TIh(XXkJimVg^&M&>NdZxMoncHG@wj`x(XD$bP$DQqE+Jd;U zrKIQ{+io~Yk2$}<;ItESEX3wG_saM*CfI?YoXaF=-!jEzzdeSESIT<{idi$nd}s0O zKDW;RG~MZPyGQP}&PdcyE^hOg>@uv_8_N!j)AcPkd>}l>*GxmD695s$>^>lT)F&Bya2{b;| z@;3odUEl@FEmeF!#AQ`+^31kEVokp1Lt^LaVi| zrpP|5o%bW=t^!r z43&M{>nwfE3ie&PB@+7`NBr3K=hP(F+nq=bde+45Cb z`!~j5MNg?TS?ixmb8NsKwCzqfc9~Yro!j%u=owhX#}RYvBX$ci=!aUF_(Z=Zo?Q&C zgf|eIvX*51?X}5xU2Y?T;6ngOCReS7fng7G49CpGFFiNq;^1uNOb@rTc52q$_xv`=h~g4pwqW}W z)%f?n#gDmi2*#Q|gNL*D_2wv2CmYUkB=k#C#s%aj(VNv|*OaP&EU_Mb) zV|ahU)n@$TV619n%d}*#6n7BcjEP{+qu=Rk0bZCUM93>c5!)#_xb4jjI?p{CaAryd zOm#XEX}2!cZ7#W1nx(KNccF`>MPb%QkZb9>`84q=svkL5L_>Jp#_o8IQP;c2uRe*~ zP1k+30ELMsg(MzBr=z1DZ-TmMfPpUGt*a!Ir^@dhE-T9U5Gc99!4pA?c?p4bPxSIf zN~_$RUbcxMIRt@N`S9z7zJ#dclunwNuvtN<#TLZs)Q}@e#FDivyrb)*1F+<2q|ti) zmIsmCCj}0QBB_zt$5BzFp;6kt^|=S4D0px(hDwh~U11^-)hJt`RlLm@L428!WExr=m?Fl|-iE=&YzQ+7|2bdwq8>Au|~ z%tKjw!2q`7Wgkzzm3W4hr4FPJ@FJb7s3mJ)-*+8I6t`V6awSXoMI$r_UOR`eabYu{ z_qomYMvHZxGTe>A$Kf|VR2yo7(#;3W)$9s;IJulq+c?D5L32yHe!)w@qeC=jnGyK( zi~bHoBk7q&!=EQpb2pxSqf#q34HIVQB@@5Yn^cTY!08MaZ8P%Q*>PWd?`0kyb}GeI zmWpMfvbEr7i>R;ltCj~gGP920*E|JjRWVR281t}`zB^(Umb8@QGcWL))A1}f#hcb7 zK}(`l~m;fv9z?cJU122`GRky?21O zIPNN<4z}LYV?rTr=#J?y%R`%|hDiEi52*k%kydyu>|FDbJ0=HH)Ti0_jtFCR|E$o_ z_*0$3XK)}Vv|^>=8z8aBu8Ve#3MFYLep01Ola%JcV5;~HZ9A1pJJP^S$aU+9s(vv zuy{f?3ds-b4ve$euO>AM7NLmy9|~cTX|lE)xBo54{iH#)B9MUw_n5Db0lIWl`CGzC zA{xzI)_Ck9N5MoTnvsz6{B>$C)K&i?h3q-xhzq%QlKDpFVn_8mO8O_x*l(BSoq zy%Nx-4V9sU{9rb-v!-PFX_pGR4J41GYknPf$c3-#r_?1~hL02@siR|EO)vEiHJ@Ia zqR4cYk23#sffgwe{X za^yf1%uSgNMMg@#qwlZNT)F1pyo~j^7VZTqV7*!9#VBz{{tCvy$7M0A0`uog*#nM1%?*^#DlC(TSiKWJx3Mg#?LHjTwrhf1lo#`ob0_>tW*!Uzb zw`3_cMA`Qtiel4{cEGok?kG{wJc%_c*n1L2Wya&;R=v$YnU-DRD7ylOuRmf<-q1X$ zP=`CJ@>RFF?0bO?m}dk^s~#o5O{ig8x0^&0k%<6ITgN~*#4j&susVQ|)ZJWoA9x~F zc$&bP#G&qA-8|CI(Pp3dYofvPL{_E~lxtkfva!B#Slv=HLM3CXypsheuI?2k#@ch+eGRBqfr?|Mcl?wdRS zdgfkvRbyh3(TZ1gi@WYE71ccHI8(n!>w36p&A)64qrkEP*LIb(rN5!k^Tm~%Tjc+8 zTz$5=1INvDg@kf63b-;f0>feZ99K8ZXlO_*E-ytwB5e@WPll?j+Fd2|WNhNTuLP{j zhbxQ$k~qomS1XL0Iscxx1TmwKXIUYqo?g+rZDnZO2Sv-ynmxYfxZ=CF^6naf^Bj|2 zM=z{1_y=exco=Z*!sxH{FqF+y&zeQKAtsCfX0~>$W2zvK`OZJqD z`|>>Nvf(BnPXo3=DI_E&hKqtv^=e*6+05)EakAFJ&sCXj3oQyq#wUy@V6A4NF& zw;R@Dz?;G%Qb^RgkDy)MGv%;HAXjk9cti_J#}|--L}bIQ9~F-Fq#rjQD!Xh~MJ`>v zO6vD#3+XsSDRchmL~QX_pVmT?)ic)vx_uy8QYqe9|fZ#9q%6?GBR_vw9>zY(^ zWo$DsWFn|Y#9W2}5J6@F)F>DZ+RxU!NfVV*ZqTWHX{f^U@~E->gP#Dp{;&_KT!%CJ zf;`qpUai-Z>xXlbd&dO_IuYxX_k*iZlXetIai5SIK+AI?!w)@BH6G}fgkjqWu!(Em|>;Rnh5Znp!Y_eUTNgMOgYYYbsed3}ZY*jHAof$!o1S+&V; zm%0Ihq2gGmT>TM=zb7AOR252O77kJHSz>l8H|$H`+&0;g!99_B;=Krr0!M34j)A`I z{>kAB64(zTy&HL`XnRQ_cRb+8l-HeN$Qys0T+~x?=CfVAQ<~yF5*+{%2LJf;I8tsz?xHB7}t8(2Zn7j zmcqnqh?G=xcQvHa>bPUXZTo_$xoaF(HbAGi1opUR zqEf4Y`Pjy%Qz1K#L*>Loam(lcS`d^3L{pTOPGC%Ss9x>~TnKA$ELE9!BMXN^?FTr51HwYHUfR8xHs0N|;gcQq zP7~f?fV@cZZzeT$W#4Su7<~yveVJx!(bZ-=+ISS39tC_h#)FT0*_z-VNrrRVSd*Ys z$30M;NKV_70UVaQcJbNr1;-Il1!?g4KaQMgeCt-(rm?vBKxTv`9w2DJh?IBHTGRd5 zy)1uh1m!Y+=x*?&dXaz1o({!RZ2b{%(4zYkKfSc%aZSv_R#fFgG?mpi^!D<_v zbwU6EU;fZj$cl^k-VlnKxrZQ|DMWgX3jwOeXDqZryA~uuk`%yPExFPXXEh?*5ou za)Jkna=+Z?wAH66pN@8dMSrzN-ANeuXzWRV)6ZyBE4KC+ca&n zOd(g^9n@~F-gtVADS~FoG#d9`{T%g1{hk~*L&FJ_;-(&_Eu=g!vo`H3AS#+#x)vN~ zKHTp|^+G!eO8^w*lJ|ZOWK;;#Qiq2#Kg}WPzwdf5$QMf;)NLKsU}bIoO&B@!<0@yJ zt((T@Laa;Jb5t$GdL3%}P&Q4k1fSLQ{aC0_TOrWBWGwoRM7{~j60?CB&TypY7l@n0 zQ&z1IVzPJom<;p_seAZbtP^4)$$UAnjCS=HQ+CCsxUHgH!@d3Z>*jT2B zjdR^wB|JJ7bp_6D5uW$p$;9@9C-J*(Wtu{>;EqJlt39v22GXkFv}Sdnvk3u^-$($^ zK0iM(JtC9(#d0w>6h5f{0A-Z!joi%>e`F94xV<%$-QS_kRp~lIV{ySnan&ZCI~b^T z;qF@++W@&un-kFNajFTuSGj@s+7aUC9O@vZWl>%p&^UE)w!FwQ3JD-&d4mdl!Fl4B zNIZVbS3-EU<%m_N*zIE^gpy4?IQMbfpJ=6fI@?ZWdxj7(Lk3U! zORk-KWJABBBd0Yk`$pry`1&fWdM8%oc`x=HRbu|?j->jB!|S|&m`46|k5rOEfwVfi z>Kz|wBzpA3**TO{O~Ts3<^)+8UOr5ov9)&Qc1{G)ffU#6b7=QN=LjJUqHeb6r#nM+ zyWQ>n<6|WcQo)tLPCNXBOYb8XN3_zC-Auao1yo)){3ZqNhKR^_{lo$;m|0Y(OQ0EB zIQK1cHwy3)Ppdhg_a~EMU4J1{v?L4H*=n`CS)z$29o_TJV*&>vXZ?+MZWkv2pv$Il zi4ks{-;79-il-DPT@4lh5^0`?O!_6hNfd!?D8Nj7quC zhN$H0EXm(FlGV|IfIemM9m((+^G8H~_PW>};eY08ZVO3cSy}nL!@{yrTRFuiZBdMg zSyc)9V5%LY3fFV_PJRs*^~S0yZ&JaS^nOiR?}1MqRIk80zN~Cft*>)d(y?!WD9=_3 z4ADnypkvpz4ndAS1`#)tk#W)dg62-jIe6bNia@vMijVD+#FRL6AQ;1+LZHpfMGPOz zb9C5s{5yUic~X2`<;7B@%@u0HK7_`2_XAM7uKG?E!W}|fk0BT(95I@Js{6%grz~&f zc7CtHXksLhcka5Eu%zPnN9Jju*uPHZT%WlBayu@7&8ilJ(bMnlz6Bl$CT~VPn?ixM zU+GV+1K#u($9x+{82|a#IDZuLmzTiBiLu*7;veSsL~mX1Q~q;M4!VMA&#Y#(7rOB= zf^8+V)aeLMz_OOpK7JeRsc zEc;nD4-F&FFm3_!!`(>NdDCvDB+C+{swPx>KUHe4&TH+J|GWl+c`&hFkixZ!Qbr(V zj7rQ73OD;AXt626Mx+!i5oZf@OtI&xb^D)3v&aM>>BQs2LRC4`1elHWdavv^xD z5buV1E9A9Py>ohVLrbo?#eMd<3{knYOF>>B{qB&4P&n;YREOKmXjx?tLN?D35cf^8 zFhEuhQv44n70e+XcS-jeInS*m1jp_`e;5R$k*B90#rIA+r~seV%e(Saq5kk`9nPTW zDnb6MJbFg^Z;>z_3)MsqGF@0FTo^rFBP|LAZHfR1Sz`y%7?CFL0XV5&gwHgND}tYY zE^+!>;EUQ>?6!+VFg9wD8ZDbTe4%HED=E3IJ#Oqq_-P0~3H*E~$vg=lhf&i!W zXPPz7KxpvkcL4z_vBBKFwZ?mY0yj!IZ5R3H6vY*vJRuwftxIJX&RvM$8;crTEa|$N*`VUwfto!6ll`?mAu-Zm|%+&MI#=$V3(=*?UjnGa^TO!Qn!xSI^nljk$L=po4MZ;{@EFv+o~Sew3>6F5 z#qMCDLJ2YM!aVXY8hM@4_qKy7g)?JfzYc$T5_3+0lnWY~8^;4hfyi9*f=NY zMsjy_c~Y)){yq{M+gP3oWK@PZCDHRwIljH5ZU*g$FpB-8K;#R8k}DaP=XFSCQhYcx zFk2b-_Ewhv&W&EWpltvclMQH-dDTP^jZ|aV9O9@;M^P?sjl9N@yxjCr^T<#_r8p4!`3d~2_N*TI-c`IbGwkwd2 zG2qDQ>ejjdDOe6kjdM)T=M0_q3v_Wb1lZF06J+KxQQ(`HQ7nF_G=P(XQ^xFaEDk5` zm;AO5?WOQ?7KClDe}b6%q*@OJ^K}_SdE^TCG9~BR&{ceNwj^x1zoX+HUOogDDh>95njbRD!X6Ad$K|m zh18)8dvFP`-PzrW8$Q?VXwVM^014zQtH{oqkh+TWm-b9wB>&j17C^_sTB6UuK1q(O$w6^Na++RybF?bu6a%%*k8?YnR-NkeX7=R-> zJ$(7Wj9--SMm&TZzeaWdozQ~`C?6-Sgenqhum^;TIS52t3!m*#25`WGf6m}R8SbWs zE!fSXjf>zC)R43RHu2CAhD9CKC07jAua}Dmj>42MDk{8%3ZILV^QV=p223t^$`R5S z%gkniS0Sl20mX6ONuVgIG@Y_L^6mk4EZl>m6-eN+I(EfdK&jNV{x!RSk&+GbSXI6M zb&;{HOgiX=YtV_6%6t>5dF+AnQ9lidK<`DBmvjHQ#l$inLFbK;I|^j{xxzM@`|kkW zuzyVyQn{HE1IKwa%yG-H=MrK2`{LynE!3sQM09jWj#hDAF&Qz{(*6iJzUph;C_j## zp37_P--w@>>`k)y&paI&+~g_f~N8qqgO>jFUfr}S#X67{%R#)wHrA}?5e29a+A$x_kI0~U?f9BseN|Hh5@2ci*Ns=557mduYHqi9<#<{p#`t}X^S+Gi$}Q=a*XTPTzS&3N5_T6Db|@ZB zY^m!cvoUf7!vQ>}l=Z(U>Lv)n`^pysa6tr$D5agd{gJ<9}D;q-pASHQ0AesEZk zXfI9U9MLP^aO-WcUJ|Nw?D@DqBq)2#!9PY!w)ZYnam2nwfXh$Hi`V#uull0U4jC|Q z!jx|t1VX_0@WV$<&o5%8{Ystdv%sAQAdFbk`y+X~>LXB6p;&cqgBKMpLUU|gB#G^A6!`&)#cvb|m@!JfBxz2rJMVu%MkQFHf4V-M$X)iP% zK;}s7loyYF4cl9ZHC+0Q!)EgZL@0tWy({18Mk`ago(LxHrkVuJ9_Y<xrw!s5E;XdDQ~RJwG=Ong39{^cBf4W|D0Vc%N(c>_8Qp3a*Z6o^8?KM@g@?h-V$ z4cM8XE)Ox$vUZEzLWf`d)56~H9)f7uyu1WVRPVY{$NMuN0+u?*{ig32N$v8Gu<>St ziXUJym$63p8*Y9{e$0Tts=bnTADn7JNaue==)h7g5v{v>hJ z=ob|&60Cy?s!$TN4dbYa|FrJrDK-Lr`hPJvk}94v$?yKSdDt=Rzb-Y3 z6(|dQ1Udbc(tm;1@hQ#pNrC7V_e{*}ZmgK1E1Do~d%V>=oy3c`Yxshf)Z_rPxg5?i zD|?20sOItN^*lJvY5wJqnv6~UUX3hz5uGxEkZq9!qat+sD-nWku~}DMaQ&Nqw@NAFzy~CyV0j`lC=eBPq+H03%h`F{zDwrKw~Q0}$ANso@DT_8pSe64M_HLl;_&XwpmY`7>V(Il}-|9)@o4#I}Kj3)4 zy%0DyZpA;I4*PY;9Q@g3nH1vKD27e+5P$U5%M3cjpINEFaH<*80AVTMG+Hjsx$Mz2 zPUqoc4`7pym^?C_4I7hUBW>)$sEx{1qr^(z6w+b%QctsnzU0Q3A$tr&eNUsDnW>Hj zL?hsp!Uxx3w+Pj*u}C(MU7T026_zDnXI>^-YOeXlXNvCDCw^8@!9+?+-Bk-ymf&;L zY=n_sk|q}Z#*hl{SqJA107+RInvC`?tA*lO7j+!SJdtx|- zNVUTo;>PV6Cr#z*sXsf3H3(j~llGF=rw>vS8AaFOw1kRX5%wPu9pd;cpTKd0JYqm%`IkOgSQ0#U-*@rST_XB5EhAPTl-GC*XqLghe|D9}bUVbW z8q_sSk5WY|W-=zMv9+gwGVqo7pO?5Mp1#!u2Rx1$`ax7%xO*zo9yc+o!d$n>Mvhta z`e1n!AS_2V`qPd8cGv^um=tW6WI-OnlKMfZG}E<3)##37EX+g$L}xPga}m)cc1t99 z!}L%qp?6Pxp7`#PyEy2z(h!np9rYLhOt-H;AM#Ygkp8{!+)jE(?vFc&Lc`@4yG+!T zsJ7pEATTl%`A?`QexsVwK>q%Ir_ol}L{A=rrWfzXfL<=mPYQ zEV4U#>7*fy^KJy}pj%oKB}s=WfNWLnsdD#3y13X3Z;j}Te0H~3AhB74f-|RqJaf98 zk<+8f?UP2&Q@4bKTB9J6KM(*N=7z`r0t+)(w406Px^NfG=TH5C;?lM1i9L0!W#uN6nb+HTqW%`~&j<7CZVfXM{=(HXwV)xK#N2zY3 zy9Ed&*dV;S3m2=O9)VK33>IIy*JFzclRh|yU9#8uSPOjRGz)hra)f{!lplLu=&T%# ztqMAg{h}5>laGTJu1BO^$EY0*0weq6LA=p8q$8|6a|kV~C&ZaE<8168nJjvim>e(y zAkt{yB{;|rmRnDY<*N5HZjgRA9!y{Fz{<%QMjZH;8e44*!MT#Ghpw4%2+Cxx^aBZs zFztoGaV7f;Kg?$s0)X2oMEd5@hy#_~xPgYW*sm)9BTs`oMANvxhNc{#Z7Z#W!|}nK za_CRJwY~F?Bu^%pixgX4AJNPg%sG#r?z6{ZIhU&qIEp) z!FmteB58DHek|yKQv|-N1u&hBBKAp4hZpk#rEksz&?X0;d96pFAzf=PHvLRk11w=4 z<=N9-D1Ng%q*!8rW3V_|E}Cr@J!U`s7S(Q|A=M&rj>VLIfQ)Q*3mGRcfWx5Nh|n3i zBp;82hTujvckKRV`!B5il%Cp`@4LEhR;}vBmL-yRc+lghIull zt5*0j7qg}hFO1x<{&~>&*8BjKcS;aRn}e{saA__sI8NSC0d$gER6p9AVO1vQ17-0Z zJDwOIEvv-B92Li}1>YmQy_TeaEq?zJkjkITdWa~sMNtQmQmP<~W-r4**xK(o6U(5M zp;d5NDca|yhF*3ZCin)d-akJ=O;oj~;BGj^ETb2}3U^I3%$vOfqqjXY>-TQ7q)5FA zJXFP;xS%+SGb%>wJUKxZ1dYC>c`1KtL+s~hJcQO2+87IBa?y;!qPyUsNW!f z!>^|R#3}6ytAlV~`jrDv@ULoZH}>dkIrGM7#&Ggke3e{L?h1K52x<<(w%M@1?>v+ zr~E!4v7rp0Z!21VfJ@$TafVMng1KORCd4s~A^vc?&qy~A^3 zGrpY3#V;LxfI+vS9P=XMaEr!)J=*$!lrB`@Yy&ejkpFz-tpiBGV-_jFs=aF#@hR4? zY;Z3FRv_c%6k=FwLxm{iBUm=8#wM==K|jyzM8&~><-nA@1Uk7&Qk;iPPS2dp8m7k7 znlNlm4))GA6DDkSKD$dvxQJe33mE*yQJ(T96&b|j%#9ZzC1IY>IT%>Z-GdIN`&6|g zk#yGzq8Gcu;vPTjI(3(?96IAa_fkd6hN2pUaG>F#9u8I`zUi+9vftu&NjWWBqwU5R zZ@e_WF{Nnr?a(tV_Jv z`vzhCEB-Yk54v!ybFEz&ZWr4U2U!psuNL#?k&4Y@7sR zqEP~H7C6pnCbAe7aIMpa6@-mf@!n6;52V-YHW~#dyE> z7Aqj@M8~n{rAAhStBKs0DYuRhozt=>c&4J_Qwc|nPH|)jU^fI#H0a+ zZ5^m=siQbxQawm#z(P#n%8%=H3ax(&il?H_Z9h&QvSz1m<|_vacM5wq>a58iC3DWX zTUWpP2nf6vT*XK@Ig|bvX~>|jz3uRth`RAia_Tjy18rK5FFNxf7;?`_`i*@JBGEr? zvf~dfY9hwIV`B3_0Y^j@y=0`QKs#lI3n6wmV?gnhuLc$O@2PzrpjX=KFig^+Z0L%-^^&hdNUIwsb-ihj-4-mlQE~ zMwahTDDiH#HHHp03cf6q)^M)Mmx}ED1##3@XPbXIhfS4^>*h7fz8EZ<5DXb6Xqm8h zxt==vul6gj8(?&+LlaRtmbXxtNFg_~dmpE#V$b25vF~?NKTyYS;&4$JQetbZx^$Y* z@cJW!V~K)m8(o;YS_O}$WNY>-LeixE*nSHLo6=c4ne)5x++H_r7!Djts%8Q;op-J) zn^*GkM$LpPvC~*GA2bo6X24Dn;>lOSg-ET`4S|^w%TF&B^jK7g3HH!vIT*h8tK|2s zHOExYa;1c}r>3~pYlb)xvh}smt^YV1B`J%zZwO4u;)~KL=>i80sF1>5Wk0LIy)d{T z#rVx&`Q}@hIe)jO;pXU9GAyowBKVk_M+HHO{b(QAE?gKg zuioc*5slGR53uA8ZdE~p4!DsyfLn$-vy8b0gdxjd5Zt(f7HalBY)Q2-NYKWQJp3k0 z_&1>dNW}=>ewNUaMUp&R^Uk=o?nGXjZff&}Us24wM*X}t{USEMJNYx9ik7EbU}IC; z*V=xCV|rRhY}$#6d)qc3nHf?_bB~1Mp#LTqU$F2s0jP!3c$}z8GtBs~`Z^tTt&893 z8r)2hb}5=hjY3np3H{TFMuntLm@sW2n7|Yl27B`mg=$1O3knI&HUrBU};ZRacm~C;K{E4?jngD!)+ygzN8*;7LP5X zOs!SP5!WAWS3_29az@)4!8-)kvOXx$d2ie(wR;(G+iQBtpl1l$T|p_Qf&!BMonD%n z*>loU>J}yHJLPI70h=#82$#jdQq&-=MK6{PloEMAzTnX$F0oz>#jD3lRuoN1JCy5# zseg95nEc9uIt5^IZFKPxts`Km^Bp16wC=ebLZ`YQ)l_#_gcOJSEJmOf8^f&m!SmM% z&#@D3jtJIQBz_{-8!+B8Nyx0R; zOTM(cCqscL{DSy}PIF-dqe3#(MGmnmsgv*YHjtL=uH;G6FUDW^<+HnL+qBx@v-D8} zSmJxV`HOHrO?J+5f=}u1BEgT}iJbS`Orajhy+-^nQ3YBXc{#N1th+j;PnZ^l`m#lJ zoP7ZhDxZb?1h*s2Zm0`kfdV4C=>ZKc1j++@AV)vPtCWh~b1Sn16ljYZ3%#%5d4}Gz zDh1i1u+BHgsGkF~Wj|ZyX_f;zVO?{Eqxkwjy5nNbA$A;Bw^-cIRNU6 z+;U{y@dLvSLu6a1NjVq}0BX_R+6_S6fBo;O) zD*Sb#&AcOm*2D?gLL?} z?cVxze#l$*w6_~#JkRE5&@POvKhCSu15$KPjj6BoUVY^Lxeh=gY4_jrxe*iqOzAHg zJ2aT^;S=i=ha#t+SRZh^S)?#zZO6Sv^6V`?1|?2S=zirsBZu3?fE{Jls^HculVJh9 zrPl?eHEfDLR!Vd$Zhqi|5n9Lf+TfFK9%4M?bV6Bl_4@_X*$z0}KISui{ByguEY&Xi zBL!N+)7j1_24BMw#bd3wTg3d9D%mv#I_4Zi{qfXEBgKvF^767`jro@=cKt_WD=i)) zCTu6oTtdYB7God%cb}lnZ3zMo#1ztM&R8q?dTp!2pSN66g z*Bqe!V^q%M!GbjU0s!4qBUJUrN&X1Kt0*vbzpsWoI3R1NuVCj1SVC}Z6iwossvR6_o>6Tp=c=qU%%Xgye-sO5_wh&&(hEE3mmnsc~ti$?ab&ImOY5IZxdDj*pX} zC03YhQ5QN%%<4`HHRUjNG0#xO6<9X{-Pe(WlhiD}yg3zSkm1PlWFqD0vup zSKJ&PSBFa)Qd5y$zj`tcdqA?>f><%?L9vNiL_s}K2CWAsUdSy<)XtSJ95RMCZ#ya0 zHp6CXi{3T_nZI1y^9zbeP|`^ke+X)$( zlWMn$0x;sTMjhQ;TH+KNN;M}0EM)oyG@p#-Nz5QvdWH1dJ%naU^d%nqyr&eJ{+GoV z_D0KgT&hVF(`uQ;zH36zYaa>m+Ux}sV)^G$0p|ge z$ZOFZB5{Ts(65{7gA{^5>>ezRc(=CWLNMyaB!i?-m5#~_YkN73Nr0k_zxcVja3)Pv z6&TC;@Up_2^UX=8V!{s1 zr>=4wQmOpC*5Cm}=jBBv-}o0>H#p#iax~wk5yKNc(}qShP_01MW2ZhNdj%S1>?U?i-qr8^+Oe z2)~&-b+r^#=AIYt<@?H(&yujt#a9cU3-QtVcf>M%S#Y~~Ldr<8Jj&+s2@oNBt&W?c zLI<`bRhD%ZK=87!%U3i&@t!s7 zGC8GjqJ#nkzt3{P`)zG$S7soRQKZyYcN0lleDfzaJV;;N;ZAutm=1F_K7lT%+|D|H zJsOY~DRp6CE<^~V!?K~)WDoc#Vc(5Gj_d*Z+3KwcFF!y@r zMBhcG?t5YSv;*>6YIq11em_6Id~yTX7zUGR4Rn(bIJNNZ8cqaW@MU*7d|r5zZqxyP zG_r3LE_F~~t(rBh1r-26K)%0Czdgy*L&w7>V zK`VWjCrwM`@iRs{vc+h;mEr3#{#nZ7Tug^C=vO{}I%%>eq4T6ORvqmw48CqV!PB%> zDm6^H=3YlKb_nbWC65V47j@@ml0nEk4%_txMuo&t0f(g-zTRM$y~u=$ZcriA)fXf# z(%esYgp?1y`(oprB}v+~xWz*zF!7{ls0~fw^%obQ7}mOcfcT%(!p)w> z_&v+r6+Z^Ta~M8=$&U3QzFBq&+x?wdUcwy?!07ekOZKIBr8ILuFZ}M<$I!lATvnwV zY|YFDoNGi7lIK(`*_D*h3vPO*d<|OC2no(@DD0BPmp|zzNh5vh-N~?Isj zVH~>>cPNi(kQx^qso`yAfW!+lpBsVz(dDiJ9i-0oTjR<--*#^nA;QW>_`FKV%c!}C zKQj(b=wp1Q!wS$RlT}bJR57&_7UgET9QO#9;p5e1pH3O~)Ur`t=H9;R8B(1v=%Rh! zzF86Ub3&q@OpgqU#bML62u-NSNHDLigpc<&AZ}VDcePGNcCDzPy*!iH63TET!g4lY zV-;hLI&CE}%5UHmyT!uzSUX-lgm<36&iQM5Ez|%H-4S^HWlaZQoa<)D>qa**jgWcug8?6?7u5AAEPV=NVXok5$bqB zinEv@9V9UXR;;NPt07Q)f3&bA^Tc-qOOwoAXyH!G+M!#(eCt;B*-IGrp!zI({FrZ@_1PEZ(Q<0$}JuphMUCV`^D4Ln+j(y1dqt>|U z!8x{ag(#lO3C6+sI{5=#Evtqcsg$Yf(9-MYeRILrAqN&n$rndU8WNYAG|+s+mUzH({u z9O^Jby`D<0{kve+;Jenmg_ zXT*EL+fO-5lL>OdB{;MGQ^me({_!KB_-uXw(FX#9wZd-aG@}fD`=^E|`iOSRHE~hE zX#%9nP)N!{+ohe%&vKz4;&x=-y3#S0im+y2VpT=IPW^1Z@!%*WHDmu&`7BNRA4 zDK*V1`!fr&0%!8i{H%~T+qS#MJgMsoun|iahy)XS5go7oeUp=l{;|>MpKyh_#QB_Y z4wjpt=sm{98g$3>O*tER* z=O8E*$IA<)BK`VvY=B5*Kx9A$0fenT@S~Gx_-W*;;cy7m`}vK96sa^Kyqqf$Led1Z zF00UC?ra@-SS`aJ_HhcGi;RAHzF;5rR`?Yl?b}X0!s(+#awZPNh{PHkwG}NiFoKP- zHdkqVy#YwUS=|Jo+{)U&*Vb-h%e8{34LDo5kx(>b7? z9KsK@qh(g1;WC*9vOXDtxz`#U9KdsEf|9Rm0YU8)Y{6jaf8Dd5}E#T@%M8 z4KJBHMmD-?bqD|C*9C>e{+_5z0-8s&0D+Lo*kgvN4uL?mtcYY?`rjv%|fnAkiJ zkSOl-oYxyAqOBh1KkLYXL*S($wIyC*w)~( zL*KsNQw6i5!78vPbnWgFZxvLkuLl6j4h-k%L~qW#fn}at#y3)n4A&1-R-GpSu{6AC zgG}hL9$l`c6{6gV$$K$a{*KM?I}jkvNV21^C#($@%5-v`5Lquc54ys^J}P1W1V#He zwVgkIyJY|yyPFZr`p)RCgx$&tHdg?F^4a)520qek;)>Y8{=-JzPabcCf$KRstsphK z03S*Gh>Wh;h3$=z{a$k=N6Bq@JpjH4$7Kz03jCuFYxKhN4qH=kigurd202Ygj z2shwd#%=a`xBVlmDdhH+==K8tOAFX}&;=7iJ57)y$eLf|h{xH4u(UrfWHBp!gi zH6SH)%XR43$2vyml(V|iX`b)}2j2=IiC6U*YhhQtbJ|gbAa=;W`x!lIXtXgJ3RH74 zE`6g%P{As)>bC6l!!bHd-bCO`6P&*$(E{&G^fy4`obkLBNn3IQNG^L1K3KV~Tiuv?F9W+R3K1p1S5u5QG%=8xFUfR&25x?>t z{7IyDAbk?{Ur9l8{UB+Z-@eWa(FBoSW1P#)jUGehug(~r#kkzJFeKYi;j4lhkl#YZ z9$&6-O^XRyooytc^9BGX-2r2d-eppuW0HKM}vN@cJQWJK`n=pc1p_F zR@XxE9yEY72-(#JuG4bYF(Oo&9TG*?TPi%U9uRpA3RVg0Yi#v(o;8Iw_~aWbKyBG4 zZn2}iO3`7Z72-$5qyR|3VcO%jzpQQma>A(DNIxywSDQ=%y2P5_TAAIqf#&g96yLCn zUnu95s7|!jvJF~1eEIiuR~NpU1d|oDNrRE{dj!@wqc1o5KZ8tlwkqMRq2I^*|y*XX4v8V$;pwK z`G_EkVTA+EnIg2;3r%jAKM0;lBH6yYp> z?-Nb{8lW~TGtyFOd74Dl=ccTuFBZnul7w#XcQtzu>Mmtu#pX&u^QuPHq}JGPe~Q`i z0=h)#RUyu8g(>Bc>1CU(Zz|NhG71dfnRsB7sXxLpWj28jK_^(HVDvpCtgK98`nv)H487*Fj0DlAl~be* zk<^_l240NoZH2iB0sE4157cflR2o?cdRPycXt6M>euT+L-m%kP!$Wr4A<(hhs5 z0vy~~Z33g{U`De>*JV1cbM9uiy1Yb5u-^+!h}Ov*nW>Z37JjnizH7mYR~rH{)-qTK znw5AEA^;BiBPR@vNCElR-;VuCPn88kJL_0rtw*fO942NVyA3;Gr3N!cD*osgW6LnU z$ta`FuZ@h#ly3Gdwwm>xCkg+OVKrmD?2K%Xx;4|#>S~+l%YK8Tip-ikI$|x#$dFfqcFH-*symm4>s9^3m^<= z3bLf>T_uGCdVwUP((}Rkx0r-oRxBkB)CPABQ9h)Y9_LVL8ruKmB$$ z)RO7(%v#Kc%yqTff6vUj&>w;$7A|$^==H}Y$mxz0F8bqPP}BM`pfCAPC%+xPi`3{aL9t~DujojkBkK6MA|Bs@mq zaJ;%FyevQPJGMh|7_Y8t0||+#YAq%1_F2zvlC3z7_4zLyphNZJOiqr2FP+jr-JoXv z{v{GDPSIsD!E1=IUUSssB@8n`;{{lcbEjJWNNBcJ* zd?fqrc7Ui0lqxbjVRGXr=sv%|8f9APexeKV_^oZ>fxrfz_5#2K?KF6+WbhHhm|`Xq zErb~*g~lbz;j&n!uRTsvN?-KweN_D1HK^>Zdr^AU8faS`DoJ6`=XjhK?g zVQD3@4A~Z~%VPqhn43H)fy3g78}hRjC83Z2^WJx_7?xJy@NmqwcW;y7P)35O%hT$j z2J@y3Xg1>!_ET%bYTxRz2+=?aQ%k2k+Suf}C}0}kJBhmc2Z}e3t=z#CcZ?jch!PEDQ2+#8g=_3%Ooa?nxIkN0jjAQl)_9M=eSSj z?y09V{@i+riRAzs3;c$gT`r-+U6G6$8>`0A=W9W!#&Z0&`qSN^^kcE#p{Cj=fhfP- zkh&g^e2`vGnEd7(>4?De`DskswInmR&Zm7%zYR_hS8PW!t{90y1&H@#>b*qTQOFS& z?;0v&(BhrHe@c)MIoKDav*G-$>1&2CL^tD>r#Hrotc!3EB4>) zFY>Zs$Pg=UVzACTq&i22@bo;h^BWa%%3{5h7=X%?$%}cnJ7z7gEDR|H=;;NtiN8%zX7@- zhGDF~pu!NJ`hbs)8ij0~;R@^3?Fg79a%;wW)W~uO6WvV&unzN0f9%zhGsH#xSu&+# z{wjM!s6N^In;8u+=m5IhoRQ1^ZdbGs0bhjZSZ@nPFzqVa-^e=2$O>>_;{rRs+)FZm z6-9gXMD${|gdwPYArgkoHZV(~pYxK;2CHe@aSnv#a(}NP=#sslct2p9I`#Ga+Dgba}>jCrA>a;5e~GB07%7{fqF=X zubOLWZ_>Oph04`IBRUAS^SOSGYQ#Y5^+M@(IY|1B51w9*itKgDiv`LA6f^5Yg zjh7Gld=jPhgFs_c%^N{$;2^MOLYP6-F6Mf4W)M;E5#%asav3}>VF+QM3q zol_z169La!?y%W!tG}&2f-BTG@Rs0hsy}}W1_o{;;bl`e+0Qx=ksIZ(TFM7uh~@nm z4qt{oB5YG^=X;jb%3Jb@ENiQC(BGe{A?e`&kEyyZJ4y)M_B?1vo7t`>)WMYc0Y1Gz zM6*%%LCsa|CGEXj$^XiEkYyh-2c&q($6QO%p%5<(PM8}5oVBwJ)?-#WRZy@1_Qgb! z)Q85_Hm@IMrH{VGlQ;l3Oh1eU{!YgnZqRW_&*nq^1yS*qR>XhGh!||NS&S&<-`>G8 zN4b3%1k{eKc1)m`+hJE8nJX$R&LDGv{tNaNqxrm-%XRBfYDbn#c!AOW6(?nHk1hU4D zV(Q=402KR)kX1X$;$FPD?)uR&8F6}-wrzT^r)w{&QeY1 z7jPxDs&QO^$XQ+3y}8R*b@tdBFzYKFgS%-J_3qQ&a?`UQE67m$ zdsAtId97P0l!bTm`~Bf?U@(UYeW02tSWnqRuJT8x14iSrlrIh3j+rkkn>DA4xYqmR za&eo5Q{GeS%}`A}Hb?bNctl9Ru1ufN72a-AyJsRZ+`_pnsu&0{Rgt4gUYT1-8>^T= z=KR@%TqLd@jh#Cv6Y}PHXe*jD*ve|9yCq>93t}N*xZclZktky@{eT*b%?>|2AY+)Q ziS5AmY9jse&qkEG))VA|(xeXGr~vAZb*9A}Y=(NxzAocT&*}{7c4G>xYpTw4sK$K- zb=z*UHmhMUJWHZNr^j)I5}T8-{Fc(z%<-;YIeTm z^b?3zBd~@m8$w1_0>F<`oV@uUshPUdmVbv$(h?%iEq|opDQa3MHN=!LfRBn79bwxY z=UYi(H-1rpeE>KM?q7T7lp1WCtX@-y|NW7sA5OC?+L|L^^o*Nss+vU~OaegVLzKAG ziuqM@-KPlROJY?53a4PG%vsf{KYbtg77S^Qvt)$49^lc@K%BkBmF-g#f=H7QpiF-+ zCu+i=>yC7>isdJd+p@voj7o4)UA?9%Pg&19-)0w!LHgZ$<`$hf}c2PA66BF13A zr%X0+%$x1u!w!`2iNJX2^mkYR%O>2+eHW)j`=KQ)y+V9Ql2jq#aWYf8#fPcKf3x|T zA(7~$Z&ctHBbpNsB;VmPaRtQku1{#=$ST}HdG$eX^4_L@ga4fB4<-mlG3vXhS{?1& z*b5g^5QHXf=Y!01eA$%6NT2-)p@-P@PFf$UugiG;Z%?|=wAe3qUj zx|2V|E0sJE&V;qnP#sc5Rh?9~;j#79dX(G2VJ>Zm6&- zbM+sIQ{X|;QnC|*=^m0_-h=#6Ic`(M4^k*pO$v9(X;~Ycw>p1j?pt?}ebfBbDO_Cx z2%B#isBPXrF)K`0_SiOkrZ}*<(01nwWfuDGM#j2Ww}$1F6dG& z0AF_wM-16l0d#w9xXnc|epGt!=u`{-1TEqK%bpZ4-F!tvB&bff1h3yP-)~U@`gS)T z?;bDlxE3|;1-aV)Zt8tI4b*88NWp}C81+0WxJ)1ADuG*X9x&y%M5)!@8Y0rEfo~3i zCH-8+xIJ%6I0qe?FVc8K)8uQU0%Op#Jv7i_TwHT9en0a|#q4C=s<^DQ;ZWL%>wDc6 zXn+d-B6Vg#nrvQ#BaAfj{z@cm#LlRjD}hunz!2|7Ugm((VKv@xG5O-ZR4kIDH>z?4 zj$v;HDSeE1wb$XR?ChlZ#E~*m@I_H;IVkTKG_jnU5C)fN@!-kER^S zT@Aej$0U!6;9OQP9KZtHDtEZxdPnMaq6x&d*CDtyMxx+_# z`sMX@gkzW>+ozr75<(FR>XoL8y9qs8bo2%r&+51qr*xgFXEop^ix+1r$4&-d?P$F& zyq%p-LSFR$7w3dbXesGn)Az0G^Mp2Gbg1mCvTRD60puGo zH!%7_%CIMMq>dk=$I(QeF1y5DPaBKU2I zz@_sHw~PZ`a*o@9U7!EmWe*4r#?<~a$c zj&lONd+W=5O-NSHBTzk6WY!ugR)W8*g+MDH@b%h!=sE++_`eQQ6=8{opkLrIoQ(!F z(KE)SkTpmgE*h~BvL=6Fn7mk^KC%>Nq)+zW{B9~~^++rBFD#?=S-2>+KwLa>BCxTi z?mg!hK{X-2;5JJksL`vi_ZH!~#@PajzElBb-&;y%X-M$TQ50cuhFNk@vw3W!n%W}b zWlTF{63_m%u+v+B;~}xR78G~(<`D&Z)_pOglSZkNDaH%?W+s@1lLI5k0|V!`p>3l` zhV?6qNKhHEF7MoptRuYdpIOL9Q(pph1&ik{)*Ndn7)C*2`8TFztqhrJ0HXDLn*Noj zzYwjU%D;^YGcOt@i~8QHp0aA!kA(T+xE=#ItK>gborT-GOK6&c?gx&Y^DaO>>x$Hy zDGP>~Trb8rnRJ_9tt!|~K>A}MevB|YL;qckPBJkaY0sm+qTIR*nUy{SH*D!AJd-tM zG;j{EIfA+%^5^$}$y?>`t8b4C|1g2ban;wj)4cP!jRkAtL-~bifT>52pjz~SMA%gs z(ZF9swO&UwiLWm-aq7DqN6PxU6uQN(`G*R!6(bC=&oUadGohg^sY1Ad(dj+1i6a^+ zH~u$IG$FptCTYFV1?&T-0KY;^R8|+1Vv?G4ksFD6o&&xldhQyl?{{~uvzr4%btzQ3 zy9Jsk^yyXd^};vlN<8juBJzx@esa;0D3SQJ({UpOEx_d%TanvWQ&2&Zd#*WUw&l$)jUrLp3jaT#PBa zQi6q$oms7{rbwlW{sD}OZFT7%Z2oBxlhvU9M8w6E5NQh>+=@FR)J4sCX;MIi{9?kp zlntT~Vd~7ULxUBa5N#%>0ujezcqb^8*X4lR(_rLYXs-50 zgb)0EJvsMFmwb7Ud=K>rXv8vI<_=z`7J6`<5{ld`O@5_MKdGm-OqIxdW}6dxTM{HKQ53NMXd!Z}JQp z#`UWf?^WLW$1_Q~;g$FEb#Z}vsA}UG52tX)jO#?hJTn}ix)KQ$2_|&FlG<95cbSVk zVuo;pYKz2gDN!#eWATT?Yd`tc8pD~FY7(B%h~98)TX2Cd<;4y8Qw^6jQC(EM;6RFG zGA?%1;Ra89bNDcUqRbPNp%}300aoh!cd6XNX2fx-#i-&W(_}h~?TarcO~Cmk9l0J& z8~tjQ*XCR6Im^g1zp(r29}r7(6+dx^W@W1#UPYb5s9{-wCBUHE z>ZrM^FzyN?Cgt{G0Qt|?USy8rRqPI%V7kUdI^Zf=MX>ufFC@mgS8m5(8tsY&(wLzPs2Hhu)5HZorwu0yPu03gsJ6y!V zd>mTpuoJ_E3Nv%7tb2}NOYnMow1jlb2__6d^B&_XF#snXo%*_(Z3Ohr{lb!g3s{nf{6Jc0 zAr|;B@dbPv3B+PaBs;u@FM4`nY<2{z6K5WEF)}S?eURiRe4;3L46NtK4^xIy5-g#z zz$K3xY`kzxET&On?=p1#~cm9e|sfIt@1Yh z9ImPmla!)W;zR61W<(i$4+NcWhb#S56LNMh!Ym!^#m#|7o2S*j85lDO+L-Ier$;oP zCSm&Aft%oO%{P$cTj$y<5?x@y=DpAwn*c!gu)H4&J=@FTu|c3dK+XA%yEkm|Kp_1E ziqlZW#!{vXQ(S5cEqUMVIgj0i<8+$e6mvm3VQlnMxW5wGmbOM1WDiaLto1N(7Cy6| zRk?N*J4HqjyciRJ0BFi@_#%XDvQM7e(bYd`VaCNi#D(XJ6*V2LBWX7E_;l;Or$A}q zlirxPYN(K~-RUP#%Ow&lXUhSx#K3-=Tw_gc9=06&_M71Uu zhpK+)eR4r!mpS^@ff(3xFXIl#Yb6VKm+P3R}eoUOLw_8K5#T zmLhTrpUuWZCH8qVRZZ?Nky2vmw-8kt25=&q>8jKgDxZ)F&W=e^+7qIV{*2ao|B>^B zdsDX?k392o5!IS?Zl^P%S}o^H_v4}?D_o~ba^q0=;DD}pkBK)N7Z|fG0}Y&V)zDqc z5Gem!y5IqrqFtjnR1_+wZwMyeRL_FMMFKmu(?7s**~e?ymY3YEqx3*n@(Tc4e3yq} z!-AMQR5ff6h~78a=}$%+>Pm|wklQk>p?TQWyp0C}(h8jmX>@vb;lnYMsA^ro0svZy z*tO6ChwhyoK(0Iu>ISuI&;!qr&g>M# z%ibH&ATLa1z>F~Hz(#?s@**O>_1%3u!NBVZ0_md$!h_~MhxZ)G8yWcS2fh5nlKal~ z3!MBHDX@My)%_H35>gdd`~AYh*+2ArX&LVxK>U z`d%H6ESD;>e;4}ikA^=lCKP6bIx%lR2Fx=L+nl4GL`#?%mV|Z=(O)(?Ave8Hpx$7V z;VPf|6|7|N*#4?OfPyYP3NDr3Os*%;w7>GR9B=%cUno1eS691F5#SfToZfk82~EO> z((>}a3uCei4EQty{Up$a?R`CNZ`#C!F-2|^dNGg3;9R4^1SMqx;nclnEv*=IwW8e` zxf@|sJWKgRYNCns?Mu)~^7O0LEByKRa7cUh>96I5ohLHu51{A8o1NF3J2akSuKd8nc;5Gcpr}Kg2CnT} zNh|2EAslD4nkrv6A&t0x?MLy94lz?*>pXNs;27n2OdyAf_iTf$YGe(N^&(FJN_Jv7 z{~s~D5MCVy%S%gE>+}XwO%VOG+xa~Ynr*L62B@Z1wOzQtWwvg5kmI%K*gDsW{my43 z^x+u}rNOz_t0~n%oxVup4)k5AP~ki->68}cT7>#jZDsX@+pUj0N7toT+C7|!8q3~S~2%fJKox*kSM?G{Q^YYOl& zmTLx|n}dB)v{mFy@iNzo7V#t-?M+00}Wl=hj(kBVT!gC*rC7b!j#tXK_RqT0}; zj9bu-IiZ*ld?u7Ge;m63bs|VNZw_K#XFH`DR_?aAfS1yr&*pDcB<}Gx5BWMi(n0E4 z@6aO)==UAKj89Qd(4EF?H}LZHNVWAK`Mg5c`~;JC=VncmZu81Z--z9GV565JY%_A} zWilf_Ac6)bYGMnv6^r1SWqGNZmsnYG&~PoG65g!xYNfL=ch(SSOVC3C9~zIY7!B2q zI?21KUz*aObj%BR%96IR^q&tttgSBi2F|39vX?sbBEvHemGhLmL~xT@eA!({G~HQa z^~W#Ma0BpiyTw;O?Iuc45=eIKgWujt&x^kP0Oki7GI z-azKt79!Fccn8~qUkm!Uh8Jxp<)>Q5L~u`}n&OoePkZ3?KDX61CB)*}N1$^;@!*7V z)Beira0PZMIDgO75f~kLctu zU&-NjNJsq}d+-7aYA$vdaM+~$bZ=t$XWL@s=iPUtHSoUIm_mNn$H4~;3njE+j)6%l zu%NGWFZZX{(%)Ye_s-Ae*MPk6=R;mG;m3Mb{erAXvv^igJ=dYvfzi}Tc_=!UxV8jX zl@)#4iBn30A8SzR!32gteGICO1;@{?rMoz=tVdq=ViN@;`Hs4Uu~ata!Kfpn z^{qrFcaiXE1S!Fcz36FtEU&fADPhC~c!8@7V_h|a;zH5zSVEMNDCLw+#1OSg2KN}0 z)@S8#TZ}=x;ipbE8JFjiIfL6nY&Tyt+2)_!^yk0F3N^U5W5B#CB@!O5dS=R(2ce*e zQg8(uV>Hg%J>?-MXW1kBbLU%kpw8rN=o6dy@qqZFvjwaaGb=0<;B~*c*mEMFTQZ|L z=Lei_%|dR+cp*+ADJc66jmjf7=_@|FuPzZ7r@2$YFv)dZ2JnI!2wQ(LU%tj3aK7LM zG<^3b-VhKx33%Zq7SWP;9v#UqV4(7ols}m(?8U^$FkWdyWh8C4yupakI?<%Ej$_pC z_;VpMBcVbUFVc3T2FZ;~VVQ|(h%=sPBFK3tT51gFfl$`?YtGz_o|v}TQ<(u60S4}O z*q!gRAQLY1;!COsx8SWH;&N|WQGr|?;*E==Y+}z{6a1HU$H$ipRfTTbrj)yp75f#C z`n1j7$M)4T7$Si1bv{kap$hTnTmU)?hfDZq&yB z@rgWe&uH0#r4A+Ro$6a&c)vU#-m?J)BW^`22LECbt2CgmpvFfbR;C0wX?;{-YEK9x zr7uxm+FX#O6tq+O!vJT#ww<9PkT<>MGSO_QQ)v?6MsXL?;6U25(te{b+ThW4L(z|A zsx`tKew^S{vv2j2PXG(hC!Ya(GlDjS(s4ujpt4L_V&uD)bZzki%bVMna9axI<1vw& z(odNZ>i?oBtDnY;y4#I)4s`sy=^qQ$9$Taz57j!~_F@HgOgz#=oOK_z6PF~9t3Q>H zgWA@Ua-e@4kkUJfRj@Q<*(f%FvB+b_c=JH8aCL4zCL|$nQ?XCU$UPe)WGnnG&d3<*I`{G- z`bpe#HLOTM8S-AVy;zI?j5D1JmZbIQbN_oURP_g#%dzBx3uB= zlA;s0)(ZB+QmjOGr_~O7b{5iOsT+(YlLa{~Pp@?RtqlhR9T=Mx7}THcV+FmDcNoTI zMn6As#=JbE27`v9>GHwCM~OB&N)MJYdxtBhVAlU>oqgcqyS5oxJcZsd?u;d~Rj0_u z)QI&&UU^7)MEUbO>+BKj-Jec+JS{M=^xx#|=`{il@?Jg(4|zO}2Nt=*<+Qy! zH|`coaIb3DSAZ)o+Ie(|dnqbasqTdL2zKE@*pKtigV_wtlEOrL+b*L!vx3eNX6F*- zrKyF%3;raZU{Dhe0e>TW%+D%)vA|CWnw@4e$-Oa(wXGmJi*xJCq?SyHxuSRx%d%)b zwqhdSe;%ZNH>B4730tUWAmYHANKd|O1>=Wu@aeq?oiIA?Z!L2V8_)}5rIyotVnF|J z@AA8opf-=W0a!Ij*j=;`?e2?(Fi*j*qh4ea4vrYTO!8tPn$~kJ?E94gUx+p#e$Jlx z5PEwIxq5qZB5YnXAc)}!xVM`cch-5;N=}I;SvSb?bNNH5OXgC$B*95@NX3rNmwdH9G%L&V(Jtz&<#$ay?diNK z2=F-!!p2VgI*L@)^>0!ZjBORs=dytgp~TTyucbZ2y|6K&-Aj#HTVUiyA_z1+qcSZs z$$zr97g$R7K8Kc3p**O6)YAvS((MmJH zSmK*P!5v-A=fKvB+1`6j6?}8+DMmj4$mZ*x)tYK8l%nnD+P>&{>?`X5K_L1EDYJee zw6q#QQ+W#TK<33)CBIGp8fBd9aOkA2g23jkR_>PMp6+9*ZJJW7wAmz@!QL(rgEI=& zK|lW7LR~L92;gL4-r=!hmORodV7HsNdvL~bf z4zso!BV+fOLC$bX?gG*t>j}pf1vsAPur#MA!NKo#K>D(HlE-mWxd%6TtEf0PBtnV~ zWRN&UMvWOZ5?9SM(S&?lM40z6#1*~4&m-|~##m8apw}jOCn=gp>kM0^i2^>xat6*f zIywUQW=+WNlq7^Sg<$uWs5S0Q-Ed(7&C`B(2iwV2l1>Q(R(@XJ%4mEQ;b6+NtXwF? z7`~(cMYPoA>V?mpt!~d+YmW`l_O+(Sgf!7ut3Zs%J0s41lmIQbTNDTr&nvi$CSc)F z@m5z@Db`@Ue@&}7q;aj&w1d&8Z7knN5X)k{T7W`jQhs!ZZC(oF5&SWDSQl+19j3hH zQe=vo6kp&R5z(V74CjJ!lF_B zo?NdqT!F#`Xo1`AfV8po#0ZxYLL5nO(it0BOr#z27zmw+$QN4Q0dZV#ssU~JC04%* zqus<9SzyzvH%`k!@KxpjseuUQ(@7@;9QMs|_{??<4+gMxk=F29&_3|B(`fq=ZiYbj zPh6I`GO7wUSqGz~ zjUn8hBl4YL9&(BW+~@%HG)yWJ25Df}GdXAj12RIsfG)s~oy^Lqm(7_dZ6$mTsPnfg zkYsAnZqxj9`k0ttR1>x@j7fQzyOejZ)GjH2Qa6ufPuOl0ji&g{Eocl0rm(n8MY!Z- z;&YFSi2RrXLhUuee=(Ua6b+c$#HGRY46}GM%reY-)-WU;!}AE8YYo{Wq%fCH5uh|rHB zKe_oRc!TsU#WMTu2G(a)QV>Qi-qKc24>4gOXxTbF+ioSbrJn>mK3&peYd;u(F6PYH zFcTFV3Q1SbsYI53+drfc+x9wnuOYK@=m5;~_YAIXt8@Mdv9NbQW9N6wLEI4XWAq01 zk8vnY71EDhIKhI2BbthRlN$CKNcdz-ir5#>7u|VRN9ANS=JO6WQZ|Rw^W5(x5E(@U zm73D$U3WTmaReEH=70-c%)c+0!XIciGXVmNN|PgaTI4N4;$qCzM*;?j8>!rV;2d#s zM4umE2)?k`ePP6g$O|+QJ-%{tO(5xla|qOY-Fs$6#Q#VM`0N7G=I}p38svD`M0o)C z04sApUKa;&gnGmmcRSSUAzI?jzVx*FY@LQqwE6SB5n9dOZhd`5e&OSb<=p@(q)!NCDz9`c%LvnA)ZHGEkd%?C(!oNGaJ$n#5&$SZm$RZa2RHJk?`t{Kh(*FvHM$GEr zdH2S*s_u3M!gjE%N}ABDSxnZ#(-mvT5z zRn#UO>qfJoh?1IXYEml4v)QZ(La-nrO=rJZMe`+$XF@@%5SM z9B_nRW+eh$L=sOlc_IkYNOa4Kk1R;Be=x<`=*f7>wW5ZTbtI#d=<=fvEGcIl{xL>q zk!}ETEm0KdDKx3<%`ER%a1Yk!U@D73Dzz0WVUrtl7$*F7@ihPbs{xM;6M+m*gsIa_ z6&(jV(KTMS(VG~mpYIc+>6scDXCuz&fAv(^Z$DCXW7opxYLW$CCy!bQfdLP;Kp zxdtA$cR1i3hD(ku{$q4H9|GoX+FUPiBd2G;V>^3}k%4$3l-26)^tL`(ghdCU~_=B7~2t2T+RtEXHLT90E3INL?-W)i@ zdId2v0`5e1QNc)5+Dv7hnZb!FV}2_X9(b4LE#;vpHamXayV7_lH2&6SA)qTKn2biW z$X25+>|ob1V@at0d@ip+EeFYX*6mFw8h0d2q@CUMm{7LRIpmAAi6Qpu8O3JX(7~;* zhS+SN6pymud5&+d1q70K&%i%Pjml8p4YZ%0Uz=f`0h&hyqu5mXH<24BO}dq7XN7QJ zwKxnm&hGc8ZjPD@%0zZRXVzu;rp6fWLAFGQig4C(nAjXi^d=6bt%ch+KA-a(&bdI& zeub{Rp;nr6O!u+-r%8UotHsjf(3ZJo5OMi;-g(jdkmVr>2$vG4E!ZMMVhcXlx!d@DMeqm-`3S=15q6h#lJY*L)*&OQWIRew_p0@k zm-}7aaG>}oDY8;OfM5CqsV{jX@3XD2XS4PZ4NrU_2nGINkM*{uZ+DRasP`w}$3GivsXt{B4>{RmD_&&B6mOhzpU^Zm3%61PN3}V% zx39SX<+W>$29v^7v_K(+NN(yKrw9Q6ivFa+mfu%B^pAv z_Cmy9^X@j__wP%ht|_u-V*i&-GM%c%cwiaA z44*G6za|mB%yOVJtrz>^{VB7I_#b(vEXZ*5wp>G7Y$rAZLVg#onZs# zaH|nu(Y)!1A540fuiPC?e&JF$e33%%K=$Omj|hPdsEI|VxLG|X&fImM52A3D1$?S_9?R?5a!Q`znG`+{cK@m0vQmKAfo=^+nr0*u>kiC=n3lQ)(van7AmWF zEy42O`;QEab0nmjM$SK@wBOaoZOq=AF}3$--MRq4rzo1)^<1 z!EFd!2_b-Vy+68{wEe`^7;ge_heZ0IypWeJ2cOI@saM!fvO6V_c}#~Me15U@EjmHA z!-oUaid9`>dYFT=bNfmEgmVP=aMU!R>WNjAhWCJ)sst#_P;7t|eh1v%A5~ zmh3G7QiSeBC$xyq@lm*>UEslIO^WS6$4zP72Ny_>>3TwHw9fZ(2eE)a$I9@)RtaYoM1Em$wko0cwifUycY)-J_%u}af} zD*X!B!Bx3-YtuF0Gf(;&+nw<`4!cn-)Vnj!TvI3lOQZK%>VSBDvavq7XKNC)w2iP6 zvDoH2P#6mRZVav3?}ghg6t{k#zPB!)fm886P|(K3%EBA()(tMMlX~vCzK3kMvUrkf zjzLne^h2nr_jx9Unx82577}8=?5v-??kcxyj_q1c*6d|woFE{*1$a5HTo`G>9}Y*q z0dLy{8YFh|M__9hK$52Mg;TaL(Q*?V>xkc%GN&qSE?8YgF0P3VF|6(6DIWo&8p!bW zamhS5I!x~Z8ZuJ4Pj`e&1rb%@;Ue`yfEB=0{NHdtOCh@Jl+CJ6eOMY`c2>*=?wsUp z<%giiM)8-o$H_S|X_GE1p3<_ToD(4a_@_(+eB8N`p~n-Z$xMU2$uE+7$YpcFGdj4z z9}mpcIhN(4G7ysFQE4dKq8$~r$o%R%x^7XmGZU;=IvoF6V?gv3uR-~(<3Y`a{7!%b^5VS;RvG2+uD}~6-8f%i4vD@;>C)~B9; zbrfUL>t*$!>N1QY*>qw39C7(w0|($7lDtH+6SNjnd>o@0*Ow=)UEqa4i{PE#DbkLJ z)M6YY!iSpect?DLX`!IuBd_fpL|y#j)u<3GToA8@?nB)I`f1{z2$yr=*^FyMVey3{ zd3Fyp2s@IzLN!WvUMJ{LzZFgi~>#(bDJNwxQ# ztpPryV*O6*(X}7j3EbE%;UGitZiOkVwzq>Kg;HwVNr5^uFwvhn z_|0x%%?0I-dYu(>oZr}dXU2xmUGDE{wDZ?!PbHSHYJ=>RxRemK%rnM)&Wm!6`8$k) za4G~wzf%8KBB0r-o8aY}AL!=oyOZlEohk}Sme?uT)zfBj_gTn`dg(l_@MEsJF1Fp( z=)oXmlFsjjv|cI|w))Kts{qZr8<#M72Ze+fosQ2FZ1f>tJ689Lmilm#k2x>2KO_|t zCDl?wd2#MYOMx0Usl%S0*+Tkg3^&}1>YubHoOlhl#)(rej%D5ji|6}1;y(H1gi*}x zEEdil4owr26g`zCy6gbq<`|}egnsS4d%agYobizevI!}7FhmPe|2lI~)hqx$ z2G0126EZOH!cajhK&laSNt*~0eFM<%Yr90$W ztr#J+@?APK`9Bfg5u2(I3Pb2gP=6o|BtNz=5Udh1N9VAEI6b?%0Zgn%NI}Y!IW2pZ zZ}#UwkPvOu*j2(Oq!smb4iZ|CtvvaJ6el4uWvdQ$Q~h zX>zr{jE*$!XEn$rfZ)X_i5G|QqHA~r6TH_dab8``T@Kt$@g@&3Joq~~MO)KZ1j1pFxAm{#Xov%+6QFbi3#f> z+(}Rv#M+)_+&*{%G-+TETTPdfnx?e*KR;#7kpcu?K3Oj@AV3H-yPfkR4G}j`4KuGY zB)uchWs)GZemOo-bcTxgxe`X+uccO9Dqg*CfR%g|IJw#&49r}_oTx|Tx(yFZkU(hj z1wDAt=ghh>NXp);q}xpE(ZgG})y8K?Y)S$2EX_}Dos}Y{9=FduYMHZq0TrCiILFeZ z<#fE@6rRN80yA(#WOt^kak??AFO5_q(uZ)N?%#6b0!)BKUzH4=_z>?mw_5dfxRVYT z*XBU3yPRRkTVyd}q{IUb2ybW_xJLA@c4_xtpV0bk^5emRdv+rCksx{9OdHjk4L zaE6sBSdJP3WbT=%$h3RE3w@;zv=w~pa{5j=_4Mg62#Q6ya>-tYOF2QZwhs2{x#=`( zRRM60cqRDdja_X74eRQ~-N|FKG1r2)xhPo)8-B$VT9T4Pjvy8fuui#XTMx$HJ<7F}}gl-bR1Q+ie#q^#m6 z)>q7GWIdA)Fa9OJje$b2F(VL$`NB?$Y zlflkd$zu;xwyCKVg_r?B5n!|R(hqPkowVXy)nRn}Ygfl(-&9@V&)${EZC7uB*U36p zW!CZF;EZY2GF{;n(I055AGdGf<--GR*Kf<=;t7%eA@!VM+`o@3XPV<@hbHr3&@+A7N~UJk+r!6IvYuO$I#nZQk&(uIn>TS1=(EqFdZ07l5DG`OE;KB9 zlbL{mgxPjewr4*$!h0}v-O7k6!iL!#Y03vPPpe;UV&I*MrO@N_%t|@?H7?$BWK1G-JY@1+nn5~&4K=z@4wY)PnNP_5^J$wQY(A$2zjnRxwLJ?H-$s)btM z01JAq5#{~E-J9w@IA%1v6(+A=fViw&Q!+z=<!@7E$PNy^Qi| zcjE$FvHCa!-;TF9=0iAr776@J|TIy@-_C6Z$#65)#>lYC# z8n?-!#ohfF?y?GlR{!XC+P!y>d{>3K-#3nI@0docu=8`vA(aB3a?MqOjU(6HK{7BI zT`e!$b)jgzdhWc=eciS>^XQ^Ve;vBjooA(CHY8( zKf9l+J(%Nnvj!+Rq`ZDK!gc)|Wh*RhYlPsNH>>7ULo;yjsFvv8d z)^w4ia|?lZ;!MpNBmZi3>aWDjE*zsXtOIjYx0)<@=px}AinS(gz@%d@DEkim(>RAP zU&m{?lQJ`cs%4N4kRrGq(|`;RKlw|mz7(ug;3tMFCs z{MB`k{ePYxM7e=4g`ng& zZ3_EDjPWZQOaqQtF00HX5QFLzRi&RU&03sbxggqC;WzXC}iv3dDKjKxw>S{} ziVHMN5q!E8q^aOI8+2SAJ)a{C13mWTnj_-&A*CRge!%QTf+*X+Z8wodb+l}7+eo6< zF%eLKDj(?>0(p$!&qvD-8fJvQlf-ZX-EqgSTC;}%SnerX8}Xq@LAt!3&mzTilTSjG zx7RI7*u*SxntB~5B8G_e+80NflfQW3;c;oZ1FxT&2u2mfAG(it7clyda3n}g`jWxt z&z{QEZt%pUO-KYyMir_aw2NF=F!<~cka#Vmn-<(d-ih}ps6LLA0bsAhg{USdBPqHz zs+VKDa?i8pPHr;=%-+XqnIwVtrEL=~Knxy!*e_n%mm%Rc5bNo2O?an;*P2}?X*v{fhSZh01u$AzY?Cgh07CRpn_=+PzP0#F#%SrJl)mc zQp^f2aW-t$jz#)vZ~1laM%^N1SSjgzp<1EOX=zKMF+k{lxJCckD{CrH-v=+sf`^Mj z2)j0;(MWW;qloRh;lx(D=@DF@kaQNamYeuDdT!s#2sH)Ov>e}d%LFZ`Jx}t2%F4`7 zle3maVnbK8sMRZjsHX1_SAVYzZw1H5S)*Gz-8JGJ>|4{Nh4eEtFxDLk@8;qp27ghDs#Ta z;=bXeX0NkDeH$=0D8*88!bkk|!oYU+SQOIoX?Io75w=-N?N9Lm;G;5W!@I}m7jJGG z`?n`?2Uv|x6Yu1SsLmK?3ee1JtjM;yJ`U`Lp?`AwXLup}sLDz>W2_q~Z)!MxCD4gOLC^ zmARzUhmxiXg_Q7NvyQ?h^`KgX5_sdHC-4rux;GTvD-2`E=3_T_F?RgIY&J1ONWalbA~ zgj!S&i%OqeB6pk}p?xDqY)lx1(4cj6-*X&&;|dRzRsxGNW6Otx(O1w7%p}Aj0_0s~ zbk(!cK@1J0UMeP4+-%r$v;P#~%CNMK0fuXg?RIH$E~1xEaO|!UtfoGCovBk(Tn8`? z3L-7<12Ly8zQWjqE5RM17NIE4M_~XAeKt9^M=UXiiXnue52fSs7qui&!tfDgW5&T| zxnk-d67{V{K0iLZbe`NKqD)XN3t=NF-sA|$5|F3i5@~TYQ#CbSi;zg;D{jFc(RW=<#1{yuiH^X@JuttY6Ah+Ewp#C?*G@UOoT4-d{sENJpfnstY34MEY5i-=j}g0d4Fb5U)MSj z5f42@>S6gQfjccBy_v6r{ZTob1hx7oDy*pQHgEFG!1>C!^I#Lpl)4O;Vpt3KW@7o) zR;g9w>@(;6ZZ@1q^>ev^S)#(PBM;+){XHr{aQK0lUs)b~4sxAeJ1L9PBkra6Hh4&t zV}O7(TcdGO;+nj(_ybh_>DV~-TcPa1<PRwhL&kxTb^(#Gx0Sa{5FhIu~|)#p;ts z4fqu01dnTiBqUHK!E9!R50Rf*KfhAzdNCaAtNEz2`7c5}`YfBZ>9{oMglt8XG@#m| zR(SXnnF#2B@=Y@BOz_1U^#e<71AWN0(Q4wkCzG6Pg7tJcOWhXu{rVvuVPw=WCGl0i z93LjgqsFlVX)l0&c=>|wW5@-izpfFpomx@`z7f&dz2&t@_qw{_B9M)dF?yFk{zHO+ z8s6Efe!hpmpoigD;7=#b8L$AG8E2Kw;SZPUK7FElb*j9IhCOO$SO_d74+T* zOXvLC4ceJvkfJW~q%El$w+#y=a;lo6&ht(&C&;C#;(}+7Q+lhT!gX+KiQ*F;`cu{$ zxqqQkQYb%`3`QfpR}-jmx2qS-hSefM>Uq@Ms%&+!Agpps4VFAf$%-M3gLYo4U7iXpl2=7-jo z0%ZAzyR3C;sihv5QkBi)l+Bq;r-j>yYQ-Q#QPdfDs*Fhy5|-NNi;PvC#%TwP=ar(+ z@!+rJgZ}K|e3;?%3c?Zs;_ZkEfR0QEwW2dFD}VoIz+=x>`m04IKpk@L&TQveR}{_B z)BSP0Y;&~_X27E0W#PS)7h?Rj@wo=On6GqBw|wc7z^yvF!jUpNPA?!pa5$}xdP&Bp zFJH7DDTg5S!w{uHn(eLPb9IWVO6&8ttB@v}CoN_CF6*@b#xqVXp?oK`QHH|;=TB?7 z_8$`9bFN@PISt6HSG?=XN%NZseW-`NI(ZDCui&qpp^{d+h0HKvylST?c;V+sKc6_= z8z14l46vN%L1KDPnGroK8)mS0uZZ}J(lW&03@5$ZsHjO(dSCDeipJjT`XOP_IBJn;R>KJN_9H`qP10JAZP@WZ+WPZi0{W)cOs)j@E?_)~Z@&cnVdL6Y0;^N}m37muqB zs5uk*D8cN*VIr!^uV4$U`WblVDR456k%t_p2PxAvK^hUwOm%BI6_#q}54eXUGX zolL5t91rSOXFlh_@hwJ6+HAvTp)AOsduaP~&7(2i7rXE)J||iI_M>uAjsbE=bN2H( zuBPs7(RqPuC&YeNdORKs=(FsS{vpJRdqY4Uz`%pi&Gd{hK(egFHHFLU(0OBAH}9j> zO=CvtRIzSYOJ{`RsIjREA%QW#n@&G3wPkxAdETvZ9NtW8uNV!T}6=(Rvmbh z-*?@`tRVg;5+L?^H*YC7<%&W#lZb@k7*LH8ril5~E%VecAFDUH=9q+M!-;L{NkVd! ze;%?k2PrJdr;@=^PRdd6-FhxAdCaFgFjp1oz;#= zJ`4@?eD5v9w!>ulRf5^sMSr7LtdUzW;hZQt1nFXzn&)rY3))>Cce$N`3+@hvL6)<$ z!(KLia(-j!`r>u>Vc(1-2o;>~H0n89du&AVzEuZVpJB&ECj&eG%G$&sL$spF60i`` zJ)G(*-Y5O|`How8%EQE9HzFj9^8(S1h%EB3OufhI4-DJf`0^q2OO(Ep@%IELQU2kG z74U<~GYUZ>Qu3QX?R8A^wx7cC-HJ9{S}w5#be3Z17qiC6=D(4M(9jxlsl|$**{&?! z=;UqCFae^_=+0GM;(7QfH9p}UEKL0iB}H-OoDI=Q#9*eM)mRQ5!{f~9kxqbGG-ouA zf^^Z-#q1MVY?RcvuW+^Csjp`$CQTe7tShvc9+jn$ZFYG(URl!)R<)Oad=^Hyhkx#e zD2eqjF7GAx*HXI)Bttf28x+E~=A_f`vRBX<124TTvh4tQdB*Hd7rzyJ2r-_vTS$;_c?31u$C_bz2e)?RfAUk#67 z^sqvk>ayq`8K4H$n1SOJ`q4CFtxB*oRs3IANcKUcewxK!ST>W@xY?ChJbZg>z|Gm&G5(zI?)jf?fg2T> z{EuSZ-uh^Ms`X-vMmJyW4h@Y7ovK1pYiY#>LUvHK2LZ*MX^YE2N_S z>PV%H3Yx`(piJDa`wd7uqE$x|>r@aIZjt^@!K@4L6cA(A2Wug_ z5@ywz?*8Kwb6iUeMCvHYw?RSe5=>Ddod_ih&bK`L+^a@i7;CO94{@*uNoN41dDl_d zTufNh0c!^=Eh2W`kSL+xqxKLJYkVdqGC-RtEktIVY#gj}osf&!AtXUl(9baAKmwu%-S~a77*Chyi zv{&jS`Q|=o5P4jj7@!E|b(vM+8M-ny@_Ar6qOIE9mhRRxyxAlKf66v03J12q78P=4vG{@7sAfXs+Pa4g8&2s6n3Wt=tPvz=J&*&_{>*-5 zgh;f!*8vu!JbZM|m%U=H6D2g^ZzFFsw`IS@m#P|rgH%v%A@G~nxbQ9j7CRluLQql{ z#;)fd_K47c_uQ1iFF_gsWlxB`YO;JyU+BTLMZgcqyx}d-!L;enq@7V_U|a^>jI?Vr zcR)t>2!?cmn)F1Ncoe?Zh&mW)&fXUMk)Wdne+n$H@`fWxMOghn@S=TPT#9<&0)-5@ z*E@DPNpODd9NQ+40<*e4_K7BkSGoP-8X~BmqitkCws&5HlX3g zC|O`8R*3l#AZ|+~CItYjYdNvB5d!t9=`RuWRLPzKu%1y_Jh?&Dw-QYy@>kh^WI1g> z#>}EylF!3h}vV1JLuIG@!O+T&)p*;H)r^GbD*u`JEZUz>R>or`|TF$Y3vwm zd6ATtMD=PzSR4S2mW_=m2;Px@nXus6R=ErXbWKj!uD=_SPJw5>_$X&Z7lxL5$fIIS zw$38;{^vVKP?!*gM=t=!M}rg>yh`xkJmUA7W3=ieFwsHR5tuZec+&xy7K#JZ@BkST zlQZgno?H}mE)I`-2niFh(qJZzO&AAolUUcTxE_G6**%K9ZIv^e|0+XH!V zH8;4NKdxd|#&_Te_POJMTGQ~R{gQAefmw?K#jyb^uzNJxRRmMD9|^%zQ`VS>W^MH^ zE?O0}2y95pJ9W4)py>0W!)nU;XV{@7Pj{ZZJ}2bA9hNXJVjHZ^S8-Wpgha0Kw;K&fsB zr27LA65ck{JK(EgJmrA`2cS>LN;}c(#g7rpxz_3;D`~kuDhmgzcrM+ko zt+5m#nFB&x21evFZsPa&2lLwC`MrOoz#cRkzLG{RhbSD49p>}WO6C^#`^WTdrKAF- zg4doOUrjxX$}p}b|GIbf-sd3z;K(YkPVGUI{K%MLpDP2eUxi~WU$BdHwWC5!_FRA@ zw`qTah*{jtQPMNk%T+p;Ox8MrLn24hZsma;F5ady15Uax?40~Kie~7yaK*`4P^rwR z;oY-2z}IVL{hChx#82!%-ajo)-=~1VmjF08@QxNJ6l9sw?ps3OU*9Gm%5*##{YA2i zDL_#?i96i!0kW^HrWKaLqJOBt(g_0nY~G-XSXg+Vhx@E*QA|@QoW+GpzH5H9_x=^L zc*f;s?10k=rk7^P@I2Z3nHep5PogjIz{(@xqrds?p&RNh9#3Cc zmi}NyvJ8P;tSO)o#ZM{h((trtW~1l5#OJ(zoQHpYQSGqpifRIt))U%@cvi*%k2;8P zpV|^I_m;Y0*w;p2l}9|*HhgS4jzU;-=%7I-rrIl3XnFcBHG;5^d^hoDC{cf^j~anR zuHU|{G0`jl%=iCXXy!VOUoyb#yD{HHJh`hFcFbt8J*BDftVfgy)X_*{w*``N?`Tw? z-R@_t2?wfxWviRtV!u#fTU71Ytn&TDdF9U50RcXU;#$z@ot8vycb>zFB(r3eLWd1A zq9d}Ysjgu*M-6&w#uyWJE~WC>>RWnqy+RK2a&2D&$%^%}afONsJJh?;Kv(k{)gmeo zgJv+xAd{5=gYAogb@2kBb!b6X{qt2D?`D#8gq{&rF+b*AN#YL59#Y)edJ?|8=!@Dm zp>ucUUrMiO@Mr9e`cSToyYZgeUCljWLB^;_cx1ZQFwI!8YVv@YwSV}jg0^VLAt8a1 z^gqm2&PJWM=d8v|?K>hn`{n3Z5L=`+MoKi;BG{`1i~II*Ngp7J(~P28vQuu~IU;eL zD0b5~={chdH|vOV@WZlq^&v%$|3!{ zHvnH}mLe5<>blxxl4c9()WK3_(0--ugo tUwy7s^{ARG1a# z<*D-wENEx+t$7vvIt;DM^3%-g04nlnfEs$&b2fP?$&uVZsD>cU$sl~ig~1$*!ELla z3@E&>=`YGh$U=}^D~@2fM2sNp17EW3HVYEogN*%z6N6+;tudU~nSdC!W59 z8enR{Y{7B;`FKJ>Q+XNCiCjR#2NgwfQqhzxo+ zd$xEsv4>gO&;%sJd;Kc9uM|NoFqcq0VG`Vx^*EgvtX1#8p4t|*2kD8td5|DXuKrNT z(Z-h8Ox!!^eq}oWXC&)!dNXUt4>WgAOyikg(oylfphstuE%WBSi>v^9qVJ*k#oL!K z%AG(qulCX&yK&BCmW_mvQ}9+ZfmxVjb*K{ zqxTx2QRF?;`nC9+0*LbG2IA|SJmvk8cPT>L#2vW;sD#-Od0qZI91Dxsb=2UVafqRi2b>>_e)LtJ& zbr+a)IH~8KerMFt=#-*cv(w>5N9G*sk-sZ^W2N8o=0P|LdJ+sg)NU0Kh?a&vaJ>u- zFQ5aoJwH{`yj$$QZKoeI(51;Ulh^q;4ic08e29VENwUx97njbn-l!*Vbta$<4_5rQ z3L!e>=4*1@`Fyd16Up2)K9W(|H>Lg%gakv7pi+n1`^40}b88$sLZAF57O&CH<|hYZ zJ&oo{P@SqBAw2xfx`rv4E&z!@E0ugxk9kg9_)FiirL54%G+UuNe}mSth#W(FhDQIO#4*^$JYlB^0b^W#j9f;v z7QGfO24ejh0#y_AgEM=4?B|Zk6Y@GpY{ch&N6Zus-yANwe0=#~PsQ(Xb*7(65_WWzaUi7lke zyHNs4Tb1<>3393zuIp`C7C_o({L}r+0aj>(%TXL-`RB=3KAt}20JI5Kl2@RJE)aRU zsh4Q^y~n$lRqc|*cK~05hq9?{rFP__fuN6xj!8-Pz-+ktpo)sN4Bcs^Jg=+}JPR=X zc&N&kwKve~={UFd$}O057R3?CV2N1S$cmo~DL5$IK|}ag=VMEkV_b5N>8(&WswVy+ zN+Pl}IB~x*Ljs=_9V-ImJCCznCDSld;l+@j?K%JYt0mU|X@VQAUfA146x$`m{EEpc&AkUh+IXi87p>?4%NuMNGyjg_Ao3pT<)32zRlf>- zbYa@ZIC&=?%CHiJHkI?Z%mekfjnKHK@9yR);)6ND^#4T(B1JV=BUm-W#g z>5}jSN800n+^%!{mh^V#^*A|Y5;{h16k?fe{i*X*x9P)%3Nk-G27=v-5X-6z*f9RF zxEnxtzuXl#Z~an#i$i zGvdvlg%FFOS|gG+cT`F-JB>M4?d-%24aKB)zU(a(oBklCLC@jqNXR|Af|ka~2YuWF zm}$iap3XvO7t4GaEXqr9Zv!@%97ia1Fg~>ZB~74GSg^odhJCu$k4_*^Zu}lmLz5u9 z3c*9}nB0JNQY2Xrjl=`gwN#?pfbev);qs_WaE2;;uw8=V@G7Yx#E4mPFL_P@l5)Nk z49wj97T<+Y2{;(?z$7p%+2#DV)u|`VRj{v0X^JY?^b{wpc3IV@WW*MA8lXLdq*wq) z5i=CpXmjona0*?=&GJB*Z0*q0tME^U%x1aj*%h(0!HgatY`Y2yvG;~Ts168X0wl4- zF7YTM0od;`_)QtvDtD#~lyZ}%djEs&H6hvl^b>1hbhU>&TZ8Gdvf$2L3TAwVUnJ5z zDpS{&tNS%iKNS8oIe%vZeve|ii2^7fBVms-5sZ2OooOO%6bBz{U%r_irC=en@=KE1 zy*#ZjrMWGi+QaF9LEAK|BywdNY#5705S7KZ|Hz$}vhI3?zx%RZjv8ST%#u$e35;zT z;xLA)d$t4w5mMu48QSaL{sIkj(XQe5iJ4iyf>F{HXWY2#>jSk0$Q*e8^bkD*zUw{^ z=_TMd6j#zI`@j|l@0rB&RTHr}>t|E1A^J_W5ZA*NP(LMtZ{K;EN}*#0(?vc^F$ z_OzTJa`dKDN*HVdgO70w#FxseW`Ri-e1oalS7B1g3p#eu)uR~OD3~xU&Eg+Y`mLxh z2t3}{TjFCQQ?_hzF)xubxcAjv)PCNJqh2by4*E4wBznp}7!Kkl&`jebv20^LV~7PH z`zCEdk}yrnhJv496L!#9kTWm(%UMJlbSaR<2UzWt>lDa}Zu5IPGTQ4!#@>B@g6;kT zN6uATI1?*y9o>NI4|@kSg@Pcs;{GzkyEt7AqLOYbwCEWqkkTMYNMUX43jv+f4^93i zehl+^W55eWLQq8{z;s@DY9t$OUr3-LBXru3@F=piIL{zF+Q^^=pz))7M?5d+bEG^> z#wPjYosg9J*%IEs2Sf}sO|DIEC2aPS#=G;l2^5;fix2H3r9j(B;9;`O5a~7ffh(=z zwh50`E8NlUDna)Q+!8+7Szr{geHB>nEMZG*2Q!t2y#?L=YQCf6r=ylVXJ5ien6Q;= z#oIg*#ZQP_&GUv#sHr1xMpQRb*vGtjAEB;H9L+pw zI2exI^H^_M=p~U8hXuV$Por#(kHxF$bFyK*4~Oz?$!PCga%v z=a-gC=jHRE03kK73%m{G)EB`5vPjN7NGpkI2CW|otX8rvO%=HW-4$dE0+_ioU_Oyv z?F72iuQDk3e5e`Wma5V{*o6QN*6rj!{`R)^CBKg;#wW5DeP-E>q!5~VK_u`kXOeZ{ zy@h;!g-hs;IyQ3bx+%q_o7Q|09ds|0Z)23Aj(4^Nle@}0E{ipI!JpoJYHUDKRD>$D zg|2KA%v%f#4!jY<`krZi_?VT6VR4fTWWRA1)j2H=c0O&-KeOsq6l_(Q6jY@B)z~*^Sqn%sQBn7})SU~Ibvf8doMCz3H_xYEw6P7=)>4PAa05> z?W{Hk+}WCZ^#KlUsWV`PcJ?gVD>lMyf!7ooV~)T{APDg{51lg=j_T51zmaPnA+$7U zCIY@Y=7MXIQ64RI%bV!?UQy{Xna`Gd!6keXX=>Cp26A{l;pxZrDwsQ{VRckyT~n{@ zAdvdGshv&t3`QQfLLMrr4yla&mtwJB-1lXm7Xh!EF(N4;l84U6qn$Mo& zsPC&(d6ii{lpPoGppP64G?`m$0M{5a*aUD|+!Rj7ABXsMp)=IH zKwbG)j)Um#T)Cw;+OXx>$1*EXZ|L#wI@M?^Dl<>HW~D`yF5$Ko+5Df4h>=0!9RRCzKP6aRBS;)9hf7k@~PX};YBh2w)xQ}DZ2n|7uGh&(k1OZH`b zq2LvYx2L+FiKZYyy?8oY|0Rd_lWwo3rmP>LbAjt%v^}(ob9RlPFfoN^ElefS%oK@T zqj^j%eOc7epgbG+#`RNz!1^MoSxjuvOBmpqwx&(aXbZhZ&exC4n{O0}kqzBFO zrP{u|TwK~^Lw!Namo5g4ZRu4)g;=QScM9B(s}=hafU#Cf=FOqg@fVkus@H&Vw6_&H zXzbakuL4>t6ad1Og`T86;*aU?0)iLAmRk7Qhi*UwERIQgvNXB+6Xmxgo2}S>?Q$Kv zoQ-BYtB3l&{K%i7j~F`Qd0BGLJQ*?erNufN4SIn{_wnBZ3ynxg$u~fBYYP?N^&EgQ zc^KBHWeq+_h|8q{jD>mz9W0$B6wah)RNW6nrj|3)%o!9(=Ic#k2(GjV3YKj3YYB{y$(Tt^{ONz!56CyahSn%E( z;(XMH_hH`HK_AfZ^a@oWKK07Q^9V*-EuFcdCUu4f@2c}?MX0FnhNO-X)aXF#z%t-# z6^1#^gL*C|>eK@|N16PV`y2(!34KVh$SaZbR2W5yytXWNg2RN&$aB32LZ_SFRtD%yccfmu|aHcpO<*l&xjavtI){1zprK+t0 zpq#^*#jH~CeM{~UNboXS?jd1-H= z@LF1IIz2MeM}#*sa~ZSLwFt1QJeoeDR9?f|A6De+j(ZLfHS3g#GFfdJ*%X$->+uXj zf%{sE{+;VfMqFY%ybMDQ6ug1U>Bj!EoX3gyW%ma92v%D=%2d;Sj#fa1bKL8~0v7=# zf2@&wlV?9btvZV(%T=U*X>^C3Hb*55w79IH=FckUww8*qPgr?;{^_+wj)`iYUpB+J z_BeMZl^;9fp=y10F^T`q1u7Apy+@sl9@#?fEERrg@&CzSZR9IfcTzAQrHp3QDXTx2 z>%MFHA=rFeHgFOmGLRZm4G4&MhAqo4A(jif=1N@EPcS|b6{eveZ1kfkSeZvl=C2N^Pa|$a- zL;D}(OdGQScqrNW&lTM9I9ZMr#%ltpW1Ix$}^wzQ@M<2q-`sTXHhezocFkUj9oId2y<*IRx zkvmHW;$`8p+9(EcSCV%ebHI4VivaQBv5wySMja06jp$zqUgJCUCN3HV#Or>*}70)PA#+IO=TD@mI(-7%^Oc zVo{h2e zotf+UIF&hv5G7-y3Nj!E^n?<#6JV`tmlN`uDXXoyup_+8xAtj>_Wt*Rm9rRm!m_K! zk?P|G-i>Lv-_YLupt_pcD>0NmgiIIU30<*HxA`x_y`gh=UThRUMvDDHF+=@iYP)*) zOEZc+`H5xa|^g86p|0oZIga^XW2+kp3`gn>e!k5_l(Hz zfGFlcARx-Dj*g@Rhg(^hMliZSzUa~vfiD*GZK~Lsj85F_wrP1$1uUBD6c&Muo9pSu z0->Il)9cZ+|JbeOOGQ@dU|7{)cDL(F!g#a*6cFrBGO)v;f-&w{LvWu`h+MNkRhrG? z(>I9&9zCB&F_N&(zJ@*7sdh@<1;cbDq*;p0a%gbLB?pyET_@^$pfM2x3=ywc2L8b? z6ZpvbxnaJ0`4QxX9y|wi{`ue`0zw5YM533sW0!yT!o;+9xPBg#?16&*`0<^>MKx{Q1nOiHqzO-PhQ!3r*P-nL#iP;rLxgCp$L~?DW?lWHpqr!SWj({tFB)^a%LyPsEPdq$G`KQx!X8bm;A#4u~*; zdzD`WTE*{DVgcH|1B`}%ZsZet5~^@1L8N}~QQuyqvSQ`qXA@&32*NA@ z;;V57;4(6z&Z`~H`m2b3SyvA~QKo_01cX*vF!LTL=?tQl`o&wt;Y!|V!I2A!(DYee z*$4`R6KnGxcy>HtW-wqNTAWT_H4dWGfrRNQ($w-JLBAT(wFxUuIsPsmqRE}qFnHAW zA<`fr=xuHZGt#3P6AocaMHT@vIjU7*`cMBXfT5@&I9mU*+NIrxD$JATKi!*E1|=^& zT>3u?AbBB}*FM!#a1&(8Ia3lRN#Z@b=d_e9+TIv-a`g!Dqm7_NWMKGIgOgN=6T0@Jlm6;`F&c*M0mr z=2|{&T|Fc^&1Gg@W>yFrs$(cI<{lbhKzqy-zJyqdj-&aFBG7Ts8Gd~=K7Zn%Zw4Oa zux9qM>I~z8CwxS_Hf?b%zo4=271vJ)5i(==wpKesBGkJ`eBYVuj`+SpGfYtJ zHr5ImgZ4EG|4)9nPG~ZH#~iqHzL4ME*3~deY*L$ms@2N$i#ejl5Em10WP>vok@Muo zam?;m%N~p8)yT<6?iHFCc;#Ugz=Z&Q&;0AsBsqx%EP1NDjw~?zQ zhel*0sCIu7>0GRR=fYw^PaUgw)Ocoe;-YNEE}1oyZ@fuH;)VYN1d;`cH5 z?u}Re-T1$*3h%7tK|KL`)P2>$s6G_=4)6TM*(BC81ngxBtkm{maz$A5e#t(p>P1AE z9Z?61;UFEI3OgEAN);R|8n^2KMbmC@EU!=b1>fu`5sjw))(Ni70)ia18i|`^O>KhN zUgn}C6r28o$2sHkJipOH%g&TBuXy?~v#%3abx@voA+MG=6^Tb}gf&vx8zzK*8YiP9 zvXm|@dY2 zO|-l9D+5%`{sX@z3@`-Kas%GlApgfHFxL*i&NtSq9Dg4&vhIB%-wi+L_uw_XG!7SY z2v%wB94;#o5H8=&3f_-IvZH4fD&MX%J&8EhOTLdM${sPWoHiv6BNA01C&Avwgum_$c(Qu!@6uPsz6W5T#E%$|AlM@6;-}XrhNJNZjAL&m(fuud&S+<4a6wLbdS2cEWxaIfTvia`y@iqJ5x| z!`p5pKbgDumxi7(g8~_;T76x^#)S#gT;ftEE%f)Od6r!Y#l7w9IU=G(qE4c9hL2NN z2xaekpeOT#*LVa(&hrB17N>-n^+PyB0h{jN=^Km>{HVzo_xDD%k9m6{s#Y+^Ej<-l z0wKWPHs>xsE--Q{lxqPYGrWk*!1giw%(UpJgN7_TFXaEouot?MvzO3-#=tbIFXfM!kjM31#be;c-BM>D3Ktef% z?5=7_DqY}1(wx~QzBfnDKwb_dFxVAtXqE=ObND*kbv6_4(4#N8v7NIOS=YUH4g+6) z0&-YgZxz*i&3%`Fc*kUiV-Qtgwdc{QySnzv2jd?8g<0tTsn*d=BlTLDM{Q;8zngu( zKp@h+!3w3?=q^f*V|T8X$SshsP_Ks>UEHwQEXhLj5d1M<-P@Q-*&h>BhO z!z8?*iS53LG{sACHxbYS5IXD|;G66&4_A3d4u5OEa}}U989I}x;ud2n`xmjvRd2tw)fV$e7lKhrI z91gC6X`zH-_)-8F*?lT9&{<9I0K~C_=Crw089qy;qzJlFli+`RL`d5SJ!es+k>GO1 zgCSME6y7j8Ia{Vt$b4v8wwP#`FpGP z?ak%(<3A5fL5=f;Hui$niWc+y83%V|7S)My%~o5VGp^)v=-R4>BcMW-F!KRtKEFnT zte*frjT?v}H<60&^SR6t=GH;()8DTd@@_En=M>GO`Fn?eIQ=~RO^%4sZ`8KHD(Spb~5PM?fH!4 zA!73KL)|XAS^@p*hEZlKBBH+@Wvt8#k@YX665TBbg*v01AO8$5B-^kBvEQGlZ|N?( z&A@e9;J)yK`~8>Rmikg*+WomWNv7Bhjld>*MFj}=|2g$r5E5o24jSNP`1%ck>e@+t zj_m1@5d?S}POJUiY>}x(S--{r2u$*)8rn(g6mISqF$k~KIkwb!P$p3Fy#R7I0u`~? z%s>Ywu6e>KoI_m$qh&a$M${?0ZL#l0!}ctdKd5RE9QCPcbYd=MEt0GSJ?cO9$NATd ziTAdv${&$#0&HEb=D&Ka`_}>DH>PVSPV}`2Vlh?9vnmsTJ`$>r8>q=KJQbDBUj-#g zH*eS)PMz_kHaa->)?hf8EHk&1(`K^#QRJe!dDKzbjvIOjJ5$fY;J&^#N#^3s_Dq{< zYy)~Z;tZ;_#tY33r^8OSm05bh1QRRt{Kj^`pFB!Xh};bBzN8$CaqwtQboS-6&nmkg zH3p_o^xe8saF|NG6$0$ax<$I=p;u07Gm>3FHFwPBi(^!*!?K+0f?4X==N_Y@7v5rO?u-R(s z`4QmYR=q|!WTZbR-#BNe2S>r&V}tu!W$*+LPjQEsPTX0}k6sQWWufL)wdgXok-!qN{hAN?Q^~p@1xpmT~8F5udnFfCe9i; zG=CZdcRrGZ8~cZS-8R--_ae0LeqmG_)u5x#P|F6=B@SC5H_lMI##fIU8am*>z-|n~ zA%8#g%tQQ_+3K*%2-X7P{9xkdP(7$|5(Ex*&C>*>F@M>AT?)QK2r&_C;7;x8WDEeW z07*#?y|LH>)+;Z;B`n=HiBgM>G{qvA_zWHhS`?l+jYB9~dqyW!3st)}CBhXTM&W$X zY4tL4x4dR3KB%I{+44LsyKt9>jT{YzYm0p=_da@y4J zyk=|GlrRn;AB7zhS3>=2rl&Nzm!xkb({j@x1@&f#*360H+GUqBrul(3tfJ+u6_6k- zIme8kBH=^B_N6C3emGl7Q!|~j0VG4tr*Kf1ZFOaCW?E(1GO``xk>7K_8MgT%0_m^8 zZXFC`?5rq+VOR&DB@1s8K+OnAtVi{|CJM1bR7e>CavHRcB#16(z_YzMA2WCSHE?{9 zIGL`z|G_!+hWKtHukK$sLs{SARR8u6H|s>2_j7^O$647xdp zP~RzHDw`^fzVRk7d^u$q5fcelt@)@{_GE0H7bIHR&lbT2Gw2d+?kBog)GD)F$aVi| zf|Zo!jnZKsOh&vd466M9N89j@34cm-m+pd3m*BWw4?Wr36EI6>(8p3pNn&qsSB5?i z&{h?<{pG#8biurPmrd85mL0{cEgC}f+%O@hp}aAd$1Mn00c6j(L3w)|F)A)R7f5p+ z*l&vprwucVr$2(qc~1;&uD%IuXkIr!@cWw#6kuyZG5~~qt6hKgqm~Wnb2{h)&N;j6 z5pq}nAk19&38pj3ny&4=U`wwb>$+Q93~HJbJvH=4CTFSqm+mmzt*exF3)9j(KmY9K zXtJXLtMV4>P{P4~Wj!{Z=?DpgVZoBIYD7CK4JKAwfM%y)-oIKQ%doTrc}(;-XaZSY z3U-&Cb;Ykg{3NoT)8JHdjY1q`Y8lbbN8>=5z%Lynxzqm8mSM}tQtsNA(Ws6rgSh)B z9Et)9^n#q{V#QD*l@hMw9z;k{R!qp#&h*G#^5mWYOSt{cp zw}>waNhAO9cH9Qld$__%@Jb8qGbyc*W+l$9jxM}_{>xUm=@F)}Av=OG*6Co#0*3zk zxi5byfn2;dAu@=&~{?8YB?V?Z!XRL-cj2NddDE0}R_(d5FNM=knoj zKcc;nCJc7n7nCaoVsQ2VRZj?1Eha*(YYISoZ~V^#pm)b?aeCV7#;}Mt>q#sUb&0FdXxbn^51`@JeP(?9Z zeS_2lL)pvQI-OsdgRKW5LF=!ux=w*8dV5ZiA59LHHlAWqC1!uqA?&>_?h%A9svR?$ z{LAWFy+sx$MNP&c?*cPyJA%dkywL6rk{66r+gr>qfe7NyBoIKApW9`5*J+Q8eTw$N zBG9rCZIt%TDB+i^<91A#;(jq@!TAk)c>~;0qQ%0LjSUCh@-pPHvRRKdudA%AN7gNW z#Z5l7o1!OsBWw1zMy{(s0jILh&;zE5_Ku2^>;u&9)aY<$u1@4G8-C;;)H4%Y?F^Cv z{wDmg#cBMVMrI{3)NL{6{Swc3gIyDd@G39AXL3CK8^5-Ox%md}odajZQPxjs<4&js zK;PP^vetfcSlzD9vBZC3FPICZ58;j}{ul@31~m`+79tAoeeWU-&atAD zu_Nl)p=EZ|Dex#MY6YI0V~)5#P#X^+y2bx_?@~UP7thFbnI$v`!q5;-TT7?7%<dB0Z3x}tpB|AAFhyt7Q3JfarP0dJ~ib*vLp zQRG$0e`gY|v?Xvm0mc?ED|}UdoUu_ZP@1r{@HT`O%iR8eYWV?>0&358D$Se-$vl@Y zgGCN~Na%K05dTWFRI5n40%KvRou=q&v`9i(_OmJgXTH|u?c}343*k9kdy1W^FBx#F z1v>;hwuicDGfcc7`I|YGi??zJuG0zsVlOtlAj(Vl_>BDJ2DO-1agbW?T6?xP zcHHzAZ&qOiCH1VZ?;R=kgOQicjNk|g?NasCZXzPB(Hx(WbP+DRYcA0Tu}lw-Y(>?`fBzZMD8mkT=OAI@ zU2o4EgPV;ZX$5+?^-|-ezE07(EE;b&3LlqqZdU8hxO3vlCne(_hO=^M<{YGcNRxo& zAT)IvY`I8d+5QdqXuOf7Sf{m5|H`C@D@sZjV!4M2ttrM!6)+z<<0e}vU zDpLp$q?n}S^hA*IU+qe4wBSlIgr3)#hB95NI#^5?NZ#a&Fx>2hOq`=R%|WzW9~*~h z64;Z>`VKLhKUIr?94X6925@VrHX9+%ZyETfB}aJKEN7;pMDiAiYr|{$h5jF8bO%3Y zO;id6L=iZo3=;Ksem-nMh~Zu}flIH}i$dSs|a1b-^7O*bwY=`A}fkock66 z^)YA|AS>fM;a6q4PFB5WjfXwA{-5b|A=1oJ+g!wau;pP1?jV{_zUaJUgrKr}cnfWY zvZ8vdDE4qCsbe07|5qp%cf7*%sSoDUMCR{ohguDD<*(hgH>Uh#(j}Z%({1d=NwDP2 z89V~huhNq|3Yvgz>^uc~5h?@%^!9n!_7r$~NSq`ye48q^jY=|>2ruG^4Szer!b@)Oq}#=CPPC-#gU}Jci`)S4+Ki$J*7Aq zf}c?#404%m*VCOFT^H*jV7L+=XN?OB#KBp*ue~bkn9~=RpN7_mZ#s6{<-_8KrL3 z1!rL$pW>lO1vHMSkG@S?$(PnF3$1q0&&861!;&*v^S^>sHlqj-;f5Eh6c-uu=0iyv&{Nl%IcW1ygY_@x6h0HK6sNrJIAUgf`6Ktpl5L z3*EoKwpQkHn=u@5(z%1y-3Rv}MdXY~tsqbhSevb?vDU7}5>*JRav+aHnPi03r2&P` z8Ck;(x26E1$D)YTM@Vt6=n_8dn8)ebR1pQ?_d8aA_toTSFWFNSDc)-jUp>fi8MhmY z7^*uKcT&yFq?ki_Y%sDV76ednQ_Z~|JunQjHIw*}W(uZy*^Y10 zsBqpL$QC9|OKVhgs&o3p1bHYSI2DLqIUfkgh`PZ=f_+q1SMN3NL6J;yNBAGdG~kut zM>UI?Lyp-1n8<8$?(C$KJ|)R)5oF<#S)O<#StkBoqD~-AlAR!!oQ*I z0sTf^kN)}Tp7!q;&j1j5Yf%pMRItE7%E`qF%!TIcN}x-u69n#-$$9umHktB-YAtyr z5!EXSjv&aVa&etk6IC62@HR&sSFXAf$SA0sM%54X810#X21w&SpA!IqGSNKKB@ZuE z_xC+{6uHxxxL`xCKx&DqY|`Db9Ne+uS1Oc&})9j2gH9yApU!@IP4b!7@9_^O|S5^CHHk58FK}IDt2!(=0@j_Ua0ck1g zC5U+}77<~<<3~c2n>wK8sf!fXGVG(xvQpREiVJWUO&ZOsO=w*@p{Y@ef2X3cEOV2V z!;P~pGeSfl(3c0iW{aQj@G$^qCC5o?DSY$~MbMaskd^Th-C7w!1y9@noI zTsb|5Fk7|c`xQ>f-H9Ygr;JOs`^y)v`}AG2LN)vNACJ_ET1ZDkO;LAXM~CyTN@GkM#ECbOuhhwDKY=d%Gu={4(~ zhcPjoMex0*H1l}UDd{BL=AmspCX%l^(LABiV1*|z!jaty-R&xM8{Ca-MD8V8(35H1 z$>Ha?jWe7>q?tipw0&g!staI47uad*4_XQt=${G#V%}IpPSacdyw=F^*C(zTmIz)O zDh@Sf*{;;`+$T{hLt;N@*aW9GBK3QF!O@77xlqcjCgnurPjj0A7TLSGCqF6f)+(ce zxTxRpkCFGrFAiuc((3I#iZ=Bd$9d#3MUqz2=?gmg2(NE|sQxpRyT_K6_TwYFj=B0Y zx&^(Va_q9yC=$YmeI>d`@CP#$q}9e$vSDvv{=Yun+5SzD?-o?n*R*{5J2b}*5BDaT z4}fHrdBYn;jD)2JYoF-y$a~}kvKkg%o;WwQ6FM$_RQHv>GpHgP$4v4*Wt7oE_=cgj zeY&#ah0k8@4A~g2ra}#Wf>96otj$549;fnJWZIkUXZ0f+)fZe>3e?1%FH~U13j3{O z0pqHkBx%Z5n>MNgV-wTDo<#?}%EQ)=l}~R;QZp?&TOgeGD>{qDUk$_|$qv4YloVF< zd=j)tKPvX&Mf%i$T-W^66+h*2RM3n=lnu2oG>MGAd35>`jvye#+i?_Ro^0^i*(2cM zKwh}G1ull?)f~G6U}i=*Xh}GUKd$|U>%yyb7;HsI8s7jw(0?Z>hf#$5L;t++FvCLj z!P^$}!1o_qYXFJik5wg1o!JOOj*O@QKt~pcuMCRL8i{N*H7Z2qb3YxssLUPSeR z&)AD^a>REpCP7n$bahmlC_Pu9SA@%asWDyx%Q)j`qdKYTN;PA=YGi%Pq@wewKC7rc z-`A{-s_*bZFSjcjAB7JuT>}7K%i&|KVL7v>?!Pd4bNcC(1oP38Of(Ok(h;&+QH|x- zWRXwthrxX}mdQN_*gV+ar+93a^r-r2Rh!R2V`Q3YS|EJDsz+)k_I00MDs+7x7MbO~ z*ciXH7#42JpdK44)96$PWT~SX8Pow^Q5M&3q?GuqE$s0>9R%J>IFNaIW zH{`0-@%9ay%iZ?$QXe+T9koiJ!-q3x@*v^F5zsfrcO64SgSea)vn~3UE=J!9V*D%} zDKc9z%k`_}5S>QStLjlnu}OYK9Tth;({*u*fG9i+Q^lQUN3ji!W11_}jI)IM`e84Z zGjlb%vM98zcG%j&`>;RxD%oEios$Oe&OkNNA%LVFptK*02J>n;#In=sM&@hBE?;Zp`;{825R6yZH5&_YIzpOLfqi$)3N0v)wpW+oG-O>8 ziU6Uc?sxRD)>j2;NxX#r)GsR!=(3;Ii)JN{y&C~S-}r-5QMJT=Rt=*o6pwc^;BDb) zGH-gpL+3Dh>BnrZLJP4!o8YqF2T)GX^@h1O2}ljKocE9{+x$5T$l*%oa|B0&eZf`5 z4tFMBl-UIiDp<&RVf(>epU%1ns_k%(o5=B#=sq*oEP$%HD;94Z8e+blRbrZ3!+#uW zkrgU@i%(UzjF?#(Z3I1=9e814OX>qg`mXiDpA0h)b(ECK)u6l^Hb0T_xBvv!yj~>| ziG9%Gn|X9W{c6C1vH`#Zdd}kqAV_d-vw%(e8l!QI*~$9ShJMs8WcS{oHH&&Pi`rNt zTP8*L-(~G;BAk^hgPI}100Y|cadym>=71(ga3c1ezw_n2@tQdMdr_KQBB!M~ahavGREw;Qkw= z6lQ7M*&tq%@6jSarN6|6VLHLdBZ9ECtW190c+?`9ELuN!mp2BzldZ-kfu>_KA%)>F z7YF;wRfB|4p<{RzX{hu^lyKWi9s3rpAkGIj!;9wiIXf4XXr)X1k7t&Q^}^@kps^-~ z!o@a7Lld^OR)}`*We1orj^EYBFTA2SD%kS9uHueq!>)oyYkTJM{jGy7BxQtpM)(`+ zbzZtX_Zr5)2mR2UJnYyT*AmtTuB!xZk%3%L=zs76#c=`b8Y7={^U@YOEOXDl1b#H` z@I@C^cn3ZSSpNB!3Y=rBjFcR;_Sv}%U0DyYs^bSc@_zgRQhXpA{dAydCN?u=HwL_5 zXIQCu9;A3@xyY#GU7s?$ll*rM-C{s4j?49-$j(Q_dCv&iK>S_$u3 zW9{<-1Uo-qFr(aWRnncFMj*Vhp_=@A}tzSUe?G(zk*F6uM)(>ySIY4QM)|X--Wh7H9Dam{12pPhORzOW% zc8Dp~l#=vb4Nyv5sG++=1~;kBfdV@{2Plo2z=6T`Cg%wclxnd^7yy_=G1NSGrqn)c zkvv7Jo}k#TE~s4+4N(=7$%O0a>p(x`Pv5Z?1Qb&{8mj#47p@;MxqWWcKG!I^|Df&G zkUW!Obco3OQ`zl!B+{n$???$;3q!d#$PHKENn1i+YLHqm;CsN>T6=Uqi)F3_0ArH$ z6d)AOgUaXnf)wiBUPJoT$erQFHL{lN1$jL6JyleL)9R%}vX~kUT+%0fatqeRkfz$k z{%}wc^zYsLPfQ>JCCIKDrBr3)9tpcZ+LL{g{7CjSsPWe~O)WHH&Rio}xg#jMu*rI% zJrg8A^k+6tf|}_m{I8uHI6#86y+ER*+*uj_?@%|OHmTpegKiJk2hNhxLvqs0(~k1$ zTk4ob1<}E&fgNBLS@`tcQ?oN3FtMeZLbRKjrx7IBErYpUnXs0X&=wA z?TCs4x*^8=6`ho%kFLhTh5SUU#r=Z1eNLQO0y+7{Tb6P?hW$1FuvqU4Ckd+Htf!W2 z7!>v&j3J0A=R#sg?<4xAaYs5sIDdhF#IcBmH3d2V}5uc$qlz!WtHKDr$e zmIkvKE+bfzQUtSIYU4Yli@wgTPln$L^>dqF0X}8Tp7TE$i&VznVVx2YX+FpvH3Q;( zgEOA)qFpE6mUTL1Hb?F*>n+#8!9mlnOZ+J+r4|7tIBiEyy%%iel2-*xw9R!ye^^<1 zKWtVuSF=H0dg|_~+|Q769|xRA1jEn5hmD;>E#*Qq2dsAtQ_?0ed zb~NlhblL=eYai?a0$xRi8}(C-AL8Yj$VTTHgl6H)G-Hw6y2wi_)UNEd&~J}&MjaT~ ztNkX=@EYN=Tei1;Xkr?3Uw-u^W-|9$I%NCYe^;$Y?|@j-48S_id(8jjVi9bnJsS)qtxByOR)6#3z%&ZfAW&IC zVpddCSHZa1nv(c!=t&*M%sUH_+g#gbIs-Z1r@&1{b-sIusWWWv*OM^N8>-IZ^oL)} zfVuinRc07DBuX#Bw6h*yj)*_%ow8%mHw37NPwK@E8)9`CZ{|`|ERwQcVaI zl|#!Xd$}X$mcB$WEXl5n3%#cCx&U@;gnHA3T3e+C`t9<=O^PD;GA0Z|apg`L12h9h z$?pf@ZTK1SL0@Yx*I19|V?%kQU=vqMs#{}i$5##)eRtW_&2a_q2KkaZH4|LpV}`Ft ziz{GrcaD_YLN=K8S6X2eK>L^e%=LXM0m88NB_PnvHB6~Gpy*JvUx>q;< zi@35@=MdMWz^=&m)OF2`0BCxiQsGXUVN^$?KA&k-?indxUra#~J|UBT4A0&dkO zRs7+xk^KHGRy57GKetdID#<#C=^^r$MvSlWpr7!|`u|Vrw40D43SWG;`Cf zV7q_*Kn)F)8^t_j)v4`S3khFDo}PKK)k_@E=m&-=AcTx*Mm_D1&WJcAvI7y8p*7~7 z`W2>@D-TBY?4br3t)UB^igtWJ=KOvo{FX3s9y=@5x}U6OKhXAj_(Rqs(Tb^>eANcKcHxj4KH>Q`gp!7C~W!|WLe2b zA9?sCRjnoArybpRrwGc^j_S$}seXCLb#S#`wh@}+zD-2;%t4|_B&_;ErZ8Gt(01+Z zH$rX))&h{~>+j&Z1#Ic0l8dC7X$$uofdEJETE!~ z2o8k(qgb*z#lWOM9X(}d+3b$zytS>TF)Ys}q9qOru`s=rl8Z)c=u2B?3>mE-=hHB5 zNPmO#;<2OgaAg3<%(8ZL1dDu5h=Ow(2Rjuo=!htgBHu-EA_Q{h=b|DeS(c;%TVNVq zI8QjEy+Zct+hBFBd$6gT@GZST=Dy}Et7OQL6Zj^1&B%yVFBA^Qlpr@yzm8t$CyNyj zMoMwgBCDxUKK@?5xV@j(AcKvMmjLkPdH3wcYTgSk9wP)^xDaDk*D4Fh2*&_PY~#54 zoe(s|*_D`z78D0h3eBV`Mi=tf;i}xQJC~?h_0jxXODnUoxXCgcW!X9xZ0RA1$Lbg9P^vRxFL8oEI{VrhM0@K3@!06DJ#QQfIykM#s^A7Oo6rKxr#}<>_`ll zM2we_v_kPoQQNMIXx(vz3cPRj*OR5}-D?65VB!Q9Z~!!FC8AzEh3Bms+`5dlD;y3U zqLS+ElffyNkop!O+8i8RF6vE~S0ED6 zMbAiCqxVxUBkdz;B{{`!m=Zy-2TGo^VI3pC$wQ`n(^ztk z=gf>F>DTj+H29Dn`8<@&FOyl)AL>Y=-gjkm96%~d8cL4qR(R^xju+yh)vCsPl4ftvprkpRPO#%wInC z8(PcLd?~mFWOjA?HOXAtiW%_!SoG&he8gwO1R5(Wd$t|)wY~jIu;0V2bz`_{L_Y*W2 zY$cefGz|tga(Ls>oDBYPr;s3;&&_hoGmY_(CJV|frN}u^sn9v(<(5K}P$D^NBqXXU z+2y$f%twUluTMy-zK&RCscXPy0&tjNcw=zR?Kd>&l%Y3WrEiMz8Xt zYAp1uF>Hbm%CwPyZhv9u3o9xgyVq0O%1s&Vq1ZinREwTHroGFv3xx`D1xz_@_i^NZ zc^=UH&~4f_dtc(RzXYfWAx%Q(4p*fC#jt-H(1+mFjh{L6!Z=g*LkP5#KE!M0t<))B zV-56@r4~p;6@a=jsDxDSFz{++QFPk)`#|aY!9lIF3BQBQlWDeMiAwVc%Ap`PAA2!Bg*&ZZocV(o4A`RkYPG6#9x5!?Y1FB)WlK=)c5oYYJm z7XeZg0WM?u9?Sl5aC-eqy*71*`29jz>-v#;-GOt=R*RuaEiXu_m)kgdOpZa?oK}1F zKatl5{itXeNrO`J#MZSEe$ovE(_3QizY(y`zmV8b&^sFi9bBze#rD$3&k#or@UnIr zu0|*iAUWsxlB?i2tW@Wd%*tuk0RDYhE$ocDqM4`fk6P<0a-3MH$jzsix;%MN1P*s2 zG~^z-kWZPI3l-<0YXFN7CURM3qp!7{pz$U>fQt99km4MnCjHT%3QH|D{Bku}2;lK6 zRaxZR{N}Eqvc%$%jnZk7xhVFItuA^t))FvCF2BmT8t5nx3Nb|JTppCFL>e|`4UM+= zakh$m4?n9YXc)XdsER~AR>E*^`kybnxdB)=D?(XnAiF)5D?%ztO9Kauxy@6MRUk83 zEopgG1qfc=l$Kbt_SDqoKL>+cQxzb#tis9Ic0?OHHrcqQsOWFNx}flBJGtIJGRU*F zJa&5MH(+4{1ew7H4_@7Zyx3MTfI!pXy` z(fhg@td&lX+7P1l-X_a;7saJ`qLwWCyP0G{xj9QaLnIRJE69~N&3Zu05CA=tt+Nc_ zhgb2-pko3l7xA~z%Q?l4ABg9qtvvjw7ZUuuF-3ChDb<~KFJeSeowrK9Mn1=&^T*+i zUC~1WW2*b{Bw|h?i5~MRwg*)8OHmCvb`XOxG<$#1AC`y^f358FOz>@`g@+wTQRn7j z#btyFV4-C-{(IoOr6lPTA0h|~&u&=TH{D(wj+P2I$Viii$5f|xxhY`?XP1;ehn{6O zZbvO;%ghsoN3}1g095d+FdwBoqf;7J1Bo+c77QC8mpuQtG0kGRq7oKFr36N0EPkC8 zh18etyig>d4oiZyj2UX4;s=9$t|#A*Tv|E^O09dM>1Sk%Zkn4r1f zUIGDDi*lATp}!;04Z2)K_>;W^MD=*nh&*Hn^~&dG7=(jc2d@g8azk^?&*8%_+@|V% zr%)g<5s-WSI5V;ml2^*Dh2vA|9qD5YxCYz(pnUazibNFz2z*Y?wdk_0@^a;en3c$S z>upy{#qes`oJj=uW`CGlaq|4FzS| zM3X5K!jnDu%>cReBFKUEY+8N})`heRH96`jobGmz(fs)C2yFVVBi!`wU3UwTaB@JV zv)-8~DRL^=^$K)fjy7VkW%m(9&+le2F;qF@WITxWGV%^NUbNPti#yy1q|&x+BQiuC zdl*g(5AA;V^#@Ddc==_;udu%gbJyDGn+SZx$`n=%g3)aSPGKkSzpT2DUUCzjcJ)9~oo=%_4 zk8xmeksLGe3oagUc$(8K)ST?hm_Xkumrq6|O@tO-M?ED*cdwrHvNMSk8!eztOB$fR z{FSue&&3u3tO43i8Xt`WvGuT~8vbQ6k2l>2+TYud@{wyAe`ZV$;F|z^UyxeiEC$gS z2d!J#^~kjeC#)Ap3e(l^8@)?j&CPYqDrGg_j(>-XirO6!nm9}B_xijHH0cu1r%2^$ z$cBiBsu70gC!8!$`_sTVlxQX6CPZCS=47U}fL@IspY^c6FN$npo~6bh1GnobJtok( z>flO5%Fu9@Z^;&M?nBYm*jw443X&e7)UmJg2U6W&uv-i1!pjrGbWzm~L6;R~ppF1? zBM|Ux=07cC4@uR?E};(<-bf^Biunr9U_xb8?T6t>0gEles-?H6H1Qm)ca+04kb3w> z!v=?h?HZrU_F#>N=n$5r7FGhSp3Rus;x|U{$>eq~nP#q*+=UFyUMC+a=nW*d?JB+0 zNqxriY(nIqUf_8>5Nml#UZL0<|4D>KZx%kI)i0mr26_8R~O0{*Vid zIL#Qzsj~vouUUqO^)rp?8Uw|(ms~@L2ROPq)~P&5nCS@;Qd-*#ILBh;64X3Devn2_ zk`BT&){{01CaGqTciY3eLM zGo$)b*kNJ~k0BN6U%(sxWH(eDwLYp&;`M6VD6Se^QDHgIn);EQBu&HVSdtMSWZG>Y z%KzeJFnt+`dC-1vhrCxdPb*4HS?GaL&KwfSf%ct{+UP9yZ^f}a?oWS+7vk9j$1srAhJ{wKVwYqxyKgcOY@GI=%qhC}I%3-nnK55q zx23waVLm(#&$yF+_xPSZ3%~mH1(g!u9g*?}rvNE&tt(?5<_GQnfrRJ^<;lEHXe%Np zMl4@LRq@9LL3(ZDy-lAzj?6mAF?#6NMSCfDYr5o%?f7u>eCmovy1#-cYd2=Hw-8g0 z=Trr2qY(=1`$*xR9+P@9hD=p>71t$!qQm#|V+}@UjOi#ajfB3*KXF7xQrpID^4GxAd6hQQyoYxXh7|DAsnt zeqDSa*%AFaWy-1x<67Z0K@dvl1B%ecBJI7uZ0+@sLOct+3WM zcT;w;oBae^aIgdDL!KJjcTKeVQ#nIpYEPIMaptTiCj=PCqG5bi9X__vQ@IK_5r(&m zxE?Ky8J=rTc@BqytA?520N3%~Bm6-^i4rS?C$gG{{bLNdybhfHw!NK6Z#VeeV0Bjp zvml^oGbTEw5hTz86;DJ>$~jdke~mUC{S-(DQ!L9M+l`S5;j#S2Pr?7`qPR`R2C6v# z*#p_Wu+V30LI-$+c3mYWU)_51uaO4tBWk~lvv9AKE22<&V(!lHSy0Y!pe7wqa7Rr? z)%E3DbMc=m@k!r|qJj4Ep9m>kJ)OSShcOBUzij$Y*w!?x(M!K$h=~ZZbXL-@&lkbq zj~N8x&z(U`HIEk&e^a(GTY?WTaR<`&$R!{jx5jLH@<9Q53x)LkCp!6t+=U1ntov|m zfUD2}5PkMHGv7|HLt)ustLktAFTDc`?|r^2O#-eSTL|ne@IIFHJR7z(_r6BtBz@&m z*mt0g9eS+BJC;~?PtW|`jel<95T|a(@PlF@CR2h%?FO|QeU#hGw%yiO@v}C9ld~@S ziD4lo2y5ttSBRzlpUF{dxy86}EACKp`Eg2b2visIX^5<$lWDi0QWalASzY|gRb5<# zpN@YycMz(JVOAhHry9clH}MClF-jyR=4GzZ)j6lWQ3$V-V2Qy(6QKbEK_u;)EKa(& zEl~7gYy=qk{QvuE6+lanJ}Bn9w;L^tM+Bt{y8=5^G&C`qg>_#U(^F!u8C{`lo)8Jn zL|@81w8E*F0b-$Emq6}S5drQmHP$pQDd7dh8A5cZk;qoqPW&*#5^B4itGqP2Rh6dS zjpeDi^1q3^j67l_xKVshB;M>)%!yv84M)03a=4g{%9-Y|CI5CuBUDiY7s()=w7ZG%#A*^ic2oh-@N^f+ zm3uFF4`ACVCk&f)rGt`=`y&!$M6sB&#>oe2ft?#?lYt4~Y@&0bQy{pEUcVpK>dq_S z3VCIcWZ2tHtp-vZFkq)PlX+goh+{@!n@-{p_R*I{ zNHi45E%%J~z1bz>TGJ#4@6dVgC$9NAxn_agB|QUYn7m_`5)`&bn!1t*FU| zo{mH42*--o8Wj+?sShJV>yYDg?>7bRg5X21R`6mHg^r;N#?g@eKs=&Dn*L11wd8v0 zE2vi=5K|6qVV&5GgL}iYd6lBT@R*Idmy`w~I)D}Q<=OGjnt*IR0%%TG)%PA2a#_8z zyHm0kr75q;FK{1APxq?=NmPw}Jbl3qM0v3|W~-Z#JT`DF&ue@spb|Ngq+9unALv#c9sajHQKa22d44sqf zBC{D$#g#US@RT*I)iKSNhZqV$tO-V*F)sF_6wN})7as94YH6cv)PeSh7 zm5%`)*DhLlG`LykHV7Wz!U*%G3TwCaShi%Q%+MKSd++^g;z&Ux=6qQx=)MiH0w*iR zAFDeOu%RT<{iNE)a7$17_afwQ_5$Av#VX1+Mcj^~`#&&EVCp2rBT*v%ulF+gGcK6k zd)2GXLVqMNok)>A!k>)xWIjoYT}}nZ?A6aC%-a{z!S*Zt@-$O`<|(x;*MAg?yqa&@ z1xtwh%-R0>ri|H$B;fj}h(?@!7D3Jkm90@-RrF zWE9%_?=>I5&%jZ;gL5jh!5|PW7>Etu(Cj^TsjoD!ya!hvn|VIne>fZ{od=clz0%Zq zVg&kAH2KEB23- zEpO$w%B8`Lr-(}zkS%I`205N0qHhmxfK`S~6)Zs4s50 zMSV~am%!_3uV1ASISBf@!|{YrX1W#uYdj)$R;4b4B+fv>rmhd%A>O!*d(%}S#xj#7 z>Lr>1>xk3pkQc3EXfn8Ned>)@ST_fsH=uIK@e_Mp*nB)_1fm6iOVl742aS1W*pXGk zkEJKVssFDO42a#K`5zDbkY$`^Yr;6`iOiWrcHiZa>mS00LfNyf-SP5+>&~my74r!o zhuPq}_5q+jlUe_*RG@9Gnw{-NjjY~}+kg%a@jFFGG))ElVQY74A`QX#CP5MnEAU^IrV0X1|%*+i0 zV*&(MwMr=9uR&B#>XQy|G7jakH#5USkB|is5g<_bIZNXjoIA%!GL?`^ z0eUhAEy6(G6p(y}kvC4J%5T4!G4L4fa6w@-M9|AcT;sPjX}|`jap~vla|Wip_7DOJ z>483n$2nwyivl~Crev(kCh9QAeW9T^w4Pcc+jyPIrJ1-?833q6e{1%x;V|p05m4%l z{^iRkOsMSC^}C$KxWVOYXt)H&3sLlMde`Jq@HenozsGEATJLQTLq3D~bd*v+AZ$z{ zPYLi@)2R;cKbTA45c}#CPh}M))K^rjn}ZF*c^UllizF{{8TODaaxeW3Px}O9hc@5F zYp@0sO$X(2skPE(mJxc@FJXp zdCYz%1Y0PgJkjA0-d8E=asAft+;tRDwfzqg>eEK*RfW^)@ufp^iydi3A^rkx7rz{~ zdthc~eqduE-}+~N_(gYQ4^YvXPK&?-A%N?lC~$3<_|bLWEfp3f{i!QQuRdX|&AJa0 z)rF+OgWTADR?$BPHl>^-FQo5cLI&L#sz{Gb$E3Y^6=?EwXv*76)y=!mq}pWy7(P#c z^!a;Nq|DKLpEf3%qj7?Or*CAjo~o%Fp1y^{$N zE_JxYqs+h5-bUByEV%ciGvYvB9%mB`UTfu3d#@Zmgf{{zt+XzF_#0K08$)1 zA36coxb^J?SkX;hunS>0Ivaaf(z2m+AB(B)g$c$pfSuY%WX z?0k$(APIy`_PYE?zp)B@ z#IUt%(IjHsOl_sHb>v>ar$2^s(hqVM)um)3L^%GMz7ip68Xq1sqKKQK2OgS-DnLq* zOaT`XURJox*UKRa*8u@S52wh7=kBS$`_fizbUh!WOq5#;7H6}r2VA3a$-ev6RiyUV z#)kB0*!Q6Rjrj&EJ5I63`So2V>EGN8otaIqdZ!n;7xfET6h5je0z4+>i>c#6zOnPP z60*NoTKgO40l6M`-^`=+##lCJ(IqWf6S^2(n2@fX(GR^|m35vyt2(L!bzE4__HN|#qwYmI;u1B|O7&yGc){kbQ|5!gtVQl$`tzE$F^Z_Lww1Dh(HA`NtI5lP_zu2V2boGT5%#)*7Jfo<~M7?u~6QuP+}cb59@!Ff0Z QOaK4?07*qoM6N<$f+6L%f0U$@<_!^mIf2@n^#*aoJRE?os!h-JcciCOn0WajCmdj0k z9D*%qTd&Otp?4iODK_^+eR$Tj9(YS4fb$sy>}Q23+Lbp`+`-;>BC5*kJXBbzta$#!^SkW}+Z= zL_SGm-slt$))r#R3RTWm$d8Y;z^<6aaSm>%B<N_!c7C7J=z{=m#S~F z;piM!_s=^x0i%z#R|!Xk#t$!&;Hw}yA-zT^ki5r#F*ZZ9wtFp{QKw%Cj@Y{h$Imlu2LdB2Y{0n;w1Jy@augt`~K$ z0AGTQY`~?ZVqjZIj$`orQvsRf5i((BsLV2vt`MFDi^=!JbRJ8YqItJ968C(0dkrtw(};AA*%BZw?2ljXkoNRL*ulZk+Jdl%iJjLc+w*_ku3K}p; z=qC;pc@br=l??RB#bg4G{hj^q_+yL9KM@_=LM`AUO#Ed!8NSH>!}S=p;s^&syqKi7 z1#*Y4qT*g^8l})Koe~2@;2I!O?FV%t+ftg+`eWy~T73%C7`}IpIWpTqPbBxUn|T`M z4jiFEF~-Em&UN{RuU?O6rE*@vUtNIq za)-QCkB13l((wo=;j6R*W#{>Y2ouU2=LTX- z)IDYVkb*?Xa|SRBm&5{UlAOPEjy%OwQq!#BHi(6M`W3f|j!1CcDW{&oxTDu8R8t7> zgMBL&xNYQ*d4&19G%-9-pHddTpUls_VV-qL=-ATvW0Q7S-i{KEE*de_QSb^IXHZ_D zNYsd7RkD)8kcGNR-n)l7dw~|;$@RZ3u%BGxNunlJ1lMbbZoLu$<4G#7alMUx1)bkf zKN`)%3Hjv0=c(r-L)sVk(~Iy4#v8&rUo!sCG)n56DG6Tk&(06+Q8zb;8SvwcwWTnJbezu?Z#71G_F4%GfQn=41(F#d!!+nKMa zC2|c|k(rjpD*%OvXtz5ef;3q5=a_ziip~f98yra}advhDYd)%jHu+N{;8CW(NPOPG z&nB45v0~_Gb@HDl_CJaa?K;Y6BHW;^IV#U->+sihkWrL0 zIGEgYs~%xH%sJKTJuEM4I&Hu#z*NV$iut^uTmaUe<}y%jVCDKO{L9Q)>7==!3#~RccgtZJsH=G_o;V6HCGT zV3il`!Se%lG{E|%mz;^?r{rMrDQsoLAVM4>o##3LBB7-t!@s9S?*0AZ9a4t#FT|?$ zp$Qqkaq|58V(j9v8&|B8!tg{l`>lR0u$VfflrZ-)rtYls&J{+)O&x`hVOTWq5akQ} z6m7=QN&BrB5E{LuG{FtMBJVOLgp(Ntgb3R$RT!NeuX-rldj?MRZ9AFZ{c$9u^*DzO zCplya;D3e8kTq$Qqp7$?0W?K!;qnPZKZnXoH)s+om7i}8*Rzha%>Kn{r*Hsk*LB7Z z5UeL)+Ef=Q3Z(b5QKmvB89lt;kJQ>B_c!VHjN$zFcj74!@KL&l;mzJ1?xkOc2`GAJ zANIaYqlcNhjg?o#1)`QG?IEHMnjCNMH81af)6l29N zek)^ll_?_H-HbPUrPe3FMjCKPaDg{1OaeBt2R`t;0{-#o4o`J2R+NPV`HqjA$sIG! zeC#|Fwhv1e5_US;w-jO-lp6GGRz1*2!_UTOOh}v}sH$Q0;wBHXp$ictlKUYOG8I{T z8(;GN=Z{oG*Jdg?Xs;pf?BO$ZN-t1P@Af70OUBzxMqqA*nK`HfZCf#ywwN*FVTvYpI=kk;oRFuZ$?Zi}zIZ9?C8D-|1z^ zgTgQv3Q(Y)@VVf2g-$7(0zRq+l^b-!(EW$sH^Iu42BElJss(^Rs7d9F)jUNdn0e}) z#N?s;U+1m}5p-ZxgFFZPOluLjOK0m_-*dL)&l0WZdB}|KwYdy?G~lVy6g65;Fl?gw zT8_<=u#>Q4Rb93!@kjUes1DmP5?P@X@mtNtzqb{~Y+rj-Tpvn^&3q$`V=-sBV)%yR zra>~wL#mT$%FqpB4{2of(*d;bsy}At2?@ee`+aU2K3}wBg2Cn9ZJ7(ZeBYnaM5|=U zeF<>1+d^vZSxlqTapQM|*5_wvMp?pEtGFM5OWtk^BDeVm#(mudlKKu1^2#p?@X&^T@Jxz2Q@VB#Yv#7L4JC% zD>M`-Pzl-A{OW1?5L;V!yJXrdL}^pVv=qw83~=nc*;a+`L8o2rkZ^mQgreFB0ne<> z23%s(>`vw->H-|+#H!dgs$kZjfV}+ZPd41$*i2SB5V#N~XitBjk4@_Xw z_@_fAd+#G-hs22)G0>bnGfzQfGFH}ifqi%+lA@KAiH#*S`5-ytGPkw^j5Te{yDsfg zwkWnh2)X`X*V4E=4bO2RjoI*+C=?UQ<7F;UpQB~~x&=ZBMzxy#-oBay9JCl~HQ~r& za4umM+RflkY{N!n5@_VhEJ5#=WYPkBzZC5dQW>JE$1H260l+Jr)K>^;f)BU;ZAiwG z#%5he6DlJlIQSQN+4uK6d&U`96b6eDgF`v0maEOtTvLU{(HTvrI4ufY!Y+1Nj-7JW z8q$Z6E=Rpn!fQt0Ypy>4aBq-1_;wiM%RLtEL66hHyHFyO;L>J+QK4J}1oAGn;l=8m zZs7#;=(%gdp>Z);@*wQyaIRpDGDJ3fy}t5NhD9I2rA~bTMm_{sS+N9ePce1x9)>>P z%fa;C`7SH-`B6#!K6$(-Y}r8%l+|egs<8`uj zhnL_k9z{zaZ*PM?9|5}xJQ~Hs8$#m}J@7eO6|iTFx^`?a292Q}%7sDjq{Ys<3ftSh zZ3pB?GcLw^t~q=pPAKhLL-r`%Pjihm_`hDkc)C%jBZyJ{%;<2+0=}cc>!8miIz5)~ zsfO?C)L~f;d|%x9WRV&hW>)RlAcqwJx7^2&jJK?fal(jz<%jnJ;$PY~>_JANVB)D< z9rSFsC?eX4!&lwt#(wCx9@w7wm_nU5s!f>|i(p8r>1W){1Q#Q<+Oa&!^8CSUQ(8WO5mDOiRg=t0AZdfuq8W< z_(u6*eFY6avjW=}L%o>Ck>%;O0-x^kT zczTI4%Zt6C;b(3fP%2OsRj{p3@w^T=jB$x5Yx0?Yp;sAdQd7=6R@mMXu2md70;PHZV718}z)9z1 zP#SDFODX%Mpse@QMi)9=^Hcw=rVZuyJ$S6ViR zzbe71_>Ns=8q#jakJ0kkukVjFG4w=T^QJB}$LO>fT9rUE3F*(&?F%v%G0pRz7|8dN zdzYprvim#fSMV6VHNMNxs8eI5QdElbL$B$v*#`rmeL<E_u;*o$uz$+^2x<(|WI z2;P3Fjm-aRwDqiecT?i4!Sv8c=ZPW1&UUgObc{Z*bjyBFVFK z<^udK-c7eh&mfC=b&Z(r4j7$GLZUEBus^wslH=C9L#ar8Dm#E=+p;(T2nZKYn&qD%*!PsTEuih6X!VzdDM{9 zEC5Bh<@1t!x)rFJiP^j7%~syB(J4`6urGMk!tPOXFR84Eg?p1+maeoMTCJ=C?>Z6* zbS8IWO6XWec<(NQFDXxA0WgL{Gxq`Fc2!LNcwV3PW@9N@Q zBVRlh>p*yd3_4~b6L=GjF*nZ{`4tXXe_WsL-E&(lkaoFM>zjENUG7?=7Bt#w&7-tN zLFoS=A(FD)`pm&4-ahysSLZnv_8npY_TGLS{+8cJan); z%yS>%F(M}UQBx2X(6!_}kX0KdF-uF7aO82-mbXqj%&Xo7UO(Mj50S3?B)#zE$)xh? z4L14=?RT@YJN9RWhdUa!KRDCz4hZ+HPBf~HJc6_ZN*D6~qUI`DW<^lmb-!1v=J62j z{0!@fZ@f3Sp=AJoO=Z)GQH<8sR=cIlq2A8DHSbt7cDVrhRoE!`o^I=1laCXNx0dY$ z_E%q;%P)6;i4zx9?Hs?)Ogg*;@owO`)_y&)`ucLO2)nxUl=V z9hgLM(s(YlltZV)uhl28p7`Ab@!sx5c8+KnzYU3H%#pk*^kxUvIlH~!r;8;c$oQd3 z3pgNw_l%(yPOHt|sX~M)8=x0H6z-K4ew@&CzBl~w?}cx|rRIYme;SmGVy`UZw~v1J zye4@7_R=5iX#4Wp`qY5Hq-}DA8Zj_Sdi=0lFDU-#cfcBo1#yb1BqrfgtvDSov{*hs z4(t?b{+&5gZo^Up=-kaA260)Miy9C{AKpr;Xaz?+G5rI1oh9llWjXGHama!jWWL^+ zNv4r*8Va)pIx-NqE+Nk1o++C-OfToVlR$5w+d4l(&#;`O+q6zAgh0;u)ZB%Yq+lO6 zHzKMX6WA3>POdTR9UXvvDCcwGh0#`21cykm*&Uqt#&o|a8W9iSnqPG1EGUjWDdm}S zFMEyOh+_cz(@{+d5haY$h3p4@f`O%e@>9x3Lug}C((DQvi{0XBG^k@uq>pqLYw&hx zoJj@78nA@KlF=`!3yw`=fi@h8vi+{E%^Sbn3h;Uhh{#pUcRw+*<6V~W`&dV{ocnp? zU!YDSr=&!XHi@9PVd$bZiEKLG+9l67DBsc|=i?7dF4W?N5CDAef)4~yq$SY3&qVok z%?ZXpx8?7(B*{=(!(Km)g6t|JDAJ*^z(ZX82%ID;+v7DgezXPXwbT@*bp7ox9XPq?wCiG0xcG*u%<9sJtH`D+g72MlB z$L%BysuU)b<%Mz4CZ0c^Y-pL!lXY20YTt7(DEY%uA4ewnhqK1fU(s!EA;(Q)R7~YI zE?|L175R|%LKICVV>wM-jol0&i%q<*fy)>yDvI4C>{+>?xOxFCa;=Poyh9laE+RxC zwC%+^1hMbE!j|R$@}kEaP`3QCdpP-+8;dAv(8qZV0B_Z|*&?Jbs7w+WZd2Fmx72@) zo5p2eZy3t6ZbB$m3QiW3MEFY(s6P2cGofpe!^>S0+1|Y=;?&P2OAJeltAC3v!{y6w z=;3pDXTg4%1mZ{mNo69C+n_TRMzs!($;@RC4E~_bpoK9J#BOMWnRza8SgoaNr&+|& zIcpzIxh5EIdNDU!0id`vE-%B9{YJg^9zasz+oTEK0a}N@@L*s6UV^ytyr(q$_6(V1U!SXD8@|ZDD?N$5_`oOOWxoyzIt*kN1 zWZhEi71+u8Dg^7NpuX#0D!qNe9eDRLESeusmtvje$Xl=A`Nx#YaF6FhyPKaab<7v35ZtG<1tA_?8}PJm7pbSFwq5Lwn8m#eCFFnN z9J4C;i^-pOdEg={%_q-~IAwqvj))Rf#XY0@nTSoVlxNs_M`J(FkIz;r$4JdwqSjJ0(spk`)T_ zFV`On;5k0k3bPsRq7=DZeEYj5V#s(0;+=G`IE$$ID4{8n%L%c*6#sMW0OTAt=^;7%kNr7IwJTeI2-uWd)K(>Q_^lhJ_3@`7AL(l*dpYq)?&4~V=<^8+EZ9EmI;~+ zxeKz0GZ^X;C|%EDMJC}_a6nYB(Y+J$``875yvw?rjQ8DTzZc%SD6G+@~OB}c2iyKZGr(k%# zN3e~Ce3HUH{QOE$E-mNJ|-%t8DwGq~_y&%d>)ClHWuqZd;o#cA`yHLw*k)|N@HD-4s58n#?Cgs%B~P8zH3Wx09A z)dW)j0^mW3*WH6>ezvNeRaqgnKEW31U+|(GFbUaDnlWik^LpIFU$0>PDb4uJVS(q1 z5*5GXQSlpo)2Na67}}b?SzJ$#DbJeBo|c|=;y)FnDeLrQ7Qq}Rl@8cP)R}n4<_&~y zeXVHTQVwN5EUq9wY8tuQfu7vX8Z>$;O&7ieI5hR7Kikf&kWU-4F19E6;WDL~^u!U} zz(~;L;e0t0_J!7tr&Qzi<5Clp`Y5EOGm zUYf!S5GEE69nX>v%!~#y6>3Knn-3RK+hllvm}sHjA^b9&`E4p$F56308g@0Jw83NM z1qDud@`jqPd%5H`H4ATq0MVmDx6K_6AIu;EztGRvxA9R~Pf3TfBGQ_!^+a_IXpD&y ziKe;k(b+wRv4i&inQ>(bYxv~p7HX=zdJCgdEp0WOCGz#jyRp80gV^PtD(tRo*S3Cp zi6E=3Ezn-d?J{Z;eS4HH<;G;iKzobT3PxmJ*>$rkQ z^oig5Gbb0-s1ssuK^=k#UbcB8UIg$ZzA_Ag8qP1~qK612;amQ(ol>LoxvkMNF0O<6 zK*)2Q>6+`j4VKQ7+0m;+zcj2o^vau%MM#K>q! zAs`Hb2|IzU!~73r^q%RQkg<$yBNv<-4a~erc?IlUa9fm*+AEkX3D89))=S7r&&jzi zRpB+0H2~T|GVmW2H_RZF2pVP>M!pQMabNE(qEfym6XH(BHL3w1%425{Pa#72@P1JX?KmHxS97VR+lXHq@TPW^ektU-C@vFwUGl`pg%v z{a5u{V_gzrQUM zN0+)iGFVYyT(-|8!O_1;{SKrAvHd>T3*s!aEc9z1=)Z{yM!mf3&H(r3_&`PV`*#S? zfAn4aTm)GY(Z0&+LC#>C`Xo&(eVzt5F{s%o{KX~XH|w!M(I7VOg^(PeD zHo(RLRfR3Kl>+T7W>vEjyjEzWc!B#E9{hrK8Iy%Or@s*k3r18Qm;jQWYD9=(c;58@ z^iE&59lc(CDeh73@g?lmus$RXqSmb)RjVmqwQ!sDV3ybd=g%Kz6XGT%;ovQxWHlg(? z79#G!OAymyQ$C?NSyr;wdXkG@cRtv9>-+ zjv+&^AMbXHe776C<~H^S*NB#g-O*7bhtGR;8tS?+gTcEp78Cy18_OR`_KHYJF8iAk zgS8djddJ?ECobrp%XnUb0$})j6-inV6KI4-s^ruV78?og8}ms9WG`);<+Q*OUWQ?FNgGDeJfSgLE9YsxEv_k0qK&>N%J}hGW=YAAT~HQOqhfv z)g(&oy4HLB10sXw#D!oe)axM9sq$T!pOSjwZUzD7dh^$skNL5(lUvXJ1{`RJ!3zBz zuv)g%`U1uaXcU%p{(Cy^3RJyNWQ#O`0%rA`L@9G$x`VkOxnz_?D{4z4O7WY4sr~!% zVlA5aCJ=3EyD*cNd`k0tabU?SSMQ_(AG2Ex_|1q>w~wv@Ujps&h@y~D!2FC)92^Vq zds@O%WG%y3fd5RV%-B_J8fldzfq73u@n6XH1*#GjIus5QygTvfdWk z^0+kv)OGAx`ds;|FX0?9ZjG$2GSxgTDRHRKkzicJ@iPyjhvxvGcemb38w64i@E=f7 zftTD?2zw%IxbhNaNyg6D-uLhJ$lV{W;W)EsFBDCb!?47KHgY=C;5U-?Kr@3F1N%}) zltI%S@-z<&uV0J$OP<0GzsAf}lmC}`txJp}F8(LPIcvc&jFlxBrIf6N;kWp3qyTAb z&rqssl0r`9{Yl4)RRIhBWG7V5ms3oQs(oPEXCJ%vUdiXEFB2B69T@Q^I1qNI+A|V+ zzJE-pWh&?J-1JbKuv9#W<;*iQ5M67C<1z5d$I@1G5k+CO2mQF!7^o z*a?}<@62Bol_hgl=q1|o)?eT%wF~mv5=s4~n1hWfTw$ViXbC?3pMsHp$6;vlX#v`P z-7cbGst%h0G(1o{UvRWND?ApHo7r>A*mbxqUldi}JFlFLpWjD19GT1!R}p~uzC?Vq)FFz&0lwL#plkX9P|5ty%jmxw) zEkXWCI_Tsv9?)P_aGkJ$#QgGu$rOs$z%!qixYG>)fsOBr=$LveA@`FXra?GK)v|Mu9qk!F-rYRwvB{G> z+dF>!OO1c6+y+JSnBQZ@qE}BXc`{H5(KV@sN@giu9n{w>Qx_wyWs%%JFvs(D#V&Ky z34#K1!J#*%2DlW=Zw3`<4L7$aE7!swt7>h7J1{Z`%iV7*EX@-pmx%rMWzR{g8=y7Y zUr1XpH>IQSLrPgj0pAz7B)hUbfHe1HnV|C1xk>5s&4R>myb*(0p|!|PrHVje8*lA| z&dzY`;Siv?)Q&kYl1$Yj&urJ%gK`7_V(E*0dC$R8VgsCCMDtUN3rS5qRy>}=>SY; zbTs7LbM#`QMm@?5aq59{$le@-OT}Byg{P+K8e_QaoE)v1PtF3z73&FOVsnBUW<=>+ zU<+)7$k4TjD#;R(94Y=C^MP^8#$be`6`{}!E|EOSUa}?_yJ-SvX*-RjJf8C*sdsH3 zu%=94kKn1izDlD(`0Z@p9t#cRDfo`DL|SVL7pweOsn?UZzr1YbsflHD3nA-yIHeT4 zcWt~>2AdJx7rtH1M#}$ zZJo7n@wi2jVZ5Qtefm;Ve^H$pjR}yW8y7B1rOqcH%i$=!??#XByZ{`t)IA`~%S{qI zvm{y!U3=y$GV*7XqrMpQ0i`Z!T|fFUrwxM=g`OY&xyt@ZV3mwwTE{-%xUe1IQV{e? z#YrPfIC|W-p+Yu%z-1B3z2j6*ILp9;iHq`jWMb%8O-vmIf)0&?(W~Q3}~btyK% zk&vt(2N}@oNj+KjcWG!cR#j$$+fmvGq|D{>8Q&3zuoN`$5<$tKxUV&lvhfU}EmEJP zLWRmdB|L)FPoxoA7Pp54wmjAtGXA#H4Q&EdjTRY1D|>;OvLw17Uwy&Q@L;WWtc*TR!j|O|6XCM!Ts&oBzv~s;e1JL0 zq2(O@{!)p@T$Ew1viD_&SpCx46UHfcORF&f{D{yhV_}J$M4nl?|Dw-v*hJxD7{byf zVB#j5LLxN4U}a=EK6z#_bPx606|_en#J zj@>{0iwd@%W1m&OU|*WMGC=++8+Q@Ia1;ii-%H~vgxg;;+Rjo0l5?y&kBVw91xRQs zJ>!WQKi_V9B-xJz-MlrhVD`-fEpaSiC;;)&fsfRLq2_?}{YkmZ9(}XrRo3+a3X&f1+_a2`cvRK!r zDrOTAmt9C%rOfEKj!#M&e8NXkTqo>@55LrS3THHfD0Pi!P%uFl4fkFB-t6v$qu-xw zN?R2;t>yUJV7z<=lO4wC8tnkL-w$l`>)V@8Sp(|cNfmy-R6*hqit1DXrbL>YiLGPW zozM2x4Dh`XE5S?X*ExF2Of2_WH>_ee=lgi10b~dn^hNI`K9t#Ds1|2oSTXsF-Y6{B z*-&;T3`zhOalnS$Ep*oP)P;dL{n{f&1XT0G?rrQ?-r(ZnBAPtnTfr=J)q5IpYZYuO zW>-*|uGhW;3^bB0#8zCYhE)jY zS$9iB0qr#tqR31k8sZyf!r~mYf}FBQ^(havKL>QHbRGO`m*4#f6$~^K zjx5L%f4)Ao@jE~M)YwPo+ViB+C66Jn>-zq{4`Gzi&#;4o7Y^K@QzthQ7}(DG9zd$N z(d|&*ys8^(sq9Tq$cvv~A^jbcbOMu35D`SyIc8mt@yr`W#rA~%r)d2i6wa|fzsI@0 zkLri`stf4oV|oI*N`%(7fvv|uZ^{lDt#Cz6>rlX*85xGMNX06=Wpu!0BS+)Qz;(0g zG4&xa(eJJWO(m77%h(465%qAphW|75vzvc=73Fb$G_#anKc7L~?3U z^zF+Hc!s&$*gSyN_`{w)IM~86^w;@?dVxBSqSuGC=(sf;=kMbXu}3S$d);d=10vKo z)w$bVz*Tyb(Px`~+?MygR`b`f*{^)vI#c|U`5ZhG35ulT{_8l{*os)}z)gQAw=_MfJFeimj!5_@UZ9~94xvG`XB zuK`XkW5?{LIO>Y^XS?W9|7G32OLxh-aSR6qPANMEXWA-&I-H(vC&qP}qTBhgUG$~# z8iAf?_Pe-p8(h#Gw~znO?2~0+4ihg}@7sgJ?R~U*_#tQJgBRUeyQWZQ$QsO8fnKNt z^(uLiVv^5g;*SE<_Viv5*82cW@-#f?bc-_$=&JYqyjwcgV9NqOfh42ilf_rr9gV%; zK|bYx`}UzWy?A?vg-gDxg#6SYSQ5lam4B*vGYI%jVNtN1s68L4b&v6|>m2Y=)%VfY zgK0; z;P-Y*qc<(KGn^AwX^))9`AV}N+(c}2o!IoW-3k^kTB@{JM0xbyBSAJ-%OI%ROb11P z#V6qw_0{k#F1p0ca8;^D`D)?{MRZXmRiieO8 zRYz8i06RB|Y6cLBUh=6i2Q!{IC8*SzGC|McZ3%ciSOMGY60}Ay=%D6jBx#%$L|(_# z8Y4pJV$xq!OFaouRPo3PxR6}7q18p0L`RR6ji>``q5rl$|Ed}h6DZ2C#1BkABHkKn z!A1E8-)oI|3Agmk-);0H=`g4p@ro<&5_vGB$9jV2yEN;LwFRx}eW`4Oq*dCD;?$Z# zZfBG2Hv7=~&Vi>E2iLec<(VD23hiWH|zB17YEUzkMlt!L9;o%3p7Y^h)pP>HEGN_?d2h;<9jI5TwCVxYT&!ASC#GvF7-Yl3 zZsH{}%K_avzuRZLPig&K_SSK^w6%1jRn8HQAnsngr%BT|Mye8)#Gv-uZH=`<9F%o4GjPRgG<65zd zh2ozX`RTs3zaijS##&s0)0odY$l?V&Se2+>Ua9?Vk!m4BThO<=VJ#K(tO|`OUNC2Q ze*$$5W)AYyQe9)5UXtLKhvBB5+>Qt|vj9Ef5Vp;2#5q}@lq`5}baNpMWxLTHPcLnt zndE8>*r?S%WPN)LYL_e62lHM(k`2A8%v>XO?--yWRQwcDbfUSuZ-BwrhaB;Ch76PD zByzX?x_uWR%lR;rGkArxlSd61AW-Mn6hpagZWddrkh_o3%2JFGtv9qiF0dX#d=IFN z-{XSE#|)Cj6Z6I+xsD_j{SF%vYM>Zn1feBb{b)ga0a1=b+Q_d>EseIBncf^2kePL@ zRE^+4jOv*FdKYeY)eugTY;~&<1nJcVNZ9F9+!E6QvzRs||9@Z)WW*XE7k+qCq9!MV zUGIYFJ;VW23qo^B)AnWxhbA%+X_oI|264<)LRQb1<{%;{kI}hWjhFZ_aZi%_EN_HG^+4 zFCa56Jwg3PBE!w0rEny=W=^Zu;4nKC!Hl=$n_ZG*$}X3d(9UM;**U)_)%E3IX;;=T z^||_b0Be!VB8l|fjeY`!p>QI$JTMHDNW26&$rm6)9d!~>S|2G>Sr5&s8wI2uLuA%u z175xe9oahG9(2%2FqS7;@F>f+Ri40FKk4l;dB$}v#>+0+qr`Q5p)`==3Xy^gyIOZv z8K@IZqdNJ)+=6{JdAr%1u3+YXu-38o`0=0Doxej22xS7))O=sM5y%|wlR1Ziog+(9 z5rv9#aHKUxwvx`wr!)ST1e{f2gzOB`Y6le}5(ZBjbxqwiV8@hWL_4W#?*Ydoi+|kP z?CbH=QvQh(X2_P$!CU6!PV;|K4$f_Ysrki@#JmdalOE}mI`~1gW}sV_aOFV=es$e^lN7^eH7bKy}l zFOfPB#m&u_2Sh>QyKNmhLy8NEYGVBS25*yMl|}ZlrZl$=&T3J1?*ht-09-O{H28_X zDjbm~d@AN!e9t4Bn)YFH| z8zM59O0WB-^FVP{Pl=cfgSrQ^0V!RVNl0@(I)r?C8oqeNXpM6ek0|H2kkShz`F1so zNN11lYg_3twWt?mfOQpybwgSJto)4;&RlaHw|wg%W-~P%0z25j^R;U9Rt=Bv_QV?u zw_M&TJS+6$7UYn-szVZPt0*ebw>rJSb%PQWWn(d;v*=m!?Dk|N#5d`Nx(qR+JDs*% zkn$DWKX%Iq8&3z`0y9R^B+dgzV3)Q4yjNnQPm1CA2R0bb)4=wz^V%;{ZtPVt6mQA~ znLV<6ILg+-VZBnIFia1%BF(}*>~7ZPQH$9Q97Q}@upEf%^-tM}w)&c@Q3^yc_PA65 zkmAtujS+hJDnq&X`(u_U8He?1NNIH4;;o#8gi~evA>ZfF_v{Y0Xh-2trmb|*ik(K(X-pvW6q)kwe8RZ z=nY21Rgd`=ky`V{K6I$wurFg(ctoJKzp(pr@CE(oXN>K(%NX&cWhUO6-^0r0qCe#t za%{M*hk<8B)OwKjNaoWRo^`Qx;OZU>!8>!2aDE>*GLnBkj!xZ%Y3>B3Ig!ldDx1%r zT)Y(C&u?6To=yI>Dzu`gel&*if?{82St1*euiUUk2dDIQNc>3W}9Kx~pce0hrwv##bx#&tPQStLp5s6yS- zNMX`!f9`v)7dLOp2IO_f@vXmi42Z^LPo@(_6aH^<11G` z7E2aSDwLiPv7v64$qU^QuKs>?a z<4Zt{M+Hz08zm`i=|`(KfNl^O>@v3a;jk~9b+t*SEr>eW%0 zq2bn6U#EFb%*kI#(D24wmNSRAyaTQ|rD1RKVanbl$rC!N{y!D1vInCq$#+C;hJ^f;__V-bdsMH!;Vc21TjVEiOIypw)4=| zg99^{jZo{xoeU-7*IDI=$^WHd6(-c>diS!##uV+zEGj=glJMtFVY1(kbov-zJ-!_M zG)r5bLPg=b1?RbwAOZ0ca^%*|^KQnqx0-fdtQ(bb`hJLP)SZ04-Ct4L5Du$B5@B-? zHk{VJ>7}06p)3Ob%BU0+3Rf6eO{ZIv4E)^2P&woSa>v_jNYn(et$Cwa!_(C6@`}Zd z<5{fBMfQ5*NNmNeMV~q}@M7F244jt!Ve?K*aht4>s(6ob8FP*)&AjVgvCi z%{btrKSRr&{F$?atLNS%bYk4D>hx0Rqm`KUI6@*h#?t(V+phv?xh*64HO- z0q|;yZLAU>cMLi=(0g8vY0Q+@3f9q{izR`!N5gfW6r*?I}IEG4Dt5;d!67qL;^ z7j*6Zv&qDb{A5nNyn&+nReodt@km6YZTjnyA!jJPJoOv_sBjguX-#KzbVU#NM5w5Q zRU4>%JwI~Q^w3M^FO=tgIXPzVVA`1Xp@$;>SWDB}KRF94w${JElDk8wX?TQ}M|~84;@K>jemC#KH2?2CZs|EVj%g$#IzYw0i5eQVqG7?t=?=r!&@$r{ z#@S9LG^@#U{pJ9B9_pf>R$@|BwPoSceDXI8d$JF={doeJ{pGMwNyJsI@1zz}1esKM zZ{q}JSKfJ4mbM`TS$tjko=k<&eX*!dHej7Yg)s$f^ITnB7}ci8HMj14t3!q=Sc6?k zlqdwit-HYABn*QhyrM@kak}{mtwhyUdscdIc(KG$0-lnE})T;3ztZEuoq?oJ>y>9$^fU zRQn)wEQ9PXSJbT6E%32!^{EiZ;RUcz$irV=HWRA%y@7gCpL8{gaLhZ7f~A`U{B$Cr znOf1I#>{k|8A1CxA_o698jds=s7F<2v<02nmi;3{DVtftwbfX|qQ0jkqEI7sH)xvl zWF&B1unye6k<@#e0koQ+hEkf!n8zF02B4dZP zm2~H`&QfakMxn%bo2bm>*0e}&W~-t;pr@xfewWbqqwXlb;>y{bL|nxQ)vC{io+4z0 zP8cAOfLoHFH|_HAf!0*$9D#S^SgzBbPOhKBvR=m;shAPmMfb(!B?VR7o(wLr_MlnK zuCKT(s`RnP5$6gMr@gY4VDEF z4+(a~{>D1X@Hy`}=I@6w#L>0e>m0ZL-sK?`94@9c{`{F22QUo$=9bhnGb6n`vUoqH z`Lxmk82M#Ff~ZnPxhd5tq@O7xxk}&bixQmAqrsr2HH)kmM zdG9E^I2V9p@z>u{v#oy&cxQy-+3$P4&hoNKIi3t5&gY~SU)-XP0|FXuV`mS)K)j~J z*(nauJo3@>vzAQsWF9rBa}h;&wgBsY3b&pA*D0e*khg+u#qEtsX{4ZuNQA>B!t7a~ z({--|daEpjXVh}|Q&H=x?Bj6jiiojM9#uh}Rs^-MFFK45UTIYZb_Ya$h#V9hsc>T< z^+Mixc`x6wAoef^jP{L>SKj!1)rp|3;BH1#GgJwgD9-_{+=6Y&&}3f~+24~I83~J5 z0Su*7s&Sa7T>f5P(%_8{g$hE-h92X}2*(9Q_FUx&ESXDqV~_#QD&|*;z7IR~QI6H{%uO+mb{b z)tIv*-*5oEWs_Kx=usN~JRG1OdG_9M*K+Q4~tQUoe4d@)B z2p)#aoD>8Mtz4*r=@^M~If8JMLT4%zyFqRw)7xOaZTLf_bk84}5sQX&KQ_BBcx`nF zp#~FV>j1;C))+zd1wxOH=ik7ok{A5^GEB-T*m8tGw({5bX&}X-*Lif6sk1pt-rI61 zM!G+!2o?0OXO(y-kuegPHsh`FP73+_IF~0DpNa8h?oEM2hbc{EazVy$?~lklK1N-} zDgvGYAQG&J7$c)O{EC5Evg?3H@0E8D284R-IVF^pyCi3{0AeaUAP#{n}8nPTvE8a3oy`SD7&yE3kJ$uCcXr97Na0m}%iEVt9{d z`uN|;3q}r;f}8R7Ul`-s0|{3zKcXXbQ1uQ2J`2yaAoq5~=VpDvyG;p7^m-ZGk$t*f z_C=VxB<1*z9~=j&&dY+_1@Uot1=;H3HuQ0V(wezccB+9;o3>{m(xpE;*gdC=9uA8w zs5(?|&XFRYs$nesKj4%a^u|gH-DktN6Diq$bYyaVF^2^oiapqF)c5I)Q<-5oHw@HE z1UkeC?#F(>3wE{WF`aC(KFF!xGCFpo2Itwu^Uh=eCqG{%(~`cg@c3j^gB!$A&!^@L z7btkyPc(D-&P`r$jl(fGUU2O2ZSop#p1Ia{QfEFLaL=(gA6mz{08j(TEhS3x65aBS*|75|de=9kn_nIC zm@9c_A*YhD8>^7V*_XiQ03^c7=-j4?NzPpp^3k~{_1;}!L!pFMP|^VoIP()zpAG{v z5HlLhKzXe*OWAjZ)`1hY>~FiHrGMTa0sJq|+}v#Oi#X0TmUq1i2Xu$Pty*pR zwJ<)YS(j>kKm?np&6m&#-n#wesN*IQ{?I>3_r*7s^UZnMp$4>x#aJU&v42uep;$Sc z-GR$wC9qt5NlFb*(fg|Wp_~R;hi8`xd8sJ_mEw`M7bQ`1nm|lRG?Em z24c+mFnYGHT)j7LxY}^pUmkIBHK`)_uTSg{c{hxfX=!6k>86J%P4L$)IpM&qJ`Sza zt;+=wnu#!6;NYxv@A3I=bIczJzkP=M>bZivy!&?kJK9XT;vFVxO@_$A#>o`N5kxMA z<9qIwBK= zFJ}P_R+uj432haVwxB}`U-u*`zS4u|LdPhF)eM@7)qTLOv{;N9ex zQ1Ee&OBXTKi(%T#*I{4dD44ki?GCHxCp|z`FrmhweuzKN*(F1}FWGq~UwuAxLTtuNiX z4J6J=^=Je`nJuU?jG6-t;}C_#XA|ku(~#$-*+X@&r@4er%Q~y!n7i&km(-B64<=FO znQ8?4ZMYcG1JT*qU6{Wo9W&Cuei|{zw-D!gd=12$!KS$YH>t#7y{X*}kAD1!s3f(T4aI$U}|I*lw!CJ0f@PIbMV<=tXvGx+h z?JcR-@_dj*6@*8M4Kk(N&CUxh)8&k{MQ^3sJ#3UVR0`r6gQV$C_f*kDP?>?a_%JS4 z%Kqi{s7wGUf9ocDig<@C#KsUxR^pe!Suu&*0k}%1=OBllugGp?n$&b3MYlbr#k8zX zo;QIG(_M)%x{97d7uJZfEHmtZ7qC+mxe;p_3lT&=DGBOLjlA!)&EgPAfkLZ9 z=5rj^YLmEJaOyeBr~Ixs8R*cSOU_;cHJrC&rpY&!9A;B-8P@~!emgfek+5+y8l+}D z+HdkCi7)tK%HigPzl5y2tL?{NRtc$WkNyj^qbgd+QS>%^_}g3=Lh(Ip4yjl-l%+wU zkkG1TwKEB{G+wN8LRU?H*X19n(S$k^Rs3gA;MaL7iv(1 ztowZW58ew{o(HsUi zTQ%mzFZV#(7yf#yI)nxY4ln+yLDzS01RDx*k(({B-#)DxmF=1XbS`n^K27xj6@E*0 zHu85Ju08mxO`x8LUiX0RLZ5KPE)2mxISP+eZ?Mn@z&UDmN*+A$xKkQ;-Wb22FXcyD zfS7Nq@xjnb+@zY6A9lP7qwcTSQa(yFx>c7m{_5B3__9En1iNOGXrZp?HCF}1Ub7d`{62mfC)G}vLHld>+xe&- z0sV!CvHLV@D)jtuTKZZWSh#TU@`wViyZrbrM|yI~jp(fH84e4P;h13EoE(h!Bg&Cy z%wR3pf6oZuMu@oX9x}#YrwPq1B327?>@K|ni|O;Mx8h7V%UU5%z@}NMiLAeu#ISu_ z0x#V+xD#3OxE&cx2`Dc)@qU>!lSj7z?s}3bf?%be=?l_l2Zcd4tl5l8(Wb1Rqw2zw z&K$Qf?-4R-XvVNa&(3mQB;MtBBP)XD{hg)Uf{4z8YkpW&^^h}u zJ!UrT+U5&ZZ$h2?d9I_`;TPVu3}+iz8LXG)_xXbv7ea|}V1aB*YktZ@^hGNTNCIa<^XDMjRWofD6G{jI?2+?_~kr17gQ?VO%psQ8$S=`SO+ zERf7K7k2OmFD4Xu+{91V$Fc7(T0yJoSvjYu1}Yjc*5MaIuM#;yZlQ0A|LC3{0pSNA zQgd@j>;vgvUK}^J4-|UF#YMYU$!5^c)CjCH`CQ(MVC>}g*2=;K4DVw&aXR`c-ue>$ zc_4MA1SlB3O$%l{P*EKX(m}uV2|$~Iaz)D4>*TPwZkts+RbQEAT||Dh@BuMYCFaQO zk*VZw-4E1DO5Xf@O0t8A3$4TU6UyLuEGbbU`Cm~3eQG{K;yd?bpLEmsCX)ALX4X)w zT{O{i8I%`}A`R4@HY}VQbUw!mbyCW}JXG!u#j?|1#mTRWgJJG27`xz!c36fn`{(;r zKbzw8xpQ{#trp^WlQ)>N0Y~U|0C+Xh@~kf0f|H!YC)wV6=g2dmK}G44cs9@2Z^P5n zXVk_wHP?loj92 zRHEXzJ79L2^LLJbR~2}kTxcn8Th(aW4o0c-ee+I+=h_0`K&4@(e1O&I-r6R;f^nIH z&71#T5HMr7QPtLF$FUaD?#?7bPnv;9W3gcrfK-7t+YZEA0n==owbpA!;INx54Eade zS_#)YY+4y4X36+no`7_ZAk7VdR=GJz>oeludh;~iX@X``2DB%b6Lo|d8G%UFV1-R&R zZ;T>NE~!Q7qN9|F@tG_i|1H5UzUMC9kvKB$T=VGz=7&~gTR0v6jU`*PC#!tGDLDYPxt+0J{ zF#gay35}s<#`~Hx_}xp$FsSDd4M~yYeB5M{9)*f|GrZB%qn2Re_Zr|sni`#q?8aq9 zMW$KlMb=lq7e)%tha`NaN8xfW1N(WBtiBQHzVvl9b)%Dxma#2CRu9Qt@f#J1;yLR% z-t#11%|ouYFKF63>~PCB)EU(2CS|-=w zsVw%$KfI(kV)}x*_%itzV%Ow--6)b-sMVwzQ?w3lQp-zk^zDY;ezt8zyHO!NvUsg| zUo}0WRgk4C;ELdEqJZ6I;lk$aM0SI!E%l2TO$lsm_YR(GOM?^|%>gx6#%Ayvj{Ycn zRKrq&d@Qa`6u0z}Rbnrf_VM}{SPRLGu%y$JqSi=Rz|~h=z)E~{f=h@BSX{WbV%`(Y zugwa&Tr;1hX4tea`~8O$pV!WZS4P0xKop^b6&hun@kB)yQsm@Z^($$AJWqRFu9;v^ zaxW9kFknBPfgUk{meo}NL{%j1xHqH7LEfM`7viEdl?K4}ajH!Gi0qTnc*jmqU6t}D z_0yaKp4*FFg_Z#;U8W97ex^)#td^Wk6<%@Z)##rOUe}gu*ac>c-X)`E23}!-N_OLrvO;}8wRQi64gKgSiJPhxGtHbf zp&TI9fIU{hN1J!;o-OJ()-gro!#V9>1y=2ivFjVj`vtdJ1*`6hu+HUmty z4rWcUTiDu^4Y%&bJ6 z;}|5|d^d`@_6Vo#VEg}CaD^(Pi49%g16?0rpk-mRH9Iz+C^m!@&v*9j(Dz-9< z&asielz-QCr=|!3K61vh0a{UY6Q~PF3-^;cFVE(sfr!}+2hvMzk$AJaA=B6p-?Z1> zg7zQ7#N)*;`fCgAm}>WvIIWk%1egDO*mIRoh23nwMzNK~Mf9q#AuA=FV;|}%07*H> zG0)&a_aSZv&@HXbCNg500*1jvbpq#EWUoaU!2flFxzWp_$XchvaJn%{*7?V4?tGVw z<~WDjGT|B!;}U7bVsh*4orVaZ7CYhW|Zre@b9P=So-PWP=WQJqSgj5{3IsiCTTRSiE zHqmL<6m4|0I_5m~eG0+co0&vltb#t#qZp1I+&iUk_yDL%9uxme4lpvM;4j_e67Pkx zYlYh+y6B(*hwzf1?#ZPdA3<<`Q9c%&9LLLhMtvHqFy5D_?*pK6J<|53CgS}CU@=fn zrDqkF!70zobz4^g_`;1;)P*5(SZA$y;Y z5FlWa7|NBnmDPkSY&x<(Iu^&ze#F*FhGXqk-vACE#+&(`5`T$M!6u`4E_(ARooT74 z=3txS?O$m#?X%h|Hi9*;o*45GZz=jnJ9R_NKaxE6dOt^z7R4hM<679WdSVoIL|(6x zj41GzyYi45%|FNRau5~u;kLrlSi5MixSRrSL8Me(DNxCgVIUyCP(azULh^T_54J!A z1D)!dXjxEg2D~E?n?#c)cegzW<$l!CgAiAc=S?lX9ry=w5eJf5&tQkEnFLo+Yj1Zy zc$+j@BA?5~49FhPty%L_5!9{kbYxuY-Oe$a3#wRmBY zUS=(_S5q=;^Pul8Y?KHz5#%9spc{6Il)32;m~X+Y!W8g_59pQbe!&3WomLR!(r27b zR0u(#Y%dch9M~xuK~bq%AmD~_L!bb)1NgT5tfpf1gysM*PZuc8NtFI_$X{^l22fX1 z{NCgML2V*uUq=?3K;4SJ6uu~}U7TzqLUkb7&n&G$XfE2q{+zqRAE~%Qr6Pl+cF*jx zQUY^+UJvh`-^xEWaE4=5Y8^NIeek$VjZa}Gq7cCH@%a|Ap9-9^Z9j08e?o^l_h~;a zHpoOh2m#`1a56pMT(jjWpoG6j)8FB$pbSbGWn;6tvJ3BMun`q+vq2_wc5eD7{L^}I z%c5!z0RV@tBWHH?Se=><$gl>}_$G1e^X=FS}qkHFS`do`|_?nks-6!DE4Loi6f!=5(I z{Evtl*owii>UMdtiZ>&eH1oACid;aEF(hB5$-q9!z$3($zy-$z^0w2j#8q53s>8-V zoKbS)_l>2i#ne&qEv~@c_()3QBwQ=PBPX zDg9Hc8KQ~qxJo`;3L}e@TH(gR0xV;sV%+q37;2T4$CMPGH=MtWSl1dEZPV#LDL!t0 zJ)G!+6c~ATZXIKlSbm&e3ON~;Ct{53q0MQ|^!TBOEv&12VeB|jgHkYHX+l-;)s-e? z?Exg{5}`O!it&cZM7;(bD6f;~4yp-zJe`5*l;CAcQwzef9 zQKG2ckEq~3Uk!WQOJCQFCJAQ?la?TzFFfL4Zv%gayti2ez4|CTaEpuP^Dyf(g4}1y1Mqy8@yjyHI zqz?kFt;LZ_qRl8*8Thc+tbLIE1%BoGgt+ow{j76~8e!MZCYWY+fQW7|h$T6a-t%`s zn-r_hNDipz4IMMm@oFwmK;c!wq1~n2S6C*-kqm0!k64D->xu)i;QfZt<91BsHTXMF zbk{L5_VfUf#>29+a(|4ei8WJ#sxta;k^B|*ucYN&G4W5*r7BYqg%9G(s-7_J8};BOGawlx zVVm)5TRArqajXRgHlzMJ@&8Q?j$ZZLdgj#W9|UV*NAwD>ORbZH$rD&=<}>CCel@Q+ zuBONAat2C%e`@HQMUiqo8&c$GP<{vW?&C-b3(Ssnon33eFxi*FEZ1_s45)P?kXfD) zqKr@4#+qKqT+gDm$K1ITVnQYPn_xJ##2B9fkmCvox&DF}INwNb?fuf%d?lR3%?oAD z_w{w;s%u4q5XTy=L8tOxj$qT1s{znh#3CbF%jm@5g{*Vlvr8CYAO`bN!FgF2g z?&-d=&_>cP9k~v)LNGBu%LnO9l6Vy_CJqE%1FvbDVx$mgM6HN>3KozLiu734b&!DSC=PI6uG&@J#lLfr(v|hBB9OMi&(?NszK4#M@kUXAS z0#{q4svS;T>t_HnYwW$^z-Qgpt~0M}8`*~#42RDv`cu)5G=c2qe27o3(^B~gW9%FzYELOs^^JEHUi}L+3^ereyY`~&UbdKy( z*s#}9PNu0+MtPqLtV+PHo9pKFIDj8KYj#|5mH>|&ck8e07E@=5zE%-1@HZnzV(QM5 zT5-yfX&6csl0zhdzb&SQg7$6)9`yqf5>Ck08DD+@P>i3>B}B4I2UIw5yuX4ph+l}0{=)1;l~(VkAa`2H@mQNCpQqbJl@=>Bc- zY2?pUXF}@mtGcwqAnqZ9-CqQ)+N0}R+4CK4is;%wnrl58F+m@>F%#3Sf|4Qt4`)Q1 z?*%>YX$IHel&IYEqpa9ejH>G{!T`Y_Qx0;#i~W%hMSp2`t~pDexY7hYH*)Kw%?F1RHWjL>ll(?XYIGnH|3KApS+PA=8U8? z)j>2uzsYh_*uQY!yD&;?cqFaEUh(ieXza!GI52`3vVp$O0>#+}#}HnFuN;#N{Wvf} zV)YVwcHlg@$B&{#kV!X*u(ha$mHzoz12F>KHmW$x^3!+|T{bdKCTZ?5v-g8v`WZ5od5ldM<-|Nuykdp1 z`_#&EAlZKw!=F>{8m|;M!p_OCIC{1a)fGA6g|(rU9t~Z7rcylZ6p%cx{zg9mAK{gcRUti%LON!W|U=k5TFksjVJd)Rq z59j<>d^3Kiu93We6VXGJsrlfQ8^a|x1rkc4#dzE88vEDS+`g86@aO$4HniUa?^3W0Jk)gh|S#S zS4aq~Wj3D5is-f!v_Z{0OKN5Zgjcsa5Q~?(cxGA20I6h&22WA|sG_AE{BkKGVVbZV zfRTO}EU5yeHUKE2J9yq3L1YT9KLe5Eash^TC%W7330KIS9Zo1Im!E^{j27|8H%lbv zB?bc4=xzHFF;w-AO2Yp6e1|tm)8dKnxOW8}S@ZHu5MjqkS;h{F1=Jq1b`sPqgd&zEN_u0drzS)3%*Jp#at^-{vKFDGpkS`Bl=Y~vqrCfBwB*+luWVs|v>nCz#$on%6oZMq?Lwg1 zsrNI9ibPriY&;TW$@v9+cm+SV154S$fYq*4OmFYH~k8;*5>BtVP@`3n7b<&f5WvP6ETZMHL`U`KrONXE*PaZNb$ ziJ%V|yU{V`q9laFjk1!&G~CgJ=PGaN!%5B{Cw&XGCKtU(O2ky*N(Ta2s9?5hxDwA0 z=yhb;f^GgoG%nh(@cs-6IS@7{(2&j@R@(k6${e<1^>eMlQQanZ@lgk%M7sdPW8;;5 zVuaQlWm;$1TDVjWJTpB8P%BX!S7v%f8R+Jut;jx(J3)8OeHrUTeuds{tQV#Qk_rJT zh$}Fu4g_rp;ZgVUDT_{9>y}a)1#9dW?+iGn5(S#U^%?q#>^o>$0shaNC^GmWF(z{` z)Yw?7?mmBtZC6$voVxFcIyhWcOi$z#9eew#f__32o|dK{GYkhf5wz5)WAP-CAU+)a z_u(T?$f=4U(mGJdu_y2P)l_vO&^t$rK~3YwO*+2p*uX?p#D?WJ_yQ`8x_#S6F^d8# zax5?AIQLR@{6$VK39zxV;Yb?(2_#I#mk)+~%+Ka5B6J%xx&IN~xni}S{1k>1z9H_* z&>mF+aspvxhsy+Wd_h+%B7X{&g5Ms{LInk;=R-En!#4*TSkUt_hscV9Dgyjh?(EJR zZ*9BNz@D!U3^#5tfeOpqq)MId)RKcgn2uiex3L~_Vz6MGY0#eCU(OSt&DcUwW%LIT zw}I=@DeR29Epjua?<38U#bRqzFAKzIJ;&w{x2b**+`u}->RVBJa^|y{FIy{!L6MBA zL2m1fXnjB>oiE|P{i%ph(1H%4`yS)V{T7{)i*0BmZUa3pwsR2%>E>S#VE`<@N3St2 zkz#Tshz{ua{i)qyN>`tdoLj#SeV-iy%Tz!p*kO@r0Z3!RAQlWu_68)^2f+E0wy;(3 zCjT=<%*zk7S#?zyusNgi;ky*Ya@L3+`w|oyGttPzOV21(oH<)PztNm+FS9Yej(Y1W z*ef^DxP}|K6?7p}NJww~jD4|f^yvKwEV0Z}9<{IZQcabHUKP&ZS;XNy)A;>rROrxe zzgIunbf=bY_rX`->zhR!o#Gsp=+YGScx*e`Q1y+s!J zGD>6HYydl0X!yIZ4VP;ZI$ag^a`YnB_=K8%a5OToDo$la92&ssn8U(&MEk4WMlOsf zevt$XZU+bbdy)AP7)4=KrHOclQTsz>Q`K}e+YyB1t)`od_Jj{(g*eAj({oWX*4W~* ztS}10Z;=EP6_eSxw937c-vBBjne>7C#qF}!Ha8X&uV#SIeuHxj!$D~{k4*6B#4;fW zEJh_FTT5~_FuU7>vbY7{!?-$&>J^t`6bOgH#NG>qVU#4ydHw*w*6$!D*qySd^BbYC z_D09-eq%XF&)!BXgNF`$@J*p_y6WFYE5rkGUiGB6HOAwsImzo5A&W|3ch7@biDj}p z-nd2uj7LP6=e?JvkNb{B*hfyn|MZQsJk|D*CfF$%Vab zs1lr3Bqu{X&w~5<@wE>!Cl0ytU*E9Fc6CI1;;;iYDLGr9c2&FBNB%gQd(kJmD&cz9 zaQOfZiz8q8jCgJZ_@mykR2N!BoH9?-WWP)K*Xnoy&1P43a$SNs%%h%jjV~w9Mj>!J z^~+t{Q|cd=n(&&Pjk4~S0T5|M(o~bhm(#35nH%h!^i_P>Cxw-1q`Fx8t6RV$%)NZa{^;-G z;=l;e2Cllfx_*TbpT;TQ$(tLGiHf%Mxc|WJ{$Uj(}!H z3AroG_|oF$+A-}CDN@qLDm>GwyI?mw>TvXjeR^dvBoJ;$)@%e^LFFo7d2NmJNm-yP z2+n7ECF{(k2gW*1-C~|4^9g1Kq)JvTSluLK6r*ddNT^}!gm_#0w#TN$WPsj$ftkw~ z;0&vLnbMTF!07JVYr>QU2#j~(tg6m1yE`31KxpZW1z+s}bVnrEar62kY2K!dVM_U^ z1^L-Ichz*EG@rrBC3+FehEO9RFBP!f6_t`VY433>P&5J&6aC(VjRs&sb{N7$l-fLIe@HL3lHZ%1w?CDWVgdk4;lXnzp7zS$+cW67H{J&q*rhVAE&4;NQ@>K!KD|TdR;NMH z*Ci6+F1_1F4aQSlJd5_lv*bMdD#xPVX=Srf4OQZ|V)dP`D(=Jf*_FMUpGur`tC(9r zs>v@K_55T3lyjd{nX&>3V;=@ny&|P&G*xf^>!uce;e@;URu^*c_$d;z774wG^8HK& zCY{nF82yJ^|sELW12To(SgHCu8`{}}LwYjFP?-mQ6-P_)tFiBjpn$MoWyF9lFLom-P`<$a$CkGsZz=a(koq(1 zvmQ5{T^bi44Bl+!p>;avYEY&;v-Y(bvBWNNl@=*(L)DA-TUC5qTfW@6SC0{2Y#1hP z1h2J@F^#rI^)h}rUENGZ=*WZmfA2BV1gpc}7D+i4c{~R};)2~+)2Q~=Obbf3QQ3}z zg!gA&5yXDx5L&*K{nqma;_CQBRHS@qQ<{zR(oA|p_x-bvhH|0Jvku{71aKsQ63)Jr z>qy_X4g$YHTp?;9V)VQ|ZJoTCRFu1gelUne2`>%{6cAzAFafZF8gx z;;4%RGG<-;K_O5U;V`ow5ibfhnUt>iFhLFBnF;isA6%9|7*1F}P-pmw{AL zAYx3ih5pS6KcOux)zn(48>A<8DY+x6q=1l1Y}x~N12Rqu>>{w`TqN%5&kY`0{3!0E zBGJp9yMB_Ub0S)We4}ZsDD%4TDL;Cl+U^pJcy%l+3|2Q2A2VdLSu4nzvZRf5Q-DfW zw&5sF>p@QW21V)DqB@S*m~e$QlF3GxXZ70R>CNtIN&ClSjo@@IT;19L9LDhX@pBUxB6ljN((2p#)hU)zOt=xMw(rbo~DMJ6lc|m5A z-uVQ%%d=q6%3V)ycmlYopjE^4g4zyNf((qtG}OVP@J*nu`h)BR9`Unp?>M-}w%uqk zjYn1@J&xv7vV}SnU#QucUp&kfZzU)*-==NMs(Td+>>FXORQXRcogm+1u+J=rNhvtD zyjk_9V35uA%}3A(JIY6;qx*{1HfQ>MC3!)A({4agHkM$Bv+YUKia& zQ-W)+M;W>>B|8#q3>sm%dj1AN@N=Bz%RHt+fK8_)-QIz1%bdllz1!p`w_ zp?LY07P*DdSYLn=HEX!Fcm9r4RV&uo*x~Pavh0ootoPhxeN9m})-J3oIEJZ+Yx)8a zvIdN#pA#-W+;Zd!8?kZ(+{0#WiF+8o`3Bk&8iVfEgllTq-eFpfwvP&Alu(jH~giWC)4xmgdO_P&iOxldHkomv*17zHMX^}aawn_owCpVZ-3D(6lP43 zttsqV13-ypT4d8w4e3$nnx-DfRFN~AXd-}2UtSsZO#XaRMURCJ;L-gv>~REhSwB?( z;_qQtvG+_#H~Meb0ITABGc(yFdcW9 z<$LnT36rTfenWB+T{!CP=n_K^VfWt$5qs9q>CA$d$_0SujP4CNSGzbr6Cc7PJ0b$p zEW>0Fw?W;NIVj>DES*L+?y&OVT8+UZ^$-s&q%4QZQek7hA>WlyGf!|%B0HD?Tu%Nn z?K;uN&DgWL_O4yg5Igb=^tvfwMwk^wcp5z z_;Cou0efASnyJexpJW)cobGchgp#|2F86yFhv^2xDNlAg5K(iFyEo2WvcIq<_DfQ3# zVtG6Ku0mM#8YmUIloSxqh`B>EGn&}hIU-qlPC_hdKEJ$RpIlDdwX?`<3Jz2`6H6eC zTVLpq==Fj$ypuBeNxDBp!bLQi6a!+i@V3}gTY9S5!Q@^y^V!}K1t ztX`ebMQIRz1>8P`hKNw2_rd+~ko%#V9T^~bIU?1>p^$qqfmKjL+M51QTYoH6kd(6< z{1Il#FJC=e(8+dFxFB2MDRM2o%kN$%xFJH;a;s2gzKQ?FJ`P-*Cuak-zXQMi{6~@# z5Yl4L=iZ+<7r=|K<|R%Hocy+6oQ}|st?aX~>W!3Jl%lU|l3V$`pjT@v=3S)=`O$c0 zMl(C`P%MF{+XN1(-HSrdKji%lkNod`tuZNM7Rqh~69AQvuvygT;9>LO0HFt^97Nlg zM@l5YRk+L;{va7D8OiaP<}0oWMpucJ97)TMLr6kk`hjCZIaD@}PJi8YPJ}N(`hY$z z3r1g>m~H8$KyF@feQ~!b%1%{^t|5=PR43;<{U9Gt_7>wlJ-PH2Hpc3)>G-JR^ap9| zFP~fEO;u$~MrqPYQCR#s5XK*oLb@-)mK0nIwsitb)#^haIDv-b$?Z3P!)~` z&mEcTny_K9L_L*F&^E$6R|a19$9z>!U6qb91yJt1r^B@1SFkd&=Vu$Z2%lkc<)QSt z=>kvmzWn{12(H7BZ1?D7VkUn^4FBM;{5QSxS^E&(s@dD7Z;2+kl=*TQu8D#vNw8+q zFCEN^xYTnrNXe?xY5b9-7qNMq>{re4Ghf~qN~^?#)Z2D;0qK6jOXO@W+_NfM6hH0L zPxO}>XTaM0jV(#DOYHBpO7in4Pi{p?uy|zUizXlq0MwP8|60nlSf1!vC%bFKhdDR> zI7j{w07da}f2q;1@^C~bRN~)UAU7RLN$ry`z`O7IcFSu5vc6`3X~hdDuhf%mP7ZRJ~HQ}Y_}iRL@T zzhL!mzUo8GZh8uXP+L&;J#Y{FKF!d_gbN04N<>~G0Ob5O$!wD|-7^S9O`XD=CY7>a z-SI>dpp+RR@LZB<&;K8&D*+I^~!daM@P)U2cPEsNdan04&+`9xV!|&xeyh=26a})${$omDF zNn@Tx4dsn4he?kQIQKy`79W8Jk%0<0n@HE-Z}OD>ATSg(KbEpShaTIq96g>sk!kS{ zJT5brZ)`;e8@rEzz}o6USsWPqA`k4VFd=3qc+JxO;qk@&`SB4pAryyp#7b|)L7*eM_(nf#4O zYK2J<(W@UfYh<(JhM$;Xxxz^l^3My3g8o8F58%MCpnGyaSqcZ(M2}9ERp~H>09QaX zvf`OJ-)MtJQF_qYYT1YU8kAht&AwSqKTXbWn%X2rxVrP;yfMC0UR0$R1K7Sn5Z%Yf zJesspi1x92SDHt7YS6NTYz;*biTo4A5DKP~{`052-u|JC@nM|#8L3^!%*@@m9Bv;y zb5My?Z|YcNKslri5@bXP1=Z|cQao(Qm7r=t5vD^rD;~03?w&4%RPORB+ia=zx0`xto zrZ;6ZfnBkeQR&B^c>B`n=O-%{G11p}hEL}Kd2a8wNKUINnl~IJg5avGgRhMSAIhnt z5e$;mt+x3^@T!%`V{7IbTAaJYRf$sqWFByZw^0d>{m!Wd&oV^t`fTlZI|1T|=RO>* zzfTn;BLMdY;x(M^0GOj%E?L|&uDR%dZ3pSftmHeS=O~{&1!AqkX(m*tP-qW`ed_XN z!mFiK@Z6?3LJ5w|ATx4L?$v{b69D5)riMpsr;yQ2n;Y_Z*!DkF{T1W*rW)UoWpvQ@ z6<~+&y+AZKu~=0RdYErzUJTeWwBT#?Tp?gaB9T@;g0j&Uu|!_L8FWP)3YJ399%_fF z&16MU(!|0P@+{g*%zMUX{=KkwBdtP(%zhW;>^ap z>p8Sm??(agbg`EvJLbNgdmuG$W1XG27#PYQ!&)7)e=ob?e-I*)?@QDxIBKln&)|?9 zqs)GDBV$&8XCKoeOFHVR6;i@k0X0$6S;`P~Op&P1fKpT0-*(U1by67uNsY}F+6`A< zH*ymCBYJ7`OdMC?A>_{l!2>`T00026Y}>YN+qP}neztAfwr$(C-8byH*#-b9K-RyI z&wx7H-5dA5%Yv?5F_do*I`>!Tno;-Y(j<&h)^meNEHjgQ%@C#6)QUXYgYacLh{Auh z_E~5-xFb^^m5+Wo%{MiDZ_<`l7@;QiAK0IPH-Ve__hckI=es(B3DySsuR%$?z*a0# zeb_OJ5c{NZqc7jYJA!SB9!770oB8{58_=k6Ht}ak0(%=CG$|0BS&OZ5w$WoA z?T_-wD!O&BR@bT)xc%TP)HoOe*-+kYWDDBU5WhbUR;g|D1FpZy({@2(Bp;A4!FAk@ zSD>c%SU?i&9xOT-w-P}r$W6qzLbC^CL^I@z_s-DK=)^+J0e`0;9wc?9d#=bv3d52{ zL(vVN@E?U!i=8c9f#H^v+Q+_yYtb51Qun`}{E$n_LRzZ`IPMHcG%U;E=?AU{*GS@s zF|0*h=kz;1CZOU9^6U&)p5qksYC0b$Wu>k)`6C=70D-~T`%}1DTe{R~7r5mL`_BZo zL&=c(6Sw#?t!>$^^56+Fgn^wEggr-}c<}o~h46+M<-qVGprc=F_&t^Led>-pe)x22 z%nZ7vvc*6p5r=Cp>eot}3p-1ezTnAqzLFO|^DSBQtuO$HHAT)p^YlVd(tlHMP$si1 zLs0l}d@oXglc1m==CF#MEnJ87s>4&^?dD}l7x0DsNAhKa?2+D(-7gH8;DV>B0n!oM ziu3x=yPR7_0LhStcOiRCc{7K0yugdF)(qMw1JKgUkk(GzBFXr;`~HQagVnkBjLQ@p zfEJ`=&HW8lce|X@?t?A%8bdY8(l39LDa@U(VO;g>rZkHQx3IJHEQZa7H_fIJXaZC# zp(s(-w^qdP}KE)`39OmX45QB$YepVXUec6fE8c>^c~%nW>Ak3el**I zKm?gweroqE)asK_8;1F6-@bYrWLB%1EQmo@%xLO`yXXoTah+Iq$r??qCSo}Ius&vN z%{As;6*ftB3*n`V*u(RG5LzOgnfG1x=JxZ>Ob3S^}~9I}_TT zsnfzfAI)VBJWX6yLr+U}(m~`Mb7pk+pzM+=AU{>u&^X>jfHW*hcL*ZUY_Hj@t3z$C za*>`8j%OGl%5v=F+RR5M(=NgR!jo03u>-^O8XteDb}y%s^fLst9wT+MQ&=p3aNjX; z&i_!uo;MYEs&Xtw^TThkV_nqC{NM6ti45gwoJ`yPUL}@5Y_{HOhad{cI!AknZd26# z)i=3mvil2|(KDVT87vgTd55PXRF%ig==?@r81y}U!!;`0=~XL8u*;CiMW;=BH|}#< zBvW&F7$aXmP|xSyU3)g(u%^S+V(R>c6EBKQvtmVwCPTd-E`FgzZ~;zA?!*^^Xq#NAf8BIj&IGu{8u2^m^qXD6hL{qJ#HUrB|JONrfmyr=MyV02qQqJ zF78&XKMz2XQSv4U9Ly+^AMOv{?0xggs8~!dwLNW|iQ4HF{VSpb8^ez3dn2cpOLw%t zpV9dyRX(@HQor;5ezaTyeV}$CF*TTfhZ+7((4zACnV&lR7_wziE0e04t%ByRoaPA= zg6*FkVGg=tK9TDPvggJk884wnBsc>wXc^vv28X*31g{h28`*6vhI+tor8PJ$pJ)pH zC@1=7{(gUfYJT5?!Vy1TST|ry^iX&BW_vb_lBF-aay#n%Z^6YhNhug#eS#a22-8uS z?;~Kz7!V#Zq{}TXu~2TnUlX1@A?9UM7ISpyBw69*M;x1YkfUo0&8Kz9IsOhiuQ8YGUubL(mz$sorwLG!uG}4|cTk|A-M0`cApmg2;B%?DG*&OL9=kZ{d3A`c z|JF9xAcu>9tbe{bSiut?TVDvK5nOG+xcoKocjBUH26Hqt=FT|4oCr-o-$56{iPjdy zX)RPVUuE#9lBm)Qv!x$3B!;d6gl1rzXirx;vpglQk?0cyM{~K<;U9MbX$otFjtZ)M zrIjD1FPhKidlNLRuL9{46I9!9}A_Szkrvccy@3q-|C$J9GfC3l{fm> z#V@d}Fz^%{FHscpOjRk1f_M=9_((%2{?rMm3z`<;n{QzNH*84IZ>HPn0e=T&;hQw9 z?x4ytZ*zyuiDd!By5Foo=Z_z9*%gAu(Jd6&LD^20@OZ7bwI z7#@@|034*%)_Yd9e>zEFtwk!-9`e*HxHuW;G#g+5ypaALGZ8>uJ%#AY4JO1sz}nJD zy|PsWp7r5B-yr=uxCJJ&@>#&&fQ~zgHuFk8do~bQaf@n*!A$A>tr@ET9xm2ILH3*1 zqV`+eR#(z9@E6BQBlwne9?goyhmQc5TnV=NurN>JKwp7j5i>1PF~Tv}GS6Z+7)%IV zXzoWiO81>fYL~z0M`BaE@4E#O20D;|%Y$xfjA8KBoU1^%p)f00fbjV{i9ys`Z7xBQ z;YszYwT~*qIBA}@S&T1>&dk8W?1G$nABQMk?|S?T8b0R%{K}bow{(GnZ*h)I6dlF@ z=2&HrCdXt*Y{aSFMkpPuvpOW08Gjlc0$GSDX>$H8-THS<)JpP;CL}|3{d8-utv&y? zG4U1q*T6P)NGnY{tspvpQnra8>N3>KW^@b*1inRcQpm;vS|HWyx!k29s3Aw`_&Mmh z40nIhzmyKi7-)}g4`kR)2>}OjISS$dN)2Jl971h$nd>l$8>BwYn zVqrmvCa+QeGgl+Z?j`tcEmK`}*k`*Wyockb^$Z;xzTm5H(V-$#YH0TAh?%;w5+JN= zNug7I2O&sholCJPFM=syFnmKca7u8Ui#0U2CIq6$WG6X8U8#g;>VYPD$26gthV*4o ztTmA!n~lTI(+@Q@=?p;*JTO{;zG7|@?yi;`O{V5%aBNPjym;4O^d9AkcE=iv=&)-E zWkq!^e`OT`V6y1!gmdR(f_$q@=H6fFiUlCA^HS5(RS=m!;uVFEkkozxiqK$hfZySD zwu8I-Xr%n*EkjzVD)iC_CnscvX!1TUp&6)45^Gz`4^+;2=T>5>-v^T?I5}v?iLO| z--Uw&UK)+hs0Z&&x-*w*V?S$awO-|0_dZ-=Oe9gWh7k|!1P=tzGdWeAzYSY@ z>`p`ZF{$)Y1UIOboI5*-TLa4nig$RGeD05Z zOFo8t<)uFq2r|w7RoSqs!Pj$T=|9n~x;!+NK9F+%KzV}A#PVG>>mUF?j-!Cf&d9sR0`j6e9M?*ffG$@VIOO z)Pmk0=|9BZkX!;xV6ilxTj+F1oRjl1IT6%hs4trSsnYLvm@Yk7g6E)_r-=N9QlhG9 z)G+hEBM*1$JHD5|I%Mhh^G6FCQpfE~a2zjyiWo^1s3Vchh5XZ?^QtE@3AnG~zU_^X zTB}wg9|Qyw4vMgY_R4bLBub<&<>KJA>#IGeQ~}vgqQ&mdQvq*3&92GnJO|-~!#u;g zLYT@D;l;lk05G~jepHG%B|TlvV_FKanLbaL7X;}pCT+BP{z#w9Cah$=8@gX(}s@l>|evm%dl8BJT^fiq%vh_7C!K%uisUdfauUe>^cmLuQ zZ~>`HPxPK^N{U>PgSX^hVFdmT@UV%NVTj<9tJAS6vTqyeuJg7%UzTOinZ}+|d^!Hs zp{b778L@ma3(0gveITIZ)bn-E9rV?$Gcg=q6!py#c_;yN!8NY1piS-@EmT}>@lVSe zI}TE+T?R8-`dmfh90NohcLwGyLTkR#ECctk?eQbrYxm+ zq|O%_l~YfJYYh8*zwhGHEcrwjIu+aN?zkf6zbN{KiL*vet`-1nkKFn z%C4}l#rO10rD#g3atu5rC}sm%&~LmM>dZ0^S_9bdIk>AN+R6yhezw5kycxT=`fJKl&l6R<$YeCPY;mM=K#8y?T61tbQf zst5=1rbPfV6F#hXH849FDBivH@;^ke}vxeDbvyV z_4eCE_bbf@Z8Z1+c~x-oG@Bc8Y}nB>dJ4`K`Le$5O5%L@{5#Q}!qL4E72Sf8dwMkU zg=O#ROokb2;cx7>eaxK%m=20m7s{zyqCNANu`cpy4-14vLhTLKf6Q|mR>wm?*P^{N zn76Gq0H3UZP_iEi!75wal5Li_-u1s6`_!Yau0wk#150iFR;6i1a^O7=TF$wdG9uLh zoeC@S1@chcG!!%b+vLZeLkYT1jsOD3N}QnSqd|e9zxoS3I7KoR)7s=XoUN15i1k$H z0A?T7nbv5$$NP!f^`K9lCWEeX1^Uc2f9BqF^DOibrIWM0B)ms4`8)HZ%n=p_#-Xub zdRQy=l-Z{Fs4u|=B|t~#&5JE(rD%JxprW-A#y#lhF+nCsjqbft;#cpG%o(e=3eQ9X z#_Wltxtrz7NR7q;CiS|t_e4|85kLY@nJ`ZVd8yC*erGOOY;y|%%VlLGAvVMSZ=^iD zF@cW<&GM8zZR~A8QH`heY;;MnLQNs`H>Ry(NP(K)y#{JF`npT|f&;#)K+LezCf44_ zLJj1_QS4z9*U7vINO*=*#ewl4WsS!3ZNr;g?W@+z(_1uYN8D~hUFtA9xD-T?QLqPr zpCh9OkI2T6t`O?77tJ>gkF$^4l?2+VhWboSVE3Ce{}n9uP{m!9eR+Md(^q5cjqkt{ z|I>zPHTL|iTGk2|#Czf*iN`0BN*{ZHyHZ6lH3quMD$zqmW&9`L!B5P2O|7DS4}*QV z)RflG&lI+8IYnXqqi()ME$Xcn3-s%$=76XVb?qnbek2%Q0(JL5PgrJ4LSkUc_^1~C zoYVgjs0~P^K`lnxCP1!|_Xu~JhBDKCfP9=_vo=Y!GHb6JpiC_cv>gAma1gRBoWH%1@Z*34JeIITC%Om7j zKDLbY3kiYGasY9ix-*YLBO}&~#Q=WR6-KJTDxO*cul(SA&=DI#UplGOqDiKpH{fLk zyNj#cm%Ukgrgcf%v2QzwN4UdhpMV!uV0y>y2_kAlX|0=oijfMT9lo-)fDuKZl>TGg>(f^ zX`meZjLgSk!WEp~i~e%5<<{wZN}1zXaEPp+0cRDt^6;%NZn%W^w&0S1z#Ndk3HI6E z<0lV;v1=VU1Q!5Q5F)$v#ymcaF}{I;z5|*LkjN`-Q|WJSoXWt;f7I_Kg4i z6??y>?|!8yF9b}%B}JjH+~~m7$2dRV7RKEW!a;rU6UjrUU#HL#0{ddnN)4Y9M5{|8 zK@Bt9uRi*NG9l?lMgnIYi@po8vG_6~u8!nvap3!-16xM9WJT;u3mPi?mN*es_@ml4q*l*oOY1!&aH#dyOfdP?flq~-ME-2V5^?8=+4s}$({0vcV505+?naW69%TPu2*Nh9S;7Uh!=zmh#&xK!FhJy(VjP6usj zHv)(Ak|hl@B=X-@9XFlo9W01NfI@5oW6Ds~^bNM<6U)CI%wNahRG(7)m!LCExV*7f zH8S9`i;g~>0A_Cc`J}?T-HI(3?wPIxW1N=a`1TZ!gyWin`CUQ^j`1nw)v za7od;t1SXogk(dB5|?pb>-x*MIyH!y!fJMY1--E`%Fp0g?;{U>Y&OR=y;0<$f>XO&YWh_9gWq&%+|CQu9>z(%#$C(UuNi%3n`D0JHB z-7w66Ih1f)R=jaR!m_2rO*-;MSHXdMnslJag>NISa9@>RJwHwuxTtZ0l~ewdi5^ij zKxFlQ>g8x#i@lX8H65m}bD1@$k6g2^FsG02)CB%wgyexUMRqxcv7{sooF*vS#%&NzOP7np{%PDlt0H~a~&K5)+6fJAZ1#v~E# zQUn9o!=Hn+nnb@HA*CC*qF?YYFK{kUDa&I8S2^0T)8wQDZ_}3BwNr2Ff`AVoWSlxS zC5b?mB94X>6$r<{->s2p<&vYWhN0&6pb?u)<$NDE6?W9Qo7gdA_ z1^53Q;=wdY9HOJmPDcW*S4wpH_R;AF0O&-sZrU~WmIXwPkVax{NL9F?kq!8k?e>ab zP2G&C_ky@bA7R2?89Ru)gN04+!qkIC{l=JzMR zsPtrfKPoq&AjIFp`$pOQ9-eMpV%Qh(`&{FKt%j+OPS3hYcyYa#4{}$-QgS+yHV6Be zvfdI#da8JbMS_0Sp2w>7Ra$I@@JK-VPG$Cr(a>&!2kw;v*l@zRmha5{C{|4BG_z}I z%6@1k3apK@p5!dTn zkZOj8uJm^NaQJEWs_BMB7($l|CSgs|IR^1N8pm>bYo=oMSCY(>GF9de#H<`BUs{Sm z%UmIEp0~dbZT#hzyq)>70up(kKPTDdEeM<~qHU`1!D?yT*CYPe8D;C>rUS$(4`}X$ zMQ0M0HU}G4(Xm@wx*b#e7)5=nab$2_)|D#u=%@{4Iw1R&1A9b7q!_qI!D|O>tiWRf z5m;cYTxHyK;D=6vyrE|V$}geWzs3<2+hQU3!MG3yB6hhlS&MQh{HGhQ#bx6^W9>Km zmd&|i@&h-e%?NtUF+av7L&CK`xaw^$d_=+_@Q_2j%bOusB}w_>I^;odYA(n`mZdmQ2`yOcVoFjmM|f{hZ}QQP(#AC>QjvAI6V=iXH?E*Lq4g zDlWin>F69d-Fxpr*x{#nl@$7Znfg3;nMCvNt&1$Mz{Dmnb6a98m)+EyZ{M!d9RA!i zYM-R%p#$)7`-dxgHTwI=;W~&k2Ei!}hXj+x3xEyDF>o~w!0s>M0itB?gY{cvID4ec z+Y{Y9BL&bwBv@n41N3NaWPPWKqbZWH$sxYYW~~%&4uf{kvp1aJB8W9zAwINP|DT(= zwKw$ACD?Fx(4{P32OSb>ymqc&AVJ0H%^mn1(7uL+iR)oHWZ0qx42 zL5kQn1d>L0&ScGlBDRV}nw&X_d9VBJp$6Mld}SfDA7w^1T%UT3!i_tDASIwzvnl`d zO0B7AGH3f>fpg<#JS+5^Q{un4hICa@Qj(L)E-MyqBhWeBZ`9%HoOZNK1?2MXgX+yp z^sIJ@MO433O6Si5}ATLniM5CPwX{3g$a z{SR6CS8fHqWrAYdU%X}(=rTJKeMJjLdv?u(r9J*{Kg}_>STn|n6{YMbm2`(&h%r4` z9WSA@rxq zR-O9pSVv#rK68hH1-r-VMko?|uU8DP(5yn=6gawLE{X?BfgINUZl2+9P@NlPaC`1P z+(lJmvXOV0ka_p;58)@|rygIxZZ^(#BC^Vi1~u`0jD;kF1NLR*8oW{5v0oF$V{lXK z-0-m^mPknzBcN?IM(C-w(B0a%=Bw0_PNQg#{rJSi{b2PEVk*B4e?W0eqM+ao;V!77 znW`#y7_O(}NKhA${(w4i`jYq#ikjuLL^Cp@&bTiM?xuvSX!|uS@ohN5eY_bl=3PA$ z2*4JmZ0t~E9nT3HV+f)4wrZUx5K>t36e*%rg)wwd_EI)LZNRBl1+%#0x6i-7M+v?k z3ON$KP$j5nnV_o4*}-ME!gs?Nknweia99&s6(ZE-?0 z5xAoJZ6ZXnB&U_*df_YIj}^_eNVOLIUjyoguG-HA&GS=5>K&1r;~UV|a5K!3 znwE8?;fP?GuPnWAn_321*e5A=gz(*$LY@0{Z@{NzdciuML8E06PXRoi=LaT(OgjN3 zJV`b+3*gy-4A0t)^;Bpd(nSmysuPE(Yue`JdYmz*9#{)|9A+@Kxu@wk)4Y~mw5$#!^b$is^_m)&f?0K;OoJK4VS4DTVZCz$l=;+a3Jy|?< zEfF%TrF&5uPKd7oz2c8o%$gl3K4Yj*yau~P!-4x)X>3q;wN3J8U@=ZD1q0aA+Ey5mGEmFLP7$7fjSY>+55B=QN z-fEBxCIfEsVGr!!0N8pyzXQfUEh9~hudPzc%uTP9mR#jKabkPLlV39?$6HXQy7)X! zCN2C#DM(|5`YcX8CG8(bjR!tjsfwCaaVN%70%}jjIH<2yFW48&h$tW1|3-kM7tFKE zKDKxw1P?M}_hUqQ=!QA9X&j#(mf04P2mel^A`M04ZKwTG=-)L0LrKL;eu{vl)n4|SwUj4*(SKb-jcynyBg^#U}C5)MsKUu|8(%U5UO za*u8pU3M|LKAxPIP*ECq%5~OPibnCL=|;Xup|)q(kxl%Q^%jI3PsJ^lI>bEv2jl2TI$kli2sqQ(`0YvUZFZn{&_Ky`!h?WRBwUYN6&rl6!$SA2H|lwfz?lh`p5fvi~o=bk6Z3 zl-;sM1tmU^QYs9G54~XZwsu=~USfWiB`x}WBz|{Ow zc?=VZQ}B)uR_z=juU$CB8`=J0w-8CxG7{+TV=X0nIXdj)~3Samd*u#UVjZEzU8*!t!t%GTPqRWvRuv^!;0(LW+?7shH(_V*RD=(ukA*zn*iIQ}=={>#d!IfE0aFiNo+wfx31h zi3hf;PMKT6V7?GQ!3aUXahaQ{H&J95Bk<7JY(1JoYOxxHHta}$2m57gPRchtPn<7^ zn*`pqhV>f4X!JMKHD913IC_2AoRpYo%N936heYbjkltUJv(3hw=ybd2`E^M^Yd6@J zVngZfVEINdx(SL}2D!%!VkW*^|3=`ueL{)ojcT-mwa4!cHD0ZEhC;Q_yFR%Et^==; zk9;H4Yrg}1z^JWjpaHaFi@$Hm4}!$}R5t`8(^y0W<#}AWv34vJwT1aKZ=xg`7C2t* z)f#%_hB3{^O&k}$Na*(Ux_H(+8rvJxWRl*#s7sntL7pKBSioq#N_)~X>;>X&4C@vg zF}wXh6Rx58#JpV%c|`cZIh%PK-7q8K5>Da0n{RoU#D+YWm6ky$qOmIt-M+v zYv=)oJ$8Gr5GVDP)kPt=wn<$c*x4YYax=TpO8ZMXRX9nD!q`$wbL+T zJ1yd}Ft!nqMhp58r#!tIpiuLxN_!|8f3@leT=}f!6^=R0TAp`?hbWBQ9L#&%P4GH* zF5JTK0w%gfUHRDzLiSF6X4!2_MoiqdCjhr!$X^8*Ppn`*q*}$0PsB(hvbfeNM0NQ2 zLQeh7H-vj>VJXJC*2vZzH)(p58zUXqxdLAEwd>jp8uy&f?jC26G)-4w#fHmjB#JC? zeX>v2AJr3G^b)@|d+B9oo?AF@*#-zxnvP6wf%k!^522+o8|u_wC_m_iVyc7!1fOXW z)2T@f`w4x)6!<;@LgJE}9Im(5KSXe9r^Uc>T-a3W|{=**+56@&q zu5LT*h25v{To~?yIMVEvC|d4+n<>yyv6*gg^(=*wklPWQ0we{vi&ZS&j*9CN%CQ~m zTR#Lnk=#mqAgee{A3k>gm-@TUJD|*5H`YU8%dbQ+I*#?WA&2GE zR;dk6v2NqWSbsqI^C)uG*yoM~kbCxz#%fO3Z{y5@q**)S(RygqttUOD7>aA#!oIX; zmt8x5i+kLX9d#&YcFS?*mB9rMEGY1sOTDQOD3Y`?tp+_AsMWuK zMpI}DsIF^%#}NOfaw_tmoce~2;USQh+N!zM30vDTU)lf29xTGjx=gdsBcSC$0zjMj zBv=U+pIl#ebvQ~Cb2x(`h_aj3=T>669zcUrn6c~7|Dcom1lxY1b)9~6h_i^*S7v@d zfcI2GBTRWXDgLz*+f){OpH>!RqOdeB-{a$K5uz|r( z2^$g(YNREHSJTSsA7d{$L8E*ef0KD%9cF`9n1-4Yf!u4X$M_;yZ5nU#YRoqugAf3M zqzo<77@J$8h1H`8RM+IjN0l1GMx+V7M(1FTP$R@THmM`{mn&ptXXAhOCAVk1G6Vu| zZiqU5X;`x!6B7$NM`0J# zbWQ;kvjcg6?R%En;3#$ZS;$r(gTs=3U?iP^oLMh~?6;}m zkNzum8(TIP2GR6BB4Yh@U<>8G^rdOcO5LjyGf@Url#zlQfoZ>#+xO3|5?MAV|8!!T zEVA+Xi1_G9`*=Xyl1ys(NXF+GvVeVw43Xa{CSV5FdpQ#`n9$}o&Xs$V?781-fvPS! zAHZf@OfO}1qRbKD2kXS}F#6U>GPbv zk+67TiMy2=6h_1;ep>GyC#6?AdEFkIT$e&SfLO$R`{R(5C)Z_#4sM~=rN!l2&FGe3 z)0zU=i90SP6pD2Efm}v!a|)2~II}dMA_1+T<0$+sg)=x6{{m1S0BTk_gkm9WQ`T5I zQ&@St=0}m!{#_4Fe|CS}vH@rtn5l9`Q0d^6uBp{-HQb*5#vxm>{8_DRYS3j_r6XjC zG<*blOA{tdin#@Wx8%fk%j@k4!M|#K2rS@$U&k9g*H>bt!&tCY>15{%mddXIV^ea1 zuIjS)Sx~DB$Sy`=W*JSV=gkAW7`X+$Gdn8*tp4nB)D|&h&x%hgXCyd%Iww}pK3VIn zuCQmu&RV7XT?JB^(chlYyX?0sy>3Xexq><+jk^ha;z{}FKs*MP_;9iLuH~&wYq@Vc4Bc9TV7K0}eNKE%Z@q$RSY`vE64pkb z7C?)$Q?mE*(q5*y-u3{8@<2mV(p zyZrWVnN0p(Ok-7;U=L&{?=NOKo%XR<{IAp&ppU|$YLB8o7=NLN{7u~>16EAjCenKk z$}0G7#1B;v27hzPRnBrFQ|TW!K(V(oUzRG6Y+&kNrXdGPhhQXLj$$+mG=iU8tN^2h zub4H$Vgf*Kj5fLZ^)nMOdkK8p3m8~wT`)H^O08bA($t$QXDvFk9!QD7v`Mg(BVfe9 zF}Zst{Z3=4nm4s;9kF0BN7@0{G_6E+JsCSY9)OH`u`J^E#!gp4g~&Ra^UbWzLE#n` zuCK#2w#)Ro0uNh!cWSuZlTEba^#b3UQNPdJO5{S>0RN546FC%i)~^l&vQaRgV+WWT zrD*QdNz8YBOA2XgDT;#l#`w#I%0H%JY6GV3}HA5qzL6O~g01Hu7!XWo%bM>Yqt@H@Za~Dap>| zVZiMd`rG+&!TK=mi-|JG9iHcyssB{5&AS=CpCTU$*Y!;kJ9&(L>c+4WS}k*z>9o!_ zNy9baDWe=Z%F-G5bmFZvGTMcd0T^y1=?d!1-#23^J^n`nY;tXnW9Mye_>6#OL32x_tay-+9nQ z*f(vCRM~@UinaEff6T@e!YkuWyy6c-zS4>4ajBURM^x@P!hYn4^0GMB45Xf8CxkXQ zVh0jq0>K;sG~(R#zx9j-^#yXVwO0w6_r}*GnDYX66atdx<#**m&M38eL>iN_pawi% z_+QP!9)BN7)=+VgAU9$@Fk3(Wi^UvT%%V7>3)Os|2mC%CHcz{KAs_eGL%^O~TR1&&7{%L0jyUWI3?$7Hy|-;6F9)6+f; zA=*-Qu$NE}v$$s?+uR&~BwxtPKzYl2mT5enlOL41E5u&2MCP6g8a;YsTGt*qRPUW{ zeQU&@hN=i zq*t+Db$f}9!Cmm0_PVl<6pU5C(fLRfsCFnt#kUNte@^r1s@IX?)}Ux7rFHJWriMOX zEeSJ~SL_B6G!;L}G{Zd^#=D5Pp5sJ!1KP?ct|*4>v!AH}2l-HC_%uy*m3d_tKa6M^ zy?C<#2PH>4NsPVz83%?L`d$7c@4iYv@?TSX%ydRLW`s+sS(owHc;400liz83hs%)H zP$V9^H~ZsD=hW51;N9j~ZqvV*Vv$Fna%M1qie=|!A}K<69bVhYu=ass@VCE*02(M} zJL*mZ4t-%{y1_-4?sz)3C#`-S_lZ=1@bd2A8*F%sN9#Z1E7U)6o_x74%z>SE@T%); zKzVsYR^O~j&H#L45F&-Jzo_N8WBwa!CPt2+c59|rd+ zdjKB`;g{<89vH!0GInZhPCYc&_nlObgfPAZOoI(c}-W;fQH z22-nmlyWvOuY4&Djjm$zMpn;f$m4Ayd!Do!P*1>@l7Q%fO!zA{ zHa+66xrHLOw-NV-TcWFL&8ct%@lq(ow`H5P4HeY|;wUiF(BS{d3;%tA8OhvU^G(}Rbf1OH zmf(>oZ--9rSgi=})%F+@813Lphq`&FwzwUkSptb0$a7a$Sdrc9yLC2$jr}4RIxI5> z3JUyMCj!w=Y7uI+b4^|h8$Ldt^5VB0zez?|%d7Ftcl4}IfKVcPNKAmcSj(`46=O1f zPSO6Jop&VNerlG^SK8o5M(U>Lkgc)=ZP9S%*%+pwc(+yk`f<<8rwg4av)_)R)klLB zImp#Y!}(1$+V^`n#wBNZ9WAhaS@@hwXsmPbK1pz8VynAcdjK~9GeJ89s8$URn6hW> zm(S`|_s_@`c2d&jTL*BT)34JLW-w+dL6(#iFIBGkQLEm~v_hmhX}CS;~Mw_8At=-Oe>U zr5=_kI*`tvYB2IeXJyO(gf9}|2-j(k5|fV#Vy47|MYgTOTbv>nKMFu@42uuTd4mFY z)Li3>KcB-2h&xW2R4W<$DgY)WJ$P{_GLT1FZ+g4L%+kMGl9mR;@Y($^ zKx;Sr12DP&+J!b6VhRw=+e*O!NScpG<1|y3$qt`1534zJ!S!@#wfnxI*xU3AlqE~I zB3)FM7*r!DW%HVmkOC^<44>)w(K(wcdmf{^Sx;#IV0?YpT(JI>IBJj_&7nH8ZA}3pR6m`OcN(jT1Q?v*xoyrOaMHtYH>bK=_ga!Vf{A{QHHpeA2u2{ z(%wbIB3^fw97ow*`S=goHJa)<&Z5ndu!xRJyMo<_(sUI7(C#02EA$0uFZ{+WPz%ob z8%M=J86lR?Rmcf}?jolv4rZ(sRUA27HEXoL4rhu>;Smc>j?g^$U~_HR;=T3Yz({sH!G zedi5{P)l#@51dKY6qz5Jv;rdw&dti>++Zc|r^B~BM4#vbK2zD@gYAJqR`_SJjmFZn zF26dq@VD6Du*$is=DBv8CaBo?SPlP;;U!ea7uvfSo&ohs6p!GTP z@D-Lz)D%FtA=xUNz>AdV>^*iZC7-ZjF4uKm#}%6I;0xjvkLJm|qG|V_=OS%Q{OJyz zD6pIs_y_VYzam14#uW%HsDyUJuciDAOQA!C5U;oR)6!0Juniwj$|jb*!KswdSS20* zXKIz|%|sd8*;wD3_$^+1=;$fI*HX8m_;YXCem@%lQB>^ey(pUKHeu1RVSWmhVC*aTyJtBiA;YQ;w z*rb9GYl9d6pL=OYvwB`Q8)*U=K+;PP9dE~RPMYQx`j#gS3gYN#-}c*9N71TWxgkvq zsTc%}(}a+!k@B-fS%KZ!rnpXLduvt3-fLeaiyBHUF#!HdzyWi(e3}$yvxp*|^K=n+ z8?0Ol{BRsLG#Vm>KQvV2)R_H}3p!YX$4okwUhC&NAoY%T(WOAD+(<5RVq1(`Q@S4u z1qG{GGmG=UBVn+EpiYqw-0RFsg$%a`aYyg=cVi93Wb!;u7^I&P#zk92MRbP;0G|8l zmxiT}Ar}U}NQT4KCyScQ%`5jr_{vW);x=$28&&8k(ZdLBYtu8>m%IGdP0*V6KKZz& zL_ME8mWoBbX5kQ1x{qU&+}0G=WlRFHY1VzhUOA~{U)@0iRd5qb-e?D0Nhvt*y$sjI z931js4%K)?5?k0TQ&t5ob$|l>iS|4hkMZHJCy0Q8%g$q)RI~=aJ^z&~Ks5b7cjc&} zY#C4oDOm)Nj97peIbML^py=#FbU;UafW3%bbi!^ZLk@p4Ik|GN_q(;>BPAh=+ zhNHYSp8~gdMGXnw&X?|Il*hd!G!aOhx{cZ%4${LKv;2C&Bh zQ0 zt|UnVOt@|_^Hs!gx%)Z^I!@7j3$EqhbQDd$>95c)a8Zq}7p3O3v6h;Hxqy*FjJ-9t zt14uL>}I}&Vyff3r;vrQAm|h9e@?wmEdA`^_*^axUn0mff!jc$ch_KuZUZp2MVwWD zw4VL7L`6Iaz_<6eLvv6MZ+Igft)`&nvT_wXuX918LQ;C`h75{e)LZvaIWk+ie-bh$ z4a|i=FW^w>4&noVQ6kEf^8h`SnrSWd^ySDdS+YL0GARS{ehvX_)tq`bWrW^TlDlen zkOkbq?}Y6z*ZkUzt}hK$@Yj&9YaH#(5y&_v1wVdXS?OPdc$gT2n;!p+DJ9m?E_!~t z(hylOviKf8P3HFpYRe;$fT>hd(K9{b^<`=V3?h;-!6c%gF>2i?`KqTm(D$dzTqBDZ z)7l1=U2T{&@!liww=s<&j-2-3CykSCoWM4rDt|yM>Pn1ndp$86)fh3tpb0mQ;0H{o zx@f8JY4E#Xm`w2QE>nRN*d9l8LGP6%cEvX5A4Iz^xPYt&OT*CYEF@{`%*OL>F?wne zMo3-&Q#QfWzY@VdaLM~Lq3)`z)ztwS*s(y)<>^2L1sTz!juu}gf2QQT?Fk|OacPbZ`4vp;=Yj~d% z{fa_&b2V3qBbD{(a#zSEJju?0`WIX*y*e}g(f+%36EOoZ9rJ?csp7jwHb(jSL!%)% z(nqu6gPoR0)YQoD#1DDII|%&gLL)O7@=m6-V85sgPo0@E5eaDLMSof4nIC{$FzE`% z8BLwDR^er-ryfqD3xB2jNAawhoUg3)vSyIDvJqkZk5BgxKp;lG<#)nh`tZQ%17=H3 zIz1Y#slyV9dvGh(q>cNqMLgFLqoMnNc zo;|Vg+%-I*GiC!o#YvFX9KM0k5rt?>9HME`}HQxpR= zL7ff{EO~1rA0NBmAo#IHc{StgfMoBIcp>N!H4hfTa3SSQj9OD2J4-)?&G{*CY_u#u zLY|U&C_2eYKL{nLsRv+vT#^wb#1u2s3MX4 zzL!gV4gu4H;@b@UB;e(@wAr2M$N*Xe;5{h3N(&$@dk)NyHF^#Z!KPCCu5)t&k;~^^ zyFsfHe=7E%np#ooxK|@ccJ$;@=1bAnBC<@1gj~TDLwEloTk&Y;yinga^-x`a)NXcJ z=Q3oW46a9fE=6jpe+Z#7;uW${v}IY$<83Jckxq9&e|MUYuFL<7gU;G&Qu`TU6fn&$ z01B%Lr4!+k(248y9=y(wxw!<)6wf!ZI{>r93a!&s5I9;&wgW&TQ8VnBU>)(t1w zV_Asu2>Iyr#+PdycHYxm(6AHP*VMC-N z?}5$D7s5vW%tICfLB%d3xhe_dR7%(1TqR~t6`$)haYh~-lE?_BPFN>qw6UKD1exAX z4e(7U!PyJwMb&_3s^HUC#P{ct@jBEk3&^6&Iz|A6h*FJ{3TjquCH9wz)`s|ERj;)3 z&gB{zk6H?Oy&v$K7%YrP)sG!_&1jsdpu=_g*ZV76n9)hp)^4{WVd{t5`F}a^qb2a6 zh_OiNZ+s8CeVK;b=SIY&$$&#L=bIc^!qZR8W@oKs9B?WNd6 zR)?F8egUaFmFXhzHULs!TftlWU%!)4R0JdBV` z)>nWD)(5wC@@gL!*~Ed0IL`>-V~Fo?e)z3Du)TK~X74z7N(ZP%<~Oc5T|ii?JkE_F zJD?7nO8}in(83%yVX1ZplSVNDa_?lrXs~ByXlL7Xf_QJ=c3y8lUxvIl2(yt91>@q) zrEVo7)0zNQ_-HbyIP2$p+&6*|P;(~yj5VDG1TK%78o4rns)lbak9>}8^gxj*6`L|q zfoiJ3m4<$Yp3~a{^(ulZ_HoT-i4awX0eJM~DY|tH zL+IbVR5O@jXB|kFox^Ig`B^#6m3A3E4py3%-E+DdaR&4X|5RMSG7k{P=}*_zYYIuM zM48|yuJgnPnoxOci4=+{ZwZ4sb-7dxKd!g>9w#FM&`j;8%vz z2`{`07NkLAe*_4L5M({KZ>;+S%v@;^doIZHp4$<>DIg{8%cD6_*)lXdIw-#B8@zN77>+glk_luf*7ckW$LxfO z16-DN-q;h#L$h{2^X0TMKsq_dPe8dxJU1A8wYQWD=J56Q^4qMtK2i(A{9$R|w*XTn zQI@lU(Hx5R5XEcsR05Kz-bi$q!8`$!c^2o+3yRk`l?M3(DzM>C-<=W)vr&2}@(wtv zNg0TuJi(PPu9>Y6=cC$cAR=x1of1P)!Bx9(uxt?glo4tkp%lbd6MF<*x05Ou zdGaN4J-Qzpu8j6DDr{xnAK9rMf~6KNR6;-!e+o@mak|F8)@q4B&(SqN8(HN}Uztr`5O^tG zdXPio;4QB;enWvL1oqcBIX1&{RJ0X8rIs-CAFCh~oD!L*S z*}z%cK16b%k{NppR zY)z_2n{-o?pMdRWf!5eLg_iXl5EPpnPpg%kFq?2ito_aTCW!B^+G=QvS!D2pDpvn(jq#XvSrxM0)*H|*qf~=*Ll|UMIU)BY6XK;bY*{JK)HenVCcig`t9T`xrFryddq zmj0_QtPT5-LWpmDbmGR?7b7|#mdti7Zohmz0?O<|V(~ufjR%_Cz~!SSN5j*I<*hT0 z$I=RAY;^C;^J(Ga(mud4tE>5FXPrQ#-OO|!5GRm`)VKEsrrA8RS!pybe|Xn*jwleg z_AD3Imgp;Wwb^cdf4DY|qwT9AVn$k}FFDFq1ibK?S{6qhdG-p-_btyMvPuRf!zgm5 zenc^fFVknQR0-iWd~Xh}1k8FZQwpvA2SQnZ{AB}vtf2Wup3BAqqc1MVJLkYhs&K_& zY0JmDiyI1D`D=xIsC54qL+t4-Jxy2MSt4}KU344#0o(YD=qJ` z7f~b{DiVm@$8RqDQ&oYdwtHfZ;&3uaZR_iqPt~uJBNE${Ua!B@IXqi1T{m}`L}dsi z2*9^=RhNlSFPu@&YpY+~&o9hQK2<*7u z6aw-*5H<;`z#)~Vmt&I3xeVgS*exl|c`5T_&0TCXT$8u4iZGf>&-XFlz( zXQVSbkk$hDdvWVXto9H)fynMb0@TvLgb~}ds3jARw1xjCHKUzfJj($;rU(DaY0kO$ z2M*mv{`e;j-*J#x`Cp^bcSx=^dWS24)&tyi{~%;@Be>S){HX-oq^g848`rE+R6@Zv zs`zH#kG*dg;|+rcLJsOV@3T2`RP+mi3GsXwRPZ8;Fq_k_$m+Uo__IEL`v$a9ks!6b zjpPdo^2DlIJ$U+R3rS;f4XV*irrs(6P$GgOH2{cEzg;sd`!E5k)NUgW=c)4nlQSJ;90i`4F9V{y%1KUwATyA`2 zliXG%op)qt(LFD@xnxs_pQZhhVgj#xaIN#@L3U1qBG|1s!WI6ldI88MUqhF9>#c*m z+s_}hMd@wj<7reCxH&Gt)h({Gs?~NIq@oJIw>jlCGK$4jbWLP+UQ5hw4|>us8Qb** zXl&m#4^B9(6da4W&_*0RiYc#QuzpRglDDhIzkYVa&xCT>@c>m(>t)y4^@~uBy7#80 zlGNG#_c^6 z!zVKTDxQwTKqmj#kxcx{^j+la#NvHA zPn0&Cq=g%jAMObqvS-jTQFL3;->1#?bwdp^9nBly5vv%WYD z{!GFp#3ddsUIu9tBKz9cndgz2-tC9EpL^QZng>*_PTGroiJ2=w~kg@PNzKl1Ed~+xt{1m1oKlrtpC9)v~HJ1KAWFrRuEWS{gLy*^^Jxso3#E@>wizwX?Ct(&qTa&Sjhce#b|y7ysQ~fL zA6LM+K7j)=u*&_{aBk+vxOW5D$3Ng)MQHOOJnbF2x@d z;Y@b3{aOCLvDGUxkD<^&#@&ZNkPG_UuS=yvnZPA8WKl>Z72Llf3{8Q!-{jT?7=;Ch ztfy-^VpJ1bf5?4rgsqG3LeUDQV1I7&{ewq+toZivH~U!G)CDP*gEROoLzKleW}ACi zDj%AURt0}Am7*@x3=AK5n_l^1hy!fZ#5rdU;Zilc42u~nO z!IO_F?8-m0HLX@fqf1r3iTD}Q$e_em3aJjtI?l9}e?mkS#=m7mVzkh?>t_TA@Vb)& zagX^wRut-om|7+}OjeGh>wdJOS{?Iftd2vK&$#=0@InU6R{%jb&55{)a^teSgp6!J#+VCzjqmfO{|zDl}+HfE+$*I?paRiO%)SSO?;= zD=fBFnO#emur0UT=3|PraDRW?*T~MrAkXH$z6CX)^;+^*uTos9-V)B?GaqlT!Eyjr z(~wdi#XTeGGBNlbyL=t?j5si?oem;H(LpC?ZTEMiaJ4rpQ}^GAud|j2oy`OVbzT=B zn3wgS!2JDnni9AVzf0QrH$cuO;w59V{ky0Pcu;pOy(c!0IDqyfpDRlfoRu{8RiW!R zi~HbM5+Xausz9IueZ_Y5H@@&xk3eyVz-eqW?W_dHy)Q6%YK@69_cPGGJDUw(OQ-BH zzbdfkJBB~{n3a9~A_+ERm^GB@tmWy{g}1dZF}9|*Jw!pP;h~SK@`GTyW}oRNxKyL) zj$#|zdn+jxolzJpU=cjEPZXL4gzyS2Za(eyip#Qr zm-jSZXnW@Ut)+LTbU34@qKdnM2)sG1#=e`oVT}+|6-FbN;3qBIM1e;mKKRE$97wm< z@^8a^gE>syJb+bbxZyx=ga+sAAr}BWwv_D(nz#YC0MNY9{Phd;r%AC1x#68oml;-V@&K1 z%fIJ7Vh9ljcG2)@8x%dYBOrvU@k!07{ul!r$nnUi)eXJ_%AfhTaoQfg#KffNp#pC? zZeWdxSEua((tDsnXMJWcqt}bFM@_Z)vH0!ZLm&nzf0AfYtt+v~r%!|#It{%N_pZv!13n7dCORv% zUq=cXyOQ1q-g|OCUDZlr_=h*)_Zw48)eP%IMPdta-;&4U(czuiryVnjUI_ zNGk0T2z(e_Bt}Vje{yK5bpwDJ3d7+v(&)=hhuoh5b-Z`%6>pR?04<^#p>rOA6B%CX zQNEOLD+Ybz$87)JBzij|oIlBr)NgZ&EEG(Movl_e(0RG=i8sD(y=6n<7|^s~^dMxI zGk!cRUHU$im#hP`lzRIrwBn%?vss5Nd?2?(01!EY?#^Hpf0?O`7%ku3lLX@j@5W}( z=uo4y9=(potv4m8M!w(>dNnM1y^j;YZWZ-1t3ooY+Z?sjT*J65=gsY^Ba?h&i7 z;hnkoT0$*>UMfC%S9$KKW&U9=2+V_yd(yAipO_Gq_pwYjjx5}H|GjeiGVQgjhkPn z(n-Qk0^PpBmT2>%Ho3>{-P@<)DCh9KFP9h5Kb9h*LCu{pMscj-zHw8n4?hRs6(8Xg zA1J6k9)X$ns@T4VZ~9^S4tYduvw{n62(7Dij`dDGRg_G;VZg6}^x8+=<36y5BJf+r z`j-xI0U+$r^J7QKO9eWAYxsdG_(PJ52bbVHBY&VK|L{wAu82UYZ2X4kfat;Lzo2Pw z=i!oLcJ!*x{RuvIx55!aQt@emw1n{_rG6eWBJ4LUbjs;DLBEbz8o&q<7FhY+s_?PE zr&7J6uw0NpJt5MAbW}J+w@C%!Ik3#2{Yq-6Z-eEPZkH|WjR?0G8Y)S$$D^{I*`9o9 zcPhc-xM@(5c$*);s)!YX^cl_^QTb((ZR(Z~q(%Fy8onZM1*DE0mCR1B1GG}&5=dlD zP!=yk$YG3Tc!83@HX`>@kRiBVsZW*?dKxETRuHBSM2t0ywa@cNcyG&qsE|T0o^*WN zbqU;x@;*ydg^ecwz;y_Eh?`9f`KCee#)p!j=OO!JqZV8A?3CUW)78m01WloYPcyCZ zgTw(POi*2pb&X+g$u4lGLHlkBrPbeOof6Hkvx_eRNDQf#c4zaqp?r;SBGUk+Z8MB_ zs{xtV*owzXV-RlMW}^*+7U?r5&Lpfu@d&*@&GtANBJJY|F_u7MA0A8_4hgolaL4w< zO;9w#IxJKHydLEqkwyUUH8P@3n2EN3HcXLkx|po@a|u>Rq$CG7sd@=X$vW@Z(-lA* znP<<^>so(c^^-RTMkP~)UH}AHO%PcZ+Y6z)e;VnWP(@_e1?FS&xjptR4v+;zW5EU; zaMpjt8CVoP}je7pw4hBo5V7N+S@~p zjzm{Z;Ty0N$}5W}r#$7&+Imma0ocb>jrFuSfPSah4+2eX}u(LZB9HOuvVm(>S?a2j zT1cp2BvPwu?9b?6HgBOs1m~PjB^?)#<9cC9_@c9IlQJJ(A_A1d6%yq+>t+67hM`;f zWl+JgB3hQVb&a-Q^YrAaCI?Yx{-ip>Sb7f;K!71gPbhteThbh^r$+JOQRpD85E1`- zifl}A_M3qbW$8F7bgKWUzvVHTKj37s3ynPGCvr&$#|oSAE(frf>q4lDu?A1}u=^3K zZM~E#L#uoPo3LXa1uUn-Os{{w&HV=M$0i~bzQt!(HlW@tqL#~Dnu_?lKnJb#LayN> z39kR95+HL0^+|G?N?epss3QamPHDUy?JOEY^2hC%!&Fq=KjJZUL2_|c>0?!NrY}IXTI!B zg+=N65h!3TOmcRexcUjj?LhS*C_GfdXuw<>0v@2t=lKg_UdKAdJ=IFU1Y>X^taIz6 zr9VvrGa~-kYJk@EF|yjG;=1?Gv4n>z0>TAO?)#0zZG;ssP?Fa!V$vlH7dM=quF;#u zwvDXkaOM3s&e<(x_VEz#=g!2ER8-HwbrouKvymOpCRLa9xrcNmL5>enhE~COoU+k~ zjcIixw(WUC)$INR5Dq=l6)=-dr7UW zjP}GOPkXjYdDz-_?>~oA5MwVBAjfKO*We`!igu=EJf!$XSy0+9@HS} znCr583AH)I&E}gSnSpRO*_>?#z#5(t#n>3WrJss$u@GqqQE|NuBY1i!jpxrDc`wbs z15*p33vwqukPArKr*U_g+tX&&ZS&EPKBOOomzuW*0b8z-yy|hPcA#Lb$6AXt|I0J& z31*M&_dAa+iVay7t&R7A-caCMFZ;#R)jWPbYYW}U!NcXk5r{ZWV`sxt^?BwRDL(;C~ABA6V+*;jI%xq7bEMs zxRu{wsWOBZIo!%JPq6p?61dBz6&(-w^|@=8$tLZhRA9bC%PL{DOtc-X)J#yk%!w7* zeQ&(*myZyc4>oP08})ALh?voo2b7>mdg=Mtn#Lh3=+3^S09FEi;x&TpxE}S0pI{Xxbs! zc7$iwvEIg&DzZkX;e*xC&}KEJg^DdX(-|#VKZZqi=)o@8Zuomj(IUM;c<40`AgR ztuf+i(f~-LuAy%BKqpAG#x!Ond53y<*eF7Rbpip$L@mFE-RJov7`U7SnR7s<)2#_m zrERM@$_XE2#Nq?J+*pqQ1X!9(4ifL~MF!!MQf#EBD!kxaU~OOrPPVC(LultFLih`< zzCL{-k2R|qXPXe&d>okQNf%Q;qZ~5DzamR>4d{6l$vY(5;k_**VU&rY>VLPT^eG%apUV4$XBIaQ+{qvbD2Yo1xdl5euo<0De66PWtm*rPY5By!` z@~;wM0K(~%ZjY)tTE|p2z%D?uvtW&)I}UMhkh;DcNKr<<$v`()X`fI9^bH#UqH8Zp$ekWoTzKa|<4O(&j&PAUsYX zfF|{Hl$JUHzbHkw$XE7#c_bmG1wmh(t6xC9Qb4;4~vS78+`>xYG&-p z%}q=A_DbzxzP=C50AJYnDeO&MPzWkg)Gc=(#vZ~$-4 zp_Z3HG-AG3(wGk1D>ZsYP#82NR_{FWWJ?x%phFPKd{zDlI5BB4UC`Nj#5-m9`K3xy zzv-a>fckJaf7QjEPW{0T7{nXljV30|f9Aw}aiDZ6kh&nc>(Kfrx5xgPFf1k8o3_ds z8k7c6!9je}Ro~i0s$-|&@IBt8!;KGgX=Ir{0V*c+7WQ?x9DO%1w*$wU27ytBIQEsP zw|u5g-Hs+95QZOfrt=>i_wd4v@&Ht1oITfqgP``0Hj*d|ly?Rk84`eaKg2p$Y7NlCTmz5W?LQ1WJ@$)#z&fbT0CwaQnnWF;RJEJ9Bm|Kk6UULT9Y%A6W{DSJfy2oA%6SEaoM zC(`#fTI_4+1?u7EzoEw?;`iN1>&Pu>xfV*asilL}DR=GgUz$0adusVxJxQi-zhBAD zX%j>4bQ0~xf$@U?s23-mecvhtF?Fu-#eG8dzM%N_sP|;hyCG@m0Ha*YQu{zl^?R6n z3nDRyx~Yj}6#uY*nn^D$KKq1Tkfh$fy(@4>d~_bs2WW9`Q`4YyePH`km3qlEGL29& z+oKy6A0Dois5`y7LILxeN$uDj|E{4Xxv`p@$_oY0nED^=PyMoyRvOtVaAB_JvI~kv z)27sOW*0XYk2=U0eJ0i)5h^eZ)l%wQgQ$Ib?YRRfQB`B8rS!ZEO7##+d(mpo&}Eoh z*Z4hn!lxh~HYOrsuVs~L(S7ZyiQ<^$6o9MaC-x{i75H!AbjzeTKnUp^qhMcud?-WD z*JYIs(|erkgI;gCmjLOj!Wbs);k}}P?kSn~ic(DIYilFH?uXelXo zzE;gStr&oxH{u++K*+6e;$j_xA%Nx8_Ds_z#O|Rlf#T`W!1ozYh0wN(0 zzbhnwNoE3^suThW>7uxjUI7Og8QoNYOq>d63Uh*7rq2V!n z)}qRGaXf-}X#b*}LV@TqjsU2umGq_Kj9q4qr5U0g$zGM4SUNb`;!)yoZ9`DA5m1Ss zD{Y*H3&*tbcO6LyrbnE-7^Wc&YU-~&<;wycIU@DP#w*j`Gw?*2fQ3`jID$HwZ2~ij zW`4Ia0k_-f2mb5nd#Ckm^ycU+34Ks_QM@F$vjab|fO{^+6!nXn0Wl`g5P)K(=DHH_ z=hmM?%MTy9-`;__C=d%4Hw|<80@#dlrUbtPv!(G8R|&*yCq2x15SzGJxIoaUf^UcnBunR${A!%kcR0}`vJNMp|Z)bd`A=Cs6Y4GzM9L;ZJ z9+|u?wt6}Is~%B6UNe&JXY<~H;>+-gGe6#!9*fjskYL~)+9IaX=~Q3?^c8LD-VLPiI0 z023Cr1H8pB5}O|GIWHUd(&|yUU_qSlr2QIa7tH^rS?f7DwZX4I>rH<{O=fz(h)niG zbACdy9bcR6dS$P|A5CmdasZ?MEEl z6;&eERHJ_4bxIdp$Bi3oLe6|%H><<-9VLR)Qz}A#AQpi;t71A#x#I?KE->2;Y>0&} z+`2;(Mg^VNQBDNb7WbH@#zcRl$d&It`bcY7dG)g~?A|b;A?A9gA8J-%JY}exKfz$_ zu85LuMb}QZ!=!S_gMGkdod!}5yd9S#Y|GPjUmn=Qkt4g_U1&5_1rl~x5`u7b;YIm; zSjWf$%FDp4Q!6!bYy*LhttKf~I{~W}{5)^wv)3GYR-(!>dSPk=V-b5+kjM3a7^kN> zG2CEaAEq-m$(N=C_G7LLhsqGU1MugroC!1xfXvp7>ct3{umT78;-Kh`V z0~uW$w{3@Yrb5)KpV+jl1`CLLu?(mbE294jmr;h`5yP!}LF;9P1!d>~Qm(o@u0e)C z!>;}Y|E-VePDzBt9Drc{z?DVCf8~cf)s1zM1a?5n*tYeIiy7R+?}6^1_eR$s(6s?X zOOl5b%qN>*WCEf;l+OlOQ##W_b>f8<5ptWB$4-LtSplaR?HlOF8_o6PUXF*qNtsr_ z`VPXrv4e$E&8*s+qMPB`Qv*=`q2?H?)LW#@poUwH&@iJ9&5-mwae)S}$?>KHF@c3^ zqz)C*Hcm`aka?Hz51nR;+)r7eI)Hx5fg(ARky+=yL;`wsYeNt47szXiXm&<>pX$g8 zO$G#zTLWmz6vWPzxO($&YEW^AQ@+knW{_i^21w`n>D~TxW{lQs{rJOVK)*i;&A&3( z`YtFHk(eq?84v{vZQYI}bZXFQJ7i2aV=5;*5rMH8g1j_KjZuQ7|BSrKtQ3YBy-pT( z`LjHrGxffEunhy1V$hpFZympyVu!lbQlghhy~6=6ru+6q{95}WwQ436sX8>!viUl| zT?xNZxEsZr1INevbL=y|wN3PyV%V+p^9jRlY9-q>VRy?XYL-$ZAm%XV{?L^EnTv%a zfbtz4j-|hxwF)MY5-*=2_owZl#@p(VCr$JChn3tb{VCg7XIW7DTY%qgN1eU0o44T1 zr9P)7Ar-clzCgX~d8upnNE-YTT`M9BWDji)gXA4Q-*IODIDA&gHCY6V&iio=)~p1h zi@>m<{!g0hu3KySv^IMm#4EQIy~#68l znBDeeSD}I$2*?Sd1;F`-9TqUfY09FA1Q;jm!BI=3icBjjYr75!_3sp4vrFC|)ZG>3 zkxZpQ;vZK{fQggE>Y)AX108n-D6MX?m4}*G7OzZHJ_^q_6zl!~`QvuxQE3&zHeJ z4IF>Ybfw!SGl+bq{2JoBE;3^$OdskHDJ2>oRZF_f4W6#KK}uwHJ1M>mOTgXvu-Jl( zfR7oZhP6yavM+p%J#$P?LR*`-uuA5ZXFenV&rti_%KRJ&Emo%GWC+`{wjg3v9v)bx zCFu;|bfhnzpF)##-*)Xw4HmVIcVX1@J+fpa#RXkfcTbNpXQC0I{F#HY69<_>Ywokq zw@-ocClRv!iEA&Jqc1@oz@}@AYk91Y`>#^?g;gI7Q)JZHSg6ealsV{?UEV9kjhcd{q*9WwRt(9W zDTk-?#05R|zcR}-FeM46 ztDj0Vgfg)|(E;M2c7P@(^eWOJ^hP1iM*=nJDy+oEpYoqi60lm3cbG*yh=)jP$=30a zxX|0#PuuF#_K4Jk$^24%7PU zH*@~Z)kM5CSuO>_S-$`^_Q}_0>je1ESv3_a(}y*q=T=ZAOeU`=S|)5+F!U(ly?Fko zuj-@lM#;*XaSLvqd>wNy60@8`tYfW1gsUNeWac|!1IvLt2IOED!ev0&N_ze6ydG!* zvYkk+O0{GpUyj%I>V|-Yhw7Nv_=Ui5VfU-_>~GoL`mht`qm=s<`u2ie{x_LmKVcNc zcwXekYGdRg)-C|D8V$aBmb*Hzf|nJC-Lh;tepCWGQvcB!EYf@FBoFj(sF0>%ajlCZ ztm)@je9O^)_om$6d1Xc$V2E(-HTb!j=3ni9zNunbrtynv#&==}@@*if!i{Em3U=H- z>s>HgC0=n&OnWdGl=cgU3~y_L0WNt`zR1qKJZSs8e>AT~d>jn<+JvziT`ks{B?+@9 zD;z2BLN1ik7m=c`F`N;`i#8V9M{TPBoI}Eb&m#gn>KD=M7A}0t3116#hoV5S5~J?^ zF7iCXb0wmePqaf8?{%}tgPkj6+SAhuHo29iTV!o(J0qjESIxv7D|mP~x)T*L@8?-K zeg{Zfx#g)mC=5M?FV?jT|5#KA%{NzewjP>jvMt{z+`;6Wm-u;#U)dOV41s|Z+EbJa z9Rfl_E>V-EGxo|sXesU@*lYd%!u7@RNJWlJHyROwJU;egUb9uIk}g zh~Gz#LLv&$?rUr_Cv_|War-n83p7*Sq0+n~-@A3aN?ga~TSrHWiGL9VF4!=;>~=tD zrCx(e?HP#ze@Q+}Whm)7#pAS>A*frUjM$DSZB4o2=jBfF>i`UBgE8AA5sk&Z8qYw> z*0qG@TuA6KVYH^u@I2p7AM*3;h`nf-r+9H>~ z$;{P3$N3c2=UpD>TXWzB0B!o$nU!d7wA08>2rjtW@<WK(7Pja1^By{R=ZKM{ z9U3t>9wh`6vTNIfc9A*#GocKGrcC7Lz%=nk_&kaP29V<$M?w<6hAuLs(*uAEy$-6^ zt;^!;b{J(y=*R;EzH6kPVN!sJGP7;~|MVeIZ?jPy{sy=v0eoUb6TK zC&^YHc5y6?8=#-Ij+4@hc#~JVsH~wHAQGR5M6dp0ow{D~)Ru3GgXw`Hu8miGa5t?q zFmB4ZaCrsara?;`p(?gOVRjsMc3bKuLyLUj%(W~m1p)%`=-_@=!~apD6mmtHOJYti zc@xl1K$(TA{q4w(ekzIglNFN_U(0Y;=qjaScMCUxa%B?yN(@>Q-uO=vmjhv;9(l3z zu;FQ|Giy8u#~j?>4fmg5wUXpAu%X%H<3kdmwNr{6e7LS|O@tBpkz2C;r1Z1|XC$Id zWx5b=uWGr^`r+%S9pC?O9-_dFp6v-Lmn=D3PJXwb%GRdSjgg}now<(fVvU(DA2D-) z=Q`qn#le~KR*g$S0>`K}elBtAcbC#b((#A@DYS%mRcTawc#_(Wfafa`d+ZR)8(&;e z!ma%-@wXZUN0LCfYe`R+?@v4@u{cgEXx0LGh{-5z$O+$ftOo)i&*GwhCM+EemfleQV>F>U1=)uOpiG64DR0cUF2!)G zFsh!mpji~hC!%m?9M~HyivB-%s_-o7<8*GM}ydJT+n+8Ee@aU~D!Up`Z z!BnUOpE^^GK^MXgvb)1D3x67gVP|PxzY63D8{MASG{zra6-HQ7T+8z`8KuKa`Bgeo zH~-+p)5d-7kc$6eEOz8GpT`nOza{3SpA|wH+9%W9o@mh)Ect{L_Crp_)xqHIEb&L^ zirB4yJ(lB;NU-jBt_nSi>@@1i^2eJZx=DI=p>lr58Owmpr2DIysA;@I6QDc}iMU$N zrn;WQ>85n=223^>&7EqGXipUu1B{?3j6xUIDGJ}XlCYkv0Mcdt-QhvMTO(TQCqVlG z@(m{a6N=6usl=}LrrmZKRTIU^fJ?H;8G|AM;wzUh2acoRr*GI#k(eWaUG5nj4NA+t z+h-%h%iVJhnkF?M4((OJh*^PaH@We8{!hkNUUYhj*EY$G;!H_=>ZdI^Vg!Z0F^l$#djmSCx}M_( z*iuCo77JwXr7A!dlUfTI4;lBb7WJ>ojjQul_B}LmCe*j}mXlmgEoBfQVza4sjFpl+ zFH0Z1PF5ni%cm7v8VDwAXL-CIVnjCYZgCn^1?$unn;!LJBE(&HTay#E7VSL!CW&{9 zmXD_iJ@#8Kb>{@aWeSJd$;|$*_AV;*Em8v1=xGt*lXuSgBT;3N{3#puhl~nF6nL7* zm}rVD(4Vn@|2AliDd7btdR7>67B8YpLg`01Qw9JA75U}pqMqH!wXC;ERZSVi5i%lS zF8(V8jTs+SU`t&ZO0|or@yeQJmnu>jG^SSloagjt(i7A!ZbJi1~I2tu;1P=otQDAsm6hNrsIn$GT>$i*W zy*n&a%k!|lgB{lkWfhU)+9dTMpV1LyK!lj3x5K2_L38sk@W#0av932a*cwr|wtW`ERQVgg@MD z{P#Vc%;V@y`GolA?EQk9jtV3Lp|13xlE0NJjDpM{vR*a+a5Rg^m?{#^&YdN4F}BCS zAfARXR@wrI1cahr{lcwu{hKPouOsGIwONj|@hY?FBXH8_t10pQvmfvhrGwCrv z?&<(Ro@x0zaV7T-a3pAq?nR6Y&KoZ^ziDA*hXtRax3Or!y!issln*f(kk?+DdxkIJ z8dpUwjj+-Yt1ksLA{T^F(7D1`6DP8?uSOu*1sy3m*wdLWU?Z+LG%T1KSw4^DYWW4x z){$(XOUIqNTpYiu2i*Ng#Fv@;TWMBWtZd*XEB#nxv{1l775<2gG}K4#V}(5w`$vy(1bh zj2m|BJ&e2!pV#TEm|nYxfz1&~PoAN&ZjaM|zaFRKL6Q7Lqn}MPIxo`sx3_oHc~`H` zEt=epKCN^UM8}TSqgeZ-7a8hs_+y{Au!HXEaR4rgyFbdar};V|58ncsV~(7>jA4*+ z)Y<{kw(3cWa|CT!b9nU3Lh2bEj2eE6hEC&|OL#2c*pGJOv{n^gi1ONTOxuA8#)beb zjPJtrGb2!ha9~KkpdzX@s^!|GJ*niIBt7F8k$-^S^+I zCZ#Xkj!kpra9pu>HIRjr5fEP+NHakF7&F;=pH;1^$>$n1BHvSW$exG#v6axSPRok# znSat|d11ah>2pON)FZ^S=b~PQY{z={V&2k%Hvs<4pl~lXOHr&M6j50saRaM$XbITF zpOgJ-hL5NCtY}A|4hQs7y!qUc8{_%3?L8jktA3pIPal49QcIsIUeC~g%!0J-?0k_C#xuz`=|-aOpu~%7J^}kxxIGO zl)Fc`ToL(l8gUrisOd!DLeMVb4@$w_q|HvLXMNnj>W?d?ulOj%FYSpxeGj)#a>9Vy z2Eybs{E#<{4Z|aj4MMdCdIed72}d?h+j&=tUs6Mn!UwTtm_fk<()BXKFjp*IcKZx& z422u$dvVI1+e=N4PuFm&1?V4v%iH_s_Ab9ufn}DMaZ?v?Um~RlgG0yKU$l~kS zv_RijJLR}o62i~=Tjj~dV=8p@-jsPSpDmL!=mxB=Ov6qBegX!A;EUe8S!pModk4`y>n1E82)o27g`TF9{hyKeAch9w%(fM zP*F~qhf%!o#kdK$)=6>WO4%ob>q9JB%w1Vwrl|0r)ClulAh74MhGLxmHIS^rYQa3ye zu{bgu`_jjdO>cJJh4R^$ZzbpSTmB@CnWoKmd?R~GU@mE%BH zx_55V3hTezSFObtF9;m$L*vVyaDUJrX3)y;Qq5_|#2gib5q2uLE;8AFF)?^{b@D7G zr%SE9(QmMEz4k#_@0Utx5>!po`C}K411VA0(%`8;-#++)o`Vfm&zYA8Amz8acba2K zx4cZs`)gP44zz1%dJVNrb$CEPAlNB-o6^25gt_3VEP}rUD#nBc(!9ETs*EO|&jPz3 zc5cr>;(M%Q7Mc;Mv7?lMCCDNe$nq?-x0HfDtddyl$4@NMcvx%LYqb|_uOOCc_4Vqf za^)Du3gf>+=j7;&E!#l8kVt2+X+>Ne=GpbiPBA~olZOk+D+kR-U$b`&C^?-MAw;NZ zechE~^ln$t6cQmD=K7RZ7P9T0I7&xMsel>zk=7FDH&@*|r039Aw@Bmi@18`eI$bm;eGORrYk*;$yS^4Zn-HWx zC`KI0v7IM6LZFmSw~Z&DXi3D8nc>_wDwyEkCnLV|4CL z+ObX~jhZdcXEvaV(Ut{_dB=SnAGLRyQ{Qq6%FEXP0RR9=L_t)6>}g&jdiRw1dm z%O!#mbLTW6-xOz$L49rL`CtJL5Zx*lNJ97f(L*w0001B*|u%lwr$(CZQHhO z+qP}n^>5e6qQ(~VDcSnL*NbKZ`u+GN&lDdREUqtg33r69ky%E3m;mT#y#F2;3`FnB z-vri8B7^J-K2Xa11?6LKY^5zmovEavwcZdA1Tm+6A?jR-nV{dH;;^XzH9*S0JGY=v z=CRq{>f5`bL1T>&R0i2F1WMn-MDCB<2Y!-=(0S^olA3q^L6lJsU~p47|2$uhCcc$Z zDbPnom16`>fMUs_-VWNy(!DC#!KpB3j4oxmS?t;n#PSOlq&fLy3H7x8-H8IZ6h*Nq zwdUk3u6kg9NOb#;oN#zEB#WelzibZ+C1nA142Pdn>3v-w)N?3cZ;Lm++3L&FJZq0a zcNXWxtJ7e$PJ@IZ?ERKydljl3pQV5n%hTb1gI7Y?9MIS1c$*q!{Ij@#i;P^Ldu3zM z5P(KV7tU%il%3w&@H=7SJaiJSyaa*s|2m+U1-bgh7n$a>lU8%|@tBVrD#Sed2w1Dq zJwM|!j*9}&7JRgK4@|x0puWHvsD(i1OAKx>r>HJn`Avc)tg_M#9)vGM9CU+apB58UEIl#m z%Nh|6{n+iJ11F7ky&0{5&#k8 zt3(zN2n{R!sHdjhYw|JZ01kw+kw5NgHwjxWS+<0Z+8)#^hJl24?S7b=CtnSw(Zh0; zuG_Vy2t+xhL%NsVS8sc1R%YLjGsQE# z{y+vwX%@Hod^md=Wl*=-Rs>d7j@Y}CNAH{TTcYFj7ck#dznNLMQl1B5|e3 z?xABsA-mJ8?;Mq*iFP1iihSl#@1@eTsS)7C>{s%1j+!59-j|9C?Xm@@%as9wZ%H%^ z-P=3b9D`@GvTLKKaSarTxoK>wXm{y0j*<`@orx}fx?6z5J32R=GSdC!8e_dzE0cYk zYGWn9jAi=`^OzUdf5qS1_KP+$?PWMZD zMYbwe16tt~bl&`h3V)(@2P#tz$byXiHXA_-r9`VT&#H@T^=Cm`5l_{x0pZjr2VjGl z5YKnwe!9GGhbgCv47RIP5Q2jeV3$g~)ho&*X{(NDcx+2P@~Uj;E6PjwpP-Y^NF!I5 z=)2*vLa*W#F1yXg5y)@#CzaHGVB z%}F({THB1c5^#9)0hr)K+MSy^vN%9!TYT_^ta*$RL-)|zP~vOl^#Ble#TQR;k3~yV z&7a(QZ$ddFCQqq?1;Zc(1eB0IFy4bT$q7!iD%8WUQ&SyNz)9jTP0QM(&J6SyRxGmQ zhw4dxNq6u@rVpP2SDwWpq*6}oq>|-9O1_XGnVt+=wj9wK$gB6dy_88p3o|OcLC#7PsY7xH823TcdL? zC$2YvRtAw8-BbYbG1*rjK$j}#S>8yOY>f74X-3L0Hmk4!AX-FJrwd?%_By7YO)&!$ z34_V@B06B^lc+>fr7UlVSx1xiq|#d0E6bb50x1k{gNKrJZK zEjBJYEO;}yKFwD!_0{~V{Q;y|}k z+rj4;v#14pZEK`4qP*FSIbE6EU8k>0R{=fSgxmd>1TSzh&pjK+pq{0<$#-)kK3t@5 z(J1_`uzd~u5s;QVxS<6o3J&~f>pkw)%Z`TrQ;Z2oCYjORXdWz3>Ld>1TiJ?M!Gy0< zKQTi{*NVP(nQDuieHPKak0M6euS-W z%mFokrOUCqn@xTR4h1eJ!Hr3Gk}~6lH4D%iFfit0X3?gSgkawEC=~1=+C2)YGIjK! zAaFeslfyN56j@37qcd%^tT^UJ234^I$ZPc98wtv|Dy+>#J5qvAXu78h@8lkb>_xnr z*GI$E&QRTN*OoK7E!`J^pXfmzn?%XTmBoueb%CCl^~xhrK`!SzUpo}~7q?L2&vjTn zsuJgHIg$_@PJLn$tFYLh6kQsUFV{d+&E7ke!IBBvz2B{F!|_y#Sh*pSR~0h~DyqSl zEBEbwz{P%aUkYIz$Ho$o4qn)4KnCPBy&3ATpURXowY6JBz=*W~Tq5P<-O2d^gtU%Y zx{zfl!r5FPMrUp-O4dMLBw)w7(}T^@WK4~bJSh)dnOc<6s4G%atNgQvOldxO49p-?7_>Fz0)*N6GYpeIL^bVMZF$-E!AUJyreZXFuWXG z?FKYEoO?W5p!<1^FRT2r)*fGW8;S%NN9fa!u1mI!^GpTkZ;RcEdSHOR@clfCS9H|R zwdlKOgKwM4SZxvZARWvsEw-m)wZyW&eUr$;=oUu_U|0h%Bf0gY(Pctfghve26ZJaN z4=0*dhVY!C9kLl^o8Crbz{{_Y7Wt*-Kl3iD)ux+?-xFjBEve&apU4J{Aciy>Y)=-u zdS>MTa@c~rPz(}6pnW7v7q=zTZRAm0*A*vnZ*F7`Tv}WJZ<6{rqD>wI_|W=b7Zm;c zd{^{`;chtPQ2sdzE{6)aQ~#K-KFXH)kKbhU5jr!6Sb!a(&24J*do^1nW7>sD+sm$^ zA@azWTbL4SOvr6UZ5>^NHDN7MIn9stBX$9Yk(y(fC_;ixm&s3@(QDS4SERTg$6{5$fIe$@>j-xcxU_omnY(JWLZ(btUE=eJ_*2d zlwIJOx1%CXECN=#wkKDBte%Ooy{B@u2g@s$;RsQ;8GY+0r{3V=KZAG&_<$CQe++?Uib)^CJ zq-w&F>@o@YeQ;3tB*k{5B!Tn?RDGmuk`6Z+9Q?pv4oqE2NzH`AoMY5gyUIDXPa7y9 z_DFeL0SiTp<>oCKtexN>{-CEJQ~rrdg$spkL6f$+thttYv`_*TJm3J5v?X6gvpS#Y z5C=$H&2jOri*K^5E=y96*OweICYR{$x7?n+HkJRdlNuZ?ca=3;n0bzOZ+-Up?JI2m zef#z2@lK9`!6{)*C!Z*gPlEcsEJM=V=Kea+GUNE8$Qpua*wyJO8i;^cJOq7d+AJ20 z-MZDh*(lKOTHRx>wU~8L3Ts_o%6BIY2#P&G$gBb#d*3pn>+#p|kKn_hQ5cEM%}5HO zZo~Aklu8eY)y@c&H{Dr*c)JX}2sl6A;mM|-FGe#tJ-i?Pt*I_&hzE#F6sBrqLyGl9 z|9tji0h8ZadOtR&te>^B&mtkVrtfF5LTQuA0BaP#vFYH^6*dh7#w3o2h2;!%eWY7$ z2wITmtE1CbiJI7fH*C#d0u0xeL-$wk>4&ly)g_jqLL= z_8qKJf-;R&*p}BVp_yZ#()(Oi=*^E@IibeWm zBhC+*!ZHX%8<{484HR}|Hd#dGx#P_dDe)((C#M)m;FuvaC0U9L3HUd&1*d0dEA=4K zR(9KhP*OR(B@+&I{P3Qavq{X{JV+bS>a*)`Xc$k*J7^4OOS=`M)l5UTe0S*bm}OlK zHqF-%LE7AX*o7Dc%GeoX3b8v=x3pNa*2(1Mao50Uh6(Q4rh^jvF%78HY%JE8`o?EY z3<$LD!&m*+C@xh1#|sNDZAQA)Za4erp_%;FOcLxWap*0orXbJ`iVN=Sr%D4f%^hi@5jz`U3+EoZnhxwkxb{Gftr&lmY6n3~NUsu0+T)|wf zqb)17;n(sSh9%0Z$Sts}&A>)zi2_-XsqjiC(d6M)Bk%@J&%HY&^vVJtp>R)`zfy`TSaIKw=OXJ)VO103 zy}tdj9e&R2>$p1I8$eyu4Ki(khfn1$h_yb}4s$ybBBx{HuDy2*=!OmW5M)wdk_&`0 z-F!QPlZGEx=BL6B<21}0`#WKKP@;B@8s+D7dG7iTyOF6?24#!Am^MVq)uy9yQOfkU zN-28N97J2U86}In7;{FbHh1)u&!B3e_J&b8 zntaf96;F5+*7k!Bv$1O{b`gI9K&R~hC($zE!_~;)~P}v2=+_RRpf%DZ24HR(0*X_+9Ww4a;H{+OX^uGC1za$ z;b8Pf4o1Ul7Pbaq#%7If>x{`N)L~-R?2X~wH~!a&0?&+C9pwOW<3W`-e_9`Z<#9dz zm<5)Rymohk1UvAsL*B?^BtvE{*IQ4~sjTJj%!-#`H6$kf-XChzFa+O>yWm?vKrfxhhEPY|?jP zS`Ma<2s#Lt;deQkB=&QNnWIE_>gsYEYZP$L@IW38F%58L5is8mA@$A&^DqzzUnmhxUOfwD z>&dLuDCjrrP_Da(+C<|3XYA2X{^eSfMs}Xj@w42m?vY&qYnE;)Lquet?7Mk8DPHr~ zE$VULidf#LF#}aPB2~TpQ;G_{kh3I~9PKI15$3}s&>UUbu-cbzU}LtLi0<2I5U_8ir-@JW8XN7sJJbR$5PjPLtPasiij0Db2v}Lk zPW%WReXqHzlXc|@XjoNJ4T*}u5Y|h~Y-aHp4=3@kPfN&k-$1Gcj4c7ewM^`}h54DW zbUg?ct(7}79)rSaJ*L=;o46lIb;dkq0>L$7INS9IE-SWMiZH=n_Md{CgUetDi1Ldc&iFq(Q_t3?Ec64Tk@kOe8dfz`Xa1h&FIDg~ zj2#+fPA*Ig{o=akQ6=<9(x+vo16>2Ht8OBhJqzNye99J<$_3pdL3TbqW4NuynDi!F zgyfGQsepoS`w*b#hn|MfX+a{1Xb2>(q$oJUk{%~(JP>H{3Ik|zeLmCl1q=PL((3w~ z1Y0vjy)KDG!e(??F?4z4K692)wH9Rk2p&IOfNftEmE%%+kk*J)M2C-xR~hXbQlle` zb+%XZ`v_hqrjIZ}!0tDC=@1uA9GB6GQE*H2_LyJ0Uh5v1aaF!+7wpp8Atv;~fid^CkQ4*NBms_%VJRuJ@l@ZXkS>u|+tfQ9 zGAqABudY2_)Zb{6E>GZq6In45-Ax5(=c7rbe{Qjo#h7 z9L$`Us8>f<*el0*%_z~!nmIf@!m@iyL;gIhM|9Cpd1RdrhLHxuIuOVgzuQ$=%pJh+ zgV#vGDQi472);CXFVbc;z?*nBr0UHztiNitz}c$LsIl)ofIsHi6BW8@h`)W#gd`8n z6xnVRFt0A0tdcw~>=#Ke7l@#ljsk3Fy(9ju{w{$WxER2@=RwQgXXR8bu>2QQkXhK2 z3^y|OY8B7RNBQuHZT*S!;+U^jLImqdyfQjOYLT=4sOFbrs6?^R6ko^y9If=-o0In;ej6N8FR1du{KJo1OX?G6qLeqt27nqV~57JZ=}`?}fhf-#}Ugv3i_L zsSf{&k_^`HAJj#nwht=~ksWCn=!L8Jfm*6#m(_jZ1`7;N z57{dllAJW(j|(Xrz@Kt*bq#TC1i@pfe&)1^>-nQAUH;)eZ#&I~u-ry4kaD~nVS2#T zI~~?y6WeD2;6;2Ea5n3`By(8z%k@auuS%F6t)17s3{hZX(8dq5&kl4_TYPxPSsfQZ zskU(>CJm0qzDc%_i%|D?Fd`+aP2tKnIPWG?tKTB0CjfXj#Gl#TnT;uxyK1&Gx4>G( ziWdEg@_iz|_nysiXY}r#uh%V%UAXQzmDFXt8Q2f6$9U&@{HwT3LiXD@gRs_;cx#en zr}^4`TK3sewJVkZvG!$~L(XhC6&;W^p|>_B$Qpko$U*G8J>*C~|0+*DhO;(l%RNit zH-jgg`YnH8%W&>F8;Ld-T>E(5$x9e7y9KGg2h|#?{dv0$t|Q=ZwyZk9uXddX=7(!b zGI>m4l``{>d)v#{JM9QqcT>%dLL@YM2g-~r`nvi{WLLuulrCvG>m0;crVFh8P*pH- zv*R+eBd8PxOr?fqOI_yZhM&GCwY`Xb4SQ&?*PWcIg~pctEQJn~`)XUT{Jw}hDQ324 zztbod*_U+)QGUR$-cgD|gpI~na2p?6Enu72qKGn|F&Op`hI7O{Z@Prd*`abV{u$k9 zS60!vh+B#HVEj-a%B#EvpK3&Hu7Gr`&S&o=SB7cV#E3w z0OQS7u<>>2pB$i|0`^KB4;y7mNialJiux!|ABsvbp3vL>YKltmY11?rdjJa=`;t)! zFT?Ng!gZPryl~9O4=azti%7dpfTt1-(DOib{V0_Hf?tI5+P1dq3`PNX$jN(06PE!^ z;lD0@LjJio8T*r^1Lbyg%OjJOA~ymoEI|ONP}M1K_-lG*c%W)eYC00ieIF;FnJ0t(oG6f%wf4D?1lgu8D`ZEwR`C z22I&lmFs8zNS>OWRN;$2PXd|w82}61n!Zy#f|)al))y(2kAbG-3leeH@O9A+(oY4U z$dW!Os-`&V;iq5MVDqm`hs=K2h#8z1E=>G2zbx)ebt9}6UMq0U;LsW9M&K&y2KA&| zySjM0_+LTWmL0>Wx`Iq$O-yhTnG))F;k5$LXJrB__K(0Dgm6bf9CU4y)qar;tg_Wr zh2kWb!)}V*g;lZ%2XJnhI7m~dm`c1&JWmDPW?Rd!#y^gT;TvlH=Tl6~9wf;kAf?)X zA9G?{%R+c|#Ld}zsR|H!3{atY*@<-6_0;A^Rpu3ngj1QET59lGrSoYkW(8J=H*&f> z6Q@7fqJF;yaZ=HelB1yrsNBig;zs9A-tYSt#HCck(e!W4p6clDcX*8nW}bxA#G>HnxJ1wzOf6MMNOJYn}bW%ccVl?sApYOFR z-VKc4d#gr$2z(J0d9m4=!yIemNJj?(nrDK|4-tc|xD3r9v1AXaK<%-hydz6#Ae+OZ zI{J_qR9Kv2+uZFM+4317AwzIOUQ$!TMJ>Pi8lr$K03i8GxfjBUER`chPo(t5Z89E| zBOJtX_?8mPbVF}#jBi0ar(<fQ1eYx{LOs-I;{$)mkfp5PW#Pug-WOp zh$HS9H`kIq!m4YaQmt3eS~9O9+G{<*`-&3l6S8Ja-L6lUv54AtK+yd=OHkh)D5b`^ zIu1Xc+GZ*6G`{#LA7E~L&f8$ZWKVZV>qTBl++nqPpFVs zd=?g@+!+cXtO@x0JHWL~67^7W!hcVW?EV-1mv@vVh;tM%joDsErgCP`#DqzEm-T~( z-lqgcgsC3@`b^vP=HjzcU^0BZ8mv_ZAhMg!E@BT%muy`>u~{|MS{?U+GD$FeXd5hWE0e5j4V$zT~!g@qtoR<0T^%b#_S zCQbAWanD*XOMwDAIl{cLbyD5q`yM@q*{OUr{Ln-yXzDEgAjxqKZcy3G*qn1E{o!G3 zz!NGAO7ysC^wCk;BtKaj`+;ig``g>&?bk7lJ34;#P3-oe?VyMHPzs&`#DH`&0ecsn`v@+!dWz|4u?LvwaS=VB*PQG z!nEl66c|Me-#3?vQf=`ErYO10GdqL(*9zw;v@r!&r)`$?JXev)r70l|+b zkqirhP?rjAbM}rqTaIU!CNk6#?Yj_AxsIv@p;W92oG-9YwxWt18{QFk?urt`E_s%# zS9EPkE=pUpOQmsR7twzbI;}nRKB>6A`+)ablusq_cibG%o6`~f~ zx5`M}tWs@m3#m=aHboEX={dTufqT+9|E7Idl)+k$o064MiyL5L!?rBS>vJDA3Jf1j zJZnGMm5}dtlj}lY|A<*(*^F!oEkb~UxR(q1LuHnp5cdy+5 ztauyPFO0^qqwiIC=(LKu3}Foq0M*Fv@onz^{Oufq*a*>U3o}IKAmn~wGEvRqCP#h~ zgZ;@LldA>*-4b~Pz?`yh0=!g*3WC~L^ri21=(F}qAz=_8ls+dW+8ot`F6?x;`4$Po z>}^tLfwDWNO!lvNm0ACJ(*3u5R84lFC%iai%K^Vn z*;}Em$!ZC+8_NxmIit#_RwcaaJCTJun zqp?TEO2^+Zbt*^{f6E@~Z|k*c{&cmy|3=*(xz`NOwAQ-IT{5w9gA;@oZ`%CdWs>2- z*buS!)aiA;GYOGeQS#tm+;M>oubu>Wc#)dg%`u+m%Onz0QNqAxZg>^{Cr`5ej z$(@0>j}h>jk*=Yhh_XnwlI}7s(!%fh@Rf8huRSggZSHkYe)?{V#kC^CC_ec`=nGm# zqwqf8Thi=fkym`K>b|I>0|`!MujXm)g{vH|=UGx!m^l+RTWWvmXNvgQ97~;!KkXM_9Idy<3!f z#uFp+jTok=-Gf>;w%y)zBYEPz9GqYK13R?Y-V#AV+}?sLXZ;|>o{RSOpDqHz#b6Q~ zFpH!Eo*3M;4i_gd{z@j)>3&4#UrUp8*mPomI7#tJ7E2|>L_)99YgrpNFsuA2D~XAO z%mE6T&W>}8Z=&X@!BvjhLE=fbG|DWbH^$rp`yuvY%sZiN)JZDr0r)!wl@<)jv#C*`T(1=4 zc5YDNukXt?TWNwe#G5CTUUFQ1L#7APqGNwbAjC(5fG8^r0@&g-!Vf>B#g>q~-YI1K zSD?fjqIK0RSz%%UvbDL0i%Jhfld8>n2yA(bIC~}1tc7zCym3XGR#fhs0emWw$nu`) z;}p0&`NlaBq)`>v=&_0axu-RU`#OpLIvw{7<}9D1%K7s3TOOl4)STfYzC@^-fcHW?YvL0Ls5)VuE6F)?Zzu>r(WJ4677pP2tjd0|p#(wqK6qpR(zPN3ObHLByusE~Kez&vd zui+fmn&dr&CT3%#MgX$)HdnOlg#Q-Yn;RmNmz~t7T1PjuTDH%*slF;#Fq`iC>ynQ^r$T9z&LrU?xTUWsMlb3f+}Rs$>G@$L+Ji{yQ^SHC9J?T?`Sxu+NREhiSR})Vy-n^KV7w? zc@rExPSQj;eZ!b(6;o#{>LWVJF8d*TXO_O`fEnjAPKL%;zoUrP@js(2Faj;0S&yH~ z;**~Ov+aa5KL*0o2-IKp=2l$5qIsP>P|xLs2-A zS)F_4$1}Q@hLE`)--HuD35q%Hz>rn?aja>Kzz6jk9ORN=i1Eqwt$f2MwC)QMpq0k({d6eca{fAAxJfa)S&NRB<#N5IT*3P@YPC{nBk9Un z?X53CU@cdNoZ5;ZrC}Y}5DY})Pp-L#XiV%otHeWzzFh!BAfgqD1BAA$NrJm|Gks#B z9v;HWo(Dd%?Hz&HNOHozDNYib`HBY==|cNd^DbE;?A}1*ay;{JZ*=tE&vJLdTcNeN z0c^?ee4ED@kG(z`*AfNFuGBz3Ff!Z)YY>SwVC37Xq(6|ci4IR%LIXHD7f*a@w?6UB z9$SHAmW*sn_SqSy1ES;3ist43Bd<0jWwrCQWI(4jgr6nWe;#3XOg&c&{`RSov_m%* zr^{4kx$Y>DfxPK!n?vb^SgWH{6g8%RUWtY^`FW!OX}HY6aWP}5F^!$A6@#JN>(>9A z93LM7qN<6hh^?XYYH(3dZJw;){p`(k?Y0EM`RKu9+-Rep(=PW2^XE)d8zm6Pqc1UxQAfBBxeLAjTNjBkg zR>Zh0z3B+47DRF9NPovHkZF@*hOE%&b%^O>i1FNUTE_A?d-bN&s98sPwvfDELUA`!IpnZcP|z!c zyD@!--)gJm?q6l2=P>bFP3a?4Ycd(nl*1DO3XDsbe~(l?479|K$o&g}+5T4);Iz8@ zt8eR>0&$+k%ZmL{Bv6}y_t=WrLaJ{SQM%SDbMDzOjSgsdODkvUO?HM02XB`#=m z4RaGsAvFmeSFYU4Mg+ipyweqdZBj71uNp^)kM=wx;e^weLD zLQH##O<$NDbrI@#u-NX}%-94s6y+DTfR4COxZQfFF4GF~S1>AY>;H!NhKn-+;EqSz zgO<|~atQE+*moA{@(f~cn?|YNz<$uGV{-$*S3=x)p|*wEu)6JEwE>`lqBbF0l5~<_ zw8?OM-}8;0CH-`xa$qgjezhr9DV#K*Ho_$+4Xj7cH)kj{PgKm@Caq(Ct+`4$A?4(v zIf|$AUd@jF6jB0b<%j&ZS!X^Zh?lKS7odcH1rGjxAB{Mw^J5L**xQPA3Ru>h-J}Gr z+hT}zzn!x5gW*I|O<7Kofq7Ma)zdo`0y1rnjnske5^S= z`_b2Pq5vvOL7g_yDz77MAi3;7S-Y<*lNvvQl4P}VENDbd@540=hUymM+T@2R65*Ld z{U!f(cMA$rU|R^xh_i+Bq4JUM0XGX+&6kVlrIYM_I*)N+FkcHgajjPk2mMCd$@$`I zR85WAlIOlJ_lWwDmspn=*p-AdRal@PNmPyWV3gC{+Bs2He(;P2IP0zy8@;!b*jwN@ zx6C{on(HzR$B5?*umg97jL(mH)Z_EFnJl5#Mq={DZo>exD*wdA;o<$q#Y>`#cphmp z_ASjI*y99tLWxmEa?Xvrayl@tlmd*H(hBTo=eQ{N43-@sR0XHPhmk(6fFDnW%0&bR z`kM&6q`A!~SYun!78lLh9osUCd1U{* zHSpFotNKDT80f*pM~ZSc*UdhWVEO~)E#3TuM%}dq_p^+ZTN*nXXNQQgQh%^+qaF(> z?626MceB8)?O6|`BhH;7sRHaktJrBrrsOP&i5V)35UdmAmhLc}x2QF9$wCSTLG>iQ z7vY+~WpS~XeEe2GJQ`XK;4%+|&(Yot@JgqwlLt5jenlsMfs65#9+5&8Yla|OwPol; z9Zb;5tn8%BIa6pk-M5U>pFm`muP>h^iTk`aM7Y}n@oCP&A7QOlKk;hJSlsw}pY_#G zN71(=9|2qOfxTyJs$dH2fUp}gplb+G8SS~H&4n+3?CmlC6x^E;t^u>}ECUWe)v;qm zlJx!Re);ohv3K2mdDUBw)qza?WZK1#U4nQylK1CX#7a}s;J>kdpiUOWfXoxp* znT|1+i)^lM&jD8PtjCvKdTWQ)CB6P>%XJH8+4J5!yzZBkuB#HRdO$>r&)cywqX4Lt zuki>VRu{DbuUTH!uAfV*imZdEzTVqFeseDb`m{P7O3j%9Fw?xtJcTXu>}9BRdHeW?Wu^G^H{&zAap88YWzvn_T6uab_8q1y)5IfQ~g{OB^6OxWCzHAcFE4;2c&m4WW7WJ zd1($;kA|1(em#J5$%+ntnj!9~2pVSrD(7_p_>m4?lj!3N=p#inWjzRocmcb_4bi%M zn=Q>bj$B<{L*GR3Yd>jA@tXn&L5)wykO+fgXH?e#% zH%Y*<-a8K3I7bG8a$I|&2`#)m6Zau_0NNi`zsj^M!k*ZI)g#W0Xp;jL;vHz<$(gsj z3*Z1Sv;Ej4Ts4-fYZF$0EiVW!W?p4vy&&pVUsdn;di%*!*cpeiK!dEresMf3bH?qx+nbRKL%HB6J0;Jp^ZZ4DQ4d#e zVuWc|qR$#FEv2;K!<<4$`iqLP2F=zV{0E;_%}lhq^t7jeO)CCfU#&=%+2FWWb1O(A zgP;do(NxCQA=jZa-n(tM%MPjHQucW@og$!fd=Idqn@u)29auV!L*4WyYf3aU&fKqF zCV%d6EjlE!Y{2hG;kRng2ro^FZ~hZS(SyPiQOX9X`0$-R0vc3JiUofji4J=P{jFO0sWGW`mJQGf4`<*JQb_&Wp1hi&%6Nv4ywB`^G1z2)*5bQ^i#oJ zla)%Fr@j2N6P!C=;^&4h0wwW-&wF9?Fgve3qmkUt+$|S zQ4K+mras`_Jb8O=F?8JF4i&&i^dN3b5cnR=}r;4#Gdhf}XIk5T1IU3tBQ5|5+I+e5$4=Xh;PH8R1hN`T+izRvm_(6gnM zTUFARfs0$Gyw>6}zg(sq^#bHvP>g=A=b^WqFLt0tb{R`#cbD9sLf5wGjT5Skr(XsD zuLpR<&DBBp@Sq%07SiTlex70hJ(EA)5{c8f_$B*}=$VfiM-&Ny?5<1ri*_`7Nn z^!usLpMGZQGlx%vk2t}OK#xL#jdi?8(>H!J3VP0lSByI}hpGu9Q%#0bGr&`hxeDt5 znL%RwNlfKNVEuY~{hWkXzrkXHV@UM(w4ja*#5Xdxfl}n!$wQY#w~CR7IB8--dnCO~ z@PQIInk75(GarU-q*#fj4#d%sfbU`y#ri`=-vD&BoI!NL5K-D4a*H=#PNkekWzZ#q z(HeVH>`}+M1e`k=R4!sQv}!*7p`mm}kXK9dLGw?B4|>O~%$CMvBj_Z;+%q;MCz4?) z`_qb}xZW($@`krRb(j2Cc}eLjYEkJVvVm@wHnz+k!nl^q7<1O5WXjWpm8m0Tx@+^%e-fg)^$ij_24dir72BuLZqsv>#7YdW2m&%n@NuP=ioBl z{$b0Q$b}1X%{>%8qqd&m>^36rn z-vb}f7p${{m@VV4Yh84Rc7PS_{l#(N)uSIkRb=P^SLRoPxw}qScMtFeMQi}SYunZ1 z`I${n>SE*ZAVKp{g`p$NmvTCt+*x49k>ifDVEg3=qy#-KkKS&yE&l4S%<_5W^tKhp zypw0%13ZpMo|i8ux!oGTaxB$_kedVCNgstuxYgB@Ic=yxM(x1X#;!EMhe^06v+V}c z9)v)8(f;$tcJc+qPH=rjhY?;BH;>LtNOS}8x`);giedbnEQDddGpFC?m(b}r&l9Dty38vp2Q>jJ0 z^!69nYr3bg6&36Sj`W7kU)`+Q^bO(lF}RG(N-}ew3rT+p!i!@tTXsyQh!jU*o zskV5R5^wJsAukR_UEyOx1sFsvyCSynan-Ayl2CijSj@wZQ-99%Bjx%bDRN=H8#yTw zC=V1@FBsDzdf^|+MK{`J1Q4^wVAL_>Tcua=J}u1$QO%L5BkA#On&3CW{e4x#QW>y+ zR=BL)cSBOv;&iUmr8{x91*Ko-uqZWjGo3Bfajkvl9N)OYJA5^Z_ioez-9xc3Ml!fq zSFt7Tyu|!NMhI@D*AjEi#hvA)zd!!5=(2Ze9cw6Fr?YEn(Fi(CD zBBHk(1LlUOUQP25eYA9GEki1R>sqqKkYqTDJOE=@FG@of4EKofZ)|Mn)55-PaWNPR zc4V+kcikm>ppVh^V{p8k1zl;G$Tk@36XRa5gJ}5_`5>lE(3}9mOsv1!YN4gJe+6Nz zJ<%cv@nZqnJ=$HtTo8rPkJr}8)q6()FnGEoYX<^|PZoUL4(5HnSDwE!y-2g{HBxGz z?;PMtKDvCI0%#0gcY2}$A6ULQhcqF>_uU3Q=ENZ#FnlWoUG9~FVB0CG7ku074M*+R zv^gxXLWTBhx%uD+;H!C}v0^VhL0r2O#^S-Lxfy)SJ&6eAs`6aO(_>ZN?fw{*I1*SJ zCupg7+cV{njrfH%y!I3c<-7-UWjkYG6hFGAnZhBPe8QvA19feu6N0%yxID2gd|!2nh;*hAo1d?0fT zh_<0|%RW>&1cgW`^~Li|4ROTz@jI`p-=o4`-2&>fw@^q+51l?A&#MSL2$zuc2eii@ zhG}_VKI7j*f}TXwk(E3iIW--?yMwfdKHm$zFB~3J9#857>fY<=a2ZP;N{5@i!YCme zUY8&vfa;s&1~64%(=0}tgZFlGQuCdv)Nd6K!tW?CYE?5yZ}1B_yfu-iKz?M@7~Q3L z#xFpf0!5$E#r4ezDVNOYclfDm|6p-?0UCwe28YUCi6y=V<_(bzBTT*R;0}6)qN}z6 zIg~USw}l9eI04sskWRrsO4zLBSd~4q6vgWCE)q`dab3Y|52Uf#RbNxHgTFhF7YIQ% zGDflg+#S=K+~hVKYuPy+xDnqTx+};dq?Z3eaEB0wx{>FHU~DVc(c1={5Eq8XA*+DR zuYK8T`GnEB<|5k}RNWmZ^k;7V5y&9Ah~&E&)06QqkQ!VN4REWHitEueuSMX>h zbgZ1M3K?BT=K$6<4_ zxnYK%mO-0nSi01q?(`vKh;+Z>vHcg|OzhdwZ5T1+H4shAX}kELxp4E1=cUk1{V@pB zlma6T+>5#JHUP8^9R%eI%w=kh@OSyZz`{YxM{ldi0wju8Gu%sPuFZLjfir{i06r^G zEE;u7ki2xNNyb(0_8?qJOSvJSl^CIb{+fwmfEmtG=%J@SZi6ICWBQB0)?@_=br>w= zG=xN~B4YozL6;Z3S^1t%i7EC{IUY_wBs^F!;K`A(ML`gjTLcEG5A~z2jr$JVeKPHNzr{ux2jv{gAu=H)nxp+INQ}=>8b9eQIUf zqX4|!i4-y+C9xLa0r^~3Y&a6jH)Ws~c3-4k>(b=Qya)&P6U$-~+-CM`=5v4W_b@_M zt^^PG>a_EU0xKWPvhX6i<|8g9YW+{*K{*J)36%k6#&HDNb1=~_C{>~dX_y!RTu+vU zHLbcvZYCX>_jR3wGIss(bDJ#ETfQk@QM&4KGxflOt$T(lqqk?F~ zj0dA>9iwBeOwl=Dp{Xc-X)OL=X7dcVg)wro&cP^(NZB~yXr!1wOgA^gxYFGw@81(p-B;d+Osx8yCotpquB7-9vBlG%#x z9xK6(CMHTC?qKIwPa~OYi!`+hp0slETvwV7K-PAz5swkTiV}OF=#i@elNmtU82=bEWh8#I*@NNp7OVL6P}A%Ot%Jw)5^p% zN6r|G{=A{Uk#edG@dOFk_K4>m8=jL?XWLp!2h8aSc2w(d*LZhCyISDw1MM%DKuV?T4p*i(qk?F-Y_jjx+I=8W&80ZuvxKg z!4D_7AfUO_Nd)c5kCPG}*w2M4HV(LVk3!-Z1Mq+UQy!-4d z*{fzzbR&)as+##x+Lub}vg?#{)fF8}k7 z2%`w>3K$JV&NBu{?pb^FD>%hwSS6Md)B!vBOgyR6`NcLKWq#O7efU}IsQNqcrNwL{ z5&Eh1mosDQD}0VR>8pixOP$t1rO}pbK_|0|HdU|T?xXTyYtG7cQze`a#H{D`s={pb zomGtF0}6PkKF1;YyHGD{o$5@1p7=-D5R9EEvXYhtS~tfetTbE?HNG&9a8usRi=tYu zu`3~l%~l@|LFV2DF1Oo|rW2N)dLT3t#J63}X$27ycELzoA$Gue_(Q2qheoJrshXcq zKIsU253Z%P0jY`cp>5dx`0t+MuoAZWNoEv8xIjlA$Rk^#_B6s9b8;A zPx?p@Gb4QniYs)an#f-svk+$ca~xb0bBi2546H_I@;-O6JG#W1!B1>p`AHR}mIg?fV4ss`7bkkvJDyu-32ctG>{5vSf2|vhiMWfTErX{h3 zEg<7Nz^xDi916nI&ftg-m=EgsGf30ghq6^zIqM;O)+m+zOp;JM6eVGjA1g`P+ZSqV@PR@Q7vdpXs8w7oTP2SYK^{rbaPM zgq9VUV2SY>abe6c!ox|CaKACEL))p(WrK5GVe!`-&WLe#StQ{{z78UnT4U-L6@8s= z(f;eEJqUuI7JDIY2#bgN0l1BTbI`z^m^S$|#4F| zlm}=x32!Kd@WgC|1O( z@nLM(JkV+gbuB@{=+SH>R)w(^liLLq2;Uci8@IsIHwRsF9Je#rO$ub=lYtaMYNauh z*P9as<3eRRC29C>^UoDogs?VOq3-GMiIF{_`ubO&!>$EVOpUxrp4Ow5n2G5g(T>I! ze+H!U5GP6#<&C0K0rmYz>)&G;7K7Is))i6;o=kU+@yc*37Kt2<;3y0vsiXbM;=j2V zD!_(djrJ0sw3Ul>w514QJIT&r9X`!jUwLX73e?4ambRAcY~J!+sR6QY{Lv8a%{WU> z0MzCT%ZFw5@1UA;3oI2P7q>8du-rWz=nwcyj5 zYU{L4dha@0%}OH0<_Im$oHsaQaH2iL;f&R$Gkhywc~|QP7Ob=uYD%wp6j1%??{W(y ziH#Vbu3ulKC$KtwfG{|J;V}e$mY1IoB>Y;T4H!>CJbRXALFFD0p;d#nO{D48Yd50F zW#EwfVZpIqtXrMbqH>)20o3*=xmu{t^2bG z{d%truc7(QOIE%Nz8w3I^kNGvuMN{1Yv`vOaWOH|dEuMe)j8Lrs52jPf|(NfjFt(V z>WnH}+-IgxyCd}&y z8jUra*P^agq9DO0;6dNaVEodZ5)k4i3PY2$z)pMS!0bC8D#pMU0xtjq(9+h=T~hth zhwJz7Jb2bAiDtuzDy3q$YvI{|eWdF{73KjG$jF}l`g$uc%BLGMNH$*RL>R~FKIPu1 ze;=9nV|)s<@wss8tPTXHYFUdW?-kr2dMbyeFc$|PV6jAGARO9~z$|p}35MkT0oua4 z``l7UPy+ccU&j1_RByf2BBZlGETKy+|~XXRkw_#Vm>OUh^>EzZpDyJ55Rg zX1NP}YV<0 zbbLOaMR_Lg`Tiklwp`+GcL)oCv9$T}QvfY_MtJ8&7t_Pr@!4IjJ+rd0)=rhsZVB;7 z0AiT&IKSnpe_istx~JVajMsPZ!aH|Cdjl-NG!g{h>gIi)q1cYVNKe!8Ac!q@)Y;!c z2T$n9(!=n;Aq|^^D^y`cO+fB zukQHvQ~>3RVi3PS)w)5LL)pR0$+GK;+193pM9zgj``s(wM=XOrKC zlx zU_8bpA!6+t6_To5bXSNRUDbc@bJ46}{kx=-v_<21>Siv78Yp5YN`dU;65j^Jj)AE9PqIrvcb;*TYo#}iuK!!Faf-DwkN)< z?d@Ko_Y5->OVGl%$WN8u_HO1w2F z_Bc+aC0bNsM;lssOx;E0!Wfjsv~dy&7+?S5AqN&{`=i}=n5&eU2PXyqq9mFprJU>n zex4a_Cw9Bg8M{)9qPa?Nh=KkNJUcif1$w`Wu|_YJvkm7ee6Z-gkrQ{l&f;=9@y=!0 zNMKYPdro(tK=u}f+EGpy-1)`7FXnCfz{3zGxYdcM3)_Y7a!!CHB4ji+3wQw(MK0sE z7m?7c8oy}3j}R{|7g_db$if|roB@im_+l_QMjGy+OMen*KVq#Z2+D*q23wZ_71DK< zL(3uigSCddzF&v89H5{DI3dUO!xvZ)#s}maVM1r5H!Y*PVYJab-pj)aM%GirCT#N; zky`Xy*QPA_eyKwl6Q`#r&0ipWc*amn+qAs)SrJ7iH#r}`e2Z@0U;QLD^Ym^cIplbt zwDD&PgQ49SyD0VAr$OlV_U~jeN&Y-;%XlS2cN@n_(P8?oTPyYT%)qA5F#ZL3$l`3v zR|TdjI8EeRRc<3PcvPeo(Ma=T|1PI0Ao(k%Hd`PGU916Q)Tnba?6$cZA^Hlcby^0N zdqT@e74QZ9ihHla5`JXsYp5NtZx71C2Iy?%ua66Q!Nk4A5dY&>+Sw(rWbHv!q#ybN z&Ji)ioDhMqEp7uP$^r=a%|JWhj>IzCtcmJlB@6BP9G9d#7NM!R=&vGZl@*$09{s^s zb%NkDI1aOssvV_CO3R`Twf)kEl;$BA_d=%>OqgJ(Gt@H9R80!FeKF)%dIE*!Cq7JK zqi5ciB_(mPgq`|9#1%ey`O1tB-zbxA1oZm$%7ki~Pw4n*rX|mkd2o|On0tgKvKTmM zzQ7*H1_L<;8SeWRH_QP%y8zWppw3SY`;`!xbn^tjCfC&gotL{ah3}MSuyuc`s*6IT z5{AG)_9;do6)4rndvX|;vH9mOJ+3)|Io+dU;b{;W6LRn3!vZ7hK|AM!o#qXMCiKj- zBDsdICM!P3avK4{3JYa_u@ec4*notb&-Sl*(}i8p~)Ez7=_wMof^uxaxSciQlF9oL)>t#%l&_fQxr;Y$7@c%Jn!9;!BN}FyQV+ zJNsq22>nAYLy_bli`6L^O(*)BEz8!g{xsn00evA?7yr$+x5~?mc#UOL_oQkQ8E4aG&mto5CyN}ukpF8oC~$@^j*`0sy1X-jlw>yq zW`FuVIbeBLfO_;660L9?HY4*eP)X$6#c`) zs8hgTf$YpR8$>~g@@m}fgBk#`4ve*nS)HcKYq5>YXfE8j=t#+d*xKsCF8Hso!q}k? zax_OIbEoK~2Nq9opcm?d5`W~?7SreCKm*AOSP&uZ7Z)u3D%dDkTrx-iag!9n)a}Tt zgt$5ATFIyt4j4%>Q^6!Kt+sTAXMbv8`g3M`oSyZWIV)NbBPA}T65z`g43Meo3IE6m zb|{Q_kfDk}9xz4*H+;HR61`g1m_UMbeK+q+^}F11!clh=GUQ^OpwY0qi4$0|Ds=u7 zcUA(k@CbOX%j}rC2BcF2wZ=wP#2Q5!L{08J5pmU=o^MN=}!beIaLYI#^NNg?mgN z8TEyIP9#5{3sY{N>1$Q87H{%Fn!qK!;IiDFT1hRx;i4$3pz2<>7wR(3`;A`4dmsO{ z&D~0Rd-9V-Z*Ubb@~DnBt@Bd(2!##qn_j+&n264q>3r(W78YB{bN$EdNRYq2apSd+ zsrE+8)~Ux`j3UcU^Y4!g*z+z&IL^B=FF_boM~jx2bjd*`mm%io+zS<63%H>g+L#F) zirt2j88KW zb9oasU?o#&>I)$FD`+RJM3N5jj{Kspi$7D_TMIe&I^R6bVJzw)fD- zfJx<-URE5%f~?Q-_CBvNlg78pR=<4x!j_NwHAapf+x zJj9qcSVS0nWi=#}2^fThTB&NhS%+{73AI226EHoM73yow(Wj953TH(UhY90A5==WW zSSLzC+4b8coT&duFoy*cTlteJN>*CZ{nf86FddPLrhYG%KpV(eb?P({7b*gng{rbn zmfaUq0vj@%y)vS)vz3nl=C~X>vPd{Jx^KG zj7qQrxd^Do;1m-Age|1_eU=!l6(*abU8xnIzLFtt`8YzA828wvXhf=+UY=d`nTFVI z|6$|fZUFC(wr!eGEbEb^pl@P!z*`mtObI4ONAjq}R7X+@>3~_4d4%9Pw+0S;*S&z^ z2u|Ky#*=@pVry?~Nc;4>2FJPE4%ho!1IDHgdW|PmOGi1jbSq71wLO{w6pyEbOzq$r z$%~z~WR>Z@BBG?JdnuQu5w@^=nc;o{6rj}D>n+@2+DoX!YQIjLAKt6X1V*KB zoE=LK`%huEHu0eHd-O3|Bfjfp7_-c7Q<-joiLWn}^B!)1u?cc@HbXgFR*64D+}|nG ze&8S`(Cemu!q6@m`?Irt8-2eH(eg$^bQRZ7b}D(?>76)F_ga6<$&4yP@tNr=p=|7h z#Y2~Bx~fOGMlp|X6Zu#A?OdTJJ1YlP1XbasV2;=p^w`qclkmYl&B5*jR^cqrgHl~l z);{fVuvkrCDqjyz?Bj>sZQ!TIg9rfbQDD5QHsuDOnu0`Q6b--z;ib1F;E~AN`Fy-* z329@+Di9SoFoVZnOD)SkAw|wvU~R7$WG03Qz4Q}h-3cGCu@g~O^p&PD<&C+<%eR3v;wH6-XvXygP zZzd-xuG$5m&K&iJ)p`_wVGh<7KfxUbXM$4rl5geLk3GRM%82;C~qaGUZVS_i*Hpw6OP`!Y3)rlkaC)~aE{%{qt zqH8TtNNJazp!62WuSttZoxMznI4i`3;P5FqS54pEjxLpr$>MaH;tGnIst_|Fu6nl=)V(U^H|@Q$b+*!qpTg)g1<*Ef1b}6VJAt16Z!F04oyOw z_nxjeVbiLv+CY?YmKdQl0H41@(&L{Hs4t>Er$TGrZ_hf`Ej-oQ+loeq8L&$k zrhHm)54L#Y+)E*s;g=G8%d>N&$Ierr`$q0i)v`doZf7b6zot(Pcx4zJ>pm_7)kL*L zLEZs{24~b+r^j(Nt80Y^VwjNw8V>*Pl@f4PTKr)OKaBHTyIHm!N+Osp4<8U?m92=I>Re>52e!J`Dq~!E&bR0v}w78}tzfz}(U`WGQxR zn}i?`x47Sw;-X|yuqvzdz9)keM*S^c{xfzf<{Dw<5%}_r{ghdt9D;~<2lp2VVs|rW z%`*`>JgFlIMJjpDTPEUkff$JTZWLYSC~02gt)!V>kbiXp8>_Z40FLcl$wSH!l>jNQ zvp$ke&b-cUIQ}J3DKwi95SC~3dii>0%br-0PaxiJs2H$dNIr|_XttjyA)_YY z((ZkNti-y2=U^<83jbM)ZazhC+Z~14n)6!u?w_BbKWDL+OAo5*=8E5t6B>BfIkm#9G1Y z)gP3>E55uf>W;1pxd=xEr2s?wRSk>?w{5_VWgOou7x0vO%>s`}qGR=`#oG=XM17Y$T)JbzGMzU(54Qr?Zi!;;xXi0 z&kKAw+wT~gb_^RDGRW6pZX*gLe{UjCq^MZZ&ASDQRT&&mhrY`63fA}PcwzFfYee&q zn*5$qRXrX}k1i$yhGG+{anrd}ie>PwqLgvYv=!vH(}6T3991lNUrb6If=7Uc7$!XX zn%ryFlxdM2z+Ha0>B!>fYWf~1B5l=gkj1&}?^5(UT=U0|bY{;m&nlM5Vc1}rP9>xT zuR>}YEZou+s$lhXfkCj^*Uq7{_y(+t}jpXRdbV1+{)@tyJ7u{gi8XB5T-g6RuH_Gkh zm^thw0^rbt0qQC4tB;bK=<3b@2C-dXQ}D)aU`a6MDLz=OPy%8wj}v{3cjp^>%)vzU z2nvj9|HXKH*n5v2UY?_j@xbjLjlyqxAIS<2lAEnkpOpeE?zIR(#0FX< zI6StRDEG(eZxHO^cJ_xYtZFm%+q6``PKYN8vQwkAx4B(W7inHseFtMdoT*f81NGIVB6E?vyIp$?4GYB@8+Yp`p>bdHKEnj09g>lEapXE3kNU(9`0E4u*1ma1YMTTF zfv%fOI>#5x8gz82_}6f*eNI5KQ9w0%fWh|Q4-3D*5W&!89@KbJa)@NN%ERQP!43z&4?=pXGv zw(=c$X_W#Efhjdw{Oxgl&^9ohR)GJOB%TWp2@+m}P5h=V72l$G*rsu2p|7NgdL7M{ zS!ex1Kt@$9D!Ei>b%eB6kv#MO_+mFiJ(A0z&X}2p#-YT3&<=$50&GS0bmYY_71@zp z{k6OW#@UjP)a(O;k{fIb?KgTnKr_&MT?uc6n>C}9-wLAkhKmE^8qC8w9#-4c63quR zZi;ayiPwF;U8jqKaA(#hN@9kPkPJ&oH{J!wfPA?jh&Q`h=8irr!_$E}*1pBK zTYm9tZXhX}Z&%VbnKkiPS|5W4RW)yVEE)I?;T*$|FAMENnEmyBYTfi>OrI*;Xf6;y z4MwW?6pciTgYEne)nb;;uQLRH^#g6MX07K#^*L#aLk)4+SWdAXqL`ls!>NB@iK2r< z-_%4~?~f44nW zpX%kt%?fVLqxYlY1K|3;^<&cHZomodweb0UJ$X7eB7vr96j^ZBF%6}BTR$|+4;di( zqB@Lf!RL0l%{z6eQNs@P#Yzf36*@)wpZ6w!Le|O9- z>hFgfzg20F^;m*;QULCltd}2l&}wDH}25(<%XyESD^u+7$8haDX>4z6?rWyAjKy|<5S_^>48jL84T3mpV zEW7E~-Cu$@wH_yiFMS1?q5&cFh+67LA#GI@_0Y~~JUUc$l6)YiTVu;|Hn4s+)|oB$ z9(4YbK&Qn1z3HnsD9r#29<}8}KuXUK#iCs0*a?7~yDNiMzdX3#*R}xBSBhuxtnCXh zyNT@k*dUS5pd&?`;Nl;PB*xeiciI=fPWi{Hh}(iE9PP*vlU$)k|D;>eNYzwIB^Tl;fX#37i8`~ zO;r=0oVZ2KuwwCQr3M+bEgVN2a0Q-rFJ^sYuLI88D$jO_wYGUEg0l_9Ph|J+mQ3^P>NlN@1gnXfb!KRbD`i(q)qA7HRxc!W zgM9<>y$C8%-sQwddkjT{GKcF;cov~16RPFfJfFFbr?S7TRKoaxM0#%mDYr-tew{37 zfSaal2q-UzJDGH-82fx?5&N8&qE=cw`8}b5K1r{Pt1m;XUFxGfOMgtv{prQaf3*%* z#ud@%B9ztmQ|?xAbj4>2wdg>_*Hzrboi*UKPeBfC0Dt{QBa(sxxAsH0v|01k7kNDV zYQSW)ulw)S@!ND8pal?HyxYBp1=Riy6b@)^eajJOGxEdEcO)G)=WNvwUz3&s)*{A_ z6+sWE6ei}2aFc&lb6alJf|D&qmd4M_rh%&NOv6JDqUl5ASmgsHX^U?AOb})_`iMw5 z4z?*wv#pUNvEiZI66jC~jhv+fX(QdsC8W~^rR%Hxtd&T{UL@AVq7Av!0K@aY3q#A{ z=+BQ^5p9ayARbQNK7)9Ad6oeybY-bQG+E{Mkj*6QQ7#y?tB7z-Pm%Y=jmqL_D>;W_ zth-)>KTMn4UQ>0mU$#3hk0G^wKx0~I6UcH=0s@UcIs?0L#qMdLSM^8qXi9v?<4)=q&}Lpb%N1b-X19OL1v?&2N1%6Pj!EU0%--MK z72F93eN})`T?4JLj!(k|IAb)J2-ttA38=jmIbOBO_W}SL*w(niJ#%~#-dH+R$a*kY z#NO1!fc7$N`8K+x`D>RBHyO8N04+)As~%hRM4*QzT#`l1*Z#{yY7M;Y0YIq9ox(pH z8aryz*JL_YY1jjVGPfB5RO*gOTQzU+)3a=~C|6USC0k9*y z3&lYGk>`S4OmGH>H6VRAR?X)wlD}3HMzB{WpbqNcYxR5FIFX?ugEHHqf|kKEDPuj| zftIKahJ1F&GF5==OZ?0sm^u%4G%1llMmQ)~psI7siuk!= zTe-@CM(>&PJ_Hwb?q(_KM@4VFHF%5GhoybB({t`i@M+mr-U&|1r5^n;kH9&2RELZUQiivX&YWYNAc zmjHU#r)%L!++?Ixa{a*c*(gwS92WqU&1wTlP2v)WbXN!EYveu-?hahbp-IR!-yv<= zg;jEn6-EUudM_XK1oA=SZvADlfQ+WO6qM7hE=L{~MXOged=G6tiYWcaxlzrdtf zj71pBK31&L*lcRtyP09e)9Rl1K#w->V1UuqRxsJsCz#PTugv*dlv<0T)n<^FFh!rS) zY5qN>^zvM^v^b&n^Ph*x)l4P0V&*fszmM znMzEA-^LfXH-A|W2gm7^cWPSIUG~1KUv#=kSdF!C@vi z_Ws>uvq_aSB8$is_V&L|xj73^+PN+P2CXa(_<8IPOU9eq-32XWI$sHmLVkKj8qLsn zRyuM=eZMC!{7I)TK&u`t^QeJ7Rb@)o%1(g){T4xY z(@lGVoaK27JS3xzv8$)jvS zASIYp!i;6hm!$ctQ~A&t)(JEg`r$SK6HIPgp=!#cB8xvcUUs4e46}L0`ggOUG{d4#hsSKgA4G);r+TLsrP6oN}6fLwF}5KY2H1fEPxKPZurpnnpfk( zl%C-58Yj&la+NLD@Dx@Puy=t)3QFpOzhGO(x*OdO+~?@m6}t6*Tt;u7Y(=)nXdwZv~XSmQ2xM>a^(F(odUu(?}wJfkE=1(*J z2KOz(!BH0)tQIYd=yeth0!_WvPG)UNDx_ZJn`v^VfTCZe2rzpY1v|ydeY^hyI`9gz zr+rpz%?#w))z@d^g}>;E)RTcK+=QL{EQ%QeF!5n42i!^BEXGIK7ei`jR-B!cqU7Mc zM9UiUBNUXr7rd*hb<_Zfj;*3yxyfj%yztSWqK?`xyES0wtDy0n)>iV|vo1z@NeuPx zWl?~VGr;D0wnV$NSdX3<)0Pj^Twd0QpFqk1ir?OYyr&dgoT(tA`Sdiu@WehT^)2=#Xy9n8YSi7}1gIZox7?xF-6L@vG;E(lpu5eo-~Pw#H{fxHLbRU>vV#F9(s^y$9nMD*C} z6#4v`dqj+}UJ2_^)ZnQ779<{*>g30<#)R(yKGo2AU+mn2sPi@}@5;%T6{OtRQXK@r z;7-OYL)_$1N&Ar)PT)s}%l0QnLX2cL2VCe|mqzrmZFxC`vjfXOuejA<{}J!Q9Bnxl zEGFoJ3lvD1_q+DPL0WcMz4s-%KX$tO!K7|9mnEB+=X}h+=bqKCy3nu%g2%l7YAd6a#tlaFqlydsENZDHce5v`8wYDWW7+qr-VS;6R-jK**pyOi`GOBdbRN7V#~o# zFuj!Sn$^fQ;n%8IFk1SZX7FL)ZPG!u6-DFw3cHuf(^*&={SQD4Mfv0`nB-taAI-w( zz)VW9N$gtdVw|Md^kSA1F%Z~()ym+tjl_3{ zpN3!%AQQNYhPnKT@^Z1yT9g($4iu9<;NPbFxbkroFerD z$))1ubJ61^*>{|^q(Ak9^wBlLA})jO8jmyZ5-T4XfERD--3nD`Wg(Jx8YL(75h;}y zr|x1l2vDXl9Vr=%EO%eL0TsFG-TOJPUtXnvVQ6H3S=nNay_mwrLc;L9!cnPMagYy_ z{y(U_6pWp9G8~T7xsIa+uicf^ZJ_~gr<^(zA{&0sQ(^MH*5d5(1oqlryR@Mh0^Avy zZCr6>VA{ifn}%9WAeqX-YwG3vXvqxNj#tac*5Rmg#SFy zg+@P;-vU;qRx>ZQhU|y;6Qx{kammbK?BC_X7e1=PgH)QKHUouIRlk3|$?_b>J4}3| zw+HtuIOo5M3MUk;+Szl7%joX&M8GS086==$u3Px+K&hIj%vL5IfQ$FL6U0sUauR@Q zZ{Kbz%C8CEi3cCmZ4>7){oX;x! zoRq}T2C(p_ja^C|;6H?j_l9~j51n`|Ms{#4>;ln9eZfk&6J7l`HN~O?%jugH!g_ZiAf1Ex+L%0WE* zMHL7}wpwt4aGqRzF?S-NI8@IuB=0oSikE+zoq{i5AU`Rx68DGak(peeE{X|jencRT zG+8c*sL}+nLX@hOf0h_fCeZD5)l|&*x6+HW-`Yxg=3j}3%`KVjEE3DcBK2YDRWKv) zoI_M8*Lf|rGezh8+0HbwDW-6(%wVX2^a<{!g!Tu1M zQ7pSOaAB}CY2o{%iy}tujLiZ<+zc>BXU;ON)NaI_5q!A+%hk(gHnHN1qbjg(5r)SL z*BF5e<=xx4bJ14^UVeniW4X~J0Jj0Fn%&qsCXy*6o)=zg`=`B&od$n`aNW2TF+8i#J>+xn-=y) zm5eg=#Q@rTuAaLS!bY=^<+rDS3<(+#3#7#It6pO-z6ciAS8HH^L zT8nlm0oI+uG*1QT-}|YLrlGrsFVy@hjMcru_tAXzG}!zb0)vWl7<;*k(bM=+Kv$zi z;uKNNBK-h_{9$CZ4RtOLDogMotZ|>UJCt|m47pVlH?$}K(m!3+Iq>QAO#Ah`;4NHw z%23;5=t{)s9ea}Okqrb-@$S*sKI9m+2ZHi za;`CFFO{@*KM8ab5}RR71F9PP%ZxHONhnW!oz2t&pM;~j>SF-ZY;bruD)dXDl~@G` z;RfU|Ai20f6%XfpMQ;>h4ozLRC}q?~5S&f)gj3BOS@|HAIUJ?63IoNV{tE#;t>!=q z3&Zip4^Gi zUeKALq<#c7AD&yg^FFy$*E|blw1eEMio-5VVVOw(>l0fqxFvg*6O_`fcE5zU;_cRu zavY}2Wyp(Q>lA(m>%8gD`$(b&M4eN`3dXyb`s(M2=!0IqJ>75AUO*xk`;iMhmiAm( zm=l?8Pw*s-{|`I0Ny#qf5^J=0R!wCG7B6C8-KNTG-vsCd07XO&77G}v0?ZHKRS7^q zzE5^CAh~`9L@l!Cb%%3aanLFzf7LL~QmP=;5Z*#1Vhki+jyfGz6|CK^M2?%*6uW9m zw=U$6?(-?1qdA7)C?|igk>EGpKv1l<_)tda4hf4XKyaQ!mC=T|W`nBNeao!yt0mo& zCjfxI1D${LDm7G{&@yW`O@RMN4ep0PqK~zoY!oKiH?*TNttnpu23|I zG~jN$16cVDQ>5@78XQ=7BCC;-Md4j&#ulrhNq|MZsz5O}W=E2*aVXE0=)R&HeF$=M zV|vXn*Uu(&-|AjTu2`n;xjO)L>sjcmj<8k}r}ls%aV-lCNRY79gz0VCUD zVlyYIUPBwPZ-ZlPRDG2>9M+}0>>dtY`nOsytM0|>wXCh(2^2%3IBz6)R&-dF_+~e9 zlRi^_`F!ZZs{V1HJ1?O%jAr+v+KOGBn4OitJf775VG5^z5@;#k5<9Kr=Hy;AR=6Q8 z(=rJHh_wMEs-sOpb5RRJVELKa&TxtQB|*8+;flMkcAS24adb+DA3@o~02A&G(Ha>p zfcJC%MddRF0B=%GpGH8SABCaisYtiHWnw|>-|U*PemG*NEc~emR(5B$^j~N1(JB_r`jPGNR@kec)PUG^&Nq5Qu|JIE*Ct`ZoFK~M-w9JVCS{89LDorBTy=mhPauH)#q{yU;@}N(07Qk zo}$h?WWMMNo5{UEVGDNoVPh4$euZQ7fHx(WUlQKirP2pTUw&qk2)28jT#&|&9gfIJ z7<=gHp0hz3shqKebOx@_nU!9m38E5ouMlvnK7;-d)ZT33EgO9N&~H|I$IVGd)iQ-g zbgA=FRlzfxNL8fz5Kv-p%p-~2%5R-x8B8@Rg!No%83F=D zG;l(0u9i9rdr{5Ei~r~qW?^2GCI&D@n>JvUz5nSUhxz;YiVi}OhC5(UYyf~Fl9ys2 zrr)oTbh{U1X$m2qZv1;b1~_u#QH=&9qdF--Nj|CqjVWq16WIHyNyb zm2b?p5O&kMN!zjOfj;DzuaItO+B z{nZdSKt0&@Y%D@>9+zT1o2AfjrM^N+nPn&h$dBZaN&8O_JOG3N0002Xwr$(CZQHhO z+qP}nwr$(4e*;}2cjZ%oMU!`m@F$=$H`Gk;_u#K$;M%u zopQ%6f=QeJuONbzYQ=S7r!K$s1=QeqE#VO(2-*x6eIPHN*Vg8dq=o2#=NM-Zs6vu6 zy6*ogF!NmaP@DRLOZvjYEjfU>98*>iDPaIS*D_=rgkJq{HGM8`g+sqG0WmAZSX*TF zqdcvQJlf1x`Hp#{?VTxnIiue(X(8cOHAQ;kqjVk-GZD9?P}0C{06;*$zvT3gO2J}x zRI+1_+eIg`TGLMC^hC{6XZo;r%jQNDou@UoJkh2(qU#n`J#78de}^J=$6xb)WEpo* zSED*h6~aPY zF-M46GE8F9$^8g9Ff&+G#F|aP5<4t4_+9N$5OamsnfcATM;fiMseIT3*=K=Xia0+~ zr}*ATSK?BTnm&ZSV5#aH({@>&+gUiNc_IXynjefpiKbflbX^!+6O_S;Kw1km-<3WR zED84i0&CLC)@T9vk~~e3wM{dja9;*PRwT=BC*%*YIgyH>%o;U#$}0`GGh|_u<@l7V zXyriB_urAOqCFnfyFHeNN99$Ryw24;?Xy28b)F%KUYkyb&aNtV>z+8%`wBf0C3(Uj zKw2jRUV)6+B)j0*;3$nu!kVuStwthppI~ zBl3XTi+A4QBs`;=o1u5;;o<66#4Zppfm{L1{i56+LE3Z;f)OB6f_AW%#>1!yL!pL+x(1UK;UzVjC>_O-6nI zpy3BSuxZZN{AE(iz;c93I^LvXBC?jq!VSoh4kI{*apB5Nm)snXhDrPAqwya_{;|t} zYLAYA!XaZt@5^@&o+$+0Sy}5;l}$rA7S4My# z+ay?@A=Bi3_Q=qqQ5J89kL^lay4iU7I*5neSCUtw%B!wNp! zwq_P6zYHcK-3Z)#Glp33(?v7*vol>t7Mb5^y>>gJmUKuYbtL{TmKXa{{Bcza|8i z=MQ^TLBfZTkv|2)S^==*OCnK;p2|)=u=cxMbs1xUo}Lv&TXjd zrbpK=>Ffh7lq|H?;}0pxoImF)`j~fj61F`^eS)=`1~y1Om@|?Xo(3P#kL5zIKer&! zD#>OmB6S5nY*;u;x0PJEZ~-bwkgH%CD*KdXeCimqJ0Jz^3sdI{)|nXTzqvU*bgPnq zBhhaBx0-^fM%WgA-%|1NSqwsBYGaDQ57)u{BlRo~__-&)J$k^6@Wi4ZNlWf-K~3fd ziPieCPLOTg2SnvMv-z<;x8(Rud=)VdIHOPyw+lo~q|Sa*ip$oj|CMhZbz@y58BG&H zcz|-v{KXc}7*oyDiM1q-T}y`1<2f~@!0C0Vy`i%CFzN>aet5$u>M}|~o`H8fQsxV5 zD^KB3h-njRZR-He{b-)Xb^k+iDx#F%lm6_@-a>ukm{y3h{ur+lsg)5)Lgx^iLJK+=1hV1 zLZ2ch`b1i_j~14pZJ(Zm5lGOrU-U-JYS9Hf#cPgOUhZtrEzfTrMWI;=OHg9pIS}M^ zixp$mb-8Mjl|FaF!KL#K?q}na%QL^4_gZ-hGI_Nn)=Cy|jy4E7%x~}4Gsa#(D@kwN z3Vqh8m36j_N}D*sMz@1uSFYT2TJ@EteuV+5i{$ZHzOW1_TsA))dB2=w8k$O;qN4)W zIHM?t33UKNUg(B;f;n9=)XOjDU7+oLws6n$p^}XJAFQ1$OXO;aGjg&rV+`a=AR(~DO<3) zpYW~VB#sbD>4sF#pv<6d`S=bsAG~eUB6?1hcB+Fp^ z02sk8RA$7;c>_$*zXt+@D6(3(ldAh|^jy1*(}`fVC%lq?;k3LKr3L$S>D?O$HNtY$ zpOyq%%7OVwkM^UFc`Q#Lt^>bb?PImXiFP1<;1O3|vMaf=En+g0KQ>mtzDDk4#Pmz_ zH^eSfJvW>`dP~*8{tx6Cd{QTjTp)53w?^nZ1Ob)UFaA zdQ01h7S^YHFJj8WhnCJThbFKCj@3dkn1iy5@+q-5TNnz`&q(=!=Ud0&oeeAfdcW{7 zk_t>&-2p_)RXFKs(@Ek~Nw<)C1AjLkA4*0*nu4epis4+(ouVjRc;Lz@ULD=vr|Ow> zN_cuRgJWYjJ?!2~9(!h|1T3XhNyf0|X-R;%SjDA%%JzB-b=-}2*4KHa8aM_PP(D22 z`T297^7MOMiy)nD_}#S9C>XfZtE`=trlJsWhOa@A!<%A)y%?0(GQM-_05`QhVP+=c zo9q+QvEhBWp#WiY&sZq1j?LWvS~n%rkP)s!tlOwhm{t0rqInwr$%SgGmn+f9F+Z#h zK03$?OCAgSsCAi>%5SQ{(}~Uq&8C&zU59DJGpQ3 zOPXbN93Vm-Nel)e&P{C|uT}m=M3OOO9obOZj8E3%yK&k3%q~@L{|5k7(aqQd8LDe{ zSa%7ZvWNC89=!gjJ7`IMZ74sa!HEa_W%E^4LZ*G2$E|zolt!McJn5;^|0Y9b zFpO8TlW|mK?a^oJR=G%8D8?lGt$7DcHIh@g`5&=`9L`#0hKCw$loF=1HfCYYT5=et z1NHu;oQD9wseJ{sUU$T&W2xiR4j*>qKG+ir

&8QAbiW>3=!CvRCoXmPHMM0M0*hw^Y`}}#uuBKcWIaHt&vXX1G+~I_XRvbz z*9clDmPp`q^h!fkIqh+$=G}LV8$$@5?T{m-tUbiPySzrwi*!#n4 zq@|FomR0u56mLkqnruHdmn1Zko;DJeR+LSk6EQw8fbEzf`6)B_S&y-vfGn zv_nQ#Y){wF?W_Ty=|5X76p4i7hP1(tF`RkCKMEK#E2Pl~q?^2c^Gc&rCdd&HQQq>K z19+XQJM}ky8G`?^fA@g(IHH?dpdG|BX+{k}5VFIy*d3)-svFTt9WXe#WfDgd@!AD+ zc6h@@ zoj=_d;>Wm>mYwqf42+xYaj=ubqkSGq0I&o!L%mm<(=M;{<)JTWup)8YyS)#pZlsf{ zK|*=HB_*sN=k=^-UNgg$JWfT~k=Ey0jqcLB@Xv)$tNjsD9fPyD0U;YKILNf@u$eyI zO;8-c=I5AHWU*;Yn0LFvgWAG*uupEC6P_Qa(QgiUFG#UOWTs`yreSaS(_@GX_aIHR z-1t4!t7TUOZSYY|5kcUdR5(3r9(0^AgX+eKaMNjSuJzR12T`EMr~4*4VHBj3 zni*TZGM?PXzPn_l zb@x|1C7G^}{rr1=#(fxyGJd;^Z1Cr*^uOR5#r)>4zX0yAhOls`}#$d_J!p6r-==C8bZ8k7P$uxXv}o* zVuo7G=r6964C&>NMf_aB9&D+mC1kuCzpj&^`MzG z#>Ws1#B;?dA9(go9+1jcM|`;1szW72hMt;q%EXV?j6sm9qDZ44Bn5J)gO#};g|?X* zlt(#ljH|nWo`vCCF&&VK=a>fE**0HtrXH<-3h}m+AQUyD`RUmu5Y?CM)jk9V&u$y8 zEHK!?Xq(_>Q~d==JVeEizjn2uDp#l+g|lcjN)wq+f{Lh6G=(2CIN-5htfz&l2F5$H zX@c{5{6VnV+Wz#8z$AGP-_E&2$ypV0RhrW`i*RrErN25xK?QNY#lxU;f2?>dmCNpG02J!XK?FFk#-pKS?lPcWKhP`zwx--%}ei-IXxG<~}a><+r}c zP+ZDAv*E%^1iTpj#X`dZQdbCRMMCcL0Zo7 zuj`epF%6H6F}aji4zenqh&>A6^BNoY11Q+k?18`&Hys3l@Pq5nnD;|GFE_*2-MZ4^VkinkB%D@l)d(|SiI58IHS1`yCZ`+JMGpzSq$>T9P&lDX zql(osTWwcsPBtaZ=yv)Lxr-B7;p*Y5@&pX;44s*CGb?8 zI5hDPRW#-UuD<))4eruN-kI3x+R>%NY*?7vPShog;$(D1ViN33@5E;_O{Y2m5Ceb< zZd7f#lNI80*C|N%FZR|&#hM7Cb9@e>Kked_J3oQ9h#BIOYm64jnxl-__94tYhMY~A_U=7@@mmmY))bsSfxlLFew zTzVxNU976)JGEj19oytX5C4*aXw2d<$O1?dDC+1IvjgDwZZo?9OD}7iG1FjP@ga?K zmz_y#TGneCA%?X15AK>%xPzKiQ_Ey&TC)B2$ugg#PHldX7lrcI&(**2c_JBm0B397V9Gz&AjYw5w#mgG_36s@}HG2dRHV5 zLTICu&_v0zd6#+u<=S9lj;K>OHC(#K*`Z)HCa#vGYDTN-9RimQwX6uVI|RN6OK&lB z>ZK=~-o|R3VxOuLE7xm^%J_9tqMQplqH4iN;8Ye1^D93}YTvKZ5#V6f3V9Pd&9`wR zfT%U1RrneA;2Q?34^jOdT4Yn*eN^9)_PN;IYC!8Z6v!Rp#-jP?rMkuL&;D^qnjz_h z$IdAuH8)jah<3joZi{K3jwSy)=|_tt6$@m_`^pLw#v3g6HI%m-4wuc=%g2Afp?aKDZW%a)$y>-vd@%KX>zB(iNRgUe5 zyT)b)FmzH#zF*0WYGIjmWN>oesfEl6=JRl*piz<-HUjxbYsi0pWgsk`v822iOi^JSTI)LPU)^FM*dFzA1?F3i z?=v^4``ypn4m5v^^gOg(0h(|I6TVvz<7{^U&(3m~f1{v-QhU=_56tq!FPx&hys0F> z$$+jzoQk{rL z8?q7i0`eX2;j!Q?7>BYCV}l(IlDnnLhiGuS&{rivO8Dh5gm=tL$Lh`(Zj>#@Sf+x9 zI`gu~t!1X9w{D+AUDvoQb2^)8-Z0M5r% zV+3G#bZq#P#%w&svKT4mV|tYUg2;J88J)&wG)YC#nFIR{_^|D4vI@ejt|Ez2gTde2 zPvo6T$EMe4ZpO!!ao~mlCEHIHG~^5b(6(q)Z~8q8;NRIWPynQi_{7%tiLy@~$;EJF z5)j$pTb4OPoa|dQ<7-s-qA2g&w*D)URPEe17B_q<9cDeqcW6jd=1->XE+-Ppwh&-|xg{B^+Ks(3> zx>QPMeOn2S$7OG{O{;MGrExWAC5?dyYLlzEO`T0uL0&^CGqA`diF*C`%i8{M5U)^H zuPy4xDhlfq_B?5b;sU`69O78-sKx#qR7$q<^A9CubTiUxH@t60E9|Vu8JE&lL`Gjf zaM5b{y)htoQ<5F2VP1^w2dW?P%nQ_Y^sr_vdoe~NyLNHE%8F`sjMKKZ1wy4#*hv<; zBxD7i?~tko&N+qtke7pCr<&&#!B=PIS~_1(!4MuW&tVn+r5kSEWdO3#=U+5v_mD{=fPca{*G`2tTdR zCO~{*vx9E)uxA0=q(W&zX|+F#g{5KygxG#$rNa=+ZpBQlvkmZD;$YY-_UDwPJYxB) zVa;-QJF1k6!e|eZd1Bj)vWL05c2Tse)1akOH_MxuD($~D4sRm7&rkczRX_@9@$5@w zV!K#B{TOQje-m>g(8El!4QHddp8pyjhHRwfItD0tXD2vnjJTqBb4w8Sef-S6^^CE3 zBW58HPk~7O_1{;yOrFAiXi?F+Jf;G(2(H#518Qfe{8mVdGdVZcDjul@l8^m2+JLriDk!`_q zw=IuFJB(@B#k3G8T*6mR3uz2jS7H}3MHc#lVi0T?Bv1 ztGcAf%=VHY{-uSmrQa3Lhf3ueN$!!5^3%*)?wPU;NA3%{*%6?Pjoy_#D9*W zkORjP7gM-(H9&>@oSOPc{1iYA{TM)L!phge|3!#oYLCD9*~uZr1#|S zJqE{)HR1D84)_&M>%vXaO?le@<|Sb3ER$L@d8k7*B_{aR=rra;;LHqnMw%1CegRKH zi9|P01d)9N)=b#(!EL^pH?m?|PUj3GL|w1s7Cb`Om?d*4Y-=RY1rm&xA1*7^pH+`Z1&Uun z6xp1LMdhgqQ^wglhzwiL1zEt)^v$DJC@}9dEsyCiuhEg5}>)Lax&&Om{H3sf}r! z1*KX%FskVoi;BRVSj2k!6+-rw{y2>asBRn%cN_Fvu}I$yw}41g@ee)e&?Dhv=~7>$ z{a;T!@mhR`umGrGULfJiZhi=*wf&~;fC&nK)V$7%TXPW<6461)V;wM zsZ>ELn0|%1W}2XES$6Vvko3ZYSXD) zK`g2TPYwE<#xHwRE+-~%xx|P3&lZHNn-uPzt$$OKZiby>$lOpVk}NWnuNeM5e{U-F zuBty=a`KzB42~XQ&Gn63Mlh?k?YI$({m?-!taz_5tTHe`ZFFrK;2kZk;qek;C*&8V z0DT15ie^>!mb&u=kIpwao}xs0?jr&@RuVzD*Bzm3*PeE6SFjBLvc>IT#Z*iIl3{yR za!mV`M87+osK|R`5U2)CAObJzW$M|#Tka!a0n%JqK$l>w{e+nayE?pz<#>dqV7#pG zFN#!Y2N}NxiR1|>)Y$Z;n-#uvLb;J63KyYnD=n=&GdOd(cbT1UBh(BQpT}LPzf1=B zQYKk2cQ{yh$;uMuf{VkWd~j~UF1cC&YcjXBdMFiVuOA5|Qz9Y~dryV94#H;a)4C1T z2guR44h`MND8>)*aNK>_&R!JA?r^^iK)NEi7HpDqQ@jyx7!%KQ4+n-VM@L4cRK&jD z9HNRC9!y;ng!y|xjDK&n2GGXM@*9V(=<9HW5qMPEdg{Oh8=Q)Y>RBvKFgF+4FpL{^ zo|&JCDDd8{gY*Wut9eLLM2VE|`wlM@LPWN#e-OwUW;EC1Rn8(5d{V09`N*NEk765L zIL%*25jdiK7L$u?+k+LAJUpLexNxN~e=}BlQ%kP~?V!ci{A9$Cd!_>1Mab}L#5QO> z38W)2i}>4^&ux4WZZ$+j>@m*`{CL0N4$^gCoG;jUO`UP8V0kjB2|r-WxcdP(mTqRSMozPTvn+A+PAOs!uB?K{|5OupcPhp8R8 z4WhjRF+{0^{p_SCT60u%VY_jLD-Bvzs4~89$BqKL_r{;xyS&=TJ>H(%bu=L@bjth0 zmG#EzSIEAF-LX6%A?ebBc7 z;brpFv+|tL2@nM3~(c4v$yl@qGw|02#@UWB_M3Di&6k2u_Ku|R!hePv>Nu)nIYxsl9s)6mirb# z=n$8X(zyE2aL+VTAZ1eX3>$~wrrw4kG`_K6IhS}pEp<(3CXr6IBso7(GSStb(uYZb z%Bnli2L-`nhAiC$n8!xFY@)Gx|1#aKPua>~iibLX+=W#~R-u3MOC@Tu5eXJ?d0zlV zuqW-L+QCo3@+M$l{yOX39h_3HIw=Au{-EFN!yBNbPW61magyPA2 z?;&uy>Oo9q*%W%I@qc)b1pb$tH)&uNdX4ghFwDx!@a#6Rl zF>_3#?guCwbTSXSF*dDuUad`_nQC+wKL6pCt@xrVs-k0wR;l@(R!Up@KILC!) zqwM~s7AS&!-8FWv*^UaDq;Uk?w__>i#dYQ<1Z)~UKv&-P+u zC*U3`E`PV&c)BfKF6&q@Vu>Q}%@=r?2BMsOPHHMV@!Di?_ju7+)Pl&<1AT?3cXF{L z7YOVwM8gbfCYIGJX`;eSGLcP@!1m|VD+@LIJlizJ4KJfX*gL4P1Y&=HJ%#FDtMw~j z)LF6;koZ;lt*^;YN9^GmP z=pF-bbu`x>kuA#QlPcbTB>6Rzfm(a^h7t$#$cQn1M+!8^LhtQ#))K~g>9H!o$N8v( zq%GVG`6#0b$Y`m*nE~8K3V<_Q2rAk$j|Mx0P8~pbkH&^sfZ8`WnqO9f{t?ID?o9g> z2HQ=uOalOCJN?O_MGweXMBC36803|`n?wi-q2i1w{FHSbm?quE>Fd|abaQ-<2^rzl zHiw(fYn;OLSY38%QbX2k>gA*b_ zSDa)H$XCG}AG3(|aS<({O^F4ENFs;g3TBWarU)}G_^56DkDzp8n~I2VHF0FKw|Y>n z2@9gkxC5mcouSZ|Jn|)M^qCncuf3S1V_tzV5x&(pJj}2u?pBadC8C(An0gE7a9REQ z;lX<>f2Rh8m*v?8Yrg!ATeyfSOuZ{SK4)j;iwc(RC>i@;e`>2_J}*W*IovG17lWg? zf=e2AxqkAmUIy9#1Jl`phWYeBOyk_pK!j3Eq^k}dFvE?Wi{hQ*VViw2<(_fY)3#$6ZZ6NOMNxby>l9qPhvfY)~eM#MYecVY&8@6mO(G38?$m!-Q z5Z}&njq}lq?mL0PiALAfhE2-6d24u|)$}$#`sWWaDfO62%KFpGxNez)34ey#F@;k9 zKZC>_gknmP4;R_^*YC(x?hMyDjcmCz8cDi_Tm^FH<5Y&aXQriVS<+IBjwx2j{N>kA z`Ejd`csV8eKK_?XN6c3mv%N-pBmvv&RYV%&w0;SA10zIM60~c9iRMi){(i!{RL9%p znAOyU80cU9g&{DJUTW!pLf~Lpi93G)oe5?y}T47`@3JJ`WH&e(yDvH*lNfkfBn6MNpc7RIfD zVrn@g<=c@My+a7oE&GqRtG=Vm)1l@G;75@3OYcile9diju=?49zS2DAd>svfi zqrT9{>F?84@^thK?+{T5mNk!@engp#f&0-%qAJ$=f(jed741?$TySaaBOfq=(p>y9 zdUgZ%(}~k|9qGcjjeGE2#P_$&uY#WGB(2QG4uT;t( zP&38VSSp@3O8_<>&b`O1U;O-TE=AcsGK&F&HK%Q=YA$Db-CuHrnZRH`CIL1sm`de` z9to!+cwn%e7KBhUP96%GLyYpWMkPF;`SLBh5WAX&QKWwI2!dH@Z)P_}7e$@FUC}H4 z{*F$*mNKpPlP0Du*YTvoEOwc@pGHYi_!%g=sc1Xo>m8+s?5y5)-*aY|xHF)6Sb50q zRJYga??MXpRU6+`gnN8b(Dw3@ugMMURum}lFOfe&x-#qroebDYi(U^n5J2rqi{wt6~TKBKL%8J4g#gQSH(o&rM5iyL7o3tOr zbccYK0ogCEczzvc_7HQ$H+JdodGn)<*6-2Tzid1A4cu_Pw5oewj)p-OMpHHx7}@i` ziajYr6(!?~3tc}{;jiqtHfGNL@ti45ihm?4T?5b|ud%wC2rC(?t9o+WO=J*_QvF$H zE$BUOdia&vS7f`^{g&>dY;^XY^HamsvJ&qE*GajW(Du>g76^2Zsj{3f=CTei3}2}OI-fVNV9sN*=uw)9o_!P$bvO56j;N>hY8y}v-nP@} zxd*ujEy8~&3xPGF`1DO2W`47;<0~}UVv^+MTaFy_%WZ_i7|d*)YDL@@NwU)?j{aS zlv7Isd&Y|YUSpOK*#-e4hALgc>Sb7ytRBv(N6Sg;vL&h%ad)Bt-`Aeqis?|GTaKSL zXGdV(_G9Y8zi%fUzncTgo3)H!xsb;hq&7=10=5Us86`|nF5DeFIBIrSvwD!AZ&uas zmoaN*f{c5zT)c3cGvXP}UOMyE0Y3JHc%vaJdqX`Vu95K@j><>uo$#l1{dF9W)Z(n3`x^pPKk%R`I;~TkGZb$=j*LN z=muHEtefaz(Mz7F4nxS$EPtQ~>(ZgOS_pf5(}{JO5m8Rof-KV4$jxV3mlMTY3!b1u zXuN`)bQXh}Bfrg+vKnEcqO>K*d_U_px!?orLrlq#)%|6R>)WssXJ}5I*OmsZ8SWU# zAwCK*ln2I(U=C*%tB=YShHt<ww7JF!L^$j@>94S6WHs2;AZ`w1eCSdVd zH||EE0Py90g-utT!5Rp4^P;)gCjG-lCVHxf@h29q;(tnrXKICX!>=Av*!=4KZ8+2;UJ&;p~`0jHm`^nF-hI1|#)TBKgrCP!SQ_9aCezb)M#ySr`<5 z{b-H+Eg6?3LlL@(4)+j)K9o_M4~M3pzOyq!y;1R|9YOL(v#9_5(5$Jz@<|7x-FraZ zWdEF*ukUx26}73Pt8xXZN?l!pzNGu!Yop#~X5Z4jO9W??q)ImgbCs2=#Lh){Lv!;! zB{TtNq)uUq?_cN0J;Yu4XG8Zvv8cBjB^#~e7H=CRZe^NsgDrB6k_$ptdM7tbA?>g= zCzf4+bZ98+bnxXPrc*2)K#j(ehfqyG{jT&*e=`@T(^t3zQp zFU#tKdrC2VlhNOw4|&UWRDW&B;~BZ1tQDk3ihCJ$D+m})*`AU!Pb-JKCf@JEShBS` z$NlUkKs!|~_T?1Pfwg<1KqT&!@I(&+*fM5sx5!sU(5dZI-2@T9c7u(frA~;s_K-~0 zl@TD(B9tffi_V{*d!?WWdk;oo9xZ#i^#R16%}U{Bz0FfEv)D&xIE$;|*_#*XFNcHt z$(4C85rZ}L`)BVez-~-AV;#q(59vl2HYtCN&QpLbkj>T)@2{43cPn_`h{vh+8IoMy zzs47f(c@nA4k>@SZ!~tr1J?Ym2EA1tg{v;bo4yT=>=oIkBHq4Qh@9nXOvX%egq@yGiLPmwHu=<$FB-@q97yQUAK8()-B z&h3|!0`^3V+1FKEDTk!CV5E61tnjx~XrHIG1p0r`4ig?h@@Q*jIh#UO!~{?OaYbninqDGfu z2y7zj^R^VxdcQ)?_j@*9?{kLTOvmj?q{yJKGLWxY!wXuW#stAgax7^Vsr=fIfY=9u zg+Xp5HsP7Nw36-bNY{`^pp}>r%Q6smFCJFMpC%zNIejxcpD6QLtkyd0vZ;e1`bTga zj1f2$l;hr77YzfU^`XXJ++m?Ybg>+1Pw$+}^|r&lsz)PAAm9mA*i?~U0@T{~6*zOH zQa{B4`&Yn`5<-u^y;S;WK_h=o7t~upbCQVK^k9z@|A5$*NB1|$04`R1Df~t14l}k(^&iRnUT6%ytN92nYW8WWsNVzdUWgf zY*hb|0zTV-^xKQM0`=jpV&XC|oQSK<05E?UO>Z`qOPsuL!Y>Q8R&MFk7J+Q{ce=I`WSIWJNUTs!17o3^`~&LITwa+x=N<39d=F~L;oJ3uYGG3e06z$`E0 zhSQI;4nRZyvO-o!7sf0}^Xw;^j>16Ze}8{rjFqhDT(VR8UawJk6`^1!;9G;j%$@ee z_*hRL)eL4k4jt%51JYs^?mjYD_&$s0oU5h>t5=&<>GfkYDA&W$H&;2{Yp08)!f-}e zshi|cU-(>Z%KMglgC!?@{n!9>kS;NBp^3$ZR`eS}?A*bI5)U0!xQ37btf~G67FnWF zQFUcdReMa4;lr1{l@SX244o-FR($pmzWn)g(*iO}JQxPL2REJ{MMM!+-KvX+b)$i? zB~A4YLC#QTpm}2zUkk)bo(b3co|JND2i}WJ=AUo>Dw^SFp-Wm(A0){#^OBy=*N`I0WoU@nredwz4m8{^tfzm7pbp7sNWuanhFws*zum zwBD-Bo(#;icNe4xIlaC_y)c3Q2y2JA#x5Q=e#J$ScSOb$icNuaO_ebVv>$APE#q4T ze%!A!FRxz-AlTW}&I>rW`g+YDSDw-^3+824zoHk6dx`ZrPtad%7;`eEW`nm?GpmOT z&*u1$qn_Frm1>m+P>Z933kU#%JniHWvwR?pu>=*o3V-W26u`Jw9+b75D-0L@8gu z9Rk=9Tz$+GI8Oy2kFfJM=LZee`K#DB>vzO>bB&;z%Q5*I_jhcKgG5cYwxh0mEHx== zB^(^o)RS-|?{SA01?HHN#eMj!2Bo0pLX)Yh{M@lbSlUuhAj!@2)ixcixdq;aS4VE| zU(-BJU}aQ(R{A)_e8Lbx5R~FQHVdrWAU-jJX{-#d)U8PHm^*U- zsm2a=Ui*V;=})Yq(QK2M9udnFtDLOL6U-ZpuV3pS6hh*K+yK91R8>YyJyV5qSl(aa z@cr@#eCERWYJ2qR*y)An?2YF=rTY=fD2%&wY@H4*&kw;xz`}HyNMcQuQR7+?7A?nD z(1$MUv0i0SK72z*-UD4$O_+r+CrBA8$QbdL`0N{hh1>pk*Fn!QDq>O%MyWwF;?{X9 z4AQpkSg2D*mKppYHgiTw&pdzn8&q-oG{efd`b@u`^aW!B`E^kH#O9@eQG(~A3D``; z&*iBl*bO<0M%@xfLz?y_Nr?;-IaR>W7xZ5HCRdYBoJCM`Z&(G9Dt=E7F=)E440Wi8 zrwT=|fGsysH-~f_$^Q)ymJf` zR=GrN*ba^3YOe}(95N{!u>yrqF3cDhz;{+Ma2b7KjP}(U8|5f1v3f<{5wKv`jrf>6 zAAHj5yEdQ8B^zxHR1?{PZDR=rO41jZnR`8aI*!D4@T7sDq07MDjd7;4+ZO?z33Af` zx#ldeI}e->qw9WX;%0`#O$!b65Q_H>rL?WBiCDjMCms6(Bg;meZ>xMt1T_S(=QhT( zVAvcAIiRbR&pzCq2<{iqpv_l~B#7}iOF#@h9ybF<8QkZaE8(d*Qf-A z-uO+xaF)5!R`mWNylh0$K#Uh(`aHuZ1RDMB6Os)XYE*+=W+K7xC zg_OoA?v+CJ#D7)*NdIe~LzZZGDsgu!M`at}BafS>yHTqt>#$>SrPphu@>JosbSt%6 zQPLP6E*b9x=cr~mQ&sOrCmBTjJQa~g$529qm_P;Ve-B+k^n zeGvAay)g3X-LKEX0n+#*^xAdOC8xp~@9qq5)}{rI6jt^(*F$dS5{HgBsfS4X)Z`7r zH{ujDLrL-w*;WOe--a>z3n3z)VvDO&V825nitKRp^^cA_K}S6ax>u1m5o`t8G!qFj1&{jwyBydfIkfS+(GI5Sb%8(SNn8dzM$&eO z-5GTHPqF2z@AFWjmz?E?ZLe}H-xkDHr}-@mipBdXuiA@1cmtWH(KKV(z<4~&6`}Fg z30TozE7JYSn%eku4=0Zbn4f%p`1h3PutH3FdY}Ih)6_R0f*q*bHNgZS;e&wGIkg*a z{wdnb>n@xQtn4?=O=t12hh;!c|KxRKaVT}~>)=w-yBNZhmW3{VtsG@A)&Px|t`>Lo zL-uvxt9>X(Q_w?n5gNX8PhZYY*+F&$vMn+T_GZ#H(rF!8#P<&^ziIB}*QJbP?7;|} z$_$n*jNG&F-DVLn#)TXSvnSTPLdkK4uj`sp$`n~eNg>yAmDu)>NaC6KFiqNU?1p;S zqc!coRT`5DC;)K!1Gk;NSMNtF>^J02q6n#ig|6+iLf$kzL^@=YnD-<$9h;mG&Cqp^ zczl3w>i0Ip4i$=}^HDuSE~bY>JEA+M4#VfB9~odEXn@HuuMou+i9y1)5eVXr%HwH5 zFSjt_6t4`!E=47*j}&N7Y>&ml@^1_bA%PupjRfnTE`wH3Q1%y7Fp-zf&9f^N)`Dl} zUa4$xqM#1(EiJhf^(?~9KE_wv^j?6TwArzb<5sLpliT>ApyeJ%{oKq~Y#GRioGC2e z-_v&rZvS2u-AbG^{#L{x`eYrdJc|*;c!P@OxsW7}^np-baGwT8RO0OGl5ym7fF^D}hl0Bo*b*$U~@q=80DTcUqgC~a<4(~51*879= zFl1zPYP^<^N7JbrXUyO!-tAm6=x|tmEmBF5nj`vDjgKs(_)Fh6t#*8*ps4P!4iY#YvS?5)K7$E_1`><1U%bO51G4jXqx`Q(>V z>pSc_mzh#r&oh8})x?x%Erk;Rm+!kvLOr70y<&9v#V(+e>CmTT7vVu+?AxF)ya&q+R>eAfK~dB$IR9 zMgGk8a|%lG_qaQ0M#rhWJ;xI~BBG4f8TAix6R-Z{S<4r(GBGy(Vwj~BWQOrfu4NE6 zvyFsDW#HXPEk_H0dvIKlQ4M(whYHjFL-Eu+7<@HnB(?f=sy-}p?n4Q(DsJ{9F!!gi z%;G3Jc?v`lk7ibBKR9m(3un!CQ7hbO!P{&Di`WPBN;&KLE-hyYvkxa`u~Ztgb#0)| z+WsR4NCpYQ3fD45$y(i8QA`Nr;s zqB*Qyttv3fv@#I?pV*F0dDcylH{xTMd5tnu2Np1)lpN7~Lv$L;3P_KP9QMRVYFnvnn8#3%Ecr+@Pbh!5m$9P_ZO_)fcj{2E98+~%zY9>mgo&rLWnUN@9JVv z$GagQYIYj#;yo|WZjrq*#i>20R9Y#wY}zB$Bac?YGd>Kpza1));!@cuB)zF;ut~rH zx6f0QMDsFHM*5pwZ{-WzexNCWnBfuZv~xzI`>71&cWsIYyubj0>_L^}OYzVcg&-0| zoytXl3#5~C2xf$D3c%jR|wJaM{@@<6ZG(G)9%j3%ho`Svy zJSs95^yQ)KfF>ga)y!MvN|cUWj|H|Xc#j)UBCKKOjswq9ar?S;N>VkoTt41KC|C4+Ikl{U*ns=O!SP1G^Z8k9K@$at@ElkNh%+32m-`|^mxxKl--sYKXXU zn@^5u!fqk2yrw_d4PTJ*$cPJBJ>6g)sD%4GnCSc6q~`cX$l>=IBeCEcO8DMW#Nqm&oGJ;Aa7 zdvQwbmM6XZ?ab5C(+YbVvL#3HVV7F1(IfKG-C&^% zK>Y0s_})mJo9aFr4J0(^D*9SJgkNBA%7~QbJY=cvLWQQplypuSlL=mILLh>p0i9nE z&+=;P@(UXO+=k*-WpvqvA8`?9;54)bMLbWYJ2Og7JXOzDDGg2n|DwKrIAW}75T&P4dJ z`~7$8I@H3+q4B!Ws{8z3Z7oRzVwvTs8tHYzS|^-pj&yzsl83eF7h8*MTEUy46!p96 z%x>kkV>MR1Dp-UKz8N5?B}Q;JS+9rhn<8v<< zgSXi)9~_lkIVg5GkowXW_8~<>SL%2h7<{6+4T9t2#7pSYx`zb5dSNC#e^8}`rCIHF zz@JH4hfWf}Y#a|7MMCj#LSeYx)-1k*z@ua~-JBeRmTU$QfX$k!pR0FF2sDhVVJ3q#CGI@yk(s4Zp!2-H&Z1-^n1g5lz|k@8VML&B-Lg2(!c`dwUmT-Ap2Ir8wyfU ziI$niqWURSFno`i>7WhTC!`h9IrF497m22}1H*WftxTM?L0q&!>gY~~H7c-j@FfgJ z3R*q4h_rkSitdw1T9K;`D;vi+N`$o37oGw2c^jZyO1N)Hhf6#fYp)Hd=yOR+L{|xI zg4xT0Mzj$wPBh3aQH=si2zz(-bIxNNgHAPelKL-`jaLiLd&ifL>mjGOs%XrWzfcnm9 z#rAa2n?S~$7=*&ou@&SFjlOu4RvFHj=t9)puo1`mAW(DLZX{^4k~p)L3_ExQvQ34F zd`=}wj$c0>pUgRznWBn==afX5V}O9RqX_9<;_RQ;D=)h{K;>~a5U1|~j%d-A>??$a zo;mCiHFm{_mSVql=CWJmB^lw2#1T1~B7%cw;lm>a>zH(w^bZNB_`E)^zlHh@1IBk( zebi0!Zrra=EO6>M^4M;BHLIAjP#z%)BGF*R0_-XBG@W9sQ@?wmWBf8<1Fccak z8eR)REec`^x=S}|XX-@aawmKPO&iHYLMeLR_hQ>#)PTH4$+N6KIs@t$`AH#v>b)RI zq=JcubE>yC-oHX(rzl;=87O{-Tr+j-7_>6MFWbkMgCXCyk3JvBdwmyTb;^})P@ce= z4Z)ChnM$DD$;C#_p|H|a3}6kyqQ#(WdSw7oaH#jYdJ1$R2Ib%MMu65zTS(=gymvvT z8C~=M@0+~&Y!w@vq3=yHy?K0^Nw;CleKhQ;^~{iNOhiDX;3g!SKu%ii<0Dc;J-ceL zC|5L~>*;b!qwzwyJzkxQi?D)qsisL1SBuVnulb%3RJ%zT5Sm=3h~jF5?jg&o1AeTV zC({?9i!*nO{}L?l2vh%2Z!_luXf!^NasmZSYiCEI=eI6sL1qc%3r??lr}*oI$VssR zp&!mB)Jy`q>lW*8dC0UcEF*m!2>mNS{JWY5-#~B!9DBLCc0T9`ptO>1j~ym*!HH08 z)6~}A6mql+O?idF+4zXQrUN1igx9s2Y&_o=sCp{VmIwmua5$@B{M}F~n|Rpw#YaAG z6m9xEe#pRL0a02NLu(Z$0%pnkbT?D{EQbSgN_!qbtwXl2h>jj4z#Pfcd&FZqq|0#& z*czTA+A^XPCrL~wq^Mnm#u@Ghi0{n3L_AJuaGMa0b{s*F36RZZ3*<8h- zMnT`y=HK*jU?v&h6uBgi{nFV${d3VvKjx9B74)Osg}SMd?48CSiJ-XVW)$wUt$pK> zka%P!hkmyeB*9@Df$ohiUVP#xt)fG|=&QGtK?6ap5@fV@F#$HN4(V%e_bzttO3Gz$WPzCARUjz$i&lIR2CZ3e6TRz-a84W}#{GC>P%FFXf80+^OXFM2)_ zZN247Ih7?24h!W|;b7~oIj;&ua=I5PuwcJq{xkY*56lHvftyT-y>IDNg~6j%hS*Si z*%Da*Bl)(rXv4MIh8w?0q;ty4)`X!%ZIzQUJ~+o@y?Sot4&ZB56cR`YH8OXwxK6E8 z_9E-z$9Dxe7nYmxcZK%#mc>p#uJg!A+Bq;py!7slP6To1TZKhL(-{3Y5)ykDke)2XoPQ>8Tso;WnV&<$r@uAhlUIx&FxZJ6*xat$O%aj>l6q)8WBl_-nh#SW} z^|wgW(@+b5`}$M-vR!=B#4(ULB?9|=U*yIxEaiN___Wn&0cFyN_!7 zT^Vy=$>+WUz$GdFv?az(lLu>H7&|d(BRT5dX42Ko_O;pg4yPPH=FZlY+(@T|;M2qWmXg??(VNw7< z?o8%Bx`3PQY+mF9&=Ca$ea_Y%3qpz^uI;cb0RV4fX5=KA2>knPBITrfauGn}LmCyF z%lXYacLUxD4}8_7_GqAv<&y}RrqcjapclP-q2?1*tX~fVa)C4=ZAaW?=Gy-f4NSYX z%LD!_6s2K;gqoB{`OthmxR>Bb3oS-NX(hk0rj|tl3D^a=^-MC@2Zt&Zg$&m!p`AAC zJy0k)Pj~+)IAN}XCTtfvq1D%EWQ@V1PXCaug268m6cf&@ z;cG>oH1AG8pnEVxf$qZn4)|nhlj0Nv>m6t91qBe@=3Febq|mhHHDG`l+u=RIm_ib6 z`K%vqe`(gb6@}RMCke*nXV;>syULs>RXC$YJAsbFak1=E-!nMq5c4S*PR?%SoTH)K z@tC`fCb0K!+QlxIt3(1!(^%G{U|QCANpASb$kE|tBc;{Bl!)PqI@`}D9u1y^fB2ef zx_;zs#2TWs_7!CNcDS~U5IxA)!ikPfr~TkjZQH1|{#IA*qF>f8kuresRlF0v>_Q={ zBwq9Vp=N8Y8Ldp?xFTadXPA5)C-@UWzQ(cMB zMG%whz@~i7{!XxrP7^LDXO!TfOWAhs+QM3Hc_CLMVy-IeJ_yBMjgBhr)vZ;bS5>O$mNfh z`p~w>L+1DFNnkc|f2T9iJm1x{I;FH>Lw=U^aexV*8=P9 zw2w^Fsuw7)qX6SJ@TfL>^u{$EG-F=k@Pe^>1YBo&GA+byoZQrzu?vH{5%tW9#PIYTl2Wq$I*S5n2>y9T7GHY5 zxuh)rP9q9tBudTsQG^G}X!R3g=bO$g^R|Q-gRnv4c2w$httZ54C6;Oe;tWpOBR z)gD}aVCxnvuV2~9g?rR5tBL4n-+B{3C%_%OBDLn9rf|VF+aoDK0t9bLYAp}Q=J2}5 z#{$V7!k1Tk&Yzit_~`;xPH5z2~*zyyG61ok{YR(PALqH=GX&D-y zH>y2;oC_>+HQMtvk_$DjQ8ro`Ct4p_$DLh&&TK55tkJNOt>?;gs8HX*Q`mgJ6tiwm zE%RBPxuw;hU3emB65Q;!HjPSxf2L&RN(32?zhxB&UzHbVUt-ZtnEj$- zD%IGG0g|O!$HJJ5gu8_j7DuPAVUY@0^DV!Q}(1GVdVH`NCAfWT{TH^Q6^7 zpH1(atI<24`;4jYY~&@e_W+dNO}Yls{ zpZTW_wiNLGu&myxCGHdSTZ3Cc9?v^q6U+!3R3Io@>rVTVm1A(fchPa&2WGaOMz#Ejs@NoU5Q=^Avl->CaZGE~;#kpbr>eX}3-lXtq zkhZuZfq4=M%+neKk)*5*Ks*1e{ji}iCg7a%5z^6mmJ!s9%L44Hm3g?&TioK{K@L6D zvL&{+H?}=riB&^(zS-)e!9P5EPk`}jmhp_!M6z$}mS0&@1_2I2vW8f-_$K;O+_zNW zpX1f=dz-$(HnD!fQ(DwaQ(X>HvA6MS=1PrS&khCMR3_aNl(Ih}6xL$yiFwTE+#afQ z7)|lBY?I>fp#3{@1+2o+IZ%JliZOo@CtCWl^7U4Qm1z%;JKDui1(y_KR845je8i>eYxr!$W?u| zyB&wA159MlEMDNR`@E?6Of>O@zH`YX#au%yea42l!*?W89EHHsh~fuSX0XZ%HR`Sm zKC+|P= zSL|BHwXAzm4$gG=TsT zPdA1971MXrMS0suuvk8L_V@&3UyD=ZrnrQiK*Y~nBh~||%8>aiVo?Esk)Lv2!#kT* zqb!6ys6L2fSK}+%7R&o%NX^X@tP!^IPbv4gh(Hls8NFulF*aaBe@Wu1uD6EX|ysD*#Pe1PQZOYj;5tHEyJNI4Henbzc6yo z7l55lC$o3B?ox_8LL)zF@aCw|kVkE$NaK|W)4{k;ya=>`fTA}by?MFvOH$0#=7Wtgq*Du#-dlBSq&Bvpnd@X3B3?Rd?m{+twi+BC8!^>2e28^ zt(*q#oT!x|CS0Bhh;DO?!mdaxBL;$=&Y*ArE-3%_PJ90;YzsChHwW>4S1V=ZL@1b4W>_wx&Z4sysKT? zb~~eKETCSy%1anTwZnClAbq*8KxB&c94*}ruX`}nO3I8%>a=Gt^(4Pw*nK3|eM^OO za2wTchSK0T%08O!ES$c6eD$W(;*NLTS3fMX1V)S=ptX@HTp8R$wTauz*wS&LIO@{QSArR|o~(i#~YgUxfm zk_jLAX^JmZZh+Mg{0jA37ZH4~_I>K}C-*pZKI9z~f>7(59K`w4xx>rk*67&>XAVbG z|H6(5CJT(C>#cc_2S9SzgC-g z*rM$ZXOV0O>WT&cdnbR5OlH- z+-h9Vc_O#F-y0ej`ks^sBj#huG9koUivwlk3Or&uRd$CIxWQXQLG@EU$ zQf*!T5#X)c>kGF&B%77EOH(?0>=+}m24?t!qkzNUT1dfc6&?zvIy@L77y6my4U9N& z6DPo)^gV6`wu%-qy7ASZ2qfWrI-aC3cCak#P=kWrmz>}Z@Gv8C0*aQ1>#CTIxr;to z%=d0h7PahNG-b7IFc%BeB|1btraX#vw%v>1s}qvg!77^EGtCuN1iw&5LovW{QS<0r zsiTphlGd*a2#so3wGcZO0mHpL;T(B}IH}tL&G;d2WIdM_=xbMbH6@%HxTN!s*@p^; zU5LorohY+F^q5uA5vFdSKNk9x7y#i=HVfW@sPfpwu~UQn(}lHfmIdE-Ed{~0d@$wl~Z3>CD$W1 zqdtNL%IAkl=|kG`6-Y?wPu%A@O1v?0_p9Z*M}_d$mk#n@TP!QfzPdYYl~dqk37gw0 zh-43N{H>@Y!h9O_tLgMfY&I2S$uk^094+3S9xaG)>o$X7n@beYMMM_B0Bkke+s%-~ zeuL@09CVK{vfBN;%c8qR1JOOfI|=!ZnzL9S48g2T;wq=MXDk8gZrl{IW$y-RN?(Qi zATR0m?~S5}2Pmkn?Av}LCfatG%Ee8A7^4~MHG68am2MyV%U-IrJ5Wou{Og-afe%e` zfYJpYj2BBIgDwR(dB4pF8jeC-yZTxT zs$iDoy>LuX_lC1)|ET7HB5zOzZm4$Up&!@gsT0YnksY&##CkSlM1-n6yz#otk1oZ39 z8M&5V)VI1Q$kGt2+rKjBZJ>E%xe1VYR}SoE4$WR2 zF{-U79CHW!Y4NVxZe^lY`w!Lfb8KJTk4$;85@2_7vt*`D7L)(> zXG2tJBEex921-{CE*L{;UDnFK^55D-3rUXv^Hqf@&DGR+*~+hFhzNGxKtPh5B*$8= zDm(aR_-Qz#t|Q(qxA&jJDCwCGEs!g;$1Q}}nVBN+12JqMcCP5Cjd(&gpv93hsGK930H-WteU?@5V;Bh%(Su>UU zOzRd=wxaTX607{FVA=S1Ugo%%7ef>nxM%jo%ouF(F1_7!X+@~bjEI|QTq^{Z-dN{9 zY0ottMT9J`|L_BYfv+gb1(N;Exbb)XgpV^kHQudV&GeZgVJ`r&a55*l?jmTxxszQk z_uCZR?C9x{B56VHyXUo#19JlU2x&?fFz=tN@c8ty?vAL5m}|^r1J7f!1uq8*zVd4~X0a zy}u9X1KpG(q$vy-!BKo*;hUode1QFx2|bYfLJ^42hCJPEX2I&OfD-vLa+g2Ycs=o9 zRDR3l7E-(F?K_@_0`3*(AHUFHjzeE51%C<7+M_PANl+jonq~bIcWH*Y=RM_yQVOY! zt-GMjrJcFm9Q1gTi>6PF`Q9lwnF77zJ{d*Z?3#E}=hMbm=+BVBqMzf04}cOqt+d@@ zVN*Q01<2ApM^I&?xtkIbx*#11WM(^L{>XfX9GdsI{S5in849gXuicrXJn?cwhc*&Q z1T6~Z1Gwt{;;1$~>9mDBj?fxjW3L&5|$8=~(8WjkzFYq!W^v-(L zw_YS5W+4rGY#6EKRM<$$?qWy?_g>DMO|49w1;Bu&<3MY-FU*2h29&tqH`(p z$441JTGxdJGcPLb!!9)&Qf3^n;!<~obBmda2~(55#h*)>_nz}h{Mop%0#^7@>`rO| zEIEL%6lB$S7pzM<>*uoFgyhwL?PVZe_bPpQOEKzy%W;5>p^sf^kZ96u22f%?`Nn0M z@x7Baru#@}6IShXER@I6>y&zB5kVQ999T!y3x1(SWT?4#)cGqt;&{cDA=2ZX;kUov zBCQn@Uu3seBr8=e^6ZURyJ>T;-^JFI(sUiewU~=2#4xRw&+Z zjTneeju7{GWCj2v4Ox!%4CUsRWo%8x7ir|uYk^qAqX2N0Gf!Qxqi4Q zwTyqTaVP_@)P0}rxZOp;pv|nGW`PniJanu3jvU_+!D)s%{z*BR1quGLUcM?2Pf;+y zbf{DxqtJ)$6qT|8U)%NGgP`jo{5vI4=Z-Dmg$o%W(z!q6IC$TB)oBzj89U{D8Nf&o z;v*8NtP>@;3ijrW#TfvVgv!j{HYZ($hpZ9?Jvl^|-t+d1v=x+e%z zS|UP+zFTbYl?DFt_cyTj3gK1jaxrdNf4D04EkW2&783~+wLr~yhrDmo(Qo~+a<0-@ zQPvRX__woM0>?%xAhOYDF?%@Y;k6s_xuKk}oD#1c|GOh!tzzyZJ+5>>@CHHndG?7D z9o<5lfLk6gTjd+K&nGl0tsu3YVNXm@&2ZRxP&tHFPWAff49Efmmg6wd%Ispd?Z1gBktFbj$@$eTMJ~@8 z`@($AbE~-taQwE77>w9M7U)K1jAc4jcaYvmnE!7Z!vg%TYg6{{44=&wqy^gqJ}zR2 zx%^q)VlhYaVLcN!#i0QPUn`IItO@pOk#~-TMRgq=ks^CzAV95>`*wZYEtv^-O81AH z?}?Ip9YTXU1WM|L(b)?h;6iGO5m>{jyj$SQ9brXEP8%1Xe*PSv`3H(u{xG4j+GQHe zKmxoW_}(mlK(D>#gD5rTKF#zrUe-&c6jhR;zHm+VCEh`)4(8d7PX`7A1GjB~U9#{q zQ|QNO`FJ9F^nUrJRjPzLcZ6U?%oLJn;dduSDtG7J=43n97u``_z@20CKB!7T6yvpi zD`Rt;d%Z~BbBv0|bP3OvN|*q!?HC1DI}PudF|qy^4bKZYWXnCx6`V=uXnOY6?S%zO zpW8xZU<4VYnTc5vf|Jt-%UgBHvpDMVuKBoa4r&wlqc4=;hW#|XMs_|5`2rD4jLQX` z9)QrrKBU+?T2Em>sf+vgcGF^kNBP_KVzWX|BN@m=Fz8(3ADTuRVbd`$avGr{C%N=1 zBbf%fk`$`(9j1Sp9^>$g1q$7?zlSz_cnND+*Lq*Q2baANFqgk5WCMgW5q=ua?5!@Y ziRt~9#sceM0l;a-u-LOS0a~9h^w3!rSB${T4g%g_&fUU7C-eYfc`tO$QB-Z1^)<6> zZOsMWiOEv=q6|!#)p$u5NoBGA@NqAY0LyNpbs7eQ@;zzlhl}CEb{};bAlODL{)f!C1 zQ^cmIqyi36$!?piGp&`1Vaz*$pHX3EUP>kgbfB^R*%P#VX(UV4CN1W_#)*l5$KUYb zpZw{k>%O$o&SOP7lANAOA&01{ndaCgR^XNY}tR{1za}($`?Y_*LWIhSE+fL!3W;oOv}-j z%&Mit)cwcc&yJgVtfS16^*J9K!uSjY?ssDQTkYD0#UeqhXHZTQrCvQbc2?_j?N?$Z zO#y2~_nt76l=>rBDw1aoIkEV@o|j=$)&0WJt{17wFDTAW-_XbM{wX4h%+a~VPRr*8v>5IT|)NyWu?3n8^YIj$reTLUFA+?b;4P*@XpqgHEK zx;IZ@`g>P9&JQc?-C@{OPxVg&+1nWMmEHn6RR6R)&n^JJ9ZiTWL`e!mSH1$~&F7=t z|DO9Z>gR@kWC}eT+Kmn!X0(>}h%Xt9iUJpUlKDzld}i154d6om4akYCoB3J8E!cph zXSHuMl56f0MS1ECVb1+>TH>dcAix$h|A{&2K&Br1VdRJz%xCcH;+jj}n>Y)xtv97B z!WFF{&XVcjUoFj+svKt2|Ag5REIlC@zD-3mEeal)kAFa9EN1FA3<`j2R|aK18~uu# zxGbB4V8VF?m-B^BIDc(oWAN6k8(ttj{Wd;Ue6~5`L@AOyif@!+dlTg}l!vG5w(YC) z7siqiFlNr8fiXf@`yM#f`m?i066VCAT^K^V6-6s==u`+@sZ9p>?T7~ zgn!}{O?>>yRVIo36cjB~kUaLaCiYOqiam6HXIPq+$gRju9FTVZ)4~ojY3A;Rr58lj zOY%=QqZV-hK-(^vhXB_{W`+n~jJg6C9-pzN9QzxceUYw9U3K;+0MHml`8wLSr#hah-gg>ReL18zrd z3r$&{Ce-y$H}`fdY=A{ONrd&MB}1QaHoH*8@I3Zkn10+(ocE4`j&)U;8d0m-+w^x) z?ZG3NaTQ>$Ej#;x`vVcwPNwkpHPt2Q4LBag(bD2V~gh z2yH{zV4X)mp4}$GhKqh?!l!`q+`PgJHy_!sG5A?u4b zy>;Q^zeZ-48w3IQQ6RvXAZ(*#7hezPgpmO>@DB&5CTN zt+6^QaaBpchN}LmLMU{->tWIg{n4gc6K718@K7AwXjt7mzEb+!ji6{$rGFs7KfE$K zcIy6g27}`P6G3=)vx}O171`ow5Yjv4GpVq9eaDyoVrx6}!`7n4bGgE6Ty`49mdVgY z2)KbvyjNy0SkPojiAEi#3K3Y}WG9KoHxU9#fW7_jOSp`QmXFrm@X2Q&P|tjfmCM)O zns?Y+Zu%{M3>~^BEm_n6{8kZ)q)N`F|&outG00-9O zfr=3D%}`{zf+dn!G^oI(sq zt*ruWg`mcp`~osU@e{$Wgmg$TL`Y3o9#Fwgp)GS4X#}@uCqnveZabMW_d`QLb>ZLP@iLlVxqY_qZ3XK} zf3GPhRAiAjb<2EIobdZ4e*h~GQ^SM&@0XVE=mGtU|&8JUHs+s1}UCBQJr^91kJa)yD zMwL!MlR5LQ#;vct&45{jiV;xMbF`uTs{w&P3$E7(2otEX{Pv6UQ`p6G*&{wx<#V^4 z=5DOHeB^7O*YPji7zZUGdkpfX4tOlm-bHNlzygfR$rv{no^ZNzF7{ahD$}t8Eoc|} z)~Lq`QRCE497##m?v=U7zYY)}wx5MfM<$Ii%S417M$cksdc;wYeq)AUwFJn~U;xII zsed)VgaiA;5+`R7>~rskWGjs!ae|XMSDU9pQvkoO#wzs!H4q)RdDgnL8F*!;fe15= z9$qG$>~9n-lAG~8%ia{(NFu3~K@&We0>$Gp_?!@FrS3dG)1_tENM{%m|A;Y`qnjuJn`EqdCc%0bQ!2gjj!G&l?G#xxoAJ`2fLE(f%t&^Nx^mI} zMUEz@(JQ=_pwh6SHZb(HUZ9d-mmtZRnJRE8OF;q_hWpU?Yz0M^>kkZtnqWn2AEJ8j zMr)*xbw$_-ahe{iPIPUF$={%$0m^^e)dKqupNlh~uDNLZhVb9&K^yR!9pPV*?4D(G z&U5{Ke7o6_=V5U0rLk6xYn8C}a04BAdI)tJ#d8)Z%)g3O-Y0F6FULbXHBsMau>*VN=c@1JoI{;2$_DnC*Sd(eeE+GDP0p>P0S8N*C% zgFSj#GlA(ZhJkcdjY}w(+UDUN-n~fRayLp~%Px*kqhX&j%R$1;ad>)X#V?rZp`_mn zFv4yiOBnL$v~I9tTzsQvvM5Ir$>!#I?zz^YBm1vLlVJ=O(Mmj8T+JBx|K6`?l!olE+<6pwZG8(P!cw%74!nql&eO zBwNXkitl!kdZEb!2Jfzi`=Y7=L{uy2?jL%K%!G7Yp=h-^*i^R>yf@JQ21nws%);$C z@X7eMPQEKb+4^QOGu+7K3#)~H@FKY%8R+Z5z%=u)mB2e); z^J)`TvJhmM@n>(uAl#A4+0R}emV$AR^yAEl0yns2rX@|75-Q{Ev&wcPVmgTN`x#5D zZ9bYlZUG_MdPDY>4Eej*fYO$c79PR6{RnGsu=TI z#+j-Sq|F8$TJv%OLdBM~>Hd_{T*TlEBrss;;W=Cb3M*jn#!bo$jcI9w5$=d!*Bfk@;a1T z?Z%h zh9WyYOfC#H#3L&$2$Zl&()$*XC^b3JMY{n_zAG8cN)_;6Rz}y9WcjwZ{Fo{n5f*mf zH-PcwTofAX*!>G;ASTK}QZ90HTHXrEb>lPSeOmN?XEcoV6yb7%h{vQLa}q?sh9<2I z&#~GL9`$fTf{TF!NDQv3mH*SS{*q*Fhia{?U>ojVkOay|s4O%haqH?QTBaxY~6*{9Z1!%TPH#uuzPrKfpt<6i?{V!%=z z>AZ!fVMMCmwSRLH?cUewj^apl?>Wk!ioxA3ghLrGRkiG-= zx_1+Rtu6tri1{;upqf%m7&bI~XeJ=7SpospPgV|uN?i<#w5nkCSv@H9&9oXqiXo%J z<2nPRjyO0fH%y^b0pk=&jP+FXnlgrMfD2x=qBk=?T7GrcX^D2@L&^J0&nG`P(q4aM zeaiN9!vv(S)R#2;VTqs8_)4LmJbeADa33!CiR9zj9v_2c1EG01h;U(UVyWyfGJOj* z1E(ZBKC5fTwWwkDoX;-=PE5B+5OX1~!d7ix7Jem|z!Q3#5xd8xs)H)BcCFL>Ark0W ziJ?7#15f9}YUUBiC2q-QCEU6Hu1pWE`n&|~B`gV6@a)2fVw#V>resULyEh;KL<9Ha zD`il5Bkv_^>sTaL%t-Z$({n9t#5StrMPQxq1Pm$Ey@~GqcT8;{s#u$brs8uh9!Xg9 z{X8E@T^}Ee_T9}tP0L~^vK1en(WmIJoIeS`t3J zCjD7k+CJi?2ADkQOk>S|rGip^KZ1kCMk3o#6J^I~{$?m~QPk>3ddq{uKbRKqCV$1H zi&p%8w|mE?`s7YR38Po@0g2rYavEB`1Jz1X$_A+6bEmW<)0z2?QX=H}=ET%Bvj1tBN8)Kr0&GQ!uxfv+Ba&jB4;@U~OW z{H@U!Nd@a<)l_Stj_?I8#<6`{Fyb5HZyA3sT*9(m>t=H|7rs5x!da1U-r#edT-0JX z4q$4@hunG3l|C8EU;Vm5=@uHg(}P>%%ilUl49TQAZt@vQ0L5fo;E7?L605vTh~Ga>rcu z@WGw53+}2^aYe0@Srxdr6kKL7rR%Gi?l=Vx8Mv{+2va=`4)g`)lIJKC`@y85p!?RA zvxKHc&Qp}&cd^ZE`b7*1n*!n;cM60ONcu4vN&?8uhM({!^-oIg(Nl54Nh#wSQ05v~ z2!ls;z`l>emwEYI$TUGw{+~>KzZV9BudB=7(AE52TETPN4=Gr$dxptdeHb9UP^=2f z(!p$LUvqrq`FVfCJ%e}LSkVD*W&giq9T3@^g>i&`me`Y^MieF2uNPVyk9e3~*sDra z!pJvp6ZjgAVyY5S^cwU(FBZ*aQw(NPpw~hR-j`7psnoV!P|}=`7`bid<^s=gTlM;~ zKb7oN3fVZO=!|m;CJC=86LWdJdy?I<()6h8vY1dDt$aT<(dKGLPxHg~u@9+mz>DYf zVFeQYsrUTPHEox!R#$>Kl4dyq;oV_~$90+j08M2{L_t(yJkzL*1Hxj3P^ z2d_uUu`hL9FpG#A=da=Ywy@dz@M@%aRqD4$Bk(KTyhDB2V8hqd$H9}ys-2xHX>xVFah^VPZDSp2g!2-`_ZMr4xSs7F!%Gy+Mf-)M1K7^yumN5G zAhxt*P{AxDIk0?IbfC6EA7FZ=93j=sUY$BFzji~=_4cP+dBC_8(gz4Ty~~VX_+3@i zJCa3t<9t&Jd;#ix2Ms-{ZH1~a|GTxP6-ZYA4r++_JiR<+yM#W?&+N?w8ARgvFfb<+ zKgf2gYhoDVX2npLWaHlfk!>Iq{n5XgVEyaPj=d(n`Tuhj!c{%neJ1UZaU<5qic~T$ z33#_jv-7hu{=M_F`=)3;jwo@G`z+ zNB2r&6%R@< zD=zW&9SY>7ye(iz(=2je(-Mg#Wfq~s^{D_iiEt;^&fWRP0 zPunDQoXn?sy>nlMt}9rc_h`B?0u9-?zhM_N&% z0gOWGCL`tu(ck$CFwPWeQ=I~$fav$B@AKL`)xeLS-_&Tih>q3CM!9%DW154{Qow)guc$KiL zq=uhK>z1Ym7}Zv9;s;im5<|9J|FKHo$B%(A;U~wAyo6?I@2iYA(p~pA1uHA#7_8kE z_e_S8tJA@gu}W-9F`5!l(|7vM1Dbiv;CeH*o^mY0iNv}gyA2n1V;QI8nrAejtc8y4W7`HWVO1Ev%lkG6;LI>7n(2W z{Y!Lla|JA|5Ri66Q7MM_5R?bjg5OV!Ng_#NB{96%B(GpNvkU*&2*28lP7VJf@cv>7 z_e^{;GR3m*?mJMIgARLA;s6Xnm#^7*O*c+~_maB_XlW9yKHM&lr+l*?Z8Rve<;&h% z1P7O!h4`}Xh?H-Nf`dm@c<^Vfn+7fO!KN)hC_r@}*qv+^vzQ+~0))4YMDp@J=RxBBF1Ks?Vz#cgtxWbe{uX0t!mXSo)?G~+u3W)pAHJb1Zs zD;Th8#c<+K$+bfUVWH)~?>Y!K-TSn35)Rd8m&Vwhjp*8gpdvtxTe6W~e3k&}>+lLu zV2zgklR_k-4y(s$=mRC?F6#$lmjm&i7=F6~{m%99k!ga^N#bB(P1x|K&%5ewbC~J| z#=X|)a2_OZub}!2AWLa<(MUc7_TfFqqnJjOc>_w!>o%U(TUi=>QK_@Dg@UH0LTcjq z5?>-ADBiz^FPYg2_6=uR81Q^sTNQnTQ8fyfGUwLkQ$vez8P)RNO_H%6ivWZ`CSSc` z_0*E@sNb#3@NNZ#-g+g^VubO;jyFapDvduo3(&&oZ+QVR}UKn_MKaZ8pk3clQ zjtm0!x1Pr1MnhL(Myr8p+Jj(>sv-%*I~qg691~?!mPPL)&p*P2vX0X@SOIO&kD(dl zFc({uFX#rOO$(T|2!I2m+^`oaSFYXVp4P%%wFa{xTClh}(1n3M>fZF7Wv@uKMvYdo zi4H7rxQM#u-ovV}8es(VvH)#Ye)Ni9v?I~;hFZMCt^MLB#14z{4qshXg#!6?Aq%+C zoYd@qF2yf&pCAcS0P?M4CZ~t2m-F)3<$l^d?nvY&cv*Uk&-vCMfn0+}K=S4G4rL@w z8eidy6wIi#32pTT4OZTFnYvcGH@8t~sWc%@h0ve7B2=5uB?g?|F-Urf* zMy^G}Gsj;E8s1l-T4Xz<1>z-zF!@tkCG^^SPt{Z^eb0oAM?@`wsk^ zAR7ANLz7W|pw&?nT042Wk}(c0JJ&#VLj2O3ce>)KJX_{KUPiVk$$}IRs>Ry*mL9tv zx_YYJ!YDFIRaRLs>i1vayh8|jK+!5jDZc5NLW-70^)m#4CH2EM+0wk z?tiY+72k*SGyDx_Gbb#vew5H*GF&tEtvAyorOG9l^E6E7m(lJ@@!ODa6e@-Jt+LQA z!+d>YuDu&9?%E(XbsMj2$O|_7+#`Yx?5A*b#FH|RW&<}IPk3hHSzE1ij&fGKF2VeL zY=N}|zK{YRyVNwlQ98VxrNWme7VweqRE*t3AiUC$6}tga8K|ldVhc=a^cH&|c6pkS8@h@0Hu$DPTihfG6x|MACph|io63%$vkWV6#l1BW2Qf0F#WRa+!6 z*!n;B78tL#zroz$xU4*AVp2%@vWCPf@TcoJV`ToVa8tgqnB?UobP>ilFF94gyubsP`FQ{+=v!&(p0Ll#Tu zMImQ!dimmaA?WeUFp;VpH6j*#Q-0v~^tzk?)VeHT5Cf`+>kzBEdwJjNX!no9D2QJk z2aaaor$}M^0Q$x&f!O)P_*hS!+Yq#pM?=;7Jv^M_AmIGvXL18LYM9Zcm-7>A2nx~q zDjz>lO;K)@1HFpLT{KCO0ea_ScgL~zJHf&P5Z2A7X^YR!3EwLqk`zTnY7<`U370%XiYPVL7Y z8wuC~G*$g!x~q=lzuPBM!b6u+P*VO{B4L6y(C0LY@8@@y(0%5B7t(U*H+Bul9a73$ zsi%FKYA~akJPT%w`Dsq=^k?Oo;Ts7TTjIL}AU=bQCGpQ&NyH!V5m>EhZBcUzO!+8~ zjBMuR`xueKW6st27SNXAt#M*8wIqmELG&%zc}2YBM;=;&2lo*8M2RU6nahb6UwG8- z#AZad=8HiN_<7A6FUi*J$e6}_t=P@QTOJX=o+N2pcHQGr%0j>+QwF;Z;&lM0$d`MC z1bvwGXt#(#3-_0@nIt)oZ#`<2l@H{vr#nI^#Nn*GFlqQyJNPZYMk;ab!Nk2_$O>5B z`OqlA)gvE0gf{C|z0ONgkdq=GM?p#G_i7saWPUcte-B3Li$Pgd_oC|5u=zdEOL^sQ zH2s8ZW=3OZxAV!GdFjo8099pA2Z6rE=noru@t~ypwmi`51Dqq=VP^sR#kuwjKdcq8 z1_PRQX)~wcd>~B{3fWLh0}C3ubdrb?9YpkI`c89m<5@VT3w(AHo#3;&zbo!qFy)g5 z*RTAcP539I6BW$Qt&yYr`*_2~giaI9L>pnZF+GfyktQ-;a#G+l(41SZs`c)CpLvGM zO5B3>QRzW3u`*L>LXD37_@7)jTb#3i<=Gx(>a%ynQ%tU}%!hVdQ>lZwo@7(iqU5@K zv>%{u69H6F3{ciDIaLtOF{YmXj=RC&2G7$i~m ziveep{XhB=h0d-Bey*4fnK^II%G_W@NM5#s9cKXKnVnp(-ZN`vAAnFT{%%X;?Y}q0 z?`lR35vouaHCpyRih*zOm}Z@$5Xs4F2@1X7uJn2o z_Eox`c?#6q@|n2)6s zwZrpPK*Z(T5#KjrVb}*@)S8!oD3x{x4D=`mPcoy8{r^c9l_hAdnOjJYht=Srbt1b+ z)2GN``Mn=v8(KxiB4174iQ_erOp(2vo#hpN(fw7qdWeB|D8*=oDeIJ+!6=8g)Qy_K z_9L zQL&U|FkE?+0yqu9xu`}GG!X=nC_a@H^xU35AWh_Ht;dIm-dWbIv5pO`2VgpsL3=r{ z6Azia7=wAHQOqCYGH|Aa9=ht`+L|?@7B#{gj!29QKXsEtFiJ$(2){3_0AbMa(W5_j`-kvTmUcY-BTSJd{TXtED7p zHfsX?g(B??YCp}QU-1g}D%df~s~I;5Zz%lITp>b#fF4SaS}5;4I!J#nUyt%5+t)%BNJfQ$ zxul7$N0eye4qelHxbbo1%lLknnQe?Uu5vt%;@T$z0GW!|*kwTE)n%*k62g)zBt!lE z+EVWZB2(G@2Pfu%%p7C@H3YxjnTyWSQi z))Sd9ixa z((rO%EX2QZ{#l$mSRuBI_F&PU74JBG6+$iIx=6F?vi`*b*9`Uin*Zay(oSLKyVbvW zs+%Jtsr^GmcRuWZ``z=Fi6fp2J?)yO^JLFQh^n6Sy@JxB$Z|V1?5pj9|K3x&b5%Js=sh?)G0;#Jf$8TQ9zY0zJUuik777#<)Uy;{?D&nyl(_R=^)s13hahN8R#BPcsxlTx)c;g{-EqCr3Z7Qq?(uv-$GX=tm4qVX-q#d zCC$%gj*eWO!!{pm0n7cbnPC*To}fYZ>5gVE#97{8%7LpR$Bbd~rH%Az84CzUS( zi>PNB!{WXiC7sc`4Hz(v z0QB679vzsRbDaO`Oh=8QMFX=u=T+B>Cj^%oX+0Z zs>C2OC;)Msf+g=ee7kYCEtgh!nJT4ld|J1bKO-gr5+T+{F8u;4Qk(!W{@1W?{F%R& z&sNs^Nb+YUEybp2~ ztjRp`O@%}>sn9Jp?g{Tab8MMJWr8XK{6s>eiTXV&XsIW$sPLu*bLmw*r{Gqy32Y0n zv>o%z-c~=}!fBkqpGOd%D+Hn8Ap@S?Me3({9=75SX0$gZDr^=I`A4D>`-@^!-c3Mz z#NCo(y9X_;2qD32eHJ^>1lUk_vk|R?x5Z?1P@MYtZb^rA{oumGmfSN|;R8vBFMEr8 zVa+6MrbqRi(UH6k_y*t$gYCrKG>f7rC@LYB*$I7LQ&edj%!`_V*RlKrPC{?)Zi)o! zSzwf%_TM^h5JB}c8Q)Y5o80S5oDj+yD=jNr`0@w`qy#_G7>g>^WXGjmjEajyxVMQEYZ!V7P3`tXis6a&pS%*+*6*z^kM!$~98iv# zZEIiUd&M>pD7Hqe8dpVGT@XP>!AQaqCpdEE@EJaLaQ_Si5KV?C59oD$nEq0G{pfU4!9Nf39W!2-4Etnn=}%4Ep)V_zu}YZ0uSSx`?MgY00e1gre{ zEGGVn%r1?tQrIHZqhgELpNnhF)yEvA>mPbzIU3Fn(Bt^^_Kv+fOvAz3Pg{G|_J-L8 z5ZA)#Cqx%<>yKYaZX_`C%MAmt7Zv1Mh2l3{f)#3hV{2zG^QeFDGTZ4Ik^f7YFXx`8(1sjaX|ZoV zMoLnzqs$wRtH`E)iPZcG1aZsHnC`Nl$AjwOG>zSCa$^00wvy1?sp(bSD)0vffd$r& zSZCT8XYIHdyE(tUKT=eH=k>ECP1I%AfEE8ltz?FiJDZbX-y>m+$VXF7shfDcf*8 z&KC<3VHAp>A(aWy$c&4D6V9zqoCW5oBR&V6J%?CT_i30=wEas5fNsTeEVz$YK}9^3 z1VDU@3Ii0i*=F>&_+SObjBI}aQmpW_``2kL@RiG&D-D|ix=p7cTuBPq@oH6i3MlXX zC!-TR5LdSS8l^+M8_#fYmJ8N$0e!L8NG7wOef?LJ_zfCk#z7H}KL$`6?hIU_qk-@N zSJu|qD!WkVH0l$cdcb8$4ocUDD5Aucoo`FSK2bTHL&S(8HjL29W^fUzG)31z65y54 z<(>-5z4@!e&?5uZK7LCn0}7v-5j5*2^(Bkr0@}|#7)kU2&d3By;3Y)cjzizUd1$(F zYEI5c`Uf?n;x0x3+aKz?!`|y!7!hm`I5yEA_Oa?NrKmmB_RLAy|C75DUII)!14Lte z21gnIQC?{n+d`!WbPz)s`r}A*RR1XYCsq5otTnS2VSMZ4c-#ip!Dku71KU-DT zfeGp#zi9ry$Jz~I7Hr3dfifn*euuXH&W75WM{U)eNirtN^Rw{#G$g(cF|@gY=pu^( z{-AvcXZ)?8M>6f7ME449w*nGZ#*lOSSl)Rkz(ri<)d?sWfxhe!<}DIFjPYbhXc{wHj4i01xE`WW5|1%zn3vly1X3kT`>IAyLQ-8$Pix z&|V5T1k*=Y^771~Jm&!>y;W%TePBNjqlAWM$wTp`a|aHL9Z28nEs{+w8W_cUh%cCI z9*O_fG+GDhfP7ywjS5A>p1Qv~9#4QSP~@ltbBKQ_p33tBv6sKb`bq#8NzwA{=_CJF zLUie9+(c2DPvTU(h+?+;z#T*xmuF*k4W6>X*)4C72=b&MNBG z-)F{KGp4QWToK~;0-(hD^#e9iZ+uA~wOi-m)!4wCytU?(4vm{KSVGQ|DS-emWw3%& z7N8Suv3zAc-X2-7Z@&?kw^U05?MO43400S@;j(X_0A2~cls!x{x8(rSe|Gr{q~798 zCl3&j538L*q3;CnDqJZYyU<_=psP&&CY2wr5;qKIOPN2ZVO>r1%(0`y+eb9P3tE@W zV&f)y7EICuC*v4PNEX=L4DybUb+;7=(|0Yu>Un0jp%oa3? zRXaZo-!){J+Iuvv;Pd`O%e)&6=aOm~-ap_5lswm}iR-(QCUWg=Lp;(KYdnB(^iiz7 zo4UrwP;B(rs9_{3*Flf5Xxv_J9o(F5Hzs0nMrm;-JzmemC65puvFt&)>E7|W7D@Qh zmgjGGrUoyXxwU@C;%f#dR+2z=a|5i09oRwj^VIG0=TFmfj#NhpG} z>I#7oS@HD8FuNM%)~-r%ZH(4{^DGQk>|7*o6l~H3+D=nDDLHvGMsPuiQl?1rm@f6* z@(kT%ArIu%TOc4#;nkSTe`wlAQRZ8~+)!jKyH+nfS01v+UaGdOO{Y@b;mQ)-47-3_ zB-KA#zMk$C8ndF}Gc%K|VICWI#1rkq{8!P4!R!X>7$8LToM7nd~B5 zd)I5Sp7qz?wZ3OGG=~Co8!A!(n$4ArygoA}msS{b$R@4c1C7kT`iLs_qOue>h7fto zWq#0vQAq~qXAHWzb9g8?a>iJbr~ytXZjPq&OMgY-9C2IKP|u!_SfcPL40ghD@ZRS= zSyPCZxDNa4gMw+kVv&Dqs$m*La3rTs{XtkK@(Fbtr{!pt6Gq=Qu9~>mqNhlAo zMARmyd-4SDY}3?c%TxRnjOcX-uq0xKye;TkihIz3ToIVvs#|k;_BJSA*wt(bZ`8rW z-@0pRy~OPK`suyeSbP>Ois``;T60u~KNpNHnWobQ_EBPSyR6XuntN!33r+k#3*>9Wd+MJPebpg9{9Wl+Rm9ndISo}t{i{T-0+XL$OjhMgJCKm|Jnvdq{S4(y; zjdLTGZ7Q|*;Ml!)&3_MyFOj~5W20^}@}25CCG0tFDmuC31#^mmV$L9rh0=9N7Wb1V z4~(>%-R#8PG(Ze?3{w@yGN`8ktvc?&q0z|a2(c5D%}Afz6*fPuWv<|2%H9t-MDMb* z#z=n(aTfX(K?^t88N)eL5M?jIs|4y5WwSNA#ex}xkQ~aoK7zjI#j&$65h<)KG)cdx z8D{0(_s1A`SL{2x8O@5&Qy0e1PsSMo_rP6R`xT=fSq?;$Z}xS}LPtG^%x(=txdwQu zWM%&833NtY+xqIKIISVs-PI&C*3s_N2?2*eLpXXo5q$@sKOEf0#xTt-=&xrN!OCQ{ z+&%rLOn&>u+Rd6oC4khojjTEJB?;Vt@cRcNZyRzRBO6EW06M>bXMa&I*P4>FUIk2D z+ZGq!kPrmjy>&?E=aS}1F-R_I@5FVdJNfQ5{-OHVtZYXo9t;^|A#A_dRRwrMY=WW# zm@=B&tEk=u7O;q*oiE6M#H*9v9Dr86+Y!{yap=%tSHstT)AHtHi@>$}ku+#SUfJ`K zd#~6xnbG!kV=(XzTy&qcKt4B5EAtkJV$lxcblVubeer!XE+!R+k!sMn#g4Pi+O&@5 z)eJapp?%xu`%3XS90f}L4#B`3!}NZ;r_Ms#eRIiUFTb1gv-uLkeQdtmNvaBbFb6E4 zV>J4Q$&t$M;^oaxE%Hr~(7KO6U*bQ&BSoQnAB3G5fak!`@_jX)=cU3=Gi0%XVe**^ zXs^Y*V`FidVt>k98JD@Xmm~6Z*jffoQbY`<_7ABdn_@i(&&y`t4qXWxu?bBM06OWfMF6I*yE=2Ml;|iF6WXePQ(1(vW?p_`Og^s zId^w}_dQI!R;6s@&0N}G&01P$fL({ks@@DoVA_PAL<$hmni%%xiuQ?NwN5t+J*om z;vZzti^KBOQk8w5{Iy>9+K5q(sCoyW!cv<_UO-F!oDn0Sz`3y3Pusx+I3OyawV45J z#88k*L|dX4(LrmTis>{A72~8!O9s8Bj|!YLSnjtLxSLVA3slA;&U9OxYQ5XVA&|0M zv$1I|>BHsPpr~3TTd2%81E{|hy(oXy24MGuB{dr{5qW{+*O1>vMc_CHWOcy`#KPuh z)aa*dgX;+H$hz`9Q<=qG0t5_KLmT`oxA#=gaX7dxG%G*=h7x`=hLx)6d0{QV8)G zvr+Z-X#XT*>QC2gQ>8nAIWpyao1Egp1(#-|YS7>mY?c))+8)YuV_O=XPe_;_PJQ1` z%TM1jp70<;e8kiQbWnX8b}!iqsmHTr#ZWAE1y1;!Y#$= zT8%~t@XSQQ`o6i)4?hi+8XJ$~cwGm3oPlqmc za~p)aV!Wt!Gj<6hmH#juE)~>mhq@9AJ}0kXZVRHMF-2>NhZRDUNhrW`w_dD z?{NbK{(|R1{S+kqXQD#1;d75NDAjk6k}ELmDM?4J#AjTFXI$>o7!n1|c>wuw$_jnZxXAMJqUqBsf1FQJAEIw)LDH z?PVLmuIxtNd?l?w5MH^9z%XXDS^F-!0`sSPemppTZ~7%TeA+ZaXPN zaD*fa!k|Pl6?dkFKqrtZs6@?NRfq$TUD zAXupD@5s^McYOuPjSwn#1VR4 zVRp~`tn7t6tW6vb{_zZ%=g@s9YsrE#5^y#im+Duzhv6qewgda&G*#%DFp8$DD_lw2 zUKQtMJr*IT7}s+a_Z{_+9e@#6+r$-Yj){iGU{$OvLA1=hJG zxFhuD0?J_tRM^UQX?F?sk%WV00{)6$;?<-AfZFyHU6@WX(J+5H@yYq1YTrXclElBp zAesv9G0^c8c0#-fh`}Z(x4&&?@V^H!$xN60_hLaL;gl8|+f89&BI#y4j@>ndVd}Qi z77uEU`6{Kk(u!2?df6-prEW9b8jN)r5UZ&k0PwWls66@QRMx*tOwRDrtujaJ__U11 z=0c}bmAwwr=yhH}YnYVwA^aGhkirqvclKAt{Rz@ozWiV@Z&vgIp=G$X&J*cTa=?s= ziPbqca}GPUt7-_q##B8cL7G{@(foT*mVHA*@i@Pc=d z6Vu0`r9jP;wu(^AfFlf9Nwx;+=2Fu1(jl3$~ zI^fy1%}@K(u&~N2WX{vBrkAoMzv$udq_zi4cSJK0Rs-%kVz}&5q%=9baviJ-Zu^g{*tD*9}{%* zMPji-Q6cd#g705}Da)KSqGh`?Yi6+yuZ}q}Re1r=nmaB&aDxDYiknfdm0k*|>4ls{ z4Ns1uW4{Rup7l+6SU1&Ls9)Sn+c`_z{VG2q)(!?n9t)zB_FNI3vQbWCt8$|LrS5$d z|2h_}01g6T!*;$Hc=ERfvQRZofweapJV^X8Ae=hr-*SsGqFdP8EyRR%w$Xvm3p$;k zw>d+pZ9kevG8$qVPXN?d7UZ`n5mrm%bgi2*D0&F&@Y{0E`?jrc4+jWI1R6g+dlXUj z%S`*VaY7F2Z=b7#@7MozjYHkKpRieEc9+_Ci8ntX3hf5{tW}pY#p^mg2fZ4H%4>!G zdZYP{`>|K$W@e&hkF-|PRYMMzfh>$K{}ETs3K&t;vn`Vc4p>-Zx=F1XW1%J9%^%Vt zBNGxG66;#ZWaWSmkK>2UEEWnPDNn5BAnG+74Z3QYtGy}q0V1Sq#g6R_WX0G>o3elo z+LZ9Dd}7vQ5lA`9GfW5}tXQ7<@0}9QVb*OC*1yK$B4*n^tSo=e(2Ay58LT_RE63Kt zJXHR?g5CVJI6Re^H}Y}uYYU%55boU5ogvl+nFi#Ul6oWlOUn!b6M=9|DCf4MtV_Z2 z$4oIv!QP5Z=wt{}ly44K*cjP85+8MrUIwf~Sz-M12EgKQf{};m%hAUja*~HvwSK1A z!(B-fn7|67cUclyCHz2D_wHNqnkYf_m{JH?1`t%)so``BmeuhBp~w*SQfv?>IwghU zuwB}IqL;N^zy_4n&DXyU7s^i=_JVS-wq-VBv_48zgayHrpp)nYHgxCPH#COPZG0NV~6 zOjresQgV$^0c%^v;6x?`)^+Vyo3y&_vU>pG_W0#gj|s92LN^U(GGc%bl@pr)X(Ocw zyu@cvQX#}14M4viM#@_37ny0jOi)_!ivPBIi7_ijZlYt*lp~L?Ix66$0UHmr8Y&fI zm#!N^!i=qk1tTL|_m(&*-^j)7u?}_w=Fo5U_+~A$cj*}o3A^X~>!R^dX<`A#Xvb3M zVRzF(T2M4n-l6a7yFkJ3L+XOc5^nBfH80cUoBf-6e>*MuTsvF-?MXi-zGl%A1|j>W zzIiHr-o^6gbgoZ+FsdT?c~w>4y^AT66KeLg=o{xDzr1Cj8sf5{FJ-F;VPuOVkE0Sa zcodMFEvv}~uf$IU;~_x$t z)|USCpWxo9xl+u5enIq(ARa_iSFSr+g$?Jw+sA7%SR*DG?m(0qfOupL^Yc?H1N~-# zKi7=MHjf-y55Zb`DkJNY$;BU0-}|BIkaWGEKr#WWZ}+?rmi&9lEqsfCZ(sGosCef{ zuqCsH9b}$5r}488M{ur1EcR6%flLIOX~DaMO>-s_CflS(h_#yn^i?GQZ&3)`!ilpd zv`)TdBCinXI!QGE0XzEOEioTj9kCPw~b&gxy=i@0kY$ z83T{UAjzwl>2vjx<961AW57vXZa~4;g+e=%+Tk><1XMSAZ<^|q zQa4lgBOE@(QtL^8Sl0&?iX?Q7`1|F+lWvb{Pp`b+GSqxMzp&W;HFTrjIH5B3TfkQ= zFs^+37Q@5y-8xi2u?^&E?*}d)Zyh1v8CBYi(MFd>?!uGyBYgOiQkqO;C-Uo?JWK)P zB7x!@8I>Ni>aU1J@VV#|SWSY^0i+-xuP+P9z`}{1O~}t=L@rqB5O5Y6L8bFwqHB+} zj6yA>MJy-GloSJ?7GJ}_U;JSk0dgO<^<(iKz0PssyWfQoz&KoSuGCxzVLOD3;9De4 zF#wbB>M>PtSRoO?`IiH;L92*OCyubBnteStyG!O>Af4K zWoTpX*8Mi7t#t>s_!RiGd3!st#Y{U-_^G%%Rio6EVkpy}c$wHucV&ApeX%~W&$03B z`(lx+@ul+?f3E^0_R@GS^@Xj3X?7>o6^8_M+fYMBxCf>sQ*tyR+)3wnfG>XEdyhC5 zs2)zpQBSX;H#fLLHA3CXKTABEs1US}zf4T# zbxT>xwOsn(h-Fsfkl9iyQKz{>drro*7NgN;G3aXYheExp888Li zVOZ9GHNBTF0+!JMjznGpp2(uEv*a;?=TCbPl!y=Bo6PyZkz4M|>rcnz%sw77e3-Yu z?im3v+7i&{YYv@BQQI!a0kyL^=YoEuP$GNULdp=1{|#Plrnb^ri2#u4BW=cgf&HCd zo$s8aHZ+atwDjYTkFxA0b`hr19+)eL*5(W_rPahcSS}gv5j8bFx)_1SL=~3U3jsa=j$HQ+6;K-nmb+=lJO^!;GvRN$QHB=h8W|2p5Rq>6 zD#gi*_LmU}E?DJ04fLois>F`V1suTsglW(^`D0i@t?k|-NRAQQXkf@1nmcG`apU^Ki)Ds=o-pQSZ7@-1~ zS~qq|gq|YMDb>f`uhCd<_QslM2h$7l06&@?trAY#7t1$F;*4eHvgy@W9r^Q^^o?vS za^vE22xe<%l4ZTJ)m`=0Vs1%ht45df2r`Me%6Qt;!Vf-2`9N3_RgCBmlpr@a+6j`R zzF@H?mhRssn*l)=S5j+_a57gKS)Lly({dJyzK;a9*K>R&O+wM{R`u%_vYd|ZDh!zW zGVE!=EOSauly`0w0wv1|>(SzJi2FyBT2lHh)Ur})MZx0Z^IU*M{)fa8%dwli^t5Zb z3%A(See&SMz+r>*EawwXo$H_hfs3|4S}^=&ZDwSAifMS_3pl!DdGiDZM&?R6cIjzy z`;IQM$+7IGp*i2|f?9~FV%EYAyD7EP1jf$+wEERSnv#LhjU-h83JGLM?X7ijLi06N zn%5{0FBcP$o3dCzo^{Ruau%BwG9gqkb3*-uXrkQkaN)lZ`o@cxt>oT7u7Q{9DVEda zR|MryRjQR>?UDy)-l8n*Zs-?WO*lu^-ZtFyX;HSoX;X z!F&JGhK6C~Id*&nZvyuCY7SU?%?(Fu;}A&`3O6*iDO5#A@TqD&CxpnyX9g{IB~*xu zY=$JP=bIG6#mgmH*kx8Dy@L5%=oTZ`VN0nU>J-oGYT_h{-?WHxi*!$;l1BQ9L@URBC z9XgbG^?p}A_qZ0Q5Oi}3h`(}<6TAKs1mQ}QJSO3TqL{SqY5`{^Trx-|p5 zUHR9_LKu*RYXJbJ(WPh-K>)wL)XG@Rgt@BoA)Od;Qm2V&8WZF-wSN5jQ5AUxQ9oh4 zp~{dJ8voyOR&1BSQfHL@!=3shaD6!LwNe`TMx>uXW2z}a6RU)TA8h{k`1!~VtvrCK zKxe2?0{V`l@jpn1spJ9UWmm^S$c+kd$5!RxzpD;zPmw+XHuV=H++aAwjU6)p<~{7q zoSahpMv@}%HhUmh-YT0?96j-h%L}wbgr(5vL`*PVaVq=J!&PvOgEm+yCJGy+HHTXx zqX6llKU{IfM^s$QnQ=Ps_4w3j@rzoLar^2GOC9BA3`#XB8<5iz|FbV_LukalNC~md z&3yHn;+^};e*;h?D9_h~zUyc!8V~&1$Xzjh+pUkk<9W}~o=OL7Inp9GV3DSV=RCwmNwD7wG~kZd)121lq5mn!*& z5Y2;QpljN&i+ZRCji!A^N0pnrj;P}=H~416z2pun=AWBsH`3rK}{~Ft5WVspSZp-z(vM`CCOj>9+k@fMg%Q4YcOd_`N!!83-r4`40*jj zf~Vr%`e3`|W9Zoh^0WTQJNTAHor#R&*5}1=1!5VH4KanQu3^z%67=%TtjJvEt0r5-{bg-x5ps zkb3wl1_U()<__-3JUSG){iSxrFlU^~KKtzT^!W((6$< zaEU!aDBT1&Ienb>fMDWndLh9cd4e64%j}4UF5#_M*r1^l$}kCj83A%5W5*kW-;pRWuDTucIZJwreO3?ir^Fb;DWLySGkC7V6Yi0K@eFR3<_kjdi z{}`w;HVXYwytzuOx>dW`=Pt;G#-14lR5uRYmcbm`&g!kM63zueP6u;>m#Y6Ei|l9-uvLOjO*7|tS3{SHp|D?S7j09 zv7VXf074+!VY;@Ai!LH=$&DQZ_Eho_`<1jhas$X!$=7{vVXs-t)_D1D17GcqB-Vfo z?pM+6fhB$<%tXpnwYuA(GDLwfO_jZAkayP zEn#f{zKSv=FCoA0nJ1t2SqeFOna2SQQx*jG3)(0T6V#jcXQLU6J%v7jwV#c$A1>iDEge4+ zZ2S0W=*(EV+VKJL8Sq^W6~9=Jnq{vT*!y*@yx6lfoAqf^s4pi3TVUXI4RWR4INaDwX2_H@b z4Y~|QWSc7uM+t9gNE+7NT@zhmDvr}zjWs;q0PPl*CDBsYm!oPh$bxHI?&W=OJsi>tQ=jR=r4jeA4+-9KZ%}-5VyVl(%Ul1O@1b>SbQp8Nq!HG-{ zixez3egK%bP!P(DWMAmqA|3?$%tG6c1Lr<~c1H~nlo-(6s8jSC2)yADDtBvXeLme@ zqPfBHf6|}vSS`jUQ(sAsj zphMRM*H$i&kUHQ8QrZ~EI;WdW!QSM;a(NOc(GSK1VG ziXpPz_?>9mO}nNFa`G(-!m*$9MC3HCJSuT3I`!Yx85sqfhQ6a=cC67IA)gKiD_jT`E z8T#$#sqivQn>Y2Avw6bI2(mknR-q#VUad3;hTmCdaq2i*5a|KTBl8Z4pmZhp}vA&NgbVq!BGG}hm zLSi%UKc-HCHsLRkC+kDqb?rCgQ+ghSuDV|^ATX>$ORVwsO2BTzD6~$w1*KkLOjt%{lqFCF}{pQ zgeaBq0f!L46cJI_(z|aoZN;ynM&0zUT0~y7h>Z-iI-mfs#6@4Ud8L1E0ph6RPeJY_ ztT*Qjri|D8M)J0r)~@^7^P-Sa_T*6gH{W~qf0??5BxuJ+`nTO|(PL;6_eQDy4lQl4 z_nBUMI}>*do|*F08IjzkqSCEl&f4#`rmr!y%(1Oi{oNEHj<7MZNpk zouD^QB$F*P&(r-ac;p~@mmDGe{N^33qBWs2RUfO`vHC;k8Y|~xg8603J?Zb zhg2Nx5hza)Aj@8DWqOJPdcHE-4KBwG?1HKKo2~mNC4s9v=)`{}+05-)F%}5!8>;@n z-LZc~<@e$T*F#{y1Of%C)v3kmTeG~}EKH9Vp2KmM&Up48#AZ4<)6dxl1wG~XlpkeMHqK-?I;H9xh+*jmHBhlgtkbnFF z1?mdq-Okwa8-TImM8g7lPQ0M}vRfsFDfTpvV+_m}V|i_2Lo7vrU@|;nz6B@Z`h_Sx z?F*s`=PRst2iDPT@0U?Je?Y@mwULoD%*-gQq$}Bfw}$o51f&5(l+j-n|Ckr5WWy#w z1HUbvd{-5n$}FgySYTG5CJj9IyCKFtLo5dT3+YP>VW3jQMxsjFF~lix=|#tQmRT)$ zqKC6gq7F$1&c4v1IM+dNlXYw+O#vmo1fiN?QGom zT@QKuT_Zu6KPj!myq4AF(x?Sn9}}jm$@1S)BZSF8Q*RtrG}JZ-1wU#a@T54 z8$OLiZ}*J~N(5F}2^@(kU@9c}?@r3eOFU9X;wutmuFe^BT3RO`9mOy{+Uh3>;_SYd z4LwM30IY?mBpV%ls}>y?ds5QQ^rO}d1jRHhZ#@m;N68HTi*-p7wAp4sIz@aXa8n;1 ziFtLf}Ia&l{5$fjZ-)|7VxGOPBk9#8KIt0kijM zFGrp)ycR%m+^zl$#!wB&WiO$V!`58m1~0-~Rf`9SNp%}e#TVfJMyn{e+@a_lMnbsP z{c|y`!>|pEmo*bh=!8K1FbAG_`7;kK&Kw2$Lm0#i1J%vHu-^8}f3M~5chV^A zyxdA9={5-(mU>n+EB)mN!^Qh#ftUc? zQ4-;cScK#6!{hYUfz;MNcIx_K3T{1H|>+~xIk4|V(U-d7!Vz)}nQEzgKo*93I;4ZzfaZG+oD$w!r?u0^Y0mMq(#QDm)hZ41;}c{X>5~m< zTgbw-b=%xgGkEI9m(SB(U1Qb<%*<9}Ov(OGp;bo@S>9IvjLnz`)D4)og*_C@xjPac zIp@%m&o`~FSuWO`UpoI_&_x6@GUm^Mw9|j{8ECk7#}2#J*#V3mse-^IMj+`jr(7_& z0&T{f*Y7W}*0)z{XlO=GDv=uwGRkyoCs^}bw(+q2T;O=nne=F-r=B_^pOG2H(M{@h zhLtScJYR((O{@QpoY_LuekL5gY&FR%%8T_T1^VgtY&Vy9K*LZ5>xwq1I}A^p-p3=K zz{!;!6i7$_xHmF)S`1$V9W-JGuM+~GO2~qG)$14sTz(~ohFf3xs#raw=B5)%0XCKl zSkoTPkwx?;l`2(zoA-r5749U6bQ!nIA7m$EAqopxq9!jW)yWR>ts7Klr>FB04VYgT z1-nJ)<{(z53mD1}m>*8hDrulzvtnCx`FueGO5?hEGjq)W-rlkT;5`uRW*i+%^!sQEH~oflQL9Yo^Y zRDxiW{(s}^;|8?L9x&4j`I)?eG-?;A6_rk2NVyN5JxI*aHGQu9pWPSkaDh(bHTw=L z3l%IM8gV(d>lUcJsSF@NZto-B(lWH52hE84?=0qeoq=7-exe-0n;$TMs6pa-aRU3w zOBc9!aClYpp<>oWS$7B=@Yrn6{IJ~o8!&ao0Xcm7pch;db~k-3lj7A8F9X3>_*xBH zIg;||59vbBBOeHuKgA11r;-Ok6yn~|g|7H91_#X7zTs^vF%QYSUrXiq`J^pl$chD) z{PYrH#RfSDLX@3UMVn+mxRiB9N?RcJuKw+hflF8nlZVawG}_AJTctVe%9JEC7iwE9 zU0~|AxbOjmJGq|52e#hoX0q!~&1Ad#;}_8nvuT-#yIKC9o~a1!3?|Zxccpe_2srV$ zFp&be`CqkaC@<>HXEnfj{cC$Tp3P+}1wK@)L30@dN!14{*8l=t{NCMc`oj2PBs z%aI_9Y-+NNbph8_v19_NWPzA<*S(&Z=^Zg0^}fF!K-^Y+wM}v6vknqj7vRTwP2iWI zBm<24l5&EufxulP6?<=uX}8RXSmC!eR2Z4kL0UK}pFt|hu!f^nSQT+{iHGEe*rYKP zBU+LdqE_Qx`jL)DR+N1O(w*6UG!4yZ+bx1E<)Wh{E1xs2{)nX#m#YDnDef>#Sf>C0 N002ovPDHLkV1lf?mS_L~ literal 0 HcmV?d00001 diff --git a/flash_rt/structures/adapters/ggml/qualification/pins.yaml b/flash_rt/structures/adapters/ggml/qualification/pins.yaml new file mode 100644 index 00000000..d38744cf --- /dev/null +++ b/flash_rt/structures/adapters/ggml/qualification/pins.yaml @@ -0,0 +1,16 @@ +# Structure versions the ggml adapter's fused windows were written against. +# The qualification runner turns RED when the live catalog moves past a pin: +# that is the signal to re-audit the corresponding window before adopting +# the new structure version, then update the pin. +binding: jetson_pi_edge_pi05 +pins: + vla_tick_pipeline: 2 + decoder_ffn: 1 + vision_ffn: 2 + qkv_pack: 1 + attention_core: 1 + linear_proj: 1 + norm_fused: 1 + adaln_producer: 1 + modnorm_qkv_chain: 3 + cadence_static: 1 diff --git a/flash_rt/structures/adapters/ggml/qualification/run_qualification.py b/flash_rt/structures/adapters/ggml/qualification/run_qualification.py new file mode 100644 index 00000000..c1c61f8e --- /dev/null +++ b/flash_rt/structures/adapters/ggml/qualification/run_qualification.py @@ -0,0 +1,177 @@ +#!/usr/bin/env python3 +"""Qualification gates for the ggml host adapter. + +Offline gates (always run): + A. manifest — the pipeline binding validates against the live catalog + (structure renames, removed embedded regions, or a + malformed manifest turn this red). + B. pins — every pinned structure version matches the live catalog + (an upstream version bump turns this red and names the + structure, which is the cue to re-audit the bound window + before adopting the bump). + +On-device gate (opt-in, needs a running llama-server with the pi0.5 model): + C. e2e-golden — drives the fixed synthetic-input protocol against the + server and compares the raw action chunk to a stored + golden. The comparison is exact by default (the adapter + is bitwise deterministic across processes); pass --tol + to allow a max-abs band instead. Any kernel or window + change that shifts numerics turns this red. + +Usage: + python run_qualification.py # gates A+B + python run_qualification.py --e2e --port 8089 # gates A+B+C + python run_qualification.py --e2e --update-golden # refresh the golden +""" + +from __future__ import annotations + +import argparse +import json +import pathlib +import sys + +import yaml + +_HERE = pathlib.Path(__file__).resolve().parent +_REPO = _HERE.parents[4] +if str(_REPO) not in sys.path: + sys.path.insert(0, str(_REPO)) + +from flash_rt.structures.binding import load_binding # noqa: E402 +from flash_rt.structures.registry import load as load_structure # noqa: E402 + +GOLDEN = _HERE / "goldens" / "pi05_thor_action.json" + + +def gate_manifest(binding_name: str) -> tuple[bool, str]: + try: + spec = load_binding(binding_name, require_pipeline_coverage=True) + except Exception as exc: # noqa: BLE001 — any validation failure is red + return False, f"binding failed validation: {exc}" + return True, (f"{spec.name} -> {spec.structure.name}@" + f"{spec.structure.version}, {len(spec.segments)} segments, " + f"contract {spec.coverage_contract}") + + +def gate_pins() -> tuple[bool, str]: + pinned = yaml.safe_load((_HERE / "pins.yaml").read_text())["pins"] + drifted = [] + for name, version in pinned.items(): + try: + live = load_structure(name).version + except KeyError: + drifted.append(f"{name}: pinned @{version}, missing from catalog") + continue + if int(live) != int(version): + drifted.append(f"{name}: pinned @{version}, catalog is @{live}") + if drifted: + return False, "; ".join(drifted) + return True, f"{len(pinned)} structure versions match the catalog" + + +# ---- gate C: end-to-end action golden -------------------------------------- + +def _server_request(base: str, method: str, path: str, body=None): + import urllib.request + opener = urllib.request.build_opener(urllib.request.ProxyHandler({})) + data = json.dumps(body if body is not None else {}).encode() + req = urllib.request.Request(base + path, data=data, method=method, + headers={"Content-Type": "application/json"}) + with opener.open(req, timeout=120) as f: + return json.loads(f.read()) + + +def _synthetic_images(directory: pathlib.Path) -> list[str]: + import numpy as np + from PIL import Image + rng = np.random.default_rng(1234) + paths = [] + for name in ("base.png", "wrist.png"): + p = directory / name + if not p.exists(): + Image.fromarray( + rng.integers(0, 255, (224, 224, 3), dtype=np.uint8)).save(p) + paths.append(str(p)) + return paths + + +def run_pipeline(port: int, image_dir: pathlib.Path, warmup: int = 2) -> dict: + """Fixed synthetic-input protocol; the first evaluations after server + start are cold (cache fill / capture paths) and differ from the steady + state, so the comparison value is taken after ``warmup`` full passes — + steady-state output is bitwise stable across runs and processes.""" + base = f"http://127.0.0.1:{port}" + images = _synthetic_images(image_dir) + state = ",".join(f"{0.01 * i:.4f}" for i in range(32)) + resp = None + for _ in range(warmup + 1): + _server_request(base, "POST", "/foreground/reset") + for p in images: + _server_request(base, "POST", "/foreground/image", {"path": p}) + _server_request(base, "PUT", "/foreground/state", {"state": state}) + resp = _server_request(base, "POST", "/foreground/infer", + {"text": "pick up the object"}) + return {"action_final_raw": resp.get("action_final_raw"), + "action_steps": resp.get("action_steps"), + "action_dim": resp.get("action_dim")} + + +def gate_e2e(port: int, image_dir: pathlib.Path, tol: float, + update_golden: bool) -> tuple[bool, str]: + got = run_pipeline(port, image_dir) + if got["action_final_raw"] is None: + return False, "server returned no action_final_raw" + if update_golden: + GOLDEN.parent.mkdir(parents=True, exist_ok=True) + GOLDEN.write_text(json.dumps(got)) + return True, f"golden updated: {GOLDEN}" + if not GOLDEN.exists(): + return False, f"no golden at {GOLDEN}; run with --update-golden first" + want = json.loads(GOLDEN.read_text()) + if (got["action_steps"] != want["action_steps"] + or got["action_dim"] != want["action_dim"]): + return False, (f"shape drift: {got['action_steps']}x{got['action_dim']}" + f" vs golden {want['action_steps']}x{want['action_dim']}") + flat_got = [x for row in got["action_final_raw"] for x in row] + flat_want = [x for row in want["action_final_raw"] for x in row] + diffs = [abs(a - b) for a, b in zip(flat_got, flat_want)] + worst = max(diffs) + n_diff = sum(1 for d in diffs if d > tol) + if n_diff: + return False, (f"{n_diff}/{len(diffs)} elements beyond tol={tol:g}, " + f"max_abs_diff={worst:.3e}") + return True, (f"{len(diffs)} elements within tol={tol:g} " + f"(max_abs_diff={worst:.3e})") + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--binding", default="jetson_pi_edge_pi05") + ap.add_argument("--e2e", action="store_true", + help="also run the on-device action-golden gate") + ap.add_argument("--port", type=int, default=8089) + ap.add_argument("--image-dir", type=pathlib.Path, + default=_HERE / "goldens") + ap.add_argument("--tol", type=float, default=0.0, + help="max-abs tolerance for the e2e gate (default exact)") + ap.add_argument("--update-golden", action="store_true") + args = ap.parse_args() + + gates = [("manifest", gate_manifest(args.binding)), + ("pins", gate_pins())] + if args.e2e: + args.image_dir.mkdir(parents=True, exist_ok=True) + gates.append(("e2e-golden", + gate_e2e(args.port, args.image_dir, args.tol, + args.update_golden))) + + all_ok = True + for name, (ok, detail) in gates: + print(f"[{'GREEN' if ok else 'RED':5s}] {name}: {detail}") + all_ok = all_ok and ok + return 0 if all_ok else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/flash_rt/structures/bindings/jetson_pi_edge_pi05.yaml b/flash_rt/structures/bindings/jetson_pi_edge_pi05.yaml new file mode 100644 index 00000000..ffa8b90d --- /dev/null +++ b/flash_rt/structures/bindings/jetson_pi_edge_pi05.yaml @@ -0,0 +1,97 @@ +binding: jetson_pi_edge_pi05 +structure: vla_tick_pipeline + +# Host: ggml/llama.cpp server (PKU-SEC-Lab/Jetson-PI-Edge fork) running pi0.5 +# on Jetson AGX Thor (SM110) with the native ggml adapter +# (flash_rt/structures/adapters/ggml/). Structure regions execute as fused +# subgraph windows matched inside ggml-cuda's graph evaluation; the mapping +# below is window <-> catalog structure. + +stages: + obs_encode: + seam: "clip batched ViT encode (SigLIP graph) + llama_encode prefix" + capture: cuda_graphs_keyed # llama.cpp keyed multi-graph CUDA graphs + outputs: + cond_features: "prefix KV cache + device-resident image embeddings" + action_denoise: + seam: "llama_decode denoise loop over the pi0 action-expert stack" + loop_steps: 10 + noise_window: "seeded host-side noise tensor (PI0 seed window)" + capture: cuda_graphs_keyed + +cadences: + observation: obs_encode + tick: [obs_encode, action_denoise] + replan: action_denoise + +coverage: + contract: complete_hot_path + hot_path: + - observation_inputs + - vision_encoder + - vision_projection + - image_embd_residency + - prefix_prefill + - prefix_kv + - denoise_control + - timestep_conditioning + - action_expert_transformer + - euler_update + - action_readout + segments: + - name: observation_inputs + stage: obs_encode + classification: host_stage + seam: "mtmd bitmap load, patchify, and state ingestion" + - name: vision_encoder + stage: obs_encode + classification: structure + seam: "SigLIP tower graph (flash-attention path, padded heads baked into repacked weights)" + structures: [vision_ffn, qkv_pack, attention_core, linear_proj] + - name: vision_projection + stage: obs_encode + classification: structure + seam: "multimodal projector head" + structures: [linear_proj] + - name: image_embd_residency + stage: obs_encode + classification: state_region + seam: "device-resident image embeddings (D2D into a persistent buffer, no host round-trip)" + structures: [cadence_static] + - name: prefix_prefill + stage: obs_encode + classification: structure + seam: "llama_encode over the prefix backbone layers" + structures: [decoder_ffn, qkv_pack, attention_core, linear_proj] + - name: prefix_kv + stage: obs_encode + classification: state_region + seam: "persistent prefix KV cache reused across denoise steps" + structures: [cadence_static] + - name: denoise_control + stage: action_denoise + classification: control + seam: "per-step row-index upload and loop synchronization" + - name: timestep_conditioning + stage: action_denoise + classification: state_region + seam: "precomputed per-step modulation table, selected by row indices in-graph" + structures: [adaln_producer] + - name: action_expert_transformer + stage: action_denoise + classification: structure + seam: "action-expert decoder layers (modulated norms, fused QKV+RoPE, GeGLU FFN, gated residuals)" + structures: [modnorm_qkv_chain, qkv_pack, attention_core, decoder_ffn, + linear_proj, norm_fused, adaln_producer] + - name: euler_update + stage: action_denoise + classification: host_stage + seam: "host-side Euler integration between denoise steps" + - name: action_readout + stage: action_denoise + classification: host_stage + seam: "action chunk fetch and denormalization" + +hosts: + jetson_pi_edge: + versions: "feat/flashrt-thor-kernels" From 0992c526da2d0aebe4782c8acefc1cfe281eba16 Mon Sep 17 00:00:00 2001 From: LiangSu8899 Date: Mon, 24 Aug 2026 14:26:56 -0400 Subject: [PATCH 07/32] feat(ggml-adapter): fold the Gemma-style RMS norm chain into the modulate kernel {RMS_NORM, MUL(w), ADD(mul, norm)} == rms_norm(x)*(1+w) runs as one kernel via ada_rms_mod with scale = w and shift = 0 (cached zero vector). ggml's own fused rms_norm cannot express this form because the add operand is the norm output itself. Opt-out via GGML_FLASHRT_NO_RMS_GEMMA. --- .../structures/adapters/ggml/fr_dispatch.cu | 57 +++++++++++++++++++ flash_rt/structures/adapters/ggml/fr_ggml.cuh | 8 +++ 2 files changed, 65 insertions(+) diff --git a/flash_rt/structures/adapters/ggml/fr_dispatch.cu b/flash_rt/structures/adapters/ggml/fr_dispatch.cu index 59f5f3f8..16b58390 100644 --- a/flash_rt/structures/adapters/ggml/fr_dispatch.cu +++ b/flash_rt/structures/adapters/ggml/fr_dispatch.cu @@ -1309,6 +1309,63 @@ bool ggml_cuda_flashrt_mm_res(ggml_backend_cuda_context & ctx, const ggml_tensor return true; } +// ── Gemma-style norm fold: {RMS_NORM, MUL(w), ADD(mul, norm)} ──────────────── +// out = rms_norm(x)*w + rms_norm(x) == rms_norm(x)*(1 + w): the adaLN +// modulate kernel with scale = w and shift = 0. ggml's own fused rms_norm +// cannot express this form (the add operand is the norm output itself, which +// no longer exists once the chain is fused), so it never fires on it. +bool ggml_cuda_flashrt_should_fuse_rms_gemma(const ggml_tensor * rms, const ggml_tensor * mul, + const ggml_tensor * add) { + static const bool disabled = getenv("GGML_FLASHRT_NO_RMS_GEMMA") != nullptr; + if (disabled) { + return false; + } + const ggml_tensor * x = rms->src[0]; + if (rms->type != GGML_TYPE_F32 || x == nullptr || x->type != GGML_TYPE_F32 || + !ggml_is_contiguous(rms) || !ggml_is_contiguous(x) || rms->ne[3] != 1) { + return false; + } + if (mul->src[0] != rms) { + return false; + } + const ggml_tensor * w = mul->src[1]; + if (w == nullptr || w->type != GGML_TYPE_F32 || !ggml_is_contiguous(w) || + w->ne[0] != rms->ne[0] || ggml_nrows(w) != 1) { + return false; + } + if (!((add->src[0] == mul && add->src[1] == rms) || + (add->src[0] == rms && add->src[1] == mul))) { + return false; + } + if (add->type != GGML_TYPE_F32 || !ggml_is_contiguous(add) || + !ggml_are_same_shape(add, rms)) { + return false; + } + return true; +} + +bool ggml_cuda_flashrt_rms_gemma(ggml_backend_cuda_context & ctx, const ggml_tensor * rms, + const ggml_tensor * mul, ggml_tensor * add) { + const ggml_tensor * x = rms->src[0]; + const int M = (int) ggml_nrows(x); + const int C = (int) x->ne[0]; + float eps; + memcpy(&eps, rms->op_params, sizeof(float)); + cudaStream_t stream = ctx.stream(); + + const void * zeros = get_zero_bias(C, stream); + if (zeros == nullptr) { + return false; // zero-vector alloc during capture: run unfused + } + const int rc = ggml_cuda_flashrt::ada_rms_mod((const float *) x->data, (const float *) mul->src[1]->data, + (const float *) zeros, (float *) add->data, M, C, eps, + /*with_rms=*/true, stream); + if (rc != 0) { + GGML_ABORT("flashrt: rms_gemma fused kernel failed (M=%d C=%d rc=%d)", M, C, rc); + } + return true; +} + void ggml_cuda_flashrt_begin_eval() { g_eval_id++; } diff --git a/flash_rt/structures/adapters/ggml/fr_ggml.cuh b/flash_rt/structures/adapters/ggml/fr_ggml.cuh index e821e086..147f1395 100644 --- a/flash_rt/structures/adapters/ggml/fr_ggml.cuh +++ b/flash_rt/structures/adapters/ggml/fr_ggml.cuh @@ -103,3 +103,11 @@ bool ggml_cuda_flashrt_should_fuse_gated_res(const ggml_tensor * view, const ggm const ggml_tensor * mul, const ggml_tensor * add); void ggml_cuda_flashrt_gated_residual(ggml_backend_cuda_context & ctx, const ggml_tensor * view, const ggml_tensor * mul, ggml_tensor * add); + +// {RMS_NORM, MUL(w), ADD(mul, norm)} -> rms_norm(x)*(1+w) in one kernel. +// The execute returns false (run unfused) only when its zero-vector cache +// cannot allocate during graph capture. +bool ggml_cuda_flashrt_should_fuse_rms_gemma(const ggml_tensor * rms, const ggml_tensor * mul, + const ggml_tensor * add); +bool ggml_cuda_flashrt_rms_gemma(ggml_backend_cuda_context & ctx, const ggml_tensor * rms, + const ggml_tensor * mul, ggml_tensor * add); From 09904456d8f1ae2fe9d0f178a84cd056d580a93e Mon Sep 17 00:00:00 2001 From: LiangSu8899 Date: Mon, 24 Aug 2026 14:38:17 -0400 Subject: [PATCH 08/32] feat(ggml-adapter): prefill fused QKV window Fuses the prefix layers' q/k/v projections, RoPE, scale, KV padding to the FA KQ stride, and the permuted f16 materializations into the fused QKV GEMM plus qkv_post. qkv_post_full additionally re-emits the rope'd K and plain V as f32 rows so graph-tail persistent-KV stores keep reading their tensors; pad rows are zeroed to preserve the graph's PAD semantics. Bitwise-identical action output; opt-out via GGML_FLASHRT_NO_QKV_PREFILL. --- .../structures/adapters/ggml/fr_dispatch.cu | 160 ++++++++++++++++++ flash_rt/structures/adapters/ggml/fr_ggml.cuh | 14 ++ .../structures/adapters/ggml/fr_kernels.h | 10 ++ .../structures/adapters/ggml/fr_qkv_post.cu | 47 ++++- 4 files changed, 222 insertions(+), 9 deletions(-) diff --git a/flash_rt/structures/adapters/ggml/fr_dispatch.cu b/flash_rt/structures/adapters/ggml/fr_dispatch.cu index 16b58390..88f0295f 100644 --- a/flash_rt/structures/adapters/ggml/fr_dispatch.cu +++ b/flash_rt/structures/adapters/ggml/fr_dispatch.cu @@ -1309,6 +1309,166 @@ bool ggml_cuda_flashrt_mm_res(ggml_backend_cuda_context & ctx, const ggml_tensor return true; } +// ── Prefill fused QKV: {q mm→reshape→rope→scale, k mm→reshape→rope→pad, +// v mm→reshape→pad, permute→cpy ×2} ───────────────────────────────────── +// Same fused GEMM + qkv_post as the decode window, but K/V land in per-eval +// padded f16 tensors (token rows of head_dim, KV length padded to the FA KQ +// stride) instead of the persistent KV suffix; the pad rows are zeroed to +// match the graph's PAD semantics (the FA mask multiplies them out, but +// garbage f16 there would poison the softmax with inf/nan). +bool ggml_cuda_flashrt_should_fuse_qkv_prefill( + const ggml_tensor * q_mm, const ggml_tensor * q_rope, const ggml_tensor * q_scale, + const ggml_tensor * k_mm, const ggml_tensor * k_rope, const ggml_tensor * k_pad, + const ggml_tensor * v_mm, const ggml_tensor * v_pad, + const ggml_tensor * k_cpy, const ggml_tensor * v_cpy) { + static const bool disabled = getenv("GGML_FLASHRT_NO_QKV_PREFILL") != nullptr; + if (disabled) { + return false; + } + const ggml_tensor * src1 = q_mm->src[1]; + if (k_mm->src[1] != src1 || v_mm->src[1] != src1) { + return false; + } + const ggml_tensor * wq = q_mm->src[0]; + const ggml_tensor * wk = k_mm->src[0]; + const ggml_tensor * wv = v_mm->src[0]; + if (!ggml_cuda_flashrt_should_use(wq, src1, q_mm) || + !ggml_cuda_flashrt_should_use(wk, src1, k_mm) || + !ggml_cuda_flashrt_should_use(wv, src1, v_mm) || + wk->ne[0] != wv->ne[0] || wk->ne[0] != wq->ne[0]) { + return false; + } + const int64_t head_dim = wk->ne[1]; + if (wv->ne[1] != head_dim || wq->ne[1] % head_dim != 0) { + return false; + } + // Q: mm -> reshape [hd, n_head, M] -> rope -> scale (contiguous f32 out) + if (q_rope->src[0]->op != GGML_OP_RESHAPE || q_rope->src[0]->src[0] != q_mm || + q_rope->src[0]->ne[0] != head_dim || + q_scale->src[0] != q_rope || !ggml_is_contiguous(q_scale) || + q_scale->type != GGML_TYPE_F32) { + return false; + } + // K: mm -> reshape [hd, 1, M] -> rope -> pad along the token dim + if (k_rope->src[0]->op != GGML_OP_RESHAPE || k_rope->src[0]->src[0] != k_mm || + k_rope->src[0]->ne[0] != head_dim || k_rope->src[0]->ne[1] != 1) { + return false; + } + const int64_t M = k_rope->src[0]->ne[2]; + const int64_t kvp = k_pad->ne[2]; + if (k_pad->src[0] != k_rope || k_pad->ne[0] != head_dim || k_pad->ne[1] != 1 || + kvp < M) { + return false; + } + // V: mm -> reshape -> pad, same geometry + if (v_pad->src[0] == nullptr || v_pad->src[0]->op != GGML_OP_RESHAPE || + v_pad->src[0]->src[0] != v_mm || v_pad->src[0]->ne[0] != head_dim || + v_pad->src[0]->ne[1] != 1 || v_pad->src[0]->ne[2] != M || + v_pad->ne[0] != head_dim || v_pad->ne[1] != 1 || v_pad->ne[2] != kvp) { + return false; + } + // both copies materialize [hd, kvp] f16 token rows + for (const ggml_tensor * cpy : { k_cpy, v_cpy }) { + if (cpy->type != GGML_TYPE_F16 || !ggml_is_contiguous(cpy) || + cpy->ne[0] != head_dim || cpy->ne[1] != kvp || cpy->ne[2] != 1 || + cpy->nb[1] != head_dim * sizeof(uint16_t)) { + return false; + } + } + // ropes share positions, freq factors and parameters; NEOX math only + if (k_rope->src[1] != q_rope->src[1] || k_rope->src[2] != q_rope->src[2] || + memcmp(k_rope->op_params, q_rope->op_params, sizeof(k_rope->op_params)) != 0) { + return false; + } + const int mode = ((const int32_t *) k_rope->op_params)[2]; + if (mode != GGML_ROPE_TYPE_NEOX) { + return false; + } + return true; +} + +void ggml_cuda_flashrt_qkv_prefill(ggml_backend_cuda_context & ctx, + const ggml_tensor * q_mm, const ggml_tensor * q_rope, ggml_tensor * q_scale, + const ggml_tensor * k_mm, const ggml_tensor * k_rope, + const ggml_tensor * v_mm, + ggml_tensor * k_cpy, ggml_tensor * v_cpy) { + const ggml_tensor * src1 = q_mm->src[1]; + const ggml_tensor * wq = q_mm->src[0]; + const ggml_tensor * wk = k_mm->src[0]; + const ggml_tensor * wv = v_mm->src[0]; + + const int K = (int) wk->ne[0]; + const int Nk = (int) wk->ne[1]; + const int Nv = (int) wv->ne[1]; + const int Nq = (int) wq->ne[1]; + const int M = (int) ggml_nrows(src1); + const int head_dim = Nk; + const int kvp = (int) k_cpy->ne[1]; + + cudaStream_t stream = ctx.stream(); + + const repacked_weight * w = get_repacked_qkv(wk, wv, wq, stream); + + const void * q_packed = nullptr; + const void * q_sf = nullptr; + ggml_cuda_pool_alloc a_packed(ctx.pool()); + ggml_cuda_pool_alloc a_sf (ctx.pool()); + int rc = 0; + if (!get_quantized_act(src1, M, K, &q_packed, &q_sf, stream)) { + a_packed.alloc(ggml_cuda_flashrt::packed_bytes(M, K)); + a_sf.alloc(ggml_cuda_flashrt::sf_bytes(M, K)); + rc = ggml_cuda_flashrt::quantize_act_f32((const float *) src1->data, a_packed.get(), a_sf.get(), M, K, stream); + q_packed = a_packed.get(); + q_sf = a_sf.get(); + } + + const int N_tot = Nk + Nv + Nq; + ggml_cuda_pool_alloc qkv_cat(ctx.pool(), (int64_t) M * N_tot); + + if (rc == 0) { + rc = ggml_cuda_flashrt::gemm_f32out(q_packed, q_sf, w->packed, w->sf, + qkv_cat.get(), M, N_tot, K, 1.0f, false, stream); + } + if (rc == 0 && kvp > M) { + // zero the pad rows once per eval; qkv_post then fills rows [0, M) + const size_t row_bytes = (size_t) head_dim * sizeof(uint16_t); + cudaMemsetAsync((char *) k_cpy->data + (size_t) M * row_bytes, 0, (size_t) (kvp - M) * row_bytes, stream); + cudaMemsetAsync((char *) v_cpy->data + (size_t) M * row_bytes, 0, (size_t) (kvp - M) * row_bytes, stream); + } + if (rc == 0) { + const int32_t * op = (const int32_t *) k_rope->op_params; + const int n_dims = op[1]; + const int n_ctx_orig = op[4]; + float freq_base, freq_scale, ext_factor, attn_factor, beta_fast, beta_slow; + memcpy(&freq_base, op + 5, sizeof(float)); + memcpy(&freq_scale, op + 6, sizeof(float)); + memcpy(&ext_factor, op + 7, sizeof(float)); + memcpy(&attn_factor, op + 8, sizeof(float)); + memcpy(&beta_fast, op + 9, sizeof(float)); + memcpy(&beta_slow, op + 10, sizeof(float)); + float corr_dims[2]; + ggml_rope_yarn_corr_dims(n_dims, n_ctx_orig, freq_base, beta_fast, beta_slow, corr_dims); + const float theta_scale = powf(freq_base, -2.0f / n_dims); + const float scale_f = ggml_get_op_params_f32(q_scale, 0); + + const ggml_tensor * ff = k_rope->src[2]; + // the rope'd K and plain V rows also land in their graph tensors' + // f32 buffers, feeding the graph-tail persistent-KV store copies + rc = ggml_cuda_flashrt::qkv_post_full( + qkv_cat.get(), (float *) q_scale->data, + k_cpy->data, v_cpy->data, + (float *) k_rope->data, (float *) v_mm->data, + (const int32_t *) k_rope->src[1]->data, + ff != nullptr ? (const float *) ff->data : nullptr, + M, Nk, Nv, Nq, head_dim, n_dims, + freq_scale, ext_factor, attn_factor, + corr_dims[0], corr_dims[1], theta_scale, scale_f, stream); + } + if (rc != 0) { + GGML_ABORT("flashrt: fused prefill qkv failed (M=%d N=%d K=%d rc=%d)", M, N_tot, K, rc); + } +} + // ── Gemma-style norm fold: {RMS_NORM, MUL(w), ADD(mul, norm)} ──────────────── // out = rms_norm(x)*w + rms_norm(x) == rms_norm(x)*(1 + w): the adaLN // modulate kernel with scale = w and shift = 0. ggml's own fused rms_norm diff --git a/flash_rt/structures/adapters/ggml/fr_ggml.cuh b/flash_rt/structures/adapters/ggml/fr_ggml.cuh index 147f1395..1b950582 100644 --- a/flash_rt/structures/adapters/ggml/fr_ggml.cuh +++ b/flash_rt/structures/adapters/ggml/fr_ggml.cuh @@ -104,6 +104,20 @@ bool ggml_cuda_flashrt_should_fuse_gated_res(const ggml_tensor * view, const ggm void ggml_cuda_flashrt_gated_residual(ggml_backend_cuda_context & ctx, const ggml_tensor * view, const ggml_tensor * mul, ggml_tensor * add); +// Prefill fused QKV window: q mm->reshape->rope->scale, k mm->reshape->rope-> +// pad, v mm->reshape->pad, each pad permuted+copied into a padded f16 tensor +// of token rows. One fused GEMM + qkv_post + pad-row zeroing. +bool ggml_cuda_flashrt_should_fuse_qkv_prefill( + const ggml_tensor * q_mm, const ggml_tensor * q_rope, const ggml_tensor * q_scale, + const ggml_tensor * k_mm, const ggml_tensor * k_rope, const ggml_tensor * k_pad, + const ggml_tensor * v_mm, const ggml_tensor * v_pad, + const ggml_tensor * k_cpy, const ggml_tensor * v_cpy); +void ggml_cuda_flashrt_qkv_prefill(ggml_backend_cuda_context & ctx, + const ggml_tensor * q_mm, const ggml_tensor * q_rope, ggml_tensor * q_scale, + const ggml_tensor * k_mm, const ggml_tensor * k_rope, + const ggml_tensor * v_mm, + ggml_tensor * k_cpy, ggml_tensor * v_cpy); + // {RMS_NORM, MUL(w), ADD(mul, norm)} -> rms_norm(x)*(1+w) in one kernel. // The execute returns false (run unfused) only when its zero-vector cache // cannot allocate during graph capture. diff --git a/flash_rt/structures/adapters/ggml/fr_kernels.h b/flash_rt/structures/adapters/ggml/fr_kernels.h index cc8c8423..7a960fd7 100644 --- a/flash_rt/structures/adapters/ggml/fr_kernels.h +++ b/flash_rt/structures/adapters/ggml/fr_kernels.h @@ -89,6 +89,16 @@ int repack_weight_concat3(const void * b0, int N0, const void * b1, int N1, // Fused QKV post: RoPE+f16-store K, f16-store V (into the persistent KV // suffix), RoPE+scale Q (f32 out) from the fused GEMM's [M, Nk+Nv+Nq] rows. +// Variant with optional f32 K/V row outputs (for graphs whose rope'd K / V +// feed additional consumers, e.g. persistent-KV stores at the graph tail). +int qkv_post_full(const float * qkv_cat, float * q_out, void * k_out_f16, void * v_out_f16, + float * k_f32_out, float * v_f32_out, + const int32_t * pos, const float * freq_factors, + int M, int Nk, int Nv, int Nq, int head_dim, int n_dims, + float freq_scale, float ext_factor, float attn_factor, + float corr_low, float corr_high, float theta_scale, float q_scale, + cudaStream_t stream); + int qkv_post(const float * qkv_cat, float * q_out, void * k_out_f16, void * v_out_f16, const int32_t * pos, const float * freq_factors, int M, int Nk, int Nv, int Nq, int head_dim, int n_dims, diff --git a/flash_rt/structures/adapters/ggml/fr_qkv_post.cu b/flash_rt/structures/adapters/ggml/fr_qkv_post.cu index 2195d27e..bdad0a5f 100644 --- a/flash_rt/structures/adapters/ggml/fr_qkv_post.cu +++ b/flash_rt/structures/adapters/ggml/fr_qkv_post.cu @@ -42,6 +42,8 @@ __global__ void kernel_qkv_post(const float * __restrict__ qkv, // [M, Nk+Nv+N float * __restrict__ q_out, // [M, Nq] f32 (head-major rows) __half * __restrict__ k_out, // suffix rows, head_dim per token __half * __restrict__ v_out, + float * __restrict__ k_f32_out, // nullable: rope'd K as f32 rows + float * __restrict__ v_f32_out, // nullable: V as f32 rows const int32_t * __restrict__ pos, const float * __restrict__ freq_factors, // nullable int Nk, int Nv, int Nq, @@ -57,9 +59,12 @@ __global__ void kernel_qkv_post(const float * __restrict__ qkv, // [M, Nk+Nv+N const int p = pos[t]; - // V: plain f16 copy + // V: plain f16 copy (and optionally the f32 row for downstream readers) for (int d = threadIdx.x; d < Nv; d += blockDim.x) { v_out[(int64_t) t * Nv + d] = __float2half(vrow[d]); + if (v_f32_out != nullptr) { + v_f32_out[(int64_t) t * Nv + d] = vrow[d]; + } } // K: rope one head (Nk == head_dim) @@ -68,6 +73,10 @@ __global__ void kernel_qkv_post(const float * __restrict__ qkv, // [M, Nk+Nv+N if (i0 >= n_dims) { k_out[(int64_t) t * Nk + n_dims + (i0 - n_dims)] = __float2half(krow[n_dims + (i0 - n_dims)]); k_out[(int64_t) t * Nk + n_dims + (i0 - n_dims) + 1] = __float2half(krow[n_dims + (i0 - n_dims) + 1]); + if (k_f32_out != nullptr) { + k_f32_out[(int64_t) t * Nk + n_dims + (i0 - n_dims)] = krow[n_dims + (i0 - n_dims)]; + k_f32_out[(int64_t) t * Nk + n_dims + (i0 - n_dims) + 1] = krow[n_dims + (i0 - n_dims) + 1]; + } continue; } const float theta_base = p * powf(theta_scale, i0 / 2.0f); @@ -77,8 +86,14 @@ __global__ void kernel_qkv_post(const float * __restrict__ qkv, // [M, Nk+Nv+N i0, ext_factor, attn_factor, cos_t, sin_t); const float x0 = krow[i0 / 2]; const float x1 = krow[i0 / 2 + n_dims / 2]; - k_out[(int64_t) t * Nk + i0 / 2] = __float2half(x0 * cos_t - x1 * sin_t); - k_out[(int64_t) t * Nk + i0 / 2 + n_dims / 2] = __float2half(x0 * sin_t + x1 * cos_t); + const float k0 = x0 * cos_t - x1 * sin_t; + const float k1 = x0 * sin_t + x1 * cos_t; + k_out[(int64_t) t * Nk + i0 / 2] = __float2half(k0); + k_out[(int64_t) t * Nk + i0 / 2 + n_dims / 2] = __float2half(k1); + if (k_f32_out != nullptr) { + k_f32_out[(int64_t) t * Nk + i0 / 2] = k0; + k_f32_out[(int64_t) t * Nk + i0 / 2 + n_dims / 2] = k1; + } } // Q: rope + scale per head @@ -107,19 +122,33 @@ __global__ void kernel_qkv_post(const float * __restrict__ qkv, // [M, Nk+Nv+N } // namespace -int qkv_post(const float * qkv_cat, float * q_out, void * k_out_f16, void * v_out_f16, - const int32_t * pos, const float * freq_factors, - int M, int Nk, int Nv, int Nq, int head_dim, int n_dims, - float freq_scale, float ext_factor, float attn_factor, - float corr_low, float corr_high, float theta_scale, float q_scale, - cudaStream_t stream) { +int qkv_post_full(const float * qkv_cat, float * q_out, void * k_out_f16, void * v_out_f16, + float * k_f32_out, float * v_f32_out, + const int32_t * pos, const float * freq_factors, + int M, int Nk, int Nv, int Nq, int head_dim, int n_dims, + float freq_scale, float ext_factor, float attn_factor, + float corr_low, float corr_high, float theta_scale, float q_scale, + cudaStream_t stream) { if (n_dims % 2 != 0 || Nk != head_dim || Nq % head_dim != 0) return -1; kernel_qkv_post<<>>( qkv_cat, q_out, (__half *) k_out_f16, (__half *) v_out_f16, + k_f32_out, v_f32_out, pos, freq_factors, Nk, Nv, Nq, head_dim, n_dims, freq_scale, ext_factor, attn_factor, corr_low, corr_high, theta_scale, q_scale); const cudaError_t e = cudaGetLastError(); return (e == cudaSuccess) ? 0 : -static_cast(e); } +int qkv_post(const float * qkv_cat, float * q_out, void * k_out_f16, void * v_out_f16, + const int32_t * pos, const float * freq_factors, + int M, int Nk, int Nv, int Nq, int head_dim, int n_dims, + float freq_scale, float ext_factor, float attn_factor, + float corr_low, float corr_high, float theta_scale, float q_scale, + cudaStream_t stream) { + return qkv_post_full(qkv_cat, q_out, k_out_f16, v_out_f16, nullptr, nullptr, + pos, freq_factors, M, Nk, Nv, Nq, head_dim, n_dims, + freq_scale, ext_factor, attn_factor, corr_low, corr_high, + theta_scale, q_scale, stream); +} + } // namespace ggml_cuda_flashrt From e5cf117ac7250892cbb6d274273fc5ea66cbc5a1 Mon Sep 17 00:00:00 2001 From: LiangSu8899 Date: Mon, 24 Aug 2026 14:50:58 -0400 Subject: [PATCH 09/32] feat(fp4): fp32-bias fp16-out GEMM variant; vision QKV window emits f16 K/V directly gemm_bias_f16out converts once from the fp32 accumulator in the epilogue, bit-equal to an fp32-out GEMM followed by an fp16 cast. The ggml adapter's vision QKV window uses it for the K/V legs when their cast tensors are handed in, eliminating the two full-tensor cast copies per layer. --- ...cutlass_fp4_gemm_bias_f32b_f16out_sm100.cu | 153 ++++++++++++++++++ ...utlass_fp4_gemm_bias_f32b_f16out_sm100.cuh | 30 ++++ .../structures/adapters/ggml/fr_dispatch.cu | 27 +++- flash_rt/structures/adapters/ggml/fr_ggml.cuh | 3 +- 4 files changed, 204 insertions(+), 9 deletions(-) create mode 100644 csrc/gemm/fp4/cutlass_fp4_gemm_bias_f32b_f16out_sm100.cu create mode 100644 csrc/gemm/fp4/cutlass_fp4_gemm_bias_f32b_f16out_sm100.cuh diff --git a/csrc/gemm/fp4/cutlass_fp4_gemm_bias_f32b_f16out_sm100.cu b/csrc/gemm/fp4/cutlass_fp4_gemm_bias_f32b_f16out_sm100.cu new file mode 100644 index 00000000..74b20a1c --- /dev/null +++ b/csrc/gemm/fp4/cutlass_fp4_gemm_bias_f32b_f16out_sm100.cu @@ -0,0 +1,153 @@ +// ============================================================================ +// FlashRT — NVFP4 GEMM with fp32 per-column bias and fp16 output +// (SM100/SM110). See header for the contract. +// ============================================================================ + +#include "gemm/fp4/cutlass_fp4_gemm_bias_f32b_f16out_sm100.cuh" + +#include "cutlass/cutlass.h" +#include "cutlass/epilogue/dispatch_policy.hpp" +#include "cutlass/epilogue/fusion/operations.hpp" +#include "cutlass/gemm/dispatch_policy.hpp" +#include "cutlass/gemm/collective/collective_builder.hpp" +#include "cutlass/epilogue/collective/collective_builder.hpp" +#include "cutlass/gemm/device/gemm_universal_adapter.h" +#include "cutlass/gemm/kernel/gemm_universal.hpp" +#include "cutlass/util/packed_stride.hpp" +#include "cutlass/detail/sm100_blockscaled_layout.hpp" +#include "cute/tensor.hpp" + +#include +#include + +namespace flash_rt { +namespace fp4 { + +namespace bias_f16out { + +using namespace cute; + +using ElementA = cutlass::nv_float4_t; +using LayoutATag = cutlass::layout::RowMajor; +constexpr int AlignmentA = 32; + +using ElementB = cutlass::nv_float4_t; +using LayoutBTag = cutlass::layout::ColumnMajor; +constexpr int AlignmentB = 32; + +using ElementAccumulator = float; +using ElementCompute = float; +using ArchTag = cutlass::arch::Sm100; +using OperatorClass = cutlass::arch::OpClassBlockScaledTensorOp; + +using ElementD = cutlass::half_t; +using ElementC = cutlass::half_t; +constexpr int AlignmentCD = 8; + +using MmaTileShape = Shape<_128, _128, _256>; +using ClusterShape = Shape<_1, _1, _1>; + +// per-shape CUTLASS workspace cache (capture-safe: growth happens during +// the uncaptured warmup evaluation) +struct ws_key { + int M, N, K; + bool operator==(const ws_key & o) const { return M == o.M && N == o.N && K == o.K; } +}; +struct ws_key_hash { + size_t operator()(const ws_key & k) const noexcept { + return (size_t) k.M * 1315423911u ^ (size_t) k.N * 2654435761u ^ (size_t) k.K; + } +}; +inline void * get_ws(int M, int N, int K, size_t needed) { + static std::unordered_map, ws_key_hash> cache; + static std::mutex mu; + std::lock_guard lk(mu); + auto & e = cache[ws_key{M, N, K}]; + if (e.second < needed) { + if (e.first) { cudaFree(e.first); } + cudaMalloc(&e.first, needed); + e.second = needed; + } + return e.first; +} + +using FusionOperation = cutlass::epilogue::fusion::LinCombPerColBias< + ElementD, ElementCompute, float, ElementC, ElementCompute>; + +using CollectiveEpilogue = + typename cutlass::epilogue::collective::CollectiveBuilder< + ArchTag, OperatorClass, MmaTileShape, ClusterShape, + cutlass::epilogue::collective::EpilogueTileAuto, + ElementAccumulator, ElementAccumulator, + ElementC, cutlass::layout::RowMajor, AlignmentCD, + ElementD, cutlass::layout::RowMajor, AlignmentCD, + cutlass::epilogue::collective::EpilogueScheduleAuto, + FusionOperation>::CollectiveOp; + +using CollectiveMainloop = + typename cutlass::gemm::collective::CollectiveBuilder< + ArchTag, OperatorClass, + ElementA, LayoutATag, AlignmentA, + ElementB, LayoutBTag, AlignmentB, + ElementAccumulator, MmaTileShape, ClusterShape, + cutlass::gemm::collective::StageCountAutoCarveout( + sizeof(typename CollectiveEpilogue::SharedStorage))>, + cutlass::gemm::collective::KernelScheduleAuto>::CollectiveOp; + +using GemmKernel = cutlass::gemm::kernel::GemmUniversal< + Shape, CollectiveMainloop, CollectiveEpilogue, void>; +using Gemm = cutlass::gemm::device::GemmUniversalAdapter; + +} // namespace bias_f16out + +int gemm_bias_f16out( + const void * A_packed, const void * SFA, + const void * B_packed, const void * SFB, + const void * bias_f32, void * D_f16, + int M, int N, int K, + cudaStream_t stream) { + using namespace bias_f16out; + + auto stride_A = cutlass::make_cute_packed_stride( + typename Gemm::GemmKernel::StrideA{}, {M, K, 1}); + auto stride_B = cutlass::make_cute_packed_stride( + typename Gemm::GemmKernel::StrideB{}, {N, K, 1}); + auto stride_C = cutlass::make_cute_packed_stride( + typename Gemm::GemmKernel::StrideC{}, {M, N, 1}); + auto stride_D = cutlass::make_cute_packed_stride( + typename Gemm::GemmKernel::StrideD{}, {M, N, 1}); + using Cfg = + typename Gemm::GemmKernel::CollectiveMainloop::Sm1xxBlkScaledConfig; + auto layout_SFA = Cfg::tile_atom_to_shape_SFA(make_shape(M, N, K, 1)); + auto layout_SFB = Cfg::tile_atom_to_shape_SFB(make_shape(M, N, K, 1)); + + using EA = typename ElementA::DataType; + using SA = typename ElementA::ScaleFactorType; + + typename Gemm::Arguments args{ + cutlass::gemm::GemmUniversalMode::kGemm, {M, N, K, 1}, + {reinterpret_cast(A_packed), stride_A, + reinterpret_cast(B_packed), stride_B, + reinterpret_cast(SFA), layout_SFA, + reinterpret_cast(SFB), layout_SFB}, + {{}, + reinterpret_cast(D_f16), stride_C, + reinterpret_cast(D_f16), stride_D}}; + args.epilogue.thread.alpha = 1.0f; + args.epilogue.thread.beta = 0.0f; + args.epilogue.thread.bias_ptr = reinterpret_cast(bias_f32); + + Gemm gemm; + auto st = gemm.can_implement(args); + if (st != cutlass::Status::kSuccess) return static_cast(st) | 0x10000; + size_t ws_sz = Gemm::get_workspace_size(args); + void* ws = ws_sz > 0 ? get_ws(M, N, K, ws_sz) : nullptr; + st = gemm.initialize(args, ws, stream); + if (st != cutlass::Status::kSuccess) return static_cast(st) | 0x20000; + st = gemm.run(stream); + return (st == cutlass::Status::kSuccess) ? 0 + : (static_cast(st) | 0x30000); +} + +} // namespace fp4 +} // namespace flash_rt diff --git a/csrc/gemm/fp4/cutlass_fp4_gemm_bias_f32b_f16out_sm100.cuh b/csrc/gemm/fp4/cutlass_fp4_gemm_bias_f32b_f16out_sm100.cuh new file mode 100644 index 00000000..fd1cab03 --- /dev/null +++ b/csrc/gemm/fp4/cutlass_fp4_gemm_bias_f32b_f16out_sm100.cuh @@ -0,0 +1,30 @@ +// ============================================================================ +// FlashRT — NVFP4 GEMM with fp32 per-column bias and fp16 output +// (SM100/SM110). +// +// D_f16[M, N] = A @ B^T + bias[N]. For hosts that keep biases in fp32 but +// consume the projection in fp16 (e.g. attention inputs cast for flash +// attention): the fp16 conversion happens once in the epilogue from the +// fp32 accumulator, matching an fp32-output GEMM followed by an fp16 cast +// bit for bit. +// ============================================================================ +#pragma once + +#include + +namespace flash_rt { +namespace fp4 { + +// A: [M, K] NVFP4 packed row-major + SFA (tile-interleaved). +// B: [N, K] NVFP4 packed column-major + SFB. +// bias_f32: [N] fp32, broadcast over rows. D_f16: [M, N] fp16 row-major. +// Returns 0 on success; CUTLASS status | stage flag otherwise. +int gemm_bias_f16out( + const void * A_packed, const void * SFA, + const void * B_packed, const void * SFB, + const void * bias_f32, void * D_f16, + int M, int N, int K, + cudaStream_t stream); + +} // namespace fp4 +} // namespace flash_rt diff --git a/flash_rt/structures/adapters/ggml/fr_dispatch.cu b/flash_rt/structures/adapters/ggml/fr_dispatch.cu index 88f0295f..9317903c 100644 --- a/flash_rt/structures/adapters/ggml/fr_dispatch.cu +++ b/flash_rt/structures/adapters/ggml/fr_dispatch.cu @@ -8,6 +8,7 @@ // (GGML_CUDA_FLASHRT_PUBLIC_DIR); no vendored copies. #include "gemm/fp4/cutlass_fp4_gemm_geglu_il_sm100.cuh" #include "gemm/fp4/cutlass_fp4_gemm_siglip_ffn_f32out_sm100.cuh" +#include "gemm/fp4/cutlass_fp4_gemm_bias_f32b_f16out_sm100.cuh" #include #include @@ -1157,7 +1158,8 @@ bool ggml_cuda_flashrt_should_fuse_vis_qkv_pad( void ggml_cuda_flashrt_vis_qkv_pad(ggml_backend_cuda_context & ctx, const ggml_tensor * mm_q, const ggml_tensor * add_q, ggml_tensor * pad_q, const ggml_tensor * mm_k, const ggml_tensor * add_k, ggml_tensor * pad_k, - const ggml_tensor * mm_v, const ggml_tensor * add_v, ggml_tensor * pad_v) { + const ggml_tensor * mm_v, const ggml_tensor * add_v, ggml_tensor * pad_v, + ggml_tensor * k_cast, ggml_tensor * v_cast) { cudaStream_t stream = ctx.stream(); const ggml_tensor * src1 = mm_q->src[1]; @@ -1185,17 +1187,26 @@ void ggml_cuda_flashrt_vis_qkv_pad(ggml_backend_cuda_context & ctx, q_sf = a_sf.get(); } - const ggml_tensor * legs[3][3] = { - { mm_q, add_q, pad_q }, - { mm_k, add_k, pad_k }, - { mm_v, add_v, pad_v }, + // K/V may go straight to their f16 cast tensors (single rounding from + // the fp32 accumulator, bitwise equal to f32-out + cast); Q stays f32. + const ggml_tensor * legs[3][4] = { + { mm_q, add_q, pad_q, nullptr }, + { mm_k, add_k, pad_k, k_cast }, + { mm_v, add_v, pad_v, v_cast }, }; for (auto & leg : legs) { const grouppad_weight * w = get_repacked_grouppad( leg[0]->src[0], leg[1]->src[1], group_in, group_out, n_groups, stream); - const int rc = flash_rt::fp4::gemm_bias_f32out( - q_packed, q_sf, w->packed, w->sf, w->bias, - (float *) leg[2]->data, M, N_pad, K, stream); + int rc; + if (leg[3] != nullptr) { + rc = flash_rt::fp4::gemm_bias_f16out( + q_packed, q_sf, w->packed, w->sf, w->bias, + leg[3]->data, M, N_pad, K, stream); + } else { + rc = flash_rt::fp4::gemm_bias_f32out( + q_packed, q_sf, w->packed, w->sf, w->bias, + (float *) leg[2]->data, M, N_pad, K, stream); + } if (rc != 0) { GGML_ABORT("flashrt: vis qkv padded gemm failed (M=%d N=%d K=%d rc=%d)", M, N_pad, K, rc); } diff --git a/flash_rt/structures/adapters/ggml/fr_ggml.cuh b/flash_rt/structures/adapters/ggml/fr_ggml.cuh index 1b950582..059f4908 100644 --- a/flash_rt/structures/adapters/ggml/fr_ggml.cuh +++ b/flash_rt/structures/adapters/ggml/fr_ggml.cuh @@ -91,7 +91,8 @@ bool ggml_cuda_flashrt_should_fuse_vis_qkv_pad( void ggml_cuda_flashrt_vis_qkv_pad(ggml_backend_cuda_context & ctx, const ggml_tensor * mm_q, const ggml_tensor * add_q, ggml_tensor * pad_q, const ggml_tensor * mm_k, const ggml_tensor * add_k, ggml_tensor * pad_k, - const ggml_tensor * mm_v, const ggml_tensor * add_v, ggml_tensor * pad_v); + const ggml_tensor * mm_v, const ggml_tensor * add_v, ggml_tensor * pad_v, + ggml_tensor * k_cast, ggml_tensor * v_cast); // GEMM + optional bias + residual add fused into one epilogue. bool ggml_cuda_flashrt_should_fuse_mm_res(const ggml_tensor * mm, const ggml_tensor * bias_add, From b70ee45e11189f1e9dbd713bee5ae3687fae9f97 Mon Sep 17 00:00:00 2001 From: LiangSu8899 Date: Mon, 24 Aug 2026 15:17:33 -0400 Subject: [PATCH 10/32] feat(ggml-adapter): decomposed tiny-M decode attention MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit FLASH_ATTN_EXT with q_tokens <= 16 over a single padded f16 KV head runs as a batched QK^T GEMM (GQA heads share the stride-0 K operand), a masked fp32-accurate softmax, and a batched PV GEMM whose fp32 result lands directly in the node's [hd, n_head, n_tok] layout via a strided C — the same decomposition the torch pipeline uses at these shapes, where it beats the stream-k flash kernel plus its fixup pass. Opt-out via GGML_FLASHRT_NO_DEC_ATTN. --- .../adapters/ggml/fr_decode_attn.cu | 153 ++++++++++++++++++ .../structures/adapters/ggml/fr_dispatch.cu | 88 ++++++++++ flash_rt/structures/adapters/ggml/fr_ggml.cuh | 6 + .../structures/adapters/ggml/fr_kernels.h | 14 ++ 4 files changed, 261 insertions(+) create mode 100644 flash_rt/structures/adapters/ggml/fr_decode_attn.cu diff --git a/flash_rt/structures/adapters/ggml/fr_decode_attn.cu b/flash_rt/structures/adapters/ggml/fr_decode_attn.cu new file mode 100644 index 00000000..35ee249d --- /dev/null +++ b/flash_rt/structures/adapters/ggml/fr_decode_attn.cu @@ -0,0 +1,153 @@ +// Decomposed tiny-M decode attention for the pi0.5 action expert (Thor). +// +// For q_tokens ≤ 16 over a padded f16 KV of token rows, flash attention's +// stream-k kernel plus its fixup pass is slower than the classic +// decomposition the FlashRT torch pipeline uses: one batched QK^T GEMM +// (all GQA query heads share the single KV head, so the K operand batches +// with stride 0), a masked softmax over the KV axis, and one batched PV +// GEMM whose fp32 output lands directly in the flash-attention node's +// [hd, n_head, n_tok] layout via a strided C. +// +// Numerics follow ggml's fattn contract: scores = scale * q.k + mask (f16 +// mask, slope 1 as max_bias must be 0), softmax in fp32 with running max. + +#include "fr_kernels.h" + +#include +#include + +namespace ggml_cuda_flashrt { + +namespace { + +// gather the permuted f32 Q view into contiguous f16 rows [n_head*n_tok, hd] +// (row r = h*n_tok + t), applying nothing else (scale folds into QK alpha) +__global__ void kernel_q_gather_f16(const float * __restrict__ q, + __half * __restrict__ out, + int hd, int n_tok, int n_head, + int64_t s_d, int64_t s_tok, int64_t s_head) { + const int r = blockIdx.x; // h*n_tok + t + const int h = r / n_tok; + const int t = r % n_tok; + const float * src = q + (int64_t) h * s_head + (int64_t) t * s_tok; + __half * dst = out + (int64_t) r * hd; + for (int d = threadIdx.x; d < hd; d += blockDim.x) { + dst[d] = __float2half(src[(int64_t) d * s_d]); + } +} + +// in-place masked softmax over rows of [n_head*n_tok, n_kv] f16 scores. +// mask element for (kv, t) at mask + kv + t*mask_stride (f16, -inf on pads). +__global__ void kernel_mask_softmax_f16(__half * __restrict__ scores, + const __half * __restrict__ mask, + int n_kv, int n_tok, int64_t mask_stride) { + const int r = blockIdx.x; // h*n_tok + t + const int t = r % n_tok; + __half * row = scores + (int64_t) r * n_kv; + const __half * mrow = mask + (int64_t) t * mask_stride; + + float m = -INFINITY; + for (int i = threadIdx.x; i < n_kv; i += blockDim.x) { + const float v = __half2float(row[i]) + __half2float(mrow[i]); + m = fmaxf(m, v); + } + __shared__ float red[32]; + #pragma unroll + for (int off = 16; off > 0; off >>= 1) { + m = fmaxf(m, __shfl_xor_sync(0xffffffff, m, off)); + } + if (threadIdx.x % 32 == 0) red[threadIdx.x / 32] = m; + __syncthreads(); + if (threadIdx.x < 32) { + float v = (threadIdx.x < blockDim.x / 32) ? red[threadIdx.x] : -INFINITY; + #pragma unroll + for (int off = 16; off > 0; off >>= 1) { + v = fmaxf(v, __shfl_xor_sync(0xffffffff, v, off)); + } + if (threadIdx.x == 0) red[0] = v; + } + __syncthreads(); + m = red[0]; + + float sum = 0.0f; + for (int i = threadIdx.x; i < n_kv; i += blockDim.x) { + const float v = __half2float(row[i]) + __half2float(mrow[i]); + const float e = expf(v - m); + row[i] = __float2half(e); + sum += e; + } + __shared__ float red2[32]; + #pragma unroll + for (int off = 16; off > 0; off >>= 1) { + sum += __shfl_xor_sync(0xffffffff, sum, off); + } + if (threadIdx.x % 32 == 0) red2[threadIdx.x / 32] = sum; + __syncthreads(); + if (threadIdx.x < 32) { + float v = (threadIdx.x < blockDim.x / 32) ? red2[threadIdx.x] : 0.0f; + #pragma unroll + for (int off = 16; off > 0; off >>= 1) { + v += __shfl_xor_sync(0xffffffff, v, off); + } + if (threadIdx.x == 0) red2[0] = v; + } + __syncthreads(); + const float inv = 1.0f / red2[0]; + for (int i = threadIdx.x; i < n_kv; i += blockDim.x) { + row[i] = __float2half(__half2float(row[i]) * inv); + } +} + +} // namespace + +int decode_attn_decomposed(void * cublas_handle, + const float * q, int64_t q_sd, int64_t q_stok, int64_t q_shead, + const void * k_f16_rows, // [n_kv, hd] f16 rows + const void * v_f16_rows, // [n_kv, hd] f16 rows + const void * mask_f16, int64_t mask_stride, + float * dst, int64_t dst_stok, int64_t dst_shead, + void * q16_ws, void * scores_ws, + int hd, int n_tok, int n_head, int n_kv, + float scale, cudaStream_t stream) { + cublasHandle_t handle = (cublasHandle_t) cublas_handle; + const int R = n_head * n_tok; + + kernel_q_gather_f16<<>>( + q, (__half *) q16_ws, hd, n_tok, n_head, q_sd, q_stok, q_shead); + + cublasSetStream(handle, stream); + // scores_col[n_kv, n_tok] per head = K_col^T [n_kv, hd] x Q_col [hd, n_tok] + const float beta0 = 0.0f; + cublasStatus_t st = cublasGemmStridedBatchedEx( + handle, CUBLAS_OP_T, CUBLAS_OP_N, + n_kv, n_tok, hd, + &scale, + k_f16_rows, CUDA_R_16F, hd, 0, + q16_ws, CUDA_R_16F, hd, (int64_t) n_tok * hd, + &beta0, + scores_ws, CUDA_R_16F, n_kv, (int64_t) n_tok * n_kv, + n_head, CUBLAS_COMPUTE_32F, CUBLAS_GEMM_DEFAULT); + if (st != CUBLAS_STATUS_SUCCESS) return -100 - (int) st; + + kernel_mask_softmax_f16<<>>( + (__half *) scores_ws, (const __half *) mask_f16, n_kv, n_tok, mask_stride); + + // out_col[hd, n_tok] per head (ldc = dst token stride) = + // V_col [hd, n_kv] x P_col [n_kv, n_tok] + const float one = 1.0f; + st = cublasGemmStridedBatchedEx( + handle, CUBLAS_OP_N, CUBLAS_OP_N, + hd, n_tok, n_kv, + &one, + v_f16_rows, CUDA_R_16F, hd, 0, + scores_ws, CUDA_R_16F, n_kv, (int64_t) n_tok * n_kv, + &beta0, + dst, CUDA_R_32F, (int) dst_stok, dst_shead, + n_head, CUBLAS_COMPUTE_32F, CUBLAS_GEMM_DEFAULT); + if (st != CUBLAS_STATUS_SUCCESS) return -200 - (int) st; + + const cudaError_t e = cudaGetLastError(); + return (e == cudaSuccess) ? 0 : -static_cast(e); +} + +} // namespace ggml_cuda_flashrt diff --git a/flash_rt/structures/adapters/ggml/fr_dispatch.cu b/flash_rt/structures/adapters/ggml/fr_dispatch.cu index 9317903c..9409a493 100644 --- a/flash_rt/structures/adapters/ggml/fr_dispatch.cu +++ b/flash_rt/structures/adapters/ggml/fr_dispatch.cu @@ -1480,6 +1480,94 @@ void ggml_cuda_flashrt_qkv_prefill(ggml_backend_cuda_context & ctx, } } +// ── Decomposed tiny-M decode attention ────────────────────────────────────── +// FLASH_ATTN_EXT with q_tokens <= 16, single f16 KV head of token rows, an +// f16 mask and no ALiBi/softcap runs as QK-GEMM + masked softmax + PV-GEMM +// (see fr_decode_attn.cu). Faster than the stream-k fattn + fixup pair at +// these shapes. +bool ggml_cuda_flashrt_should_fuse_dec_attn(const ggml_tensor * fa) { + static const bool disabled = getenv("GGML_FLASHRT_NO_DEC_ATTN") != nullptr; + if (disabled) { + return false; + } + const ggml_tensor * q = fa->src[0]; + const ggml_tensor * k = fa->src[1]; + const ggml_tensor * v = fa->src[2]; + const ggml_tensor * mask = fa->src[3]; + if (q == nullptr || k == nullptr || v == nullptr || mask == nullptr || + fa->src[4] != nullptr) { // no attention sinks + return false; + } + const int64_t hd = q->ne[0]; + const int64_t n_tok = q->ne[1]; + const int64_t n_head = q->ne[2]; + const int64_t n_kv = k->ne[1]; + if (q->type != GGML_TYPE_F32 || n_tok > 16 || q->ne[3] != 1 || + hd % 2 != 0 || n_head < 1) { + return false; + } + for (const ggml_tensor * kv : { k, v }) { + if (kv->type != GGML_TYPE_F16 || kv->ne[0] != hd || kv->ne[2] != 1 || + kv->ne[3] != 1 || kv->nb[0] != sizeof(uint16_t) || + (int64_t) kv->nb[1] != hd * (int64_t) sizeof(uint16_t)) { + return false; + } + } + if (v->ne[1] != n_kv) { + return false; + } + if (mask->type != GGML_TYPE_F16 || mask->ne[0] < n_kv || mask->ne[1] < n_tok) { + return false; + } + if (fa->type != GGML_TYPE_F32 || fa->ne[0] != hd || fa->ne[1] != n_head || + fa->ne[2] != n_tok || fa->nb[0] != sizeof(float) || + (int64_t) fa->nb[1] != hd * (int64_t) sizeof(float)) { + return false; + } + float max_bias, softcap; + memcpy(&max_bias, (const float *) fa->op_params + 1, sizeof(float)); + memcpy(&softcap, (const float *) fa->op_params + 2, sizeof(float)); + if (max_bias != 0.0f || softcap != 0.0f) { + return false; + } + return true; +} + +void ggml_cuda_flashrt_dec_attn(ggml_backend_cuda_context & ctx, ggml_tensor * fa) { + const ggml_tensor * q = fa->src[0]; + const ggml_tensor * k = fa->src[1]; + const ggml_tensor * v = fa->src[2]; + const ggml_tensor * mask = fa->src[3]; + const int hd = (int) q->ne[0]; + const int n_tok = (int) q->ne[1]; + const int n_head = (int) q->ne[2]; + const int n_kv = (int) k->ne[1]; + float scale; + memcpy(&scale, (const float *) fa->op_params + 0, sizeof(float)); + cudaStream_t stream = ctx.stream(); + + const int64_t R = (int64_t) n_head * n_tok; + ggml_cuda_pool_alloc q16 (ctx.pool(), R * hd * sizeof(uint16_t)); + ggml_cuda_pool_alloc scores(ctx.pool(), R * n_kv * sizeof(uint16_t)); + + const int rc = ggml_cuda_flashrt::decode_attn_decomposed( + (void *) ctx.cublas_handle(), + (const float *) q->data, + (int64_t) (q->nb[0] / sizeof(float)), + (int64_t) (q->nb[1] / sizeof(float)), + (int64_t) (q->nb[2] / sizeof(float)), + k->data, v->data, + mask->data, (int64_t) (mask->nb[1] / sizeof(uint16_t)), + (float *) fa->data, + (int64_t) (fa->nb[2] / sizeof(float)), + (int64_t) (fa->nb[1] / sizeof(float)), + q16.get(), scores.get(), + hd, n_tok, n_head, n_kv, scale, stream); + if (rc != 0) { + GGML_ABORT("flashrt: decomposed decode attention failed (tok=%d kv=%d rc=%d)", n_tok, n_kv, rc); + } +} + // ── Gemma-style norm fold: {RMS_NORM, MUL(w), ADD(mul, norm)} ──────────────── // out = rms_norm(x)*w + rms_norm(x) == rms_norm(x)*(1 + w): the adaLN // modulate kernel with scale = w and shift = 0. ggml's own fused rms_norm diff --git a/flash_rt/structures/adapters/ggml/fr_ggml.cuh b/flash_rt/structures/adapters/ggml/fr_ggml.cuh index 059f4908..bf33df67 100644 --- a/flash_rt/structures/adapters/ggml/fr_ggml.cuh +++ b/flash_rt/structures/adapters/ggml/fr_ggml.cuh @@ -105,6 +105,12 @@ bool ggml_cuda_flashrt_should_fuse_gated_res(const ggml_tensor * view, const ggm void ggml_cuda_flashrt_gated_residual(ggml_backend_cuda_context & ctx, const ggml_tensor * view, const ggml_tensor * mul, ggml_tensor * add); +// Decomposed tiny-M decode attention: replaces a FLASH_ATTN_EXT node with +// QK-GEMM + masked softmax + PV-GEMM for q_tokens <= 16 over a single f16 +// KV head of token rows. +bool ggml_cuda_flashrt_should_fuse_dec_attn(const ggml_tensor * fa); +void ggml_cuda_flashrt_dec_attn(ggml_backend_cuda_context & ctx, ggml_tensor * fa); + // Prefill fused QKV window: q mm->reshape->rope->scale, k mm->reshape->rope-> // pad, v mm->reshape->pad, each pad permuted+copied into a padded f16 tensor // of token rows. One fused GEMM + qkv_post + pad-row zeroing. diff --git a/flash_rt/structures/adapters/ggml/fr_kernels.h b/flash_rt/structures/adapters/ggml/fr_kernels.h index 7a960fd7..5374f6c6 100644 --- a/flash_rt/structures/adapters/ggml/fr_kernels.h +++ b/flash_rt/structures/adapters/ggml/fr_kernels.h @@ -130,6 +130,20 @@ int layer_norm_affine_quant(const float * x, const float * w, const float * b, int layer_norm_affine(const float * x, const float * w, const float * b, float * out, int M, int C, float eps, cudaStream_t stream); +// Decomposed tiny-M attention: batched QK^T (f16, GQA heads share the +// stride-0 K operand) + masked softmax + batched PV with fp32 output +// written directly in the [hd, n_head, n_tok] layout via strided C. +// q strides are in elements; workspaces: q16 [n_head*n_tok, hd] f16, +// scores [n_head*n_tok, n_kv] f16. cublas_handle is a cublasHandle_t. +int decode_attn_decomposed(void * cublas_handle, + const float * q, int64_t q_sd, int64_t q_stok, int64_t q_shead, + const void * k_f16_rows, const void * v_f16_rows, + const void * mask_f16, int64_t mask_stride, + float * dst, int64_t dst_stok, int64_t dst_shead, + void * q16_ws, void * scores_ws, + int hd, int n_tok, int n_head, int n_kv, + float scale, cudaStream_t stream); + // out[i] = a[i] + b[i] for n fp32 elements. int vec_add_f32(const float * a, const float * b, float * out, int n, cudaStream_t stream); From 29ef196de0f51f63e87563b4940399d6025d5ad1 Mon Sep 17 00:00:00 2001 From: LiangSu8899 Date: Mon, 24 Aug 2026 15:26:03 -0400 Subject: [PATCH 11/32] chore(ggml-adapter): refresh the e2e action golden The rms_gemma and decomposed decode attention windows change the action output at the ulp-amplification level (both judged against the FP16 reference on gripper-active real frames); the golden follows the verified current state. All gates green. --- .../adapters/ggml/qualification/goldens/pi05_thor_action.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flash_rt/structures/adapters/ggml/qualification/goldens/pi05_thor_action.json b/flash_rt/structures/adapters/ggml/qualification/goldens/pi05_thor_action.json index e72c460f..e41f869a 100644 --- a/flash_rt/structures/adapters/ggml/qualification/goldens/pi05_thor_action.json +++ b/flash_rt/structures/adapters/ggml/qualification/goldens/pi05_thor_action.json @@ -1 +1 @@ -{"action_final_raw": [[0.13445311784744263, -0.22609688341617584, 0.09673481434583664, -0.48897215723991394, -0.20383907854557037, 0.059894900768995285, 0.008553847670555115, 0.014760290272533894, 0.017829006537795067, 0.005666017532348633, 0.0018993731355294585, 0.003710060380399227, 0.0011131869396194816, 0.021037299185991287, 0.008911199867725372, 0.009632907807826996, -0.005290915258228779, -0.012890446931123734, 0.010225889272987843, 0.008097179234027863, 0.010170763358473778, 0.006614642683416605, -0.0010463700164109468, -0.006414866540580988, 0.0025974814780056477, -0.0035260769072920084, 0.001079554669559002, -0.007584728300571442, 0.004214358516037464, 0.005772009491920471, -0.00444263918325305, -0.00495037529617548], [0.153213769197464, -0.2348431646823883, 0.0752343162894249, -0.532599151134491, -0.2245737612247467, 0.07155972719192505, -0.005815661512315273, -0.005690991412848234, 0.00688743544742465, 0.0006532398401759565, -0.009447730146348476, 0.0024954203981906176, -0.011725224554538727, 0.026050876826047897, -0.005367973353713751, -0.003108944743871689, -0.01864856481552124, 0.015692617744207382, 0.0027790614403784275, -0.00055804563453421, 0.0023957756347954273, -0.0038551436737179756, -0.0005429460434243083, -0.009038927964866161, -0.009726220741868019, 0.004309936426579952, -0.008310661651194096, -0.011007283814251423, 0.007427852135151625, -0.002252178266644478, 0.0020292296539992094, -0.00646913331001997], [0.15412196516990662, -0.2424684315919876, 0.09056395292282104, -0.5615240335464478, -0.22581106424331665, 0.06655687838792801, 0.004199258051812649, 0.012291430495679379, 0.006383476313203573, -0.004499392583966255, 0.000584578316193074, 0.012444681487977505, 0.022909963503479958, 0.02723228745162487, 0.003040420589968562, -0.004673979710787535, -0.014560073614120483, -0.008193220011889935, 0.017965083941817284, 0.006782323122024536, -0.0006875474355183542, -0.0018831955967471004, 0.0039725108072161674, 0.00507722795009613, -0.008768548257648945, -0.001825154758989811, -0.004630640149116516, 0.0012659032363444567, 0.007830733433365822, -0.00014293950516730547, -0.0018628238467499614, 0.007962911389768124], [0.17018526792526245, -0.2378462255001068, 0.1038762629032135, -0.613089919090271, -0.24281015992164612, 0.07278838008642197, 0.018476080149412155, 0.027088085189461708, 0.002266970230266452, 0.007945772260427475, 0.015290194191038609, -0.0042078737169504166, 0.005958756897598505, 0.010867411270737648, -0.0013605238636955619, -0.008001739159226418, -0.006307597737759352, 0.004680096171796322, 0.0032985578291118145, 0.0022222467232495546, -0.005925439763814211, 0.002563274232670665, 0.008856181986629963, 0.003001272212713957, 0.0028560697101056576, 0.01327639538794756, -0.003269349690526724, 0.0020344506483525038, 0.0034366112668067217, 0.007903024554252625, -0.009116885252296925, 0.01662973128259182], [0.1909807175397873, -0.2396012395620346, 0.10089189559221268, -0.6470962166786194, -0.24781852960586548, 0.06777454167604446, -0.004258676432073116, 0.0023328678216785192, 0.0025521176867187023, 0.011204604990780354, -0.004636475350707769, 0.00019298936240375042, 0.000634241325315088, 0.0334710031747818, -0.00512863090261817, 0.0033141777385026217, -0.024761488661170006, -0.00896375346928835, 0.023857535794377327, 0.013167213648557663, 0.004921273794025183, 0.00802926067262888, -0.0002448978484608233, 0.004498007707297802, -0.008802114985883236, -0.0018076369306072593, -0.0021204198710620403, -0.009473863057792187, 0.004197344183921814, -0.008038798347115517, -0.0058691417798399925, -0.004124474246054888], [0.19594214856624603, -0.2469625473022461, 0.10335008800029755, -0.6753498315811157, -0.2431798279285431, 0.09148994088172913, -0.0007321917801164091, 0.01651609316468239, 0.01433064229786396, 0.016656866297125816, -0.00768603989854455, 0.012770654633641243, 0.006066967733204365, 0.023857075721025467, -0.00360122206620872, -0.006528365425765514, -0.02446194551885128, -0.011438625864684582, 0.01774892397224903, 0.010402456857264042, -0.010078227147459984, 0.004601590801030397, -0.0051121884025633335, 0.0019120099022984505, 0.0005054213688708842, -0.002769842278212309, -0.010812192223966122, -0.012320353649556637, 0.005433861166238785, -0.006443703081458807, -0.0023747680243104696, 0.002368318848311901], [0.21122439205646515, -0.230372354388237, 0.11299057304859161, -0.6827223300933838, -0.25735557079315186, 0.1060398668050766, -0.017443519085645676, 0.008841664530336857, 0.007297935429960489, 0.00346162891946733, 0.0036726801190525293, -0.0007329204236157238, 0.0153369614854455, 0.020901501178741455, -0.0017678173026069999, 0.0019413013942539692, -0.025271348655223846, -0.0027492677327245474, 0.015792900696396828, 0.006516820285469294, 0.0008242192561738193, 0.006501537747681141, -0.006030264310538769, 0.002991343615576625, -0.0005614488618448377, 0.0032644555903971195, -0.010776827111840248, -0.014541511423885822, -0.015579303726553917, 0.005722007714211941, -0.009285654872655869, -0.015917623415589333], [0.2255934178829193, -0.2563740909099579, 0.09825213998556137, -0.7078331112861633, -0.25863707065582275, 0.09265213459730148, 0.009679882787168026, 0.011851120740175247, 0.01195165328681469, -0.001129768555983901, -0.0039059848058968782, 0.009348861873149872, 0.007520776242017746, 0.013013992458581924, -0.004669691435992718, -0.0028344085440039635, -0.017348486930131912, -0.007212277036160231, 0.018364541232585907, 0.005154776852577925, 0.00037904095370322466, 0.002544970251619816, -0.004483690485358238, -0.016937261447310448, -0.00886005163192749, 0.015370937995612621, -0.0010434178402647376, -0.006074860692024231, -0.006481626071035862, 0.0043618083000183105, 0.000283833040157333, 0.002730634529143572], [0.23954874277114868, -0.23072250187397003, 0.10531431436538696, -0.7259851098060608, -0.25069379806518555, 0.09300434589385986, 0.000626230554189533, 0.0064532398246228695, 0.004475202411413193, 0.00021071868832223117, -0.0016137735219672322, -0.003830856643617153, 0.013515385799109936, 0.021383723244071007, -0.002741217380389571, -0.003884095698595047, -0.0355781652033329, 0.0053338888101279736, 0.0208896454423666, 0.01268384512513876, -0.0062049212865531445, -0.0011432894971221685, -0.0017026233254000545, 0.007449421100318432, -0.013919113203883171, 0.010599161498248577, -0.003683089278638363, 0.005648847669363022, 0.012240931391716003, 0.0026538781821727753, 0.009731813333928585, 0.003944839350879192], [0.23698823153972626, -0.2292289286851883, 0.11931466311216354, -0.7365030646324158, -0.2563290297985077, 0.09378568828105927, -0.004572081379592419, 0.0037944295909255743, 0.01265899371355772, 0.012127694673836231, -0.006983333732932806, 0.007948275655508041, -0.007485417649149895, 0.021325770765542984, 0.008064990863204002, -0.005761218257248402, -0.00792820192873478, -0.012324832379817963, 0.013445529155433178, 0.0070065222680568695, 0.001400871085934341, -0.00759178027510643, -0.011623742990195751, 0.007791984360665083, -0.0041619096882641315, -0.0035866538528352976, -0.010152994655072689, -0.02762559987604618, -0.005135960411280394, 0.003369926242157817, -0.007827958092093468, -0.007953008636832237]], "action_steps": 10, "action_dim": 32} \ No newline at end of file +{"action_final_raw": [[0.1252194046974182, -0.2584618031978607, 0.1081027016043663, -0.4715811312198639, -0.16625730693340302, 0.06437430530786514, 0.0038535576313734055, 0.014515518210828304, 0.021946538239717484, 0.005747061222791672, 0.007577195297926664, -0.0014543490251526237, 0.011012047529220581, 0.026697054505348206, 0.009022124111652374, 0.00391031801700592, -0.0042377919889986515, -0.014031890779733658, 0.019611328840255737, 0.010316522791981697, 0.0071461377665400505, 0.0014671484241262078, 0.006010034587234259, -0.0032027096021920443, 0.0015054258983582258, -0.0016148658469319344, 0.003685156349092722, -0.009740755893290043, -0.0009369119652546942, 0.007600389886647463, -0.0037479838356375694, -0.009390483610332012], [0.14668071269989014, -0.2629736363887787, 0.10172148048877716, -0.5047131776809692, -0.19165721535682678, 0.05824785307049751, -0.007131943479180336, -0.007506182417273521, 0.002765909070149064, 0.007244679145514965, 0.0025957580655813217, -0.004217817913740873, -0.005774653982371092, 0.021622294560074806, -0.002640289021655917, 0.00571085698902607, -0.021713245660066605, 0.007794871460646391, 0.008303631097078323, 0.0026964619755744934, 0.0036214396823197603, 0.003017223672941327, 0.00900877546519041, -0.007324070204049349, -0.0034644873812794685, 0.0009487142087891698, 0.0030688068363815546, -0.012043364346027374, -0.003853978356346488, 0.0012464025057852268, -0.002187462290748954, -0.005726518575102091], [0.14581987261772156, -0.2675715386867523, 0.10253500193357468, -0.5388630032539368, -0.18654906749725342, 0.04533516988158226, 0.008264916017651558, 0.005853405687958002, 0.005849978420883417, 0.0007846317603252828, 0.0072210971266031265, 0.007986214011907578, 0.01339389756321907, 0.025508800521492958, 0.010266689583659172, -0.006672204006463289, -0.011262786574661732, -0.012860552407801151, 0.015247649513185024, 0.00785924308001995, -0.007838825695216656, -0.005817960482090712, 0.006671297363936901, 0.003971206024289131, -0.008289201185107231, -0.0038015800528228283, -0.006464078091084957, -0.005423170048743486, 0.0045570433139801025, -0.002559739165008068, -0.0008506326121278107, 0.015581250190734863], [0.1665424406528473, -0.26912224292755127, 0.11419762670993805, -0.5868101119995117, -0.1954595297574997, 0.06867559999227524, 0.024295512586832047, 0.03045368380844593, -0.005143571645021439, 0.008289759978652, 0.018656104803085327, -0.0012989797396585345, 0.013029848225414753, 0.01944301463663578, 0.0019133611349388957, -0.008343377150595188, -0.004050553310662508, -0.00034157527261413634, 0.012200172059237957, 0.0031303628347814083, -0.008614039979875088, -0.0032254692632704973, 0.011622898280620575, 0.0021338339429348707, -0.002373832743614912, 0.006613467819988728, -0.007932721637189388, 0.005981613416224718, -0.004626246634870768, 0.006114748306572437, -0.010100271552801132, 0.017188122496008873], [0.1811451017856598, -0.27415621280670166, 0.11289053410291672, -0.6146825551986694, -0.21037657558918, 0.05968904122710228, -0.000720147741958499, 0.014394178055226803, 0.003951057326048613, 0.009166492149233818, -0.002458348870277405, -0.0013514806050807238, 0.00679404754191637, 0.037289682775735855, -0.0029500590171664953, 0.0022290078923106194, -0.017472537234425545, -0.008945085108280182, 0.024387016892433167, 0.011426499113440514, 0.004050580319017172, 0.0028549344278872013, -0.003012970322743058, 0.0005769968847744167, -0.008992276154458523, 0.0035018613561987877, -0.001130217919126153, -0.005233350209891796, 0.009396878071129322, -0.008633887395262718, -0.003988988697528839, -0.0007890011183917522], [0.21046115458011627, -0.2658277153968811, 0.1125495433807373, -0.6338517665863037, -0.20075951516628265, 0.08012385666370392, -0.004816076252609491, 0.017015647143125534, 0.015811583027243614, 0.017829682677984238, -0.006385502405464649, 0.013073045760393143, 0.00828036293387413, 0.02529015764594078, -0.004360095132142305, -8.300511399284005e-05, -0.025862213224172592, -0.009099933318793774, 0.01678266003727913, 0.014498458243906498, -0.002535214414820075, 0.004935296252369881, 0.0028997468762099743, 0.0004930304712615907, -0.0019306342583149672, 0.0030226786620914936, -0.005562719888985157, -0.010786283761262894, -0.004058332182466984, -0.009337633848190308, 0.005500102415680885, 0.0017217950662598014], [0.21137407422065735, -0.2595526874065399, 0.11502957344055176, -0.6415976881980896, -0.21462856233119965, 0.09210913628339767, -0.01837189495563507, 0.009373516775667667, 0.005604609847068787, -0.0032101485412567854, -0.0005869076703675091, -0.003446893999353051, 0.01756187528371811, 0.030138596892356873, -0.0049069467931985855, 0.001485658111050725, -0.02137862890958786, -0.002073426963761449, 0.021583713591098785, 0.010400763712823391, -0.004751626402139664, 0.01309794932603836, -0.014470523223280907, 0.007418171502649784, 0.0023551909253001213, 0.008607795462012291, -0.00849146582186222, -0.013215807266533375, -0.013685611076653004, 0.006035448983311653, -0.006928929127752781, -0.009577583521604538], [0.22011639177799225, -0.27294179797172546, 0.10333369672298431, -0.6590995192527771, -0.2087649255990982, 0.07929916679859161, 0.006832613609731197, 0.01281677559018135, 0.004098140634596348, 0.0016815605340525508, -0.006420925259590149, 0.00988492090255022, 0.009271795861423016, 0.011885983869433403, -0.0028244717977941036, -0.007358779665082693, -0.018294701352715492, -0.008869344368577003, 0.024098796769976616, -0.0007464916561730206, 0.005675711203366518, -0.0017750047845765948, -0.001103853341192007, -0.012181228026747704, -0.005984604824334383, 0.00799475610256195, -0.004131009336560965, -0.003986013121902943, -0.01077263243496418, 0.0017262446926906705, 0.0030581357423216105, 0.005382431671023369], [0.22306571900844574, -0.25544920563697815, 0.10616041719913483, -0.6666560173034668, -0.22026728093624115, 0.07681779563426971, -0.01113913580775261, 0.013008050620555878, 0.0014461483806371689, 0.0028875942807644606, -0.0009666667901910841, 1.4614631254517008e-05, 0.012881576083600521, 0.026309167966246605, -0.005227194633334875, 0.005573877599090338, -0.030521145090460777, -0.0004678427067119628, 0.02640548162162304, 0.005939433351159096, -0.003373843850567937, -0.00106965156737715, -0.006294457707554102, -0.00036198238376528025, -0.010745595209300518, 0.01108782272785902, -0.0066130622290074825, 0.0034260833635926247, 0.004696901421993971, 0.005760704632848501, 0.009224440902471542, 0.0001614554930711165], [0.2363392561674118, -0.2566598653793335, 0.12728047370910645, -0.6774600148200989, -0.2089737504720688, 0.08037654310464859, -0.00643885275349021, 0.004675334319472313, 0.016090121120214462, 0.012052156962454319, -0.01001695729792118, 0.007081877440214157, -0.00250675599090755, 0.02842658758163452, 0.0021272446028888226, 0.0036669981200248003, -0.006756100337952375, -0.008480187505483627, 0.012720120139420033, 0.007946241647005081, 0.004657568875700235, -0.006620081607252359, -0.008366083726286888, 0.0014283015625551343, -0.007338288705796003, -0.002251375000923872, -0.017037000507116318, -0.02396969497203827, -0.00759129086509347, 0.0009175803861580789, -0.013536610640585423, -0.002894690493121743]], "action_steps": 10, "action_dim": 32} \ No newline at end of file From df80e8f0d004c60bf5a6aa6b4f80cf1893483f09 Mon Sep 17 00:00:00 2001 From: LiangSu8899 Date: Tue, 25 Aug 2026 02:31:32 -0400 Subject: [PATCH 12/32] perf(ggml-adapter): run decode attention as single wide GEMMs The GQA query heads share the one KV head, so the per-head batched QK^T and PV GEMMs collapse into single GEMMs over all n_head*n_tok query rows. Ordering the rows t-major makes the PV output land contiguously in the flash-attention node's [hd, n_head, n_tok] destination, removing the strided-C store penalty. Numerically identical to the batched form. --- .../adapters/ggml/fr_decode_attn.cu | 76 +++++++++++-------- .../structures/adapters/ggml/fr_dispatch.cu | 4 +- .../structures/adapters/ggml/fr_kernels.h | 12 +-- 3 files changed, 54 insertions(+), 38 deletions(-) diff --git a/flash_rt/structures/adapters/ggml/fr_decode_attn.cu b/flash_rt/structures/adapters/ggml/fr_decode_attn.cu index 35ee249d..d635da5a 100644 --- a/flash_rt/structures/adapters/ggml/fr_decode_attn.cu +++ b/flash_rt/structures/adapters/ggml/fr_decode_attn.cu @@ -2,11 +2,18 @@ // // For q_tokens ≤ 16 over a padded f16 KV of token rows, flash attention's // stream-k kernel plus its fixup pass is slower than the classic -// decomposition the FlashRT torch pipeline uses: one batched QK^T GEMM -// (all GQA query heads share the single KV head, so the K operand batches -// with stride 0), a masked softmax over the KV axis, and one batched PV -// GEMM whose fp32 output lands directly in the flash-attention node's -// [hd, n_head, n_tok] layout via a strided C. +// decomposition the FlashRT torch pipeline uses: one QK^T GEMM, a masked +// softmax over the KV axis, and one PV GEMM. +// +// All GQA query heads share the single KV head, so instead of a batched +// GEMM per head both contractions run as one wide GEMM over the +// n_head*n_tok query rows. Rows are ordered t-major (row r = t*n_head + h): +// with that ordering the PV output column j lands at byte offset j*hd in +// the flash-attention node's [hd, n_head, n_tok] destination, i.e. the +// GEMM writes the fp32 result contiguously with no strided-C penalty (the +// dispatch layer guarantees the destination is contiguous). The t-major +// order also makes the q gather read the [hd, n_head, n_tok]-contiguous +// Q buffer sequentially. // // Numerics follow ggml's fattn contract: scores = scale * q.k + mask (f16 // mask, slope 1 as max_bias must be 0), softmax in fp32 with running max. @@ -20,15 +27,15 @@ namespace ggml_cuda_flashrt { namespace { -// gather the permuted f32 Q view into contiguous f16 rows [n_head*n_tok, hd] -// (row r = h*n_tok + t), applying nothing else (scale folds into QK alpha) +// gather the permuted f32 Q view into contiguous f16 rows [n_tok*n_head, hd] +// (row r = t*n_head + h), applying nothing else (scale folds into QK alpha) __global__ void kernel_q_gather_f16(const float * __restrict__ q, __half * __restrict__ out, - int hd, int n_tok, int n_head, + int hd, int n_head, int64_t s_d, int64_t s_tok, int64_t s_head) { - const int r = blockIdx.x; // h*n_tok + t - const int h = r / n_tok; - const int t = r % n_tok; + const int r = blockIdx.x; // t*n_head + h + const int h = r % n_head; + const int t = r / n_head; const float * src = q + (int64_t) h * s_head + (int64_t) t * s_tok; __half * dst = out + (int64_t) r * hd; for (int d = threadIdx.x; d < hd; d += blockDim.x) { @@ -36,13 +43,13 @@ __global__ void kernel_q_gather_f16(const float * __restrict__ q, } } -// in-place masked softmax over rows of [n_head*n_tok, n_kv] f16 scores. +// in-place masked softmax over rows of [n_tok*n_head, n_kv] f16 scores. // mask element for (kv, t) at mask + kv + t*mask_stride (f16, -inf on pads). __global__ void kernel_mask_softmax_f16(__half * __restrict__ scores, const __half * __restrict__ mask, - int n_kv, int n_tok, int64_t mask_stride) { - const int r = blockIdx.x; // h*n_tok + t - const int t = r % n_tok; + int n_kv, int n_head, int64_t mask_stride) { + const int r = blockIdx.x; // t*n_head + h + const int t = r / n_head; __half * row = scores + (int64_t) r * n_kv; const __half * mrow = mask + (int64_t) t * mask_stride; @@ -109,41 +116,46 @@ int decode_attn_decomposed(void * cublas_handle, void * q16_ws, void * scores_ws, int hd, int n_tok, int n_head, int n_kv, float scale, cudaStream_t stream) { + // the contiguous PV store below requires the [hd, n_head, n_tok] dst + // to be dense; the dispatch layer checks the same before fusing + if (dst_shead != hd || dst_stok != (int64_t) hd * n_head) { + return -1; + } cublasHandle_t handle = (cublasHandle_t) cublas_handle; const int R = n_head * n_tok; kernel_q_gather_f16<<>>( - q, (__half *) q16_ws, hd, n_tok, n_head, q_sd, q_stok, q_shead); + q, (__half *) q16_ws, hd, n_head, q_sd, q_stok, q_shead); cublasSetStream(handle, stream); - // scores_col[n_kv, n_tok] per head = K_col^T [n_kv, hd] x Q_col [hd, n_tok] + // scores_col[n_kv, R] = K_col^T [n_kv, hd] x Q16_col [hd, R] const float beta0 = 0.0f; - cublasStatus_t st = cublasGemmStridedBatchedEx( + cublasStatus_t st = cublasGemmEx( handle, CUBLAS_OP_T, CUBLAS_OP_N, - n_kv, n_tok, hd, + n_kv, R, hd, &scale, - k_f16_rows, CUDA_R_16F, hd, 0, - q16_ws, CUDA_R_16F, hd, (int64_t) n_tok * hd, + k_f16_rows, CUDA_R_16F, hd, + q16_ws, CUDA_R_16F, hd, &beta0, - scores_ws, CUDA_R_16F, n_kv, (int64_t) n_tok * n_kv, - n_head, CUBLAS_COMPUTE_32F, CUBLAS_GEMM_DEFAULT); + scores_ws, CUDA_R_16F, n_kv, + CUBLAS_COMPUTE_32F, CUBLAS_GEMM_DEFAULT); if (st != CUBLAS_STATUS_SUCCESS) return -100 - (int) st; kernel_mask_softmax_f16<<>>( - (__half *) scores_ws, (const __half *) mask_f16, n_kv, n_tok, mask_stride); + (__half *) scores_ws, (const __half *) mask_f16, n_kv, n_head, mask_stride); - // out_col[hd, n_tok] per head (ldc = dst token stride) = - // V_col [hd, n_kv] x P_col [n_kv, n_tok] + // dst_col[hd, R] (dense, column r = t*n_head + h at offset r*hd) = + // V_col [hd, n_kv] x P_col [n_kv, R] const float one = 1.0f; - st = cublasGemmStridedBatchedEx( + st = cublasGemmEx( handle, CUBLAS_OP_N, CUBLAS_OP_N, - hd, n_tok, n_kv, + hd, R, n_kv, &one, - v_f16_rows, CUDA_R_16F, hd, 0, - scores_ws, CUDA_R_16F, n_kv, (int64_t) n_tok * n_kv, + v_f16_rows, CUDA_R_16F, hd, + scores_ws, CUDA_R_16F, n_kv, &beta0, - dst, CUDA_R_32F, (int) dst_stok, dst_shead, - n_head, CUBLAS_COMPUTE_32F, CUBLAS_GEMM_DEFAULT); + dst, CUDA_R_32F, hd, + CUBLAS_COMPUTE_32F, CUBLAS_GEMM_DEFAULT); if (st != CUBLAS_STATUS_SUCCESS) return -200 - (int) st; const cudaError_t e = cudaGetLastError(); diff --git a/flash_rt/structures/adapters/ggml/fr_dispatch.cu b/flash_rt/structures/adapters/ggml/fr_dispatch.cu index 9409a493..b0988829 100644 --- a/flash_rt/structures/adapters/ggml/fr_dispatch.cu +++ b/flash_rt/structures/adapters/ggml/fr_dispatch.cu @@ -1521,7 +1521,9 @@ bool ggml_cuda_flashrt_should_fuse_dec_attn(const ggml_tensor * fa) { } if (fa->type != GGML_TYPE_F32 || fa->ne[0] != hd || fa->ne[1] != n_head || fa->ne[2] != n_tok || fa->nb[0] != sizeof(float) || - (int64_t) fa->nb[1] != hd * (int64_t) sizeof(float)) { + (int64_t) fa->nb[1] != hd * (int64_t) sizeof(float) || + // the PV GEMM stores its result as one dense column-block + (int64_t) fa->nb[2] != hd * n_head * (int64_t) sizeof(float)) { return false; } float max_bias, softcap; diff --git a/flash_rt/structures/adapters/ggml/fr_kernels.h b/flash_rt/structures/adapters/ggml/fr_kernels.h index 5374f6c6..01885e2f 100644 --- a/flash_rt/structures/adapters/ggml/fr_kernels.h +++ b/flash_rt/structures/adapters/ggml/fr_kernels.h @@ -130,11 +130,13 @@ int layer_norm_affine_quant(const float * x, const float * w, const float * b, int layer_norm_affine(const float * x, const float * w, const float * b, float * out, int M, int C, float eps, cudaStream_t stream); -// Decomposed tiny-M attention: batched QK^T (f16, GQA heads share the -// stride-0 K operand) + masked softmax + batched PV with fp32 output -// written directly in the [hd, n_head, n_tok] layout via strided C. -// q strides are in elements; workspaces: q16 [n_head*n_tok, hd] f16, -// scores [n_head*n_tok, n_kv] f16. cublas_handle is a cublasHandle_t. +// Decomposed tiny-M attention: one QK^T GEMM over all n_head*n_tok query +// rows (f16, the GQA heads share the single K operand) + masked softmax + +// one PV GEMM whose fp32 output is the dense [hd, n_head, n_tok] dst +// (rows ordered t-major so the store is contiguous; requires +// dst_shead == hd and dst_stok == hd*n_head). q strides are in elements; +// workspaces: q16 [n_tok*n_head, hd] f16, scores [n_tok*n_head, n_kv] +// f16. cublas_handle is a cublasHandle_t. int decode_attn_decomposed(void * cublas_handle, const float * q, int64_t q_sd, int64_t q_stok, int64_t q_shead, const void * k_f16_rows, const void * v_f16_rows, From c35ffe8116b3755dcb2b98e05c931d9acd7034e4 Mon Sep 17 00:00:00 2001 From: LiangSu8899 Date: Tue, 25 Aug 2026 02:38:51 -0400 Subject: [PATCH 13/32] perf(ggml-adapter): hand f16 Q from the fused QKV window to decode attention qkv_post optionally stores the rope'd+scaled Q rows as f16 alongside the f32 tensor; its [M, Nq] layout is exactly the t-major gather order the decomposed decode attention consumes. A one-shot grow-only slot hands the buffer to the following attention window, which then skips its gather kernel (one launch less per layer). The f16 values round from the same fp32 registers the gather would read, so the results are bit-identical. --- .../adapters/ggml/fr_decode_attn.cu | 8 ++- .../structures/adapters/ggml/fr_dispatch.cu | 59 +++++++++++++++++-- .../structures/adapters/ggml/fr_kernels.h | 19 ++++-- .../structures/adapters/ggml/fr_qkv_post.cu | 35 ++++++++--- 4 files changed, 99 insertions(+), 22 deletions(-) diff --git a/flash_rt/structures/adapters/ggml/fr_decode_attn.cu b/flash_rt/structures/adapters/ggml/fr_decode_attn.cu index d635da5a..0579a5ed 100644 --- a/flash_rt/structures/adapters/ggml/fr_decode_attn.cu +++ b/flash_rt/structures/adapters/ggml/fr_decode_attn.cu @@ -113,7 +113,7 @@ int decode_attn_decomposed(void * cublas_handle, const void * v_f16_rows, // [n_kv, hd] f16 rows const void * mask_f16, int64_t mask_stride, float * dst, int64_t dst_stok, int64_t dst_shead, - void * q16_ws, void * scores_ws, + void * q16_ws, int q16_ready, void * scores_ws, int hd, int n_tok, int n_head, int n_kv, float scale, cudaStream_t stream) { // the contiguous PV store below requires the [hd, n_head, n_tok] dst @@ -124,8 +124,10 @@ int decode_attn_decomposed(void * cublas_handle, cublasHandle_t handle = (cublasHandle_t) cublas_handle; const int R = n_head * n_tok; - kernel_q_gather_f16<<>>( - q, (__half *) q16_ws, hd, n_head, q_sd, q_stok, q_shead); + if (!q16_ready) { + kernel_q_gather_f16<<>>( + q, (__half *) q16_ws, hd, n_head, q_sd, q_stok, q_shead); + } cublasSetStream(handle, stream); // scores_col[n_kv, R] = K_col^T [n_kv, hd] x Q16_col [hd, R] diff --git a/flash_rt/structures/adapters/ggml/fr_dispatch.cu b/flash_rt/structures/adapters/ggml/fr_dispatch.cu index b0988829..9f100a52 100644 --- a/flash_rt/structures/adapters/ggml/fr_dispatch.cu +++ b/flash_rt/structures/adapters/ggml/fr_dispatch.cu @@ -214,6 +214,37 @@ bool reserve_quantized_act(const ggml_tensor * out_tensor, int M, int K, return true; } +// One-shot f16 Q handoff from the fused decode QKV window to the decomposed +// decode attention: qkv_post writes the rope'd+scaled Q rows as f16 in the +// t-major gather order, and the next attention window consumes them instead +// of running its own gather kernel. A single grow-only slot suffices (the +// producer and consumer alternate strictly within each layer); the key is +// cleared on consumption so a recycled activation address can never alias a +// stale entry. +struct q16_slot { + const void * key = nullptr; // data pointer of the Q tensor written for + uint64_t eval_id = 0; + void * buf = nullptr; + size_t cap = 0; +}; +q16_slot g_q16; + +void * reserve_q16(const void * qdata, size_t bytes, cudaStream_t stream) { + if (bytes > g_q16.cap) { + cudaStreamCaptureStatus cap = cudaStreamCaptureStatusNone; + cudaStreamIsCapturing(stream, &cap); + if (cap != cudaStreamCaptureStatusNone) { + return nullptr; + } + if (g_q16.buf != nullptr) { cudaFree(g_q16.buf); } + CUDA_CHECK(cudaMalloc(&g_q16.buf, bytes)); + g_q16.cap = bytes; + } + g_q16.key = qdata; + g_q16.eval_id = g_eval_id; + return g_q16.buf; +} + // Grow-only device buffer for the never-written D of the no-D-store GeGLU // variants (the host-side TMA descriptor still needs a valid allocation). void * get_dummy_d(size_t bytes) { @@ -1011,8 +1042,10 @@ void ggml_cuda_flashrt_qkv(ggml_backend_cuda_context & ctx, const float scale_f = ggml_get_op_params_f32(q_scale, 0); const ggml_tensor * ff = k_rope->src[2]; + void * q16 = reserve_q16(q_scale->data, + (size_t) M * Nq * sizeof(uint16_t), stream); rc = ggml_cuda_flashrt::qkv_post( - qkv_cat.get(), (float *) q_scale->data, + qkv_cat.get(), (float *) q_scale->data, q16, k_cpy->src[1]->data, v_cpy->src[1]->data, (const int32_t *) k_rope->src[1]->data, ff != nullptr ? (const float *) ff->data : nullptr, @@ -1466,7 +1499,8 @@ void ggml_cuda_flashrt_qkv_prefill(ggml_backend_cuda_context & ctx, // the rope'd K and plain V rows also land in their graph tensors' // f32 buffers, feeding the graph-tail persistent-KV store copies rc = ggml_cuda_flashrt::qkv_post_full( - qkv_cat.get(), (float *) q_scale->data, + // no f16 Q handoff on the prefill path (flash attention reads f32 Q) + qkv_cat.get(), (float *) q_scale->data, nullptr, k_cpy->data, v_cpy->data, (float *) k_rope->data, (float *) v_mm->data, (const int32_t *) k_rope->src[1]->data, @@ -1549,9 +1583,26 @@ void ggml_cuda_flashrt_dec_attn(ggml_backend_cuda_context & ctx, ggml_tensor * f cudaStream_t stream = ctx.stream(); const int64_t R = (int64_t) n_head * n_tok; - ggml_cuda_pool_alloc q16 (ctx.pool(), R * hd * sizeof(uint16_t)); ggml_cuda_pool_alloc scores(ctx.pool(), R * n_kv * sizeof(uint16_t)); + // consume the f16 Q rows the fused QKV window handed off (one-shot); + // they are valid only when the Q view is the dense t-major layout the + // handoff was written in + const bool q16_hit = g_q16.key != nullptr && g_q16.key == q->data && + g_q16.eval_id == g_eval_id && + q->nb[0] == sizeof(float) && + (int64_t) q->nb[2] == (int64_t) hd * sizeof(float) && + (int64_t) q->nb[1] == (int64_t) hd * n_head * sizeof(float); + ggml_cuda_pool_alloc q16; + void * q16p; + if (q16_hit) { + q16p = g_q16.buf; + g_q16.key = nullptr; + } else { + q16.alloc(ctx.pool(), R * hd * sizeof(uint16_t)); + q16p = q16.get(); + } + const int rc = ggml_cuda_flashrt::decode_attn_decomposed( (void *) ctx.cublas_handle(), (const float *) q->data, @@ -1563,7 +1614,7 @@ void ggml_cuda_flashrt_dec_attn(ggml_backend_cuda_context & ctx, ggml_tensor * f (float *) fa->data, (int64_t) (fa->nb[2] / sizeof(float)), (int64_t) (fa->nb[1] / sizeof(float)), - q16.get(), scores.get(), + q16p, q16_hit ? 1 : 0, scores.get(), hd, n_tok, n_head, n_kv, scale, stream); if (rc != 0) { GGML_ABORT("flashrt: decomposed decode attention failed (tok=%d kv=%d rc=%d)", n_tok, n_kv, rc); diff --git a/flash_rt/structures/adapters/ggml/fr_kernels.h b/flash_rt/structures/adapters/ggml/fr_kernels.h index 01885e2f..26708a53 100644 --- a/flash_rt/structures/adapters/ggml/fr_kernels.h +++ b/flash_rt/structures/adapters/ggml/fr_kernels.h @@ -89,9 +89,13 @@ int repack_weight_concat3(const void * b0, int N0, const void * b1, int N1, // Fused QKV post: RoPE+f16-store K, f16-store V (into the persistent KV // suffix), RoPE+scale Q (f32 out) from the fused GEMM's [M, Nk+Nv+Nq] rows. -// Variant with optional f32 K/V row outputs (for graphs whose rope'd K / V -// feed additional consumers, e.g. persistent-KV stores at the graph tail). -int qkv_post_full(const float * qkv_cat, float * q_out, void * k_out_f16, void * v_out_f16, +// q16_out (nullable) additionally stores the Q rows as f16 in the same +// [M, Nq] layout, which is the t-major gather order the decomposed decode +// attention consumes. Variant with optional f32 K/V row outputs (for +// graphs whose rope'd K / V feed additional consumers, e.g. persistent-KV +// stores at the graph tail). +int qkv_post_full(const float * qkv_cat, float * q_out, void * q16_out, + void * k_out_f16, void * v_out_f16, float * k_f32_out, float * v_f32_out, const int32_t * pos, const float * freq_factors, int M, int Nk, int Nv, int Nq, int head_dim, int n_dims, @@ -99,7 +103,8 @@ int qkv_post_full(const float * qkv_cat, float * q_out, void * k_out_f16, void * float corr_low, float corr_high, float theta_scale, float q_scale, cudaStream_t stream); -int qkv_post(const float * qkv_cat, float * q_out, void * k_out_f16, void * v_out_f16, +int qkv_post(const float * qkv_cat, float * q_out, void * q16_out, + void * k_out_f16, void * v_out_f16, const int32_t * pos, const float * freq_factors, int M, int Nk, int Nv, int Nq, int head_dim, int n_dims, float freq_scale, float ext_factor, float attn_factor, @@ -136,13 +141,15 @@ int layer_norm_affine(const float * x, const float * w, const float * b, // (rows ordered t-major so the store is contiguous; requires // dst_shead == hd and dst_stok == hd*n_head). q strides are in elements; // workspaces: q16 [n_tok*n_head, hd] f16, scores [n_tok*n_head, n_kv] -// f16. cublas_handle is a cublasHandle_t. +// f16. When q16_ready is nonzero, q16_ws already holds the gathered f16 Q +// rows (produced upstream, e.g. by qkv_post) and the gather kernel is +// skipped. cublas_handle is a cublasHandle_t. int decode_attn_decomposed(void * cublas_handle, const float * q, int64_t q_sd, int64_t q_stok, int64_t q_shead, const void * k_f16_rows, const void * v_f16_rows, const void * mask_f16, int64_t mask_stride, float * dst, int64_t dst_stok, int64_t dst_shead, - void * q16_ws, void * scores_ws, + void * q16_ws, int q16_ready, void * scores_ws, int hd, int n_tok, int n_head, int n_kv, float scale, cudaStream_t stream); diff --git a/flash_rt/structures/adapters/ggml/fr_qkv_post.cu b/flash_rt/structures/adapters/ggml/fr_qkv_post.cu index bdad0a5f..b050dea5 100644 --- a/flash_rt/structures/adapters/ggml/fr_qkv_post.cu +++ b/flash_rt/structures/adapters/ggml/fr_qkv_post.cu @@ -40,6 +40,7 @@ __device__ void qkv_rope_yarn(const float theta_extrap, const float freq_scale, // one block per token; threads cover the K/Q rope pairs and the V copy __global__ void kernel_qkv_post(const float * __restrict__ qkv, // [M, Nk+Nv+Nq] float * __restrict__ q_out, // [M, Nq] f32 (head-major rows) + __half * __restrict__ q16_out, // nullable: same values as f16 __half * __restrict__ k_out, // suffix rows, head_dim per token __half * __restrict__ v_out, float * __restrict__ k_f32_out, // nullable: rope'd K as f32 rows @@ -102,10 +103,18 @@ __global__ void kernel_qkv_post(const float * __restrict__ qkv, // [M, Nk+Nv+N const int h = hp / (head_dim / 2); const int i0 = 2 * (hp % (head_dim / 2)); const float * qh = qrow + (int64_t) h * head_dim; - float * oh = q_out + (int64_t) t * Nq + (int64_t) h * head_dim; + float * oh = q_out + (int64_t) t * Nq + (int64_t) h * head_dim; + __half * oh16 = q16_out != nullptr + ? q16_out + (int64_t) t * Nq + (int64_t) h * head_dim : nullptr; if (i0 >= n_dims) { - oh[n_dims + (i0 - n_dims)] = qh[n_dims + (i0 - n_dims)] * q_scale; - oh[n_dims + (i0 - n_dims) + 1] = qh[n_dims + (i0 - n_dims) + 1] * q_scale; + const float v0 = qh[n_dims + (i0 - n_dims)] * q_scale; + const float v1 = qh[n_dims + (i0 - n_dims) + 1] * q_scale; + oh[n_dims + (i0 - n_dims)] = v0; + oh[n_dims + (i0 - n_dims) + 1] = v1; + if (oh16 != nullptr) { + oh16[n_dims + (i0 - n_dims)] = __float2half(v0); + oh16[n_dims + (i0 - n_dims) + 1] = __float2half(v1); + } continue; } const float theta_base = p * powf(theta_scale, i0 / 2.0f); @@ -115,14 +124,21 @@ __global__ void kernel_qkv_post(const float * __restrict__ qkv, // [M, Nk+Nv+N i0, ext_factor, attn_factor, cos_t, sin_t); const float x0 = qh[i0 / 2]; const float x1 = qh[i0 / 2 + n_dims / 2]; - oh[i0 / 2] = (x0 * cos_t - x1 * sin_t) * q_scale; - oh[i0 / 2 + n_dims / 2] = (x0 * sin_t + x1 * cos_t) * q_scale; + const float v0 = (x0 * cos_t - x1 * sin_t) * q_scale; + const float v1 = (x0 * sin_t + x1 * cos_t) * q_scale; + oh[i0 / 2] = v0; + oh[i0 / 2 + n_dims / 2] = v1; + if (oh16 != nullptr) { + oh16[i0 / 2] = __float2half(v0); + oh16[i0 / 2 + n_dims / 2] = __float2half(v1); + } } } } // namespace -int qkv_post_full(const float * qkv_cat, float * q_out, void * k_out_f16, void * v_out_f16, +int qkv_post_full(const float * qkv_cat, float * q_out, void * q16_out, + void * k_out_f16, void * v_out_f16, float * k_f32_out, float * v_f32_out, const int32_t * pos, const float * freq_factors, int M, int Nk, int Nv, int Nq, int head_dim, int n_dims, @@ -131,7 +147,7 @@ int qkv_post_full(const float * qkv_cat, float * q_out, void * k_out_f16, void * cudaStream_t stream) { if (n_dims % 2 != 0 || Nk != head_dim || Nq % head_dim != 0) return -1; kernel_qkv_post<<>>( - qkv_cat, q_out, (__half *) k_out_f16, (__half *) v_out_f16, + qkv_cat, q_out, (__half *) q16_out, (__half *) k_out_f16, (__half *) v_out_f16, k_f32_out, v_f32_out, pos, freq_factors, Nk, Nv, Nq, head_dim, n_dims, freq_scale, ext_factor, attn_factor, corr_low, corr_high, theta_scale, q_scale); @@ -139,13 +155,14 @@ int qkv_post_full(const float * qkv_cat, float * q_out, void * k_out_f16, void * return (e == cudaSuccess) ? 0 : -static_cast(e); } -int qkv_post(const float * qkv_cat, float * q_out, void * k_out_f16, void * v_out_f16, +int qkv_post(const float * qkv_cat, float * q_out, void * q16_out, + void * k_out_f16, void * v_out_f16, const int32_t * pos, const float * freq_factors, int M, int Nk, int Nv, int Nq, int head_dim, int n_dims, float freq_scale, float ext_factor, float attn_factor, float corr_low, float corr_high, float theta_scale, float q_scale, cudaStream_t stream) { - return qkv_post_full(qkv_cat, q_out, k_out_f16, v_out_f16, nullptr, nullptr, + return qkv_post_full(qkv_cat, q_out, q16_out, k_out_f16, v_out_f16, nullptr, nullptr, pos, freq_factors, M, Nk, Nv, Nq, head_dim, n_dims, freq_scale, ext_factor, attn_factor, corr_low, corr_high, theta_scale, q_scale, stream); From e70432dd2b44e0462873423a167c4328269314a4 Mon Sep 17 00:00:00 2001 From: LiangSu8899 Date: Tue, 25 Aug 2026 03:08:40 -0400 Subject: [PATCH 14/32] chore(ggml-adapter): refresh the e2e action golden The host now offloads its action input projection, which moves the final actions at the usual chaotic-amplification scale; distances to the f16 reference were re-judged before this refresh. --- .../adapters/ggml/qualification/goldens/pi05_thor_action.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flash_rt/structures/adapters/ggml/qualification/goldens/pi05_thor_action.json b/flash_rt/structures/adapters/ggml/qualification/goldens/pi05_thor_action.json index e41f869a..813182c6 100644 --- a/flash_rt/structures/adapters/ggml/qualification/goldens/pi05_thor_action.json +++ b/flash_rt/structures/adapters/ggml/qualification/goldens/pi05_thor_action.json @@ -1 +1 @@ -{"action_final_raw": [[0.1252194046974182, -0.2584618031978607, 0.1081027016043663, -0.4715811312198639, -0.16625730693340302, 0.06437430530786514, 0.0038535576313734055, 0.014515518210828304, 0.021946538239717484, 0.005747061222791672, 0.007577195297926664, -0.0014543490251526237, 0.011012047529220581, 0.026697054505348206, 0.009022124111652374, 0.00391031801700592, -0.0042377919889986515, -0.014031890779733658, 0.019611328840255737, 0.010316522791981697, 0.0071461377665400505, 0.0014671484241262078, 0.006010034587234259, -0.0032027096021920443, 0.0015054258983582258, -0.0016148658469319344, 0.003685156349092722, -0.009740755893290043, -0.0009369119652546942, 0.007600389886647463, -0.0037479838356375694, -0.009390483610332012], [0.14668071269989014, -0.2629736363887787, 0.10172148048877716, -0.5047131776809692, -0.19165721535682678, 0.05824785307049751, -0.007131943479180336, -0.007506182417273521, 0.002765909070149064, 0.007244679145514965, 0.0025957580655813217, -0.004217817913740873, -0.005774653982371092, 0.021622294560074806, -0.002640289021655917, 0.00571085698902607, -0.021713245660066605, 0.007794871460646391, 0.008303631097078323, 0.0026964619755744934, 0.0036214396823197603, 0.003017223672941327, 0.00900877546519041, -0.007324070204049349, -0.0034644873812794685, 0.0009487142087891698, 0.0030688068363815546, -0.012043364346027374, -0.003853978356346488, 0.0012464025057852268, -0.002187462290748954, -0.005726518575102091], [0.14581987261772156, -0.2675715386867523, 0.10253500193357468, -0.5388630032539368, -0.18654906749725342, 0.04533516988158226, 0.008264916017651558, 0.005853405687958002, 0.005849978420883417, 0.0007846317603252828, 0.0072210971266031265, 0.007986214011907578, 0.01339389756321907, 0.025508800521492958, 0.010266689583659172, -0.006672204006463289, -0.011262786574661732, -0.012860552407801151, 0.015247649513185024, 0.00785924308001995, -0.007838825695216656, -0.005817960482090712, 0.006671297363936901, 0.003971206024289131, -0.008289201185107231, -0.0038015800528228283, -0.006464078091084957, -0.005423170048743486, 0.0045570433139801025, -0.002559739165008068, -0.0008506326121278107, 0.015581250190734863], [0.1665424406528473, -0.26912224292755127, 0.11419762670993805, -0.5868101119995117, -0.1954595297574997, 0.06867559999227524, 0.024295512586832047, 0.03045368380844593, -0.005143571645021439, 0.008289759978652, 0.018656104803085327, -0.0012989797396585345, 0.013029848225414753, 0.01944301463663578, 0.0019133611349388957, -0.008343377150595188, -0.004050553310662508, -0.00034157527261413634, 0.012200172059237957, 0.0031303628347814083, -0.008614039979875088, -0.0032254692632704973, 0.011622898280620575, 0.0021338339429348707, -0.002373832743614912, 0.006613467819988728, -0.007932721637189388, 0.005981613416224718, -0.004626246634870768, 0.006114748306572437, -0.010100271552801132, 0.017188122496008873], [0.1811451017856598, -0.27415621280670166, 0.11289053410291672, -0.6146825551986694, -0.21037657558918, 0.05968904122710228, -0.000720147741958499, 0.014394178055226803, 0.003951057326048613, 0.009166492149233818, -0.002458348870277405, -0.0013514806050807238, 0.00679404754191637, 0.037289682775735855, -0.0029500590171664953, 0.0022290078923106194, -0.017472537234425545, -0.008945085108280182, 0.024387016892433167, 0.011426499113440514, 0.004050580319017172, 0.0028549344278872013, -0.003012970322743058, 0.0005769968847744167, -0.008992276154458523, 0.0035018613561987877, -0.001130217919126153, -0.005233350209891796, 0.009396878071129322, -0.008633887395262718, -0.003988988697528839, -0.0007890011183917522], [0.21046115458011627, -0.2658277153968811, 0.1125495433807373, -0.6338517665863037, -0.20075951516628265, 0.08012385666370392, -0.004816076252609491, 0.017015647143125534, 0.015811583027243614, 0.017829682677984238, -0.006385502405464649, 0.013073045760393143, 0.00828036293387413, 0.02529015764594078, -0.004360095132142305, -8.300511399284005e-05, -0.025862213224172592, -0.009099933318793774, 0.01678266003727913, 0.014498458243906498, -0.002535214414820075, 0.004935296252369881, 0.0028997468762099743, 0.0004930304712615907, -0.0019306342583149672, 0.0030226786620914936, -0.005562719888985157, -0.010786283761262894, -0.004058332182466984, -0.009337633848190308, 0.005500102415680885, 0.0017217950662598014], [0.21137407422065735, -0.2595526874065399, 0.11502957344055176, -0.6415976881980896, -0.21462856233119965, 0.09210913628339767, -0.01837189495563507, 0.009373516775667667, 0.005604609847068787, -0.0032101485412567854, -0.0005869076703675091, -0.003446893999353051, 0.01756187528371811, 0.030138596892356873, -0.0049069467931985855, 0.001485658111050725, -0.02137862890958786, -0.002073426963761449, 0.021583713591098785, 0.010400763712823391, -0.004751626402139664, 0.01309794932603836, -0.014470523223280907, 0.007418171502649784, 0.0023551909253001213, 0.008607795462012291, -0.00849146582186222, -0.013215807266533375, -0.013685611076653004, 0.006035448983311653, -0.006928929127752781, -0.009577583521604538], [0.22011639177799225, -0.27294179797172546, 0.10333369672298431, -0.6590995192527771, -0.2087649255990982, 0.07929916679859161, 0.006832613609731197, 0.01281677559018135, 0.004098140634596348, 0.0016815605340525508, -0.006420925259590149, 0.00988492090255022, 0.009271795861423016, 0.011885983869433403, -0.0028244717977941036, -0.007358779665082693, -0.018294701352715492, -0.008869344368577003, 0.024098796769976616, -0.0007464916561730206, 0.005675711203366518, -0.0017750047845765948, -0.001103853341192007, -0.012181228026747704, -0.005984604824334383, 0.00799475610256195, -0.004131009336560965, -0.003986013121902943, -0.01077263243496418, 0.0017262446926906705, 0.0030581357423216105, 0.005382431671023369], [0.22306571900844574, -0.25544920563697815, 0.10616041719913483, -0.6666560173034668, -0.22026728093624115, 0.07681779563426971, -0.01113913580775261, 0.013008050620555878, 0.0014461483806371689, 0.0028875942807644606, -0.0009666667901910841, 1.4614631254517008e-05, 0.012881576083600521, 0.026309167966246605, -0.005227194633334875, 0.005573877599090338, -0.030521145090460777, -0.0004678427067119628, 0.02640548162162304, 0.005939433351159096, -0.003373843850567937, -0.00106965156737715, -0.006294457707554102, -0.00036198238376528025, -0.010745595209300518, 0.01108782272785902, -0.0066130622290074825, 0.0034260833635926247, 0.004696901421993971, 0.005760704632848501, 0.009224440902471542, 0.0001614554930711165], [0.2363392561674118, -0.2566598653793335, 0.12728047370910645, -0.6774600148200989, -0.2089737504720688, 0.08037654310464859, -0.00643885275349021, 0.004675334319472313, 0.016090121120214462, 0.012052156962454319, -0.01001695729792118, 0.007081877440214157, -0.00250675599090755, 0.02842658758163452, 0.0021272446028888226, 0.0036669981200248003, -0.006756100337952375, -0.008480187505483627, 0.012720120139420033, 0.007946241647005081, 0.004657568875700235, -0.006620081607252359, -0.008366083726286888, 0.0014283015625551343, -0.007338288705796003, -0.002251375000923872, -0.017037000507116318, -0.02396969497203827, -0.00759129086509347, 0.0009175803861580789, -0.013536610640585423, -0.002894690493121743]], "action_steps": 10, "action_dim": 32} \ No newline at end of file +{"action_final_raw": [[0.12940391898155212, -0.2528587281703949, 0.10582581162452698, -0.4627326428890228, -0.1517871916294098, 0.05789315328001976, -0.009118168614804745, 0.01561721134930849, 0.019083378836512566, 0.0080350236967206, 0.006184188649058342, 0.00844589713960886, 0.004920070059597492, 0.02140013314783573, 0.008404633030295372, 0.003924117889255285, -0.0030497461557388306, -0.01774791069328785, 0.016703812405467033, 0.009317993186414242, 0.003162568435072899, 0.0025065350346267223, 0.001067449222318828, -0.006151682231575251, 0.00018592315609566867, -0.0020070478785783052, -0.0008290940895676613, -0.012632769532501698, 0.0042471857741475105, 0.0028852762188762426, -0.0028384749311953783, -0.0012326251016929746], [0.15184731781482697, -0.2700265944004059, 0.09386720508337021, -0.4975416958332062, -0.1869109719991684, 0.05920719355344772, -0.009828323498368263, -0.0064353132620453835, 0.00014671108510810882, 0.0005270597757771611, 0.0031039936002343893, -0.002291979966685176, -0.004778217524290085, 0.023754512891173363, -0.002179354429244995, 0.006903328467160463, -0.018551316112279892, 0.006453586742281914, 0.006781680043786764, 0.005654975771903992, 0.006293158978223801, 0.005439461208879948, 0.004844403360038996, -0.009044076316058636, -0.008247747085988522, 0.00579757010564208, 0.0027987658977508545, -0.011956475675106049, 0.0031155601609498262, -0.0010544508695602417, 0.0010488464031368494, -0.007074669003486633], [0.13989382982254028, -0.25814589858055115, 0.09946267306804657, -0.5187717080116272, -0.17015187442302704, 0.05254214629530907, 0.004989681299775839, 0.010219192132353783, 0.01351972110569477, -0.001688823220320046, 0.002372527029365301, 0.00768582196906209, 0.020504146814346313, 0.023498935624957085, 0.0003212030860595405, -0.009923301637172699, -0.007970130071043968, -0.011813816614449024, 0.015340851619839668, 0.007163303904235363, -0.009145798161625862, -0.007841194979846478, 0.008479283191263676, -0.00021931514493189752, -0.012552785687148571, -0.004921156447380781, -0.002375485608354211, -0.0034394313115626574, 0.0035447371192276478, -0.0021622469648718834, 0.005108300130814314, 0.009533166885375977], [0.16775454580783844, -0.2645603120326996, 0.10399901866912842, -0.557577908039093, -0.18896692991256714, 0.07578115910291672, 0.02140253223478794, 0.03042648918926716, 0.0021330579183995724, 0.0029577657114714384, 0.012866067700088024, 0.004185037687420845, 0.0030482769943773746, 0.013907969929277897, -0.004347839392721653, -0.010749413631856441, -0.006680286023765802, -2.668274100869894e-05, 0.004398548509925604, 0.005133638624101877, -0.001514833071269095, -0.005666330456733704, 0.00192269217222929, -0.0006473756511695683, -0.0019386905478313565, 0.006693375762552023, -0.010833384469151497, -0.005241188686341047, -0.00850680097937584, 7.693461520830169e-05, -0.014442027546465397, 0.01909816265106201], [0.1837024986743927, -0.2717337906360626, 0.11260585486888885, -0.5920315980911255, -0.19844241440296173, 0.05850118026137352, 0.0022641720715910196, 0.00940608698874712, 0.003187427995726466, 0.009219805710017681, -0.007168120238929987, -0.0005367595585994422, 0.00547138461843133, 0.03524341061711311, 0.000594788754824549, 0.005399557761847973, -0.023198217153549194, -0.012830831110477448, 0.020483607426285744, 0.013120101764798164, 0.0053434851579368114, 0.005363209638744593, -0.0069306110963225365, 0.001020975410938263, -0.0036603708285838366, 0.002235849853605032, 0.0006648871931247413, -0.00819794274866581, 0.005948537494987249, -0.005503628868609667, -0.004608611110597849, -0.005888366140425205], [0.20225247740745544, -0.2638978958129883, 0.10064870864152908, -0.6103572845458984, -0.1930299997329712, 0.0759415552020073, 0.002225512871518731, 0.015738461166620255, 0.01841391809284687, 0.017062587663531303, -0.006852346938103437, 0.016573432832956314, 0.008369849994778633, 0.03159790113568306, -0.0018215854652225971, -0.0007803556509315968, -0.01254053320735693, -0.009736414067447186, 0.012862961739301682, 0.007550361100584269, -0.006668137852102518, 0.004641189239919186, -0.0041647846810519695, 0.003983424045145512, 0.005212467163801193, -0.0008021246176213026, -0.011270538903772831, -0.00967323500663042, -0.0006975189317017794, -0.0034966417588293552, 0.00386984390206635, 0.0034399370197206736], [0.2047695815563202, -0.2553747296333313, 0.10427432507276535, -0.6205598711967468, -0.2059750258922577, 0.09672776609659195, 0.0007960622315295041, 0.009596768766641617, 0.007772690616548061, 0.0006036338163539767, -0.0026294041890650988, -0.003348570317029953, 0.011335442773997784, 0.022261248901486397, -0.008794681169092655, -5.653189418808324e-06, -0.027853652834892273, -0.005915062502026558, 0.016558898612856865, 0.009591544046998024, 0.005161453504115343, 0.008785799145698547, -0.007114224601536989, 0.007564001716673374, 0.005259434226900339, 0.013112467713654041, -0.008463529869914055, -0.009648582898080349, -0.009051376022398472, 0.005528939887881279, -0.002176202367991209, -0.006872260011732578], [0.2194412648677826, -0.2765859365463257, 0.10265745222568512, -0.6340638995170593, -0.20541149377822876, 0.08428670465946198, 0.0097963847219944, 0.014086037874221802, 0.00032647105399519205, -0.0003711430181283504, -0.003853543195873499, 0.0008552426006644964, 0.015407024882733822, 0.02043493464589119, 0.004803684540092945, -0.0014393915189430118, -0.01593821682035923, -0.002983001060783863, 0.018109437078237534, 0.0057259202003479, -0.0007894366281107068, 0.0020676464773714542, -0.0024445049930363894, -0.010146920569241047, -0.005904162302613258, 0.008207222446799278, -0.0024562610778957605, -0.000604762404691428, -0.0026838593184947968, 0.004686345346271992, 0.003601022297516465, 0.00964265689253807], [0.22591130435466766, -0.2531126141548157, 0.10270681232213974, -0.6457228660583496, -0.21248739957809448, 0.08996982872486115, -0.00910878460854292, 0.013310388661921024, -0.005985552910715342, 0.0021230243146419525, 0.0012873277300968766, 0.0060186865739524364, 0.018974991515278816, 0.018281321972608566, 0.0002817067434079945, 0.012477329932153225, -0.022212624549865723, -0.0030888498295098543, 0.02746962569653988, 0.004553196020424366, -0.0009546522051095963, -0.012151896953582764, -0.008990723639726639, -0.001741186250001192, -0.002104580169543624, 0.011146661825478077, -0.0034014740958809853, 0.0027611013501882553, 0.0014544056029990315, 0.005902513395994902, 0.0029447246342897415, -0.0002835096383932978], [0.2338383048772812, -0.25216224789619446, 0.12483970820903778, -0.6578122973442078, -0.20657934248447418, 0.0806683748960495, -0.004676615819334984, -0.0006798385875299573, 0.015721520408988, 0.01621541567146778, -0.009323552250862122, -0.00032941167592070997, -0.0019576391205191612, 0.025488749146461487, 0.005375501234084368, 0.001192403258755803, 0.00319452746771276, -0.009350409731268883, 0.0034876212012022734, 0.0032979880925267935, 0.00373227265663445, 0.001819843309931457, -0.007980229333043098, 0.011167601682245731, -0.0028672132175415754, -0.007579560857266188, -0.015309120528399944, -0.027653951197862625, -0.012417344376444817, 0.006930120754987001, -0.004635754972696304, -0.006454023998230696]], "action_steps": 10, "action_dim": 32} \ No newline at end of file From 50db36cf426ac40ec1a7a35c2d11078df6eb59c7 Mon Sep 17 00:00:00 2001 From: LiangSu8899 Date: Tue, 25 Aug 2026 03:35:30 -0400 Subject: [PATCH 15/32] perf(ggml-adapter): batch the persistent-KV tail copies The prefill graph ends with one tiny f32->f16 row copy per layer and KV tensor into the persistent encoder-KV buffers. Batch a run of them into a single launch; every copy is still performed with identical rounding, so results are bit-exact. --- .../structures/adapters/ggml/fr_dispatch.cu | 58 +++++++++++++++++++ flash_rt/structures/adapters/ggml/fr_ggml.cuh | 8 +++ .../structures/adapters/ggml/fr_kernels.h | 9 +++ .../structures/adapters/ggml/fr_repack.cu | 42 ++++++++++++++ 4 files changed, 117 insertions(+) diff --git a/flash_rt/structures/adapters/ggml/fr_dispatch.cu b/flash_rt/structures/adapters/ggml/fr_dispatch.cu index 9f100a52..456994a9 100644 --- a/flash_rt/structures/adapters/ggml/fr_dispatch.cu +++ b/flash_rt/structures/adapters/ggml/fr_dispatch.cu @@ -1621,6 +1621,64 @@ void ggml_cuda_flashrt_dec_attn(ggml_backend_cuda_context & ctx, ggml_tensor * f } } +// ── Batched persistent-KV tail copies ─────────────────────────────────────── +// The prefill graph ends with one f32->f16 row-copy per layer and KV tensor +// into the persistent encoder-KV buffers. Each is a tiny kernel; a run of +// them batches into a single launch with identical rounding. +bool ggml_cuda_flashrt_kv_tail_cpy_ok(const ggml_tensor * cpy, int64_t * hd, int64_t * n_rows) { + static const bool disabled = getenv("GGML_FLASHRT_NO_KV_TAIL") != nullptr; + if (disabled) { + return false; + } + const ggml_tensor * src = cpy->src[0]; + if (cpy->op != GGML_OP_CPY || src == nullptr || cpy->src[1] == nullptr) { + return false; + } + if (src->type != GGML_TYPE_F32 || cpy->type != GGML_TYPE_F16 || + cpy->src[1]->type != GGML_TYPE_F16) { + return false; + } + const int64_t d = src->ne[0]; + const int64_t r = src->ne[2]; + if (src->ne[1] != 1 || src->ne[3] != 1 || d % 2 != 0 || + cpy->ne[0] != d || cpy->ne[1] != 1 || cpy->ne[2] != r || cpy->ne[3] != 1) { + return false; + } + // contiguous rows on both sides (row stride == hd elements) + if (src->nb[0] != sizeof(float) || (int64_t) src->nb[2] != d * (int64_t) sizeof(float) || + cpy->nb[0] != sizeof(uint16_t) || (int64_t) cpy->nb[2] != d * (int64_t) sizeof(uint16_t)) { + return false; + } + if (*hd == 0) { + *hd = d; + *n_rows = r; + } else if (*hd != d || *n_rows != r) { + return false; + } + return true; +} + +bool ggml_cuda_flashrt_kv_tail_cpy(ggml_backend_cuda_context & ctx, ggml_tensor ** cpys, int n) { + if (n < 1 || n > FR_CPY_ROWS_MAX) { + return false; + } + const float * srcs[FR_CPY_ROWS_MAX]; + void * dsts[FR_CPY_ROWS_MAX]; + for (int i = 0; i < n; ++i) { + srcs[i] = (const float *) cpys[i]->src[0]->data; + dsts[i] = cpys[i]->data; + } + const int64_t hd = cpys[0]->ne[0]; + const int64_t n_rows = cpys[0]->ne[2]; + const int rc = ggml_cuda_flashrt::cpy_rows_f32_f16( + srcs, dsts, n, (int) hd, (int) n_rows, ctx.stream()); + if (rc != 0) { + GGML_ABORT("flashrt: batched kv tail copy failed (n=%d hd=%lld rows=%lld rc=%d)", + n, (long long) hd, (long long) n_rows, rc); + } + return true; +} + // ── Gemma-style norm fold: {RMS_NORM, MUL(w), ADD(mul, norm)} ──────────────── // out = rms_norm(x)*w + rms_norm(x) == rms_norm(x)*(1 + w): the adaLN // modulate kernel with scale = w and shift = 0. ggml's own fused rms_norm diff --git a/flash_rt/structures/adapters/ggml/fr_ggml.cuh b/flash_rt/structures/adapters/ggml/fr_ggml.cuh index bf33df67..c97091d4 100644 --- a/flash_rt/structures/adapters/ggml/fr_ggml.cuh +++ b/flash_rt/structures/adapters/ggml/fr_ggml.cuh @@ -125,6 +125,14 @@ void ggml_cuda_flashrt_qkv_prefill(ggml_backend_cuda_context & ctx, const ggml_tensor * v_mm, ggml_tensor * k_cpy, ggml_tensor * v_cpy); +// Run of terminal f32->f16 row-copy CPY nodes (the persistent encoder-KV +// stores at the prefill graph tail) batched into one kernel launch. A node +// qualifies when it copies [hd, 1, n_rows] contiguous f32 rows into +// contiguous f16 rows and nothing reads the copy back inside the graph; +// all nodes of one batch share hd and n_rows. +bool ggml_cuda_flashrt_kv_tail_cpy_ok(const ggml_tensor * cpy, int64_t * hd, int64_t * n_rows); +bool ggml_cuda_flashrt_kv_tail_cpy(ggml_backend_cuda_context & ctx, ggml_tensor ** cpys, int n); + // {RMS_NORM, MUL(w), ADD(mul, norm)} -> rms_norm(x)*(1+w) in one kernel. // The execute returns false (run unfused) only when its zero-vector cache // cannot allocate during graph capture. diff --git a/flash_rt/structures/adapters/ggml/fr_kernels.h b/flash_rt/structures/adapters/ggml/fr_kernels.h index 26708a53..9ce384ae 100644 --- a/flash_rt/structures/adapters/ggml/fr_kernels.h +++ b/flash_rt/structures/adapters/ggml/fr_kernels.h @@ -156,4 +156,13 @@ int decode_attn_decomposed(void * cublas_handle, // out[i] = a[i] + b[i] for n fp32 elements. int vec_add_f32(const float * a, const float * b, float * out, int n, cudaStream_t stream); +// Batched f32->f16 row copies: for each pair p, dst[p][r*hd + i] = +// (half) src[p][r*hd + i] over n_rows rows of hd elements. One launch +// replaces up to FR_CPY_ROWS_MAX individual copy kernels (the persistent +// encoder-KV stores at the prefill graph tail). Rounding matches ggml's +// f32->f16 cpy exactly. +#define FR_CPY_ROWS_MAX 40 +int cpy_rows_f32_f16(const float * const * srcs, void * const * dsts, int n_pairs, + int hd, int n_rows, cudaStream_t stream); + } // namespace ggml_cuda_flashrt diff --git a/flash_rt/structures/adapters/ggml/fr_repack.cu b/flash_rt/structures/adapters/ggml/fr_repack.cu index 9d94dfda..b3289863 100644 --- a/flash_rt/structures/adapters/ggml/fr_repack.cu +++ b/flash_rt/structures/adapters/ggml/fr_repack.cu @@ -413,4 +413,46 @@ int repack_weight(const void * ggml_blocks, void * dst_packed, void * dst_sf, return (e == cudaSuccess) ? 0 : -static_cast(e); } +namespace { + +struct cpy_rows_pair { + const float * src; + __half * dst; +}; + +struct cpy_rows_args { + cpy_rows_pair pairs[FR_CPY_ROWS_MAX]; +}; + +// one y-slice of rows per pair; conversion identical to ggml's f32->f16 cpy +__global__ void kernel_cpy_rows_f32_f16(cpy_rows_args args, int hd, int n_rows) { + const cpy_rows_pair p = args.pairs[blockIdx.x]; + for (int r = blockIdx.y; r < n_rows; r += gridDim.y) { + const float * s = p.src + (int64_t) r * hd; + __half * d = p.dst + (int64_t) r * hd; + for (int i = threadIdx.x; i < hd; i += blockDim.x) { + d[i] = __float2half(s[i]); + } + } +} + +} // namespace + +int cpy_rows_f32_f16(const float * const * srcs, void * const * dsts, int n_pairs, + int hd, int n_rows, cudaStream_t stream) { + if (n_pairs < 1 || n_pairs > FR_CPY_ROWS_MAX) { + return -1; + } + cpy_rows_args args; + for (int i = 0; i < n_pairs; ++i) { + args.pairs[i].src = srcs[i]; + args.pairs[i].dst = (__half *) dsts[i]; + } + const int rows_y = n_rows < 64 ? n_rows : 64; + dim3 grid(n_pairs, rows_y); + kernel_cpy_rows_f32_f16<<>>(args, hd, n_rows); + const cudaError_t e = cudaGetLastError(); + return (e == cudaSuccess) ? 0 : -static_cast(e); +} + } // namespace ggml_cuda_flashrt From 46931455b533b6f46e189e353041c8bd297e28d0 Mon Sep 17 00:00:00 2001 From: LiangSu8899 Date: Tue, 25 Aug 2026 03:58:00 -0400 Subject: [PATCH 16/32] feat(ggml-adapter): AOT FlashAttention-4 for the SigLIP vision attention Export the vendored FA4 SM100-compatible forward (CuTe-DSL) ahead of time for sm_110a at the padded head_dim-80 shape and run the vision FLASH_ATTN_EXT through it. The padded Q/K/V/dst buffers all share one dense (B, S, H, D) linear layout, so the f32 boundaries reduce to dense elementwise converts and the f16 K/V pass straight through. A small shim file provides the _cuda* runtime aliases the AOT object expects, keeping the build free of any CuTe-DSL runtime dependency; the module loads once outside CUDA graph capture and the wrapper is capture-safe. Includes the regeneration script and provenance notes. Refreshes the e2e action golden: the vision attention numerics move, and the distance to the f16 reference improves. --- .../adapters/ggml/fa4_aot/README.md | 30 +++++ .../ggml/fa4_aot/export_fa4_siglip.py | 64 ++++++++++ .../adapters/ggml/fa4_aot/fa4_siglip_fwd.h | 101 ++++++++++++++++ .../adapters/ggml/fa4_aot/fa4_siglip_fwd.o | Bin 0 -> 118488 bytes .../structures/adapters/ggml/fr_dispatch.cu | 95 +++++++++++++++ .../structures/adapters/ggml/fr_fa4_shims.c | 42 +++++++ .../structures/adapters/ggml/fr_fa4_vit.cu | 110 ++++++++++++++++++ flash_rt/structures/adapters/ggml/fr_ggml.cuh | 7 ++ .../structures/adapters/ggml/fr_kernels.h | 10 ++ .../goldens/pi05_thor_action.json | 2 +- 10 files changed, 460 insertions(+), 1 deletion(-) create mode 100644 flash_rt/structures/adapters/ggml/fa4_aot/README.md create mode 100644 flash_rt/structures/adapters/ggml/fa4_aot/export_fa4_siglip.py create mode 100644 flash_rt/structures/adapters/ggml/fa4_aot/fa4_siglip_fwd.h create mode 100644 flash_rt/structures/adapters/ggml/fa4_aot/fa4_siglip_fwd.o create mode 100644 flash_rt/structures/adapters/ggml/fr_fa4_shims.c create mode 100644 flash_rt/structures/adapters/ggml/fr_fa4_vit.cu diff --git a/flash_rt/structures/adapters/ggml/fa4_aot/README.md b/flash_rt/structures/adapters/ggml/fa4_aot/README.md new file mode 100644 index 00000000..0ba5f2fa --- /dev/null +++ b/flash_rt/structures/adapters/ggml/fa4_aot/README.md @@ -0,0 +1,30 @@ +# AOT FlashAttention-4 module (SigLIP vision attention, Thor SM110) + +`fa4_siglip_fwd.h` / `fa4_siglip_fwd.o` are the CuTe-DSL ahead-of-time +export of the vendored FA4 SM100-compatible forward +(`csrc/attention/flash_attn_4_src/flashrt_fa4`) compiled for `sm_110a` +at head_dim 80 — the padded-head layout the ggml adapter's vision path +uses. Sequence length, head count and batch stay dynamic; the softmax +scale is a runtime argument. The `.o` contains the embedded cubin plus +the host launch entry; `fr_fa4_shims.c` provides the small `_cuda*` +runtime aliases the object expects, so no CuTe-DSL runtime library is +needed at build or run time. + +The ggml adapter build enables the FA4 vision-attention window +automatically when these files are present (see the host build's +`GGML_CUDA_FLASHRT` integration); delete them or set +`GGML_FLASHRT_NO_VIT_FA4=1` to fall back to the host's own flash +attention. + +## Regeneration + +Requires the `thor-fa4` runtime deps (`nvidia-cutlass-dsl`, +`quack-kernels`) and PyTorch with CUDA, on the target device: + +```bash +CUTE_DSL_ARCH=sm_110a python export_fa4_siglip.py +``` + +The script compiles the vendored FA4 forward once at the head_dim-80 +shape with `--enable-tvm-ffi` stripped (the plain JIT object carries the +classic C-header exporter) and writes both files into this directory. diff --git a/flash_rt/structures/adapters/ggml/fa4_aot/export_fa4_siglip.py b/flash_rt/structures/adapters/ggml/fa4_aot/export_fa4_siglip.py new file mode 100644 index 00000000..22659a47 --- /dev/null +++ b/flash_rt/structures/adapters/ggml/fa4_aot/export_fa4_siglip.py @@ -0,0 +1,64 @@ +#!/usr/bin/env python3 +"""AOT-export the vendored FA4 forward for the ggml adapter's vision path. + +Compiles the FA4 SM100-compatible forward (vendored under +csrc/attention/flash_attn_4_src) at the padded SigLIP shape (head_dim 80, +f16, no mask) and writes fa4_siglip_fwd.h / fa4_siglip_fwd.o into this +directory. Run on the target device with the thor-fa4 deps installed: + + CUTE_DSL_ARCH=sm_110a python export_fa4_siglip.py +""" +import os +import sys +from pathlib import Path + +_HERE = Path(__file__).resolve().parent +_REPO = _HERE.parents[4] +sys.path.insert(0, str(_REPO)) + +import torch # noqa: E402 + +from flash_rt.hardware.thor import fa4_backend # noqa: E402 + +assert fa4_backend.is_available(), fa4_backend.status() +fwd = fa4_backend.fa4_fwd() + +import flashrt_fa4.cute.interface_fwd_sm100 as ifw # noqa: E402 + +# Strip --enable-tvm-ffi so the cache holds a plain JitCompiledFunction: +# only that variant carries the classic C-header exporter (embedded cubin +# plus a plain-C host launch entry). The compiled object is exported +# without ever being executed (the tvm-ffi call convention differs). +_holder = [] +_orig_compile = ifw.cute.compile + + +class _NoCall: + def __init__(self, inner): + self._inner = inner + + def __call__(self, *args, **kwargs): + return None + + +def _compile_no_ffi(*args, **kwargs): + kwargs.pop("options", None) + obj = _orig_compile(*args, **kwargs) + _holder.append(obj) + return _NoCall(obj) + + +ifw.cute.compile = _compile_no_ffi +ifw._flash_attn_fwd.compile_cache.clear() + +NV, SQ, NH, HD = 2, 256, 16, 80 +q = torch.zeros(NV, SQ, NH, HD, dtype=torch.float16, device="cuda") +k = torch.zeros_like(q) +v = torch.zeros_like(q) +out = torch.empty_like(q) +fwd(q, k, v, causal=False, num_splits=1, pack_gqa=False, out=out) +torch.cuda.synchronize() + +assert _holder, "FA4 compile did not run" +_holder[0].export_to_c(str(_HERE), "fa4_siglip_fwd") +print("exported:", sorted(p.name for p in _HERE.glob("fa4_siglip_fwd.*"))) diff --git a/flash_rt/structures/adapters/ggml/fa4_aot/fa4_siglip_fwd.h b/flash_rt/structures/adapters/ggml/fa4_aot/fa4_siglip_fwd.h new file mode 100644 index 00000000..c6fdc58b --- /dev/null +++ b/flash_rt/structures/adapters/ggml/fa4_aot/fa4_siglip_fwd.h @@ -0,0 +1,101 @@ + +#pragma once + +#include +#include +#include +#include + + +// Macro to check for cuda errors. +#ifndef CUTE_DSL_CUDA_ERROR_CHECK +#define CUTE_DSL_CUDA_ERROR_CHECK(err) { \ + if ((err) != cudaSuccess) { \ + printf("Got Cuda Error %s: %s\n", cudaGetErrorName(err), cudaGetErrorString(err)); \ + } \ +} + +#endif + +typedef struct { + cudaLibrary_t module; +} fa4_siglip_fwd_Kernel_Module_t; + +#ifdef __cplusplus +extern "C" { +#endif +void _mlir_fa4_siglip_fwd_cuda_init(void **); +void _mlir_fa4_siglip_fwd_cuda_load_to_device(void **); +static inline void fa4_siglip_fwd_Kernel_Module_Load(fa4_siglip_fwd_Kernel_Module_t *module) { + cudaLibrary_t *libraryPtr = &(module->module); + cudaError_t ret; + struct { + cudaLibrary_t **libraryPtr; + cudaError_t *ret; + } initArgs = {&libraryPtr, &ret}; + _mlir_fa4_siglip_fwd_cuda_init((void **)(&initArgs)); + CUTE_DSL_CUDA_ERROR_CHECK(ret); + int32_t device_id = 0; + struct { + cudaLibrary_t **library; + int32_t *device_id; + cudaError_t *ret; + } loadArgs = {&libraryPtr, &device_id, &ret}; + int32_t device_count; + CUTE_DSL_CUDA_ERROR_CHECK(cudaGetDeviceCount(&device_count)); + for (int32_t i = 0; i < device_count; i++) { + device_id = i; + _mlir_fa4_siglip_fwd_cuda_load_to_device((void **)(&loadArgs)); + CUTE_DSL_CUDA_ERROR_CHECK(ret); + } +} + +static inline void fa4_siglip_fwd_Kernel_Module_Unload(fa4_siglip_fwd_Kernel_Module_t *module) { + CUTE_DSL_CUDA_ERROR_CHECK(cudaLibraryUnload(module->module)); +} + +#ifdef __cplusplus +} +#endif + +typedef struct { + void *data; + int32_t dynamic_shapes[4]; + int64_t dynamic_strides[3]; +} fa4_siglip_fwd_Tensor_mQ_t; + + +typedef struct { + void *data; + int32_t dynamic_shapes[4]; + int64_t dynamic_strides[3]; +} fa4_siglip_fwd_Tensor_mK_t; + + +typedef struct { + void *data; + int32_t dynamic_shapes[4]; + int64_t dynamic_strides[3]; +} fa4_siglip_fwd_Tensor_mV_t; + + +typedef struct { + void *data; + int32_t dynamic_shapes[4]; + int64_t dynamic_strides[3]; +} fa4_siglip_fwd_Tensor_mO_t; + +#ifdef __cplusplus +extern "C" +#endif +void _mlir_fa4_siglip_fwd__mlir_ciface_cutlass___call___flashrt_fa4cuteflash_fwd_sm100FlashAttentionForwardSm100_object_at__Tensorgmemoi64i64i641_Tensorgmemoi64i64i641_Tensorgmemoi64i64i641_Tensorgmemoi64i64i641_None_01(void **args, int32_t num_args); + +static inline int32_t cute_dsl_fa4_siglip_fwd_wrapper(fa4_siglip_fwd_Kernel_Module_t *module, fa4_siglip_fwd_Tensor_mQ_t *mQ, fa4_siglip_fwd_Tensor_mK_t *mK, fa4_siglip_fwd_Tensor_mV_t *mV, fa4_siglip_fwd_Tensor_mO_t *mO, float softmax_scale, cudaStream_t stream) { + int32_t ret; + void *args[7] = { + mQ, mK, mV, mO, &softmax_scale, &stream, + &ret + }; + _mlir_fa4_siglip_fwd__mlir_ciface_cutlass___call___flashrt_fa4cuteflash_fwd_sm100FlashAttentionForwardSm100_object_at__Tensorgmemoi64i64i641_Tensorgmemoi64i64i641_Tensorgmemoi64i64i641_Tensorgmemoi64i64i641_None_01(args, 7); + return ret; +} diff --git a/flash_rt/structures/adapters/ggml/fa4_aot/fa4_siglip_fwd.o b/flash_rt/structures/adapters/ggml/fa4_aot/fa4_siglip_fwd.o new file mode 100644 index 0000000000000000000000000000000000000000..d7690a93457116593b3ba92243767a997ac218ee GIT binary patch literal 118488 zcmeFa33y#ab}m|H(%BlU$&ze&(vc+(l5Gv1Fks8aqddzKV89^BwmjfL@;)*?Y7Y1y zllCMd+0flgjYUI)(rG*Cd*cpVLL0(oAUBVf^m}o;9m3$I9i<16Zu;DRtzCQX zsx5kW?|bii2`}|wq<^ofRjaC2ty)#J_SS(#D;E2GK4B-H_^xn~{Zxqe-f*8r`EgWa zA!kG+*Io!T{(VU#@IUDva`lu7apFp#v1^cT1J`+n9C==&@nPW(I|&h*B_3FLiL+k!xEuN^c0K&gnH3MelfLrX{TE+6+rRVuH~J@! zf4~38CvWs0{mC2s$G&~7{|A$=^5+|7L?1GS0j&H|n4%&eW=HwkT~_ENj3EtwT_`tCuw ziyyv-_P*Kw(HkH3r&rYVeMqvTCD-;{`&R!CKK$GMjPIAkfAHe9{&Zi-z=v-fS^MEn zzGbFASaOcWs@E@O7#If{%R*oDQyx#~k2Ri!&=+e=n|xx%>ljm?k7$KwuK11RTwaJ7 z-^a7DT&D00J^L$+`_CE{^!?(t`Ou^2r=Q*T>SrxcD^j+xM zf9cYHd|kTy2kKIMY^w$u1JJ!7Y*(l+ z^-A#V(}8*C`(BBjhaKAct3cYBGrx+S|8>A$`QQI7dj7?aqvyZ#m(laje=B;v_tedO zr(YR%?)1yi^QR9M51c+2J>U22O%^*&O_`plg zF6}%0!cBeOes)RU>A>)T)6=4#o(_x{p!$A>`f^eJAnH4iCH5XfeM`Y>I{w)palM*a)t@##Rp!1mOFcfVOSqi^EpYnJ*h z%;|eYyuBhM-aZp39>^Ft@a|8OqX+&N_$FWOz=wfRl@<8ie|W*m$)H%%AF1m*ohIHs zd4I-}-w6yI`1$?Ap1gK{)61;~(w|IErl09Mm_JYf{y#Xn;N_`jhoT zHlD1+zaJdg{H~E`dO00DPY3d$2X_zj1#WgSQWS?{B_TaU~F_{Ba=j%uGBRc7N`ZeP!U2 zz47EfLk~rC_OaI=XnOg*2eO}h>$A=8{(V@yEsWAJmxgS3`7QL%MAUOSuzsNLmD|q! zBjg*2=cf;fw};^Ql#p0+`oW?zyYDZ0@>>Tt416^Vj4@l8Hssh^@Oiy6|ID8s+&Ey~ zpa10PgSi7cKU?tb(loK=9rWwOWd50F7T~9#IyMaCBi}F6@{j%Lv*PW${ka2w zioW?xAXxd_!CMAak$j6PD^Uv}(ALz+(hNchgY7%DvWOdxNd8TiG| z*1Wsa=R0%emG$TT$|wGHNU(QpAM`dIb(ex}_TI|0WaP9(AvWo%OlXe%f9q8t@!tXKmAN5eollwq?i8P6BR!H@{0SL-mOai zA5V~O{?Qm=mity?yzd;C@pj^5-@!WuYUpR+j)7B#IDcxjI6sA--#KtIejb6o`~dp; zW=Q0nJvj2*weYDC-^N`1X5hAgvw>0Pu1$M*M0V-_{>0h9$a8P`Ld)MmU2mbTzeE12 z5@8M?Kg#a?bXwPlYnXEn`}}YJ6Xw%Z{QiDm#F;0e;_XKK{@qvBoa>x8x8iH@x{BXG zZ?53kLeNgZhdT4@1AVtHtgCn;Z&u&3$Xw`~OFO-p=GwY}jl=6I9`Y3r7--v|JQ(kX z9<*pnvg#^M2W}r&k}Nq#dPez|TqrrmJcXa@JAH8Nz|!E2ed(8f^v<1=XZ887iq`pu z>t38MPR)MEH)3GPlAHQI$Vt9qM1Ip>y7Ge{t>5Wr*Sy1XUYv`zeJikf;0*L_$Q+dQ zoto`CGUrAAk-8U;JUC(?v*O0SGYiDq%Te!{GV%5@{LHKXKgom3A6z$Z2Kt`s^OX0E zOT-up~cb+@q8#6GbqORghVB@(A@!3szs?7r&h^O_VO?Cr{`;Zmrvv}e z(D$Q*e>3~a!8c}qFe&=!hd+*<|LDo+`H!ECp8v_qOZq;+n)*|$rPr};{wvnZ{>{J|_6-~mpZ@aS1o(;bKfhC)|F?IuP|rftw*d9dNB#5qe)W$6WdzC!@DS&( z96UDrr!R~1KRS45_9x#GsKe;{_*sEE#rY3^EY5#0NqqWQ>;E{7_G1}(U+RmV zr!m@|@%eR-5A)eh@#$|`v(Ya(=$~BlQy%&&pLrPIwWn@gU+IJ2pWSiK{Jy`&&qyHp z-+qjr#NU@buOg5)zasSbyuNT@^wWuWRyk!}#pD_DD=L650l&R;Ud7+tGQZ+2;9mg$ zvUy&`r@QA@{1o_(K=jj}ew^9&EA;iJ=<9(;#Q9$y6X!qq$8g`zH;eN>!}#^D!#FL+ zIJ^?*`|-h}(1|b4{?WnC**_`6dOK2_|9B|IDHCH95Uh{eM?^pUS;JbiV@=;@JE2RB ztI@WGzTdR2>Z5)}n@`w#hW|geA9{iAu>J`+hV=XDpWN~nK5Vj2zd%UfvCr3rpR**U zXG`$UK<+*G8O6Ua{^f~Ws{npi)6N!iPs^^DSdE9Hh~URe@%#NE5cc`}0iOtF!j!n_ zfOS4G&wl6^|DENsbA0ektD4hKOZoPQw5XUR(jdub%2^fL9&0z-n>XP%`)9{)3-iy$ z*!DeJ;-YF#YkN!8o_nj>ckbA;wbjL&_ep$jmGjHy*4=H*?V_r^CEi|THK}p)zV=8qe zPs`@^#^(0MMv!BzW(`tnZEbB$?bO<7zxP=Ua{jh_t6KN8G^+n=tU)`ic6|Yw5!>Eu zwru(0G{kI)wV3y!Mb;>7Zru}WZ{E{B^*?zF?WrrO%$8kWh$)12G=Fg#0)z0yQ?)HE zoAzyM+-f%OZn5;v`=Gf>dqSPiyldAsv$<_M+sHPvwY{~yqYcHC4yyT2J{P$y&276| z%*|g21;)^DTfAe*4?%QRaP`3#U<|oAUl=oFk0LinnQc|AZS6aD@Ay)S`Gp#?-=?g# z75?Hi8)GV+HII+OdK|O-bH`8kUBCR7d+%-yRL`Cr{r__B|3hxKY@J`XyJdG(>(;HY zmUhd-A3P@-8~5DX=;U9yXl0|7zp8U4+osx@O^w)3Hs0H8eh$5T^R|{f zH8UF9n$7kdwEJX-#;Jp3+T7fxc{yJg161DFxMNQ{jv$(Mp$CCB@7vPMJMws=-7nkr zVJmJn@7UGY4(G3NbL+0gO)We2Y-`-o0*A15bIX=S92HBHfF@+|syZpGox$<~H>Ys(DU%t3b zj^@{(nfeZk+pT3GPY)XWd3&z^{AX;oGxwkU^yZ7S`~6*xx!DMoxFj5aDCmQ^# z5cGWVATHJeyRE?puvIn~xyhnYx!KmQ%!4r1lk7}R^j2eZqS_*&RV!6F$u3uP<>hW_bC2d!g#N`b?L&T^CAmc z%{JVZS&52*Bxd5@R(jOd9*6tB0RCr$Zj*NGY9?Pi(%#y-D>gZT#jyphVWf6iRqf=+ zt@NX6=1ehv-;P~dBC(EGd&};~?tQWL$d;C^JNC3h);Bb)Sl4*#qP6RmHmqt~v|;_C zRSOp_6jK@^(;`!v&CS~*vE7ZhLTZknYLb=i({MlPk#&zai08p-12=v`B@T%hNuS~;?UE@LJi{%9@ym{78iWty3cRdhx7k!?Y=(Gsn0JW z8m2o_m5%PPa6=Gxlek-*hxxx6KiA;r$fWSE$Q6M`pD$ROB`(zpaT)(|r=mYABF_Az zIrMocI58q3s4fa~avfq_Lb=FcsMuKq9Q-Bl_fgM!jK{L12;L}+UMOXX1tLL$;;xs5t*{iOC93`G~|3i6=^2BQZqb@==LrNxWL(sKiYYFOj%S;?)u-CEg%$ zkHk$9_e#87;+G_DllX$fafvTUoRs*o#D^umE^&{aj(P?iBC&hBk@ZT zMk_v~d`aS@#P3PmBk^U4dnNur;+G`8F7XA42PD2EvGCymjlaZ( z#MdR}8+_2oO`!Go@n#K+D7@Qe1;EeF{Zd(p6~@cB249dz}F?F@`P)O zEIP8Fd`R?29AO+3kz$K(Ncu}7)%?@Nb%}c=J-5W7(|9nyh?*Z(xy-^eenih5XJHyI z!j~pk7{!1wKP2vz^qNT)-R9pmS?AYOrSofolSS>{Aj`MSv2ZTqu!zpL@Nh}LuvpI@ zUE#s2bsV`(#X*sj*yb0x)1uS(Q~RP4_sIG#bz1aZU?1o;hlRip2x_0tml7`lll)eB zSjM9kw#pl#sZT9$h$QR~%$ZZZ2iPqi61{(}V(8;{)$$k*3AezPLjBZV7bK3LJma_Z`Y%aL`cM4DJ39Yd ziQkj_Y9uE8Cw^`JsLPv_nC2h1M`AmF?hj$x!ONO&DbXe|<^$l<3zaT|FH5$ z@Pu#=FwqT(uLIj_JJDDFpO(+iE9oyuO!`3d+<(&Zw*hB?c3RReNsM9S{56+7^#fD= zz4%A`u1jq5U;QJCZu5IdVw+#?6^q{nSw0C&^5@$8BpxpD>K|KlJO4|-7$rF$(VnY< z*JzvG22A~H<4Y3T{A&K$q9cpdpTMx1oVY!}#GexP=leSSC191F`!7~rJAWH6`3#o+ zVLX8;zomZ|&krnq@8KEo?*S&dr61@|U`jR)%o;C(qu{4uo?VA71*Z=em!$wS4wtcA zxek2kHqdf0l4v|siT?%1SSZ){zm@%O#b{eem+Nx zkf=Xtd~9EOy63s=)4xiYk?j2T-eRHk*m*9YNP76#&v`9V_}I^TT%>p?i_j6TSBr~gy)1x57Ujzy z5Gg8=5)bEy_Zmc(De7sk+SXpr6z}AS`h7L5J3=Hg#YZ{dPnw5x82>h9i?jQKDX{~; zXFwv<*BKAH$_)OUaf$?#i7{3V`vKzuXKs)@J@F!B$Q7v{rEx#6IDc+1 zit?$;zswO=8&Ezwc!BZHfr(#MFp2`H%l{^EM(_gTf6WoE=7QWwzwSF1&j znaAUmx|&M2%BLB+JmEB5p0qSg9y3RlKVX$NhuHkM|9Z?!n{H71&v)>&g9Jf?Hh{oT z3`)Rw2>_T(w~FPV2IvUNe=TNF`Tx&li$C3$3bKayNFU~-{RX^0bv)AzU4FxG&MC_2APwE_lg<&*?brf(O6u!IwPvJslgDJ@^9;zV5*T9*n`X#?u-< z)(9S7iP>?Y@s^k!9~b9G^z!+%u*venZTpBSN#DJ=D=zBkJfm$s>>cQ%@1u;Az~lN) z1NHkhiYbgO`_~f>*8dFaR_d~4pHMzXYeicw%EMkAiU%3vkzv^{3$y)tFNf?U%x#jW zhS-t~dlpKSAak(5XNbS678^Pq3Jwte)Iqkln!J}_649O-XrdkxCMsa-(dSk>(A>sK9o1S z{K?>0{B!&of7n;7SLwvRZ4<3m)mX14iZ#IYdUbYxZYpB82kn2QLz=h4$in?4%^R6L zjDKxA)-$9OhM(}+{h29Y_Ye11n!Da{f2F6A=aHA%vx(LdNJ||+3FUjt5cQAMUyzsj zD;Feg506hyiUZ}T{5{!NkHhYI%;kfGt@6BHWM@-18F>7yxxO`jh;HP1%q?6e5kyBKcpX@eQX~AFdv- z)~ooEYHUHYl7I35VneCJ64`s49nh~^L5 zhu3n%3;RSW0DG66hyKC*@Fzmrc+?+Bu-^}r2#9Ode$01v?}XIf&P0XQ-xtjgFUOJ| z)W#3}f6)w5cOyan(Z3hXN{!#y{mS@a{4bi5Tz;H?vRfYQ(ewrSLHab2*yo6Jo)2-1 zN0!o`iDe*glKQe-`kQD!#;Y^2!fL;xPxC$b7r6PUzY_~R`EPRbL*DPiMA}4};XSZd zSJILHW>5Lqp7K-O^2~plC;xOUKk;AU@XzG>q8>cUgKHFg(VR%j3DPRkAK!Pi^~-1J z2i8Bx(&LwobFo{=d`&3-sc z<9Bxdcz6APK6ED1HGS)hm%8%<>tSbNlBd7QJb1F!f1QbP4<6~sKgvTN>Y+RP38_z+ z9(tBm9_!DLanwFqe_pMIeW=Vx+U(BEhh3@Je$Ug&^p1Ai@B zQ^-DxlKYwTpff%w|G1f+qW(!Ie}vc9 z@$UK$dy4g#*BjS=$>97U%Fp|+kh|XU_@}3^-df`iOzShRzoakdKdfJ)X#x%5^_Ork zoz6ysgx{q6Y1(>Tzb|2YYWsI+PkqO)#X{sq6#?2`|2}{3l@6*6N%}9H@u0ZC5AZ+8 zqm>6fVWwI7Mf=OwpfAWT@E<3%2m4Q#j{bY7!=FkU#Z;`%RhSQ<;0E>|;7@>GmMQ#X zu5dmO5dQ#wD}95-OUhpzCn_bh=O4&EH8HZ=NA;8c!wLLq+x_fn+5VP49>94KsB}e+ z^^o)<`E@J1_#j98cwf*GU;5Afgv9KxK%RZW$#X}7zmERB3+1<1nEO|&|Kk50{vQ~0 zer5aT#bzKt-O20u)f}VP^!pu*_LakaMu;w8KhfV^(Ek8VEU<`X(tew;f4Gf9;&6v` z9s{QU`WuFR*u7=xhd=JO&eN!WAV1}&1%vZrzfN?z_v8M9J}>NeQ~VvE5Er=pOK84v zviQByG0TFaPc?G?ll(FOdJ*z#`w7&i=@00%e-I|JOaGVm8{d-{>p%2s7WW6~AN)^y zzZ0hY&g(h;6PSp+3P=3Du?qT!{nD?|f6|{l8BYr(srxN{6v^68(Efz{JKLUKG}Gkz4*EYdfPW6Fj_r@v?=$e8 zDkV#QU+GxJi03=(_eFDwg(ZLTN398g$2_01c>>z=sV6>X4z<_rzn*x_R2o_rS52iu zQx{L3D$oMPuadu9IhFV`=J8(c?pJvIAE~eZi4h)rgO1}-DZeEz`Omqckh+NF?~E_x zUu}HCT)Ye|w8jtiyfeNC`6Et#jOQF0ueMD;%OroXC%$-b59KF)fxMmZI%dWHV?5VW z-+T{!fyTcxj?ILfxco{_`Dr3IDorrcYdyHygQs}#Bo|+7!2VRR@0-3oXucI=e^&x| z?EOP$VwUUQV7_BMx$_O@x3r(2`4^)71kM}kk31z_<@V70r~JVYRAXv5^dIv369F-j z=wweZAA91XEc+$<6Z@BdQ$OYh?^hyF!QaCFxp;5{z1$t5^$`5aO+TA=wujtbhM@V$ z3AiUde5&}Bs;`)zD=tQJ7tM%9*X#q+OEfypkE`_ZV>J57 zxPk7&VbV2A{)^^ll@57xCp+a?{}Aw^%`cqOpajbgdxY~~!vLil-@Sq=%MaDdCkjQi#y>}+f3!~wH^!zA zo0r6R2H=n7$@4Lr-w<#9qI~3!kUhH+LjJIrr18)6<}b)b{u0g~7N{>b3wR9Ubi!#F zz@r&wiVtJLvcKG(fTi!S4y^AukEZ>RtM9P4@HdjE-_}2@FQLZ>^ZHVa@ktv+E28Xw z?5~HZ;~ngbW3jdq>r;`3Uaa-^MYB+$cSB#pBgkGce=c9((xE>W%{&)l|3~W;tv9@$ zVZNdLQ;p#%F--RVp?EHvdSv181idC7^di~chs-b*Xihf0DjW1dNk3`kjwBBQ`#YYW zC_gz1^a7$^4N-lS8NhimzuwVF)Fp*+`5`)Nb5_vDX_%CO?azz_ayZF@%QI_0NM zK>PjB-_`Ux`7^+^?co$*rei*ow;QPe$}jz=+V+4oepH6(I8U_c%r7Je+x*ILo=4ao zZ(Tm(@70Idp$O%-*N?^JP{_^zjY@$2%68a0w_VNSV$HlWUONb5!kNj1#k2vlS z(;fTxkX$Ut{{FGwpU4z9fbNzbGN}BJUe$hgDdkCDvHfxEV>jB9nTmp)-Cp<~28~bK zYU_NvGp^1z+7DE?^AG)b(d6qkl(764;>Y%k`XBw<8K2VQ;kc5a$63#(bjV zN{#;#AX{EKKULR*=i+eZR?Gd=`*gm+*LzxdRldjL)_yCE_H*ce%)dePft^#|$#@2h zlrMwqJLl)?zTe8fCq6MFLgm?BkUbxt5w$V;9}oURX?|lr za?uT~(!l!G7J3+9(TD8J4xOQS=cA!AWKo8h|~woo_)gl zZHxjPx9Gf{+x9I?_Ko}hQLX>s|D`$e7voos`)3|cNBx4 z`J=qedU@Tx!hYpu)TKnewEtA!C5_+U^-JeB zc)fbhE>GjrA%YXZ-?Fb^=<{It9ez0fLjHpB$nUNngV&?WHh=Dq%-_>rgZuLXySyb| zemUy1){}I|r}ker+doG>t}jEY5AtS?MfqGB|6hbZ&;AD^pl7gtWLo-gPWAVn6M#r$ z{~KUGrca+m@=5!M`Jc{YIi5Z7OxORv(hYe>(D5#kHQqh(jIdH3`+-7o$0fa*))y=O z#^H~+7>7>wYvxRK{ek~N{sxl8-(7Dk|Lmg4%Yj2b8NcBME-(Fa_!DFvZMqNsy5?@B=>H7-A&*{8N#!p~;Lb;UR8Xw$$2l)QoUhjJ1>7n5+9p`bu z6#3&e9p@uq=X?bAN<07H^(jv|A8GLG`=J2x2etcN(9@mv;=EQnKjHmqmZrZ%5B5s^ z#(5i^mq$rHQVNVu{gL5r{%Z{wU+w$@^UbgM@6SP>)$K|3#DeoLr+ zXZ_B|aQ7crf8t|7l?n27#_9Y)p6|U%`lp|tc;$OMp26#X2H)RMdC&S9SJqROPdl$+ z`BtNPTR!;rG@pH$^8BVVUg)mJD6j8_NWPKk{(DgWVLi==5P#AqvMv{&e+GH1^`RT}Yvbe7 z>gWE=9{Eq(M{9b=hIsbc& z`d=SU^v9^IF}D2=aDQDu^{jugzYNfMJa9e-`xJES6YMkFpVPAc%E{hLR{Ado`|+|t zTKn^G&%Q*7W0Z_zkvamvw$)iolQ0GkHI(j}PgaF5h5%`atJ5ssQ}>{FC{q z^P~H7QNMI$zGI^W}AZPsg4wB=2=a9^4Oyxc{X8lONB{ zmi`K#7h(O)n??DFezgJPlRZcuoc5#t&+gB2zd)$iw%4 zX#YubWUk}CK>xun!vMcb$q(}{q|QIDKT*H0-S}t4ztQg7(SI~P-2ZZZ(0%O%)XxcY z#0SwyD-zk$Zk#vFa_@^kUlGShGadf}?u@6=@?xd8s>NZ1`}ju*><=;D-sk((5N+uB ze)ToVAF}T6sRYW?{R{a|d_Vg-#b<=9cmq40AMx+pAB2&g<8STzXZUm8c!q~M0;yTJ zKSyr9&&p2(0_>mB{Vw89vl4ERkiL;5L8ASDb-z4oIG|Hr-q*Y3`TpK5AEm-b zR{7!%>0Z+#+{d0od?(jG4fT`$ApVEjLm2HPf0F5Rz5;sr1AdDx&s)BX@!{lqI6&=x zH5Qf+@D%>ue=}u#gTOu=^dEPm-21+3VdS^xH}a1~dq@q@f!q5C}O zAKmZq288fCxL-(5r9$XW?DsPSK2AV5fc|03@dwn|l&tnsd_nJ2K++<#XJN-_xgK2L z{;lsYC^M3qpWZ)q^Z%Ka-#fnb9kWEaQXccO9-6AWk5a(z3xGf0ClH-P;ql`4RVIpY zR{1|De!9NnvPfF_A%C^5Z+}p{wcNh)e5!_j9CRBqxkFO_atuP`0Gp>PpsEajt0!9a0(!!V6+isC6S zJ`|5e8(u43tefk1;=g4)6!VwH?%c=W0)Q|SL@#e_i)47=U zyLNjaPbN9|mVM3fQ+w&Y#j-aI9Pj0{kL(lTg~+}l@qDV^H{ixwT&MS&6SsWJYRe!R-|m*dxXJgzjLez<#kAP>jeP<$Yw*QmoN zp*_#xe1XPmB;DV?2LIxL4A-84j`*@nig&Z(YoL$uF^hS5;`N@!kL{zQpE$41xE?hU&xp8T6aVk4|ws>Q#u1`pO?L6#Gf>?;(y~NVnn(eE!5J z>pSw2Ur)R)JIeKa%J-WTzsK`K+FzW{;P{s0!TVdxe{_oCZ&)9Jd*W63Q4DbEGT!F` z_9Nbv?71y3_J0KzC_la5h5p6o=A6=&N^rbS5TP@fuQ}I};h2es{)~i3s>}VtMAf@uZAz5>ZKJc@tB$ z@;&AaDe_-Z^gbH3&#X=jWqd%sUp6d7@uHy1@ybaRTXY`J&UhvEH!^X3V?FYW@W_+z z$^-ige?@Aewf}>DgxOw5e}V2#3G;Yhy)BMsX3~ta<6|iP(i!hJNq^*eEyo|{)3`uI ze!9<<@iLhkr2o5x?voFJ7sUworev0o`&-az~b9_DSOM`U26Sm`Rg4mw} z^L~%@$0mcT{TSBWYrn zO3$agnZuvX|9QL|eU;-UG0s=gL_#f(7)_--_TOouOQmPQl&bu7{q^354AMu$r)&2+ ziOFI*@pI&xBIth1#aSMDx=zRWSA?_Uf85_?9{%N?^5eAfe}?-P_k940mnn7qJ)S?F z`2u@K>m8Cc-x8zs_;kGABNa~4i#2}5aeu&_FV%wTci)ddd=Kv9qs$)v2k&kI(|l@$X^Ki$s-8=llg?f`*5R@fyyfvo%Mb zh#yuw=Y0-;A{S?6lFssELo|t*{}7rx$S zq&Y=-v=94ZZjLL@*nBsCH|W_`_>Cj)2<{G5BggN4^QeZzHJj99uhxv>E8K6_Fv^s@7G8L zlJ!s48}Ki2d7ytGam^mz2IN=l0qFESf+*ETi7@1o6HfN`P_4f% znu`5Ed%X6-t6y}UOYa+Sd?fZaZv2YXf0}*a_6&3F3*EH_Ck}%bVxgAI3q=-q84~ z{>or`k*nDYy3dMO0`PdycFLjK>G!DNJxKpepI{j7vo9%T>D1% zja43hWrE^gki8i%y955ht89N<|3dO}*SB=OPx9OQt4)-DjB8(59{IsX$6vAIQNf?- zm5RSa_F=5z4>bUfaP804Zs2@rKTsa=)b&SxZGE?o_H#eXf&Jm_A4Uvffd63olIhvc z<$3mV=^lGyc_su@}hy`~0Uv zxNjnN8;SMTjdy4JK>Lryeh{tq_2~Wg0R9Li>^Z&vj`n-?6Yu*e_K56_Vvq2?qi&DL z-l+Bn{?5M3>`w{0&%pkQ;(K^MLY30{Zg^kDk(ckwL+<_y?WOZ9_8(~d!u?h;;+>rJ z=M!H4-Se$}>wSbY+y3(T)+PQReR;#f08{+i@4`sou2bivM+K0VtG3gGwkmpGsgaLmTn(&HTwsB zDsty*Xplcn^S_#U!?quo&$Wad|0-QSU-F)};yg>2Z;B=#?I+dsg7%ZDed&oWnWfrq zlGpWjX#cjt!+)iR|56YCa~s)`x`}f2w~8O$hn@j+17ZXFoF2^#@QM z^Ih8?Vtgv=V@5?Z$jcZT2EHX?+>BA!Ty83AA$YIRqe;`^Di*}bJh9ZiTS9` zFFK#m{i#TzOSeIdAbk1LifG=c_uu7*?@l1_rRR{`%Q$?-S6RGKJk4y-RIN& z?sTk&=7i7$KzqHU_$K+nf+3z?h5PTc3A916=3h6;E8i=-M&H*7x#wZ862G(wXpgo2 zay(5MpD*(JZn#fn`Rx1Wt&~5+AAsWeIe*B#kLT~BARYndo7`T+13;b-E!Zs2TlD>! zG`D{AC+?GppHhFA?+3&D{gejk{|Thu)_r_6$!o3m&V6v0?vJhV;P*JbPe}%w<-giZ z_szWFnn3qg{5`NVy8m_Z)BUp}AIjtYH;oJyzh92~J@5T&{l06)kL-9U#1ql?Ovt~p z$K#a_+D#xG#y^Z7t^YJ0tUu^4=pTO%0+0B6eiTp6{-A6R^7HpA+ScNIF)Kg64*-$* zKGfk?*`G!G%imre@j9S8{3`po{2}>%fm?sjec!;!@9u|G?3B0Wn_E8Qp2u7H-Sc)8 zyXD>XzefHwMo%8_Xy^4ZKed;7)SiEkU+yQ8+X#HmWr~Luy*XA?& z6Y=-1|BCtN$9$mt@P~LjiZMRUdWrnBzJ;*9x!)sLLZ5R(BL8;qdZoqlBOX>ezaxI~ z{u51xJ&^JG!{zxY$Lp*78}#)M@n?Pm+J^N7^78k0nI8;3dJ*v<`2Gvxm$<*^`;#2M z^gVGGiEG6-`L*~c8ISZjk;#9i`uF`z`5w(rVL#ySnxM;g4xFD-ZY0}&h28l+mc}#7 zjqgMMQoJ7ZC&w4P-azMvJRdlo==E;;o=V7x55jnzzZk7m6SUjg^Nno1{m=(XUhDl+(hm#&g6lhB=7;D2&XT7A^Cgev z!6Rv3va>0_%^&4+X?*zmDBS)bX>31f{$1%FH2$5~Uy!?q#NQ)*3h`K!a<30?d?LL+ zqxze)K9D>}yx$>zl>7^8J*!50)9C}ojy&3YLw_BL=kWXa$dXO=M4QiKZ!q6gf2rrM zL%e_I_F;Zfd_5^JW9*M<{!>Jj`@NGmn|2Hl=KoJE`_LojFYJAB2dxH3rT7Q^A$}lR z{2&j-OR_^v_6Gfj{k5OQ-#Xu=`wE5s^=iyN45vN+z@N?!glH0J1J?WVDm;Ir{X={S4IdKb>(!iKe*B?81PI?JqVGom|4|^S;V%c8Yz%pEKA1=I)B65c zwfF?z1E={U#AN>7Cj9MS1c<+{&H88rqYB3S>9FrJE&0RoHBcwc@3-==g?ey)Kh4K*d>zgcm>wXyBM%K4@UVn+ULvj_3=8~m zKND;MiTeW>?fD}>B+FjXc|sD%ribI}q5jNY%6|*HCwRu^-{JVJD9Ci#p4*^4ET6<1 zAPLi@{I_Gj%XEqFz;t4|E`J0yFFsAGB)<6jKcd>aW zbx4!<2*>Z)^74E+hYWJP3dc8%lKi;6`SDbF66Dl{BhXPs$U=@d@__<4-z-xBOvCJ>$*UdMpwmzW&r;q4_I%alar848aqA~}9<=9| ztiRKqU$Xv#az0``L3_gS@^V>T)<15Xluy<_e!QgX^`llxKPFDp>#wMg}(6nlQl`m5~ut=I49XAORH^$q%RC<)PAdsl1EcPY=|5;ecqAE5HO zeonRRgVfIn6V&pY-_=j_=QOGRcK=pOdjz4--_XxPs(#XXa0Db?zhHmEc)wlpm-;!~ zwm-6eX4v*eV*J@dtNnWafTuM+2iDwkAFeukxZB4 zL+=waf4zQ3KeIgk!4TI!VExPX_y;*2{{ZJ}7Ju2^Jl8+q{-VE6@;UWK?Dik^tmntv z^&Iu5o5NPFLRRqL?~$gPMGdPJ9FDK>_zOo|e*yKUn}uuED*4mRf^}OIe6Kld*m~ew zB>#KOqM}=YZ*y>A;RfK_9b8aw2k>UX;7{f61m5V-i;8GJe7A!O3mbu(99&S)47}CB z!-j1F-sa$QJFRS3=5!`j99&d{1xM67xUdilgjnj}f&wf6VmVwK}7ZlL?aoEAbh9!X?B+UGaiaLQ0N_rUkAF0ngKfU&d_?IL9IEPij3@q*pY+O_X zgNQpeV94*af5Z<43o+bw{(=G+DBLO9c-XKYaLB<$MKC}jEO9tKQm&792>#)Cm%YBq z`G4GAUnPFnUSDhleJYo9_0jqxs3%fu+ z?&L2nco-P(OLKq0UJiqSMY7vpSojs7#~fT-`~+~1#G~Z-N!#^)$^TFh45}z5I`-F@ zSRYybOW02U8|K#zVX?9j#?vmpb6b=y>fG9Y1@Ej?Zk+@e8->`00&0ejI=CR+VSv zIvqc^UdN|z)$uoP)A5!&bi8@9j-R|!$6voo$Gvy!_*A2gztN=Q(qQp zn~rbUuHy|mbbR|gI^MWb#|67|T-c)G@&~r3Xg*u#Z_uaMBjOQYXbaz8l_hRN0EOEB zcigDsJ8x345&Nq6s#gB)n^k&Q;;z{`ZknTFBlfs>Tr0n6u1YUU6z$gWustd^Vo%YZ zUs2>A-m22e62)yg9<^7+M(i2!j8?t`Z&a)8kHmD`Sf^qm_PlsrE8iSd>1Bx#^K?9V zzKV_5O0iNaKW2eSFH4MFsN)+Kso03E73^<0<2P%uN-s;?T(9GsmZ;c>Jtv;i>YuYz zrI#gUFVped-{)KzOuMlzAVwuuH*IlRBXh) zDZZ)jFH5YuSEZLFZr!irx(*c^u`}X~R({@>RC-w=dY_ILBvfp~UJx(X<#GO7mY9FP zN-s++dO*hu52)CPZ4q1S@;E;%OVl4$>1BzQo(1o2FtT z_6_k3t^D-qD!nW*V}_1r&Q!4xE9I*Lcl>U+L8X@^meGb(S$|tXDmG%P#44@+t!XN~ zEU_)DW5ZCf5nIjIi*Ehf(_MNs*8d$DF3thICsV~nY>oBz(j5KE%2MfNiCsf>Y{WM3-_do;$MRKrS;G3@o$Jr+U7^yA*zMR$s`Vc^ zqSDI}M~~|G*fA9wv5oZS!4&zsx>R~u;`nhLKm4$YjaUJ`m93U<9jVgG65ZW8K6z5b zMywFu-jn5VUsRSjaYCh+B_4fL$6xu1jvsqW$4@+=gs+%J;hEyTN~GA@#RKuSWgDtoUcAeEvq#2e_ zLN7J%eVqD(6V``1Gy5B-0NnEOdj*dCLwd=c+VoQM*{|~YOANSt*3)F~EqVjmUt-0} zQhDf4ote3Li$l-B_-$Ux>kFP);}>2_{)o-rh(&lQ=kdq-mmfbI;P%mdBkC(PSFKXY zi=T8@*V#c)j{Qd%^_Ph<;w|NeejO%g>sP56e}wv*6Z6}+Q5_E>)*u>)D9fY2zJ9Nj zo7-DzKK-=Xe#leE_LA~|pAlOx*3<7!`7JA{{hYYIC!dUKxUhuhJ3T}ElM!1d*3s`y z`CD%B)W3$;N6t_E@#Jb=ZzVQjw~AY}^6k&CKjP#sEmh?)Vz=Qe42kN){;t$)Kh;i{ z63bg!K>EXp+jobc^D)kkcBF7kPCGxw(}>-Tvol%#3gNq~_z9=I zg?F(1p*-9lO`?f@Cpz@0)GVM2L@U7_7_mmtNJy49Vw=P!8*_clqS?lfANqJV$;0~g zD$U=lrn_v+^^LIJr<3);{@g+R!}-xauqU_D__+Ey+WKCLtRM4vJ?W2|f5@gr8?(G) ztoNZPKjf=3!(|bNo`d;aR;KC`>{|&AK$QB%TJPUD^@WPbUPyURzPNY;Feh$r_6Taf zMJN5b5odi!PWysjKP75tgiv% zHGCb($BFglW}Gb|$@ZgsLy<+5^^F=u_L39V-$)m}4mM&p;oHnevi;=`40qo*;P&R; zhyxU-y`|<|bfIh|u73{BGLdBa%d3hw{?Oi^7RQGVF9F$pU#|S=BK|cczh7a*X49Yd zl=YK7j=GJ;hZEPIcQf^m)t}V<5hGlE&O!gr#n~y6)BfgWwS1{LdbF$GG#~F8OZv%) z<*yTU1gQL2qBqWJWFr2@_2hCVdk#QINpxcoev)gsCMgM4#F5T*n=>Yq*H z#|iN`Ml33#1Z00ADeXmlOL6wB zmLI*)Z9mnwL@crCT;Fn>O{@9uTI|YC^_7cqhfea1v%ddD?X}voL~qY{>wPTQzea3= z^*$_P8jt33y*(4fM1?=vGfrtw2=;0m)$fi^1rx8Eq)Dw7JZJh`=W)b94j4D0(XPW_Ej_5PS? zz3;>2X?*ggkvwjH++e+bM0Ci9{+O=!$1>}C`cC=$8PlEe4b&eq^!{kUT^5qe4}L3L zexzSpaaZQjVP9^T324bvjrwQKbo-C;Z^K=h%nyCg{XhEqPeCm{&xq~d&yU&q-}Bej z`*s|k2mOC7M$C}>_;)qi`um=}bU#9W540Qg-6QTHx|H9D?V>-|tl-^(?&oYe=W4}Z?grW>&~+}$A&Kg>ruk7E%W{w9a(x%u}By5F-g`mfFHKhpoTxZAMlA>gCp zsDr7$j^VDurc-@gR{XK7FF$@HK=N=rALK1H!?*JOfy(gwd00G5zsvlkW@!B|!j#Z| zh{xdSZTntoZrT*Ks7yCvt)i7Mm1liwY6>azI&;V|lGhsVZqz^QZuZ|P5BM9gZqZG@ z%lhH3t)>2u`ilHbH#QNS67s_ynfiX>B<@;}s66@`{@WfJFSq=dF=~5Y|8|X3=tk^> zp!*rO{_s6J6gt{_^CqHG;`(!1slROfHDZri?=#Bw!`^KpeOKzAJBG?q;_|_kW%$*m z8?mp5uMkbPAMKsjM3@ri&pk=~W96su+q6KH$A~>4o}j!=`7!lsec-pWiRhHT59OEZ z<%_NNt7QL`nqwxYCQ?+d`byik!UO1(twCHopedq?cl>+9-r?Hi5P z%7tK zI59r{80#ZFV7&4Pk9@_te9M-x{c!TPv^>hpobp|s^2gQk6z@yd8@yh@--13Z)AP5q zsQKyd3vOioh%diR`gClW+Wz`|Kg?i%klyFO`$t;5EBZtKUSq2sFOT-p`!1QpA5)9I zZ>+_~V?Cz7XP3_PgO2T@5A#XKLYI#9;ggPXVXa@tKbF?F z(3hS3Xy3hN_n|F#(8BlW)rddp+G@>D`#%2oc3K|<(cdEeL6`rgY~(+B58ERy&-}W{ zpV;l-t}dEitqwkZoaXml2Om93;~jHwcXvDRKElwCd(Ez{dx7^m^yA0lz#R@gdh|=c z_c^$`I{|#ZgS)yO06yU0xfJ@73K?&`V~_%;V0KfVF@ zcEVErJAm(W=-u5kU+!{nSJ&OZjSfD3ya~A3!AFmh{om~1?(TBnafDyZ@lpSe2cF>2 zj~|~1T;brOM=OCRIk>xfGVl}!cXd?(S3CIl@fzS-!czXJz|$OhcQ^UpGaTI2MeD;2 z4nBUI&LeJg@X@0;0pIN4?rz%8&mk=PzYch=LqC2z3OvujM~}`2Uf^KsJ|V=on#Vs4 z^sqxeew>!hbiz{p4B$+M-rb!AJjB6WUD?1n4nBT77dX$sM~~(M4|Q;NcL8uAVJUwR z@Gysd{P=L-Vh0~RO6$c)2X|9|5&iuF8GqE(MgByIgO4AN0GAS$@{a{BlXT4g?k-@S z?^oGg{1HR>{>i=O(Jopqj#7Rrz6Q%pG9{e&ny#a~_ERCmdzYF|bseJLVe=_}B=OW! zLBb+xh>P76pYs&J_eIms*#2qp2sn>5%qI_7@_+;0&oRtHhfpCUpB1n3mO5f3;P| zUu)CxV|#V{gsJ1lV>)i;?$g75oX z`u+Fo^auEU-=!Zopwp8{9e48myqo`Ehfe?c4jn(eTE|--RIxwNg1Lu8>pRwO?|#vU zKL!0u&P9$8?GMWmr`D+D)8kJ+EyQpBn&$(TSN(U2S8QoHfWm(H{#jX~sYxyGPi&)^ zqtx%&f3~j%|LAfNMIP|aOSH6*y-DTL`-7y~qK@Oe-1-X&)cj?M!a@}rv35cJ zpQW#)|3yVAz0SPnG3p-2O3RRC-yWq(sM&h>DF^9I@`QJkS5JV^w;c zxoaQwkK2ALUd3(ymk?X7me1h$6iLT^CQbVujuE>LzO&4a{;D%~@2B?L?K5Ht#HLF+ z*qickAmEoeJHE zJt(%=81>bedk;{3F8>bqyp{PMgfEZ8_KxaP;-|(^eX$3LKbObf%cwKADCZT_KJWdi z*^#vKUughcJ+CHydk)fgS@O{Q^PWd<{d$i>r}J>-JX+o_QTsyjyaD^O!+=OwHMqa^ z`$_M8p%GK=A348zpGWOc?gyE!+_y2-<7>2d8K&#;Bi?urZ+wRlQ|_C&{7CEXC-8V+ zy=iY%*1KxpJx!`TD{XIEr{f*4|E|3&ZEtA>cGkPn_MLllytPfm>G3V6RQrqbvZrx= zkfa`_{;h3)I`#GIU!TJt(SSc(YVK^pe%7I5Jzp3^R(pPxn%mZGRp_Pd>##jm<=Ktl zbmc8eY+tUHN4()C1*5)oYtdk(zO_pfdTv?!!ZpvO?WjYik-SD(`;u)6M*r>It=5nD zvt>$oQP#eAP4Ulb=PtC$121e*%M(AMFLCJ0cB=IBIKAJftY0Xv@+&p>ZrQ8JzolNO zAM)4ZI3cOp)hmt>!;uTjBZ|c9XVd&=ybf-H% z$`X55D*1Cu+fUV~bR(w3$6$Vznhg!Ayk+exmaF|+me_fx%dZ;#sW)DD?QJf-0sXsT zIrRtY2>9oj=4uSENS(9du_%5B10eM$0D5cdLWK_f+2Hb{{^0k29exxabdOV>^lwFj zB5yFyY-_7m_?4QcdSRfFRKyyO4eog0=84Md^6uVEV`;+_!Ic?;&f89jwTe|}bsLdbrn0tRamgHY^PoqPp_Vx-? zDCY;!SM9vRp;PALDe-5R z&sw}D;|OmpZGYZ~>G5Wz?W?D`_6_IX@s+h|f0nkdin?^#kJQdn$EUP?@oLq6l_lnl zSLm>R4Y=O0?I-y^OJN>edsu4LPs4mpQj2K*!(L*)sODct_V*2kZZ3^tz9wy4);_*o zu`iIH-cNVeYuq=jT&U(xES#Xo2YWcap6ttbV#xJRgK1RiYhMn2t~_|(qCzPT`!WIM zEa{X7zom*ih@Y4?6&>u#o2ai;%47Xo#P37c`byiH_Bc&SQiJFoZfH^86jpGeQBKg`-%3`mk7VfZ7H)bixz;UVci+kM=DCf3^RsX6X4R)~fOvu^Dds749R^ ze+%6F@HfDp@|WA~fqbgG6wfx9XxtxulwUW_EuWK{9;fdwsO?>IipsnE%2%rL!`@F( z%WLs>=+8WJ_38Jt!+y%7gzj-fP(Xqs6~-|4k+T+v-2|Cu(o3voZW5@B3nP z=Hvy_Z94lCRV(TK%i`Y+`PBEbsC|=GHQ02{KXEPlcX(#SKa~oSm*@AUG~}N8_giWxAfRpW zb$;I-@i)a0V5vXQ2R)t^@ku3PY&wr`?&wiA#{AOaZF#=sj2vOpd3>|WhTE9)EAh9i zpF`aEdFG#`ybnbDmH1nfr}k!nAGeqDhsTd|_*Wyp9&hU(RcpmlfED#mZda}Ky;=G9 z%d~h~@Gp+n)RYjvH2jbG8xwiG0JP?(8edED>G89OkE>++%K1qiJ$_b;m%T#sC)16G zeYF8Nql)b-mB;tO5dT|Uj$e6v@V!nuUdZ-Of9LlU(H?C+d|o_JajMpdC;BMF-&3r? z{Rn?=5%SUBgQx2fJKmGNk4W|aHXYv!?-mjB*i*+CN7+7?n#WVe8RLJbztsH7!*qVn ziSbvvzpCM`E~`8qGW}~kbl%R1@nc_qOv5KmJfY!}Cm+}FqmSzKKk`VIM*qsa9(nfb z@_a4sssEs-{x6--_?^7(q=p~e_o#*+X@5k+4?LjPf8c;#e=@1p-_fDh-`T0x|KNk3 z`tSGDpU~@XZhk)*7=Q~z!2wfvhmY}Rng9iIAk>h<4w zC++V!ar-uI+^ylgx9-*Ordwz~%8B!@Z}ilEk6!Bic z@7C+T>n^?idGqx87cTVFzliqdoLJr+i+5;vSN$#x-@ZhD-)P;syES?=s>YL+n)Bw* z)94FlEY$FtHG2K4SL^jRH0brOTBX;&md>}W#PTm)8rAT;nV$M@@YFwRjh27)N^H2T z#QYj=B7cVy<5f3fgRS7T3)X6Q>GY)%9o|3**!D?Ift*Xv)lOs{|LTu=RTJoV4k z>+kK=>wo4Mz5eoYz5eQIz5a3Iu%WRMw}1Tjxf(uoO0WOvr}g?PDtfj2m6gwEc+#YD z4Odl>f6s~Q8#kHOdl!$NGG4uIT3F~%cLiASD@N1kF`p0Q(fc}h!l zc}9=cs9B-&png=hx|b>ij+HETWTt3*1T3RMa3!XrK4{-1z=G z@Avt8jP$^=t9esIUG=mi2xY`-95&-6LRv z_-K4@z25oWI^yNJ@O@LNkM-?M_j_0deGd!q#4FQ5c`)E=k`#Wz`v){`uoaU9{Pg$%zwYZ=GTM2-}1lO`yS{huPeXzeKV2D&TO*gdOoD3&!;_mS_+C8UTHq9Pro0GU%pP23L9@9AS^mpH%`QCfqH#1;FVm9** zV&=Yk-+lMpci(;Y{rSFu|4J*E#w(Sd7{Kor{Pz_OJ-vx5J_396?r{>}z`u!%}h#ytcec(8n`w4c5{LizjocskP2irQYKE$Hv zFETFpEAF3?&rCpGO5~qtz6s0e7FyuXLxg2}Hb&UcCajOc{JepCE%=AnWLcc-QNSwz zm&a+o@ZgHLn86a{4}g!zHv%enSzNSN!Ig1g@d~brixH^cXk2(S1uu^a{VKRRF8WWw zE8;5{B?Z^u2d{Ma7W{y^4zG-_G~ip~w<;L+CH5=uyb!-{M9*{7vM~Pz3(`+D^ivzJ zRrFjxx5aPE!PNd$@l^)=zWDnLcy)ZW0k4U#G2ps5?MHd-SsPz#!0Y0)pXK3ikKb;< zcf@J`%fqjauQ%XWJZ8Xm#_u%X4RI0gBxHZy6~9a2rG9Fm4^Ka^pBRlerJs`hkUl*9 zz<%m={gmvd7W(k?1N%wn`YG8D=|kwJ%-j!!|2N~aX9XOQ^AqMznx8(%NT2vad@kk> zL{KydZshNK)#i^C&XUX@t*ZY@^9P?lpbyW!G5-6MeoFHP>8BR@@brW6KcVZVH2z5+ zo_=6IX5TWIv=2p&x7hfd4nv>o`xx??HOM=ZN?Y>nHL1cx__8 zhSM`SzhFNrIrA#}8*RVGUf*zfZ9Z}zU*kPLj`@k+-$n61-%pb3soZ)8_jA|Gu31Cz zsyfd!-p_^g*;8|2Z$0#)Yp&10pZ0&&EMg5u&zGaU$!AV85ij%gkHg;w5R>PH9DmNh z8|x#;S0&d|tQG4itgj-o^mq(^9*)s|N$x&BEjqA&K+k`R^&#TnuhD#1S?8^H9>V&d zs?J>RMC$nZG`D_PR%fo4%ImE4R-L)t3iIc6ym-H?PG7I#ytYc{8}dOvvB{Nn`uYdw zx67!0S~Cj#t#!He5P#q1vI?r-!`Ia5>nprJMeNVz%HL9FuD7b|to2%*xn7IbS?j$z zZM{ePk?-gFAtKnv$0y%+E6LC0d1VFQRkx5l97cbAeDbyxRG$aeE~la#{u92x>GJg( z=T|2Bt48?`yT9%$_1ENzchO(yk-+|<`vUK#zZ5^&Ul_k)y;h_6q>wya4;(5pRDR_X*AYgB1>kG5_)X z?H`N%g9ze1v424O&&v>R$^C;3`2A_RpQy!mJl?t>AWNPOxoaQ#AndwsmbUHKEKP~LpRQ6 z5f6M`onO=Mk@EKgAwFxB{%GSk#X~VI9x97X-Wt>T5A%Obj70Iq?=3MazKL1!P0WmM zDq~*!NBhH7F)RLyS@9n!M5`|vqxO0Am&dGlFlNPrF|psDko&h~F+CoPO)ih=@zuvC z*IMyZofS{5vEr%qUOdU;tvkHh-nn@VX!uRXZZn;%6yhv(H2x&A>s>CKnCzR2?2{)*Utz{ZrK53z#r z+eR6mou)ep^gM`!SIc+}^(SXuiTsteALl+E63F2AVBe;GIR7I0X}9SxoC%T;QdHLgc zP(wd1TUz`7p~XMI{+0ds^nV}qZzxRoyz(y==bztx#P}*`KTok;hJFI%&)dIm3i%86 z?;jWEpSORl;R~M;|@CE97SgS9vJ%#mG^{4M)&>Hcy7dlmS+{3qD7AwQijB6;xzH{b&N1^%k} zmudXDe#)SpLh=iIApdceoV&mlEcreCuOR(<^8d)5Kc6N4S495H7JtqDasL_o%j%2B z&+$3=-_YVyl7E7|V#qJAb9?${x&95de}O0dJl@vh-#4A)+z;7n27hgQiFmzy&7#zQ zFn@v1@z>@5I`}8pC4;|TKdWQ;;%n&dGpvl`1NHND=%?P|FW-Mo`oVcQ_3xSsQvR>m z{R=$$m*a^)@X2R%`QHTp6nj0d{G5N;YOH6t{Ri_G_`Li_EdI3ClH=>uagv|Wda1zp z68OCOInD3;<9Q&yzIhnyDL#H_&Hht*zg~zhE&2QP(BG|dUvq1Dd%qJ%@8Wq4o@kSq z(?3^`-p~JY27Pkw7oNRmR+!JPya!>?_|pF+4U{xc(m+WAB@L7`P}0DS(*T88IE~4S z;*cW!{Xe`&Co`<|V&vlY&t)%UdoTX@V#OQ3$X@(O`e#r4-CO%V@Xxwld*Yq9-^ynH z!;iCXXTNzq+eOc{I?SoTnP>;4_5(Zd=dein%$fN!l`PFppFz6kOynLs#JONC*J(O+4p6?_0~Uq`PZC| z(c-uN@?ytdWZzx`;VyKI|9y57yWM@`r}zBy;#b$>w}1ZrKmFi|+bR~z>JZw-*usk& z?s>8DMbs2|`m-;dPNOc!^5U5@r|&uQ*%xs=f9CD%+sD57q22p_`S0G!zBx1Z_B(I> z@)OzrNA)~G|7CfV{2TszEBo)&*Nf_l{LH~`XTPmb7damet$*j8?0@{j9~s<$$dPK_ zRO#$Hxo`Z-pVI&DthI8p)ChKf9spsC*FbhKgoVI`^4q9#V2L_ zlfT?`k&06rcwLa>zCV9fwDRq2mXJROwD;wg@dzHGcKmaoE&5keG~?{cFMs+?95umW zcp;R)zc+qHX>98{Y;QHV=ncmaj|ZaX00VTmi2)?|b!7C=cS?Y-5P^U|TNSRnLPF6m z{E}=r7zRhYs>&;W&hd^DHsBM^y&SFvo>n^{=c6e9XO6S>Bgi?3oF~iDWtps=6T-sL z%y}}L4qG`6b2Hhd>N_g+zXOiD_EY6QXH8UG`bSGQoZiwWl63plX)(TMIDc^Bze7k`^;n?_&9KIRl&-?HX z=kWN#PWbru7I-VAgiL^F=p-t)ftx;ed?k)Gr;l1JaVAAgd+Y3V<~^%uAQyTj#T)HzoW7a{ua+DQ5r z{Y9B4eC7H0@at)wZVRbo=qb_?K|4%*PCh?>K7Mrmv*@tweh4w z=dP9TL}?P4nhei&FYGTtPKuq+O8X-OQVf|IJxJ_x z$(wq3(WimlErIq6Rf9WtzrVRQkd6ZX3@2eRqPwTO+#UQVc`16K4Rgy8V|kH^46{U*L?@ZKFhZ?3hM@7)yvgT7Y6peRN~Tmfc4sxUiqG^+R)+op-&kJGhn4jWO7N zx(t3C^GZ#kCKttDFMqqdULRiGi1DZD6LYamztpf^e>yzBjAg>Iy*H+Qy}nGyT}Jby z(C^aJr}IyQ<|A}{z7YRKL=s+)CtMe^YT**TnN9PWv;&16D$Q(W#Id9i$x@GirRL{F zj!nxM9_hCEQ>1R^{ijs#<29gmin$mp80aD zy30-T0Up)> zZb1zd@(|}(DWXS@+|WHnyT@-f*u6W7(`0OuW4HMH^2~C5>z_6-ueUw=2uPPBTF;ah zv}e0My*$PPBK5)^BXAbg&J6f|J1a}qWN2R`*y|XPL==$+M(x$oV?Q2|$JNujw11zz zdiwcFL=uU@^g%iZ%CmKC!`^BTC-uwu$(s?Kfc~c1El}r7qzhUV(x=aVGV{#Iho=Zk zo?R_4CeNh2T^JUjeK&O`1Og=8h7%RAr?kQg`_u86a@Y^ck57bfZzx@!HqX4g`V%gl zd4=-g(+=#7IVRq|eW10Gc7=5LSOrD0<>`t{MQ;4)__TvaAM)w=Ks{@9P$aAK$dC@x z5q4|h)?B2c<5`4O0}AigVY-6GV1+F2$D<8pIJe7-ho1`5>Os^WP@Z&{c1(N$9iF51 zX}1negwi3#kAHwbB0uY$zcz&vIuba33#Evk*7cJPQ6Gg2`Mj}w9w&nEq}mQh+C`sb zT!WsE5gI2rgKyfw9?cy{I{IE^K|OuCfDvy?-mx7cYJv7mKEpk!+W=qg86&5xCrlr?E3qVP3QUES^asJ z&hoWe8@@W})`n-J{x;K1Cyif)xEr8uW`#m@S6C}wZp&odGZo_s#d0f9ap_D`uBf86GABdUpRMXim&M%lkYabct6x6%NRV)!E({m;Ft z4ref@_|&PDb+5a(VUoajAfvEYDAW zm~MS(^12C^opcR)pRO`q`^nCmz~f#2k#Eu3%H{XzJB&LZqHjonhP+`Mz?=-vd+TA* z9|n!kuaCy3E1}y!IwhmQSLoJHAD|DTT!73z9cwm5eopCE^WnI)mT*+Q@p&~mWzcCF zjFD*jR<)1EM_PXvatqs-U#{7*ftPyL@jhFZIyPvu`luZSPpM;r-qNo@FLkTa7t$>s z>lVLj)}_{0F-@02a#d-%Q##i_jWWV_%=^knc@0{rSA!OGp1nA31{R~Y%_uMRY?tT# zp~Wrd=`xl#6h+*k`wyxAAU>*YIcI+Ys^XmZ8sa9pQv`YKHIM<5&D%6Vr)I%+=|C-F z4&r(-?-b4APLO`*=)HEwh+?Y*#J?%~mF9fA-)L;wdDjY)-Ct`NoBnF#xEdOpuMJ-f zkKlMn*li4*Z}YrfI7lYLcS%2q$IMW>NX6<5;#sWBjm8^ol5^>`wCQ@~yBa$+WL%k8 z$xg1EFC?Qk|D53QiY?>S)Oj^DHeVYK^1n8X4I6F6^#XRXreIs`@-`eaFW5B2;2>QX za#km**~#jHat6*j32~Ew;-RaQ;ilU6TclRK*KfdL-sBS}cAI%GC|K=mD3}(azFrEN zy^8hZuNxmclTho#%KFzW-?i3nj6^vrg|1;%y4L!N(Hru}8Rb1Ak6f`_k32^E+i~P{)@~!wr8yak;lsy=b{{eV&v%NjiYeq* zX}5>=x5L^~#QwcpZ~wX(^yhuOt)2(-oeAWg1(VP7%r&mBv?O7N4>r?9*@3h1L(D9^89ioAh)WQW_PEi9CcH2B2e2E zx5llNFi!4;E@K=o{vN>fe&o08S5ng>5sPWq)!`f~;cz*GEk3{m=?>Lk6(w1Cf` zGWHn?@uuh{e^;$Ll0l{gd@8-$IuFtx-t{k^#KrkRG5zsA z62F$f^{3gg6Y|@jZlmV|Y~33>=dH2RTB z3bJWQ!9Ubf!|Dw-I7Wm;*tO;NN=_(T#2ZkHij9Iocz%+ddsg3Z6S^xw8)!s&@`_V8 z4?e+9;XGKppMam#B|GJtB5;z*z}0?UTN|ll1x}8h9+^(4QS(Vu&qsiV7yRT1`IN#7 zt0yl(o-hBWZOw(`S^)Dx_=q8QW*Ny{tbW2vkI)bH;1KMk3f0T|tcX6Peg;$>h4`86 z5c;LjFFcj?oW0d=KJQ3)9O6-0#9ts@de0bIFZ3m5EA<`E!|$$ypHC1Io(f670E$9* zdK-mjJ(vT*w-7yPBT#DDstde6>+Kg@E27op2#fhF9e=`2=bgWmM(ERv_WW}%d4X%w zS}_}<_fiDTRMgT8RDeV+B`^4Ac=J_fvv?bb!_xFN5Tb#mN%!yM5aRZ-sH9Z8Q?J~qOLZQDqpw%LqAO`ur5mr55Z4=N*+R7eEm(|+YO9m68anxNWpKz z1V<_=9-?T6lK-v?lXjT@HjJFhSosq0i00>^-RudLRE-#tgKmHM+oDTK_rLrUs=Vi_K#&Tb^dBhhzCA?7?zpDUs#(&;#hWKX2y z5&l)s+%L#||JOe)tS#IMRQMGB6AZ#N|5zZ9Lwp~QCFwgQsxLjCj@#8m=287B!qkLh z5%zh?V;eZ1bux28eyaMZW|fHlQrCA$GSA5Nrd4J5yb<-aO6?_{5nuH7LDBa`_>EeU zB794PEtk^JcX=b4=T~X@HR}6HQI^2)Z-v>D^w|?*pM7D&*Z$yz+EMn6d;a#V?$Gal z@ZLM?|H~cUdv-;1cdGI}(7aD@u4W@HyL?LyUQTam(%>k=Gb@)Ha`-BSXG}D>k|hD- zCEJqUGPc8jaR(f*#~&}wbJ%6aRrGiPmI33UJ=5`F92)9yh}8pb%GF21SBG)C(14*1 z0Y4OGbQ6<)kS^4UPu!vpt;J$;Zw(9e;?8$Nl%2)B;SByR<1c)7EH|3SB7#yl9%GkQ z!PC^?em?FbxAvjzk8wL1F6>gH!(t!B&F*#_aqLO6@HQ5?jH8a+)7?&(#V{ra<{KA) z6B6cq3%F0hG(+G==IAy0oIWPu1qsI`oRKgz#OW_dc&mgN?o3mC2?@s}yi39f2}3=+ ze4m87C7hOUpM)1AJR;$YgeN3?Ny4)dW_Uh}>Q75JCgGBjL*uz9iu* z5@z^$HL9PH6QKT=a74lh3Df;p!uLs-Zo~5amoUtW!wV9oxs33O_+4G%Pvt9}YPNvi z79*I-6I_3rfT_F#I4xl+Pw?V;DIdyII%RAkCh0jGX000q9Fh3?yLI~eS-WnDD93nJ{}^hr?_^!ipbzQQ)cndHW=MO4c{= zKLvgPu!HuqpD+gZ!u97kS#}CAwO5n}|NkRkQ9i=z*Et-&KqPp)Du#}3z<1I=l}`Zn z%0s^&(qRnN59{U8UlOMNA$cx0c=QCH+Gf(9l`z>e;TIDEugcdqS>+|H=wr<$J|W>n z)KB_KOBl`I@Tt94{Sqeo=la_x@Hzexz99MaNjNU~UFs6_N?z72U>ZNfuTR2?ei1NS zx*TuN?;!ybeI@jL7BE~ePXxz$1U`lwf=>Y^d_=l2L3M~{-{BG2IZ1O^cTKENzPd6J|QQf zZtbZ@#vaa>kNPOt^l#h~oF84_!QnpBfI%U)Lw{BWM@d5XDoJ zUg{N}D*uv%UwT;NQ@hA+(gI}9ulMqy&Rlxn_Tc>25Wi8f^|M#zt9mKUqZE~^MVtPI zi5+vO#*&F=z;8klCl7mPqv~bl}~m==_SekW0H?reLP_0t5wiz zq8L;3YBjdQ%9kK}+Tx?~$^TJOd<4W8`4LKXzQ|qr|49txe|7E&2^=J&(UKH9mqXna zRQ?NAxm*GHGG({{Iu?huB&TBfefjer&;? zkLPFEhqe5)EP1EQS8z<`??ir3y*06<_FRtqQY$)^3oWy#ejO%|{4SLz{zj7J zFBzFa20e@Bm7w-BS^f<&e3^pUzaE1JQ#jcVH7}PguMzV9>nV{%c1#2Y5#XPpylkm~ z`qJ-;+>AsDE;(8SvJe{oWI+Ah&x(S(B&8ZhWCMDtmE{wbXbZCZkL38F^5pC6^8Xv< z=TX01{^{Qn^ZjStE`Ju~ zpMw0&R{Oi37xnA*AJqQe#r%S;nF-jR(SH-N{DQ1seHcW@{I@9onSlPQc|p|woYbES zS}JSHPvegI_BDyLeP459YG!bAbL!CMgB_1FO(%!C zW~UBx?-}c$ye$XYQ`<*|QrnO2YZ`56OYMGa`#{GG==Mx^P8_7XR^;s*=$LJ9>p8e@ zNB90?J!5Uthq^}&4>s-}8Ax^!&*=l3JKKA@Tc&!twjA#O-S)P zrh3NEhUS*d{asrg9oRes8B!nV>D)Hg-!(KovbshmxH$gN@tf2DeZfS{jC14Ep5Y z=C-+>u9oRkvh$(#qazQsKe|Q#t53BwcC?Q*c08Covh$JV{f`WFw@n@D-hX`W=>CSG zWXH_VuGC|YX$*BIJ7ry|Jr8yCH+7CD8;>OWo2SQ{8;=b1Hyz!#y>r{K!DQ#$q3*V0 zgXoLq#-Y?uG6ne?J0696&0}p_29v3o9{S$dIn&>@ZHDt|p4qoOxu>DOYqlPBQ9E~1 zz0hfUYUf8OOFM z-voFhk387FJvF|)@yJAf)9mml^gw5A z4Tl=LAa~pNuHBFAfc_72S-1bAea#)InV}u~<^=rUz9!f!`jYAy9oZt~Y;GRk+jwNQ zf5-mu{at!~3)O$Hzj@2}?n93q=@~^k8b{`aTDFZIYCkY@Xme+M&%rID1Id=-{m2_? zY=h0z&+gl?9dd2i+0(uMkwdUi)feb@QLoZZ^ZxN&qen&_Y2GGy?AlY`-Pkj-e>>XO z-Im%rx@T&jasOoZ=KYVs?qH{FTcGcQ2ZUXoqeoIlCwdP*Hq{mA{|U&9jQ?jIi;?yaW{Xqi5};nCsA zvEfv%84_HIMxc#oL_N}YLqkJ-Lu13{-eP(6pv*_dh1dV5s`*r$K{x6zh{O`=h1 z-gS<*;r#m!XI*qX|2-EKhfuRO%C^(Tz4*IdgXwL6Jhew}Pkh1sV)&nze<|`WqF>n*nv2Zr@rXzeL4P>&qaT)`d`+3dFAu9Us?46$x8LS zCMfy;uc+_s)vf+wFVs@p^9x0T&mHRfj_R5(xL*kq9!C+ml+%HRS9?%I6jziIPwjYE zE1#?D;p(%lU!bql@>U4le(_zDQ`Os_JHac={w8bZ6KuQ`PZEfBs!sAskx1T7Gh^re zjS%ur5#WiPOF^s03n3m!c|7Rb_j$3fnJJ+VUrn1RL9Z<~G<48Dl%V&mx@5h|m^>ze zUIz1CHhUP~S87`as$E-Uz8W{mE|x)MDo{s`?QJ_(tr#!DLcLG(_Z{I8>Y5dB~0 z$^UP8@^L)F1lt6=qF^p(ld^?c>aRSSK}@u^{7(6_{!;0*{6h8WTXZkUe5zN^k4rlsK1wzst(W;WpRF?g4vP=H zb(RwG!MZP(a5IM|%9mUE9DY1c{;OKP-mbrs`L;Y)WWL?5Y1t8WyB?SMHlL?tzTK|p zH9mT~&T0AjT)CJh|8vqMQN6l%dR_uJ#y0XJa{SnGMrFP&Cmpp>qV^iQ+j2aRU9Ax$ z{QrApKJnM>vtH)g_Sq`)iN9{2X)XU2jsG{a{M^Mv;1}dLBtE)43w#qx$gk(GlRU$q zr><6YlOLv}@)I0lY(##l`~_(rG)^h0eC5v;MRqP9Z}}7n^-)sv%0I=deC3}KR=)C2 zeOA8mPip@{@mKzdiFk6Vci`w)|KuD)oH^9Lw|{1AaO6QAcCK0rs?6? zX%-l9Q>osgV@Idirh%!c0yJYYqoS&#!&A%(+hu5bbnNiBjXE|wIfZasig$Ei60#;L63q)?oj`Z;bTV!hmCyE z;^C3r!;}4^!?rw!!JO6yy<>#jySH=Sj-&l4tPb{#56$4LHouZmwBztUe;&5jE|S z!qKrPO&MxVRCD_s$RtWV1|uFdazaw`66H}+u=;!(g{eI8)>BLpsP&Y>srnUs0ADE2 z*5QS33i)k%8>Sxf16 Q convert + the FA4 forward + a dense f16->f32 output convert. +bool ggml_cuda_flashrt_should_fuse_vit_fa4(const ggml_tensor * fa, ggml_backend_cuda_context & ctx) { +#ifndef GGML_CUDA_FLASHRT_FA4 + GGML_UNUSED(fa); GGML_UNUSED(ctx); + return false; +#else + static const bool disabled = getenv("GGML_FLASHRT_NO_VIT_FA4") != nullptr; + if (disabled) { + return false; + } + const ggml_tensor * q = fa->src[0]; + const ggml_tensor * k = fa->src[1]; + const ggml_tensor * v = fa->src[2]; + if (q == nullptr || k == nullptr || v == nullptr || + fa->src[3] != nullptr || fa->src[4] != nullptr) { // no mask, no sinks + return false; + } + const int64_t hd = q->ne[0]; + const int64_t S = q->ne[1]; + const int64_t H = q->ne[2]; + const int64_t B = q->ne[3]; + if (hd != 80 || S < 32 || H < 1 || B < 1 || q->type != GGML_TYPE_F32) { + return false; + } + // Q: permuted view over the dense (B, S, H, D) f32 buffer + if (q->nb[0] != sizeof(float) || + (int64_t) q->nb[2] != hd * (int64_t) sizeof(float) || + (int64_t) q->nb[1] != hd * H * (int64_t) sizeof(float) || + (int64_t) q->nb[3] != hd * H * S * (int64_t) sizeof(float)) { + return false; + } + for (const ggml_tensor * kv : { k, v }) { + if (kv->type != GGML_TYPE_F16 || kv->ne[0] != hd || kv->ne[1] != S || + kv->ne[2] != H || kv->ne[3] != B || + kv->nb[0] != sizeof(uint16_t) || + (int64_t) kv->nb[2] != hd * (int64_t) sizeof(uint16_t) || + (int64_t) kv->nb[1] != hd * H * (int64_t) sizeof(uint16_t) || + (int64_t) kv->nb[3] != hd * H * S * (int64_t) sizeof(uint16_t)) { + return false; + } + } + // dst: contiguous [hd, H, S, B] f32 — the same linear layout + if (fa->type != GGML_TYPE_F32 || fa->ne[0] != hd || fa->ne[1] != H || + fa->ne[2] != S || fa->ne[3] != B || fa->nb[0] != sizeof(float) || + (int64_t) fa->nb[1] != hd * (int64_t) sizeof(float) || + (int64_t) fa->nb[2] != hd * H * (int64_t) sizeof(float) || + (int64_t) fa->nb[3] != hd * H * S * (int64_t) sizeof(float)) { + return false; + } + float max_bias, softcap; + memcpy(&max_bias, (const float *) fa->op_params + 1, sizeof(float)); + memcpy(&softcap, (const float *) fa->op_params + 2, sizeof(float)); + if (max_bias != 0.0f || softcap != 0.0f) { + return false; + } + // module load must happen outside CUDA graph capture; fall back until then + return ggml_cuda_flashrt::fa4_vit_ensure_loaded(ctx.stream()) == 0; +#endif // GGML_CUDA_FLASHRT_FA4 +} + +void ggml_cuda_flashrt_vit_fa4(ggml_backend_cuda_context & ctx, ggml_tensor * fa) { +#ifndef GGML_CUDA_FLASHRT_FA4 + GGML_UNUSED(ctx); GGML_UNUSED(fa); + GGML_ABORT("flashrt: FA4 vision attention not built"); +#else + const ggml_tensor * q = fa->src[0]; + const int B = (int) q->ne[3]; + const int S = (int) q->ne[1]; + const int H = (int) q->ne[2]; + const int D = (int) q->ne[0]; + float scale; + memcpy(&scale, (const float *) fa->op_params + 0, sizeof(float)); + + const int64_t n = (int64_t) B * S * H * D; + ggml_cuda_pool_alloc q16(ctx.pool(), n * sizeof(uint16_t)); + ggml_cuda_pool_alloc o16(ctx.pool(), n * sizeof(uint16_t)); + + const int rc = ggml_cuda_flashrt::fa4_vit_attention( + (const float *) q->data, fa->src[1]->data, fa->src[2]->data, + (float *) fa->data, q16.get(), o16.get(), + B, S, H, D, scale, ctx.stream()); + if (rc != 0) { + GGML_ABORT("flashrt: FA4 vision attention failed (B=%d S=%d H=%d rc=%d)", B, S, H, rc); + } +#endif // GGML_CUDA_FLASHRT_FA4 +} + // ── Gemma-style norm fold: {RMS_NORM, MUL(w), ADD(mul, norm)} ──────────────── // out = rms_norm(x)*w + rms_norm(x) == rms_norm(x)*(1 + w): the adaLN // modulate kernel with scale = w and shift = 0. ggml's own fused rms_norm @@ -1738,4 +1828,9 @@ bool ggml_cuda_flashrt_rms_gemma(ggml_backend_cuda_context & ctx, const ggml_ten void ggml_cuda_flashrt_begin_eval() { g_eval_id++; +#ifdef GGML_CUDA_FLASHRT_FA4 + // begin_eval runs before any CUDA graph capture starts, so the AOT + // module load never has to race a capturing first evaluation + ggml_cuda_flashrt::fa4_vit_ensure_loaded(nullptr); +#endif // GGML_CUDA_FLASHRT_FA4 } diff --git a/flash_rt/structures/adapters/ggml/fr_fa4_shims.c b/flash_rt/structures/adapters/ggml/fr_fa4_shims.c new file mode 100644 index 00000000..d3186754 --- /dev/null +++ b/flash_rt/structures/adapters/ggml/fr_fa4_shims.c @@ -0,0 +1,42 @@ +// Runtime shims the CuTe-DSL AOT object expects: stable "_cuda*" aliases of +// the CUDA runtime/driver entry points it calls. +#include +#include + +cudaError_t _cudaGetDevice(int * dev) { + return cudaGetDevice(dev); +} + +cudaError_t _cudaDeviceGetAttribute(int * value, enum cudaDeviceAttr attr, int device) { + return cudaDeviceGetAttribute(value, attr, device); +} + +cudaError_t _cudaFuncSetAttribute(const void * func, enum cudaFuncAttribute attr, int value) { + return cudaFuncSetAttribute(func, attr, value); +} + +cudaError_t _cudaKernelSetAttributeForDevice(cudaKernel_t kernel, enum cudaFuncAttribute attr, + int value, int device) { + return cudaKernelSetAttributeForDevice(kernel, attr, value, device); +} + +cudaError_t _cudaLaunchKernelEx(const cudaLaunchConfig_t * config, const void * func, void ** args) { + return cudaLaunchKernelExC(config, func, args); +} + +cudaError_t _cudaLibraryGetKernel(cudaKernel_t * kernel, cudaLibrary_t library, const char * name) { + return cudaLibraryGetKernel(kernel, library, name); +} + +cudaError_t _cudaLibraryLoadData(cudaLibrary_t * library, const void * code, + enum cudaJitOption * jitOptions, void ** jitOptionsValues, + unsigned int numJitOptions, + enum cudaLibraryOption * libraryOptions, + void ** libraryOptionValues, unsigned int numLibraryOptions) { + return cudaLibraryLoadData(library, code, jitOptions, jitOptionsValues, numJitOptions, + libraryOptions, libraryOptionValues, numLibraryOptions); +} + +CUresult _cuKernelGetAttribute(int * pi, CUfunction_attribute attrib, CUkernel kernel, CUdevice dev) { + return cuKernelGetAttribute(pi, attrib, kernel, dev); +} diff --git a/flash_rt/structures/adapters/ggml/fr_fa4_vit.cu b/flash_rt/structures/adapters/ggml/fr_fa4_vit.cu new file mode 100644 index 00000000..2424afcc --- /dev/null +++ b/flash_rt/structures/adapters/ggml/fr_fa4_vit.cu @@ -0,0 +1,110 @@ +// FlashAttention-4 (AOT) for the SigLIP vision attention (Thor SM110). +// +// The vendored fa4_aot/ artifacts are the CuTe-DSL AOT export of the FA4 +// SM100-compatible forward compiled for sm_110a at head_dim 80 (the +// padded-head layout this adapter's vision path already uses). The kernel +// takes (batch, seq, heads, head_dim) f16 tensors with arbitrary leading +// strides; softmax scale is a runtime argument. +// +// The ggml flash-attention node's padded Q/K/V/dst all share one linear +// layout (d + h*hd + s*hd*H + b*hd*H*S), which is exactly FA4's +// (B, S, H, D) with strides (S*H*hd, H*hd, hd) — so the f32 Q input and +// f32 output only need dense elementwise converts, and the f16 K/V pass +// straight through. + +#include "fr_kernels.h" + +#include "fa4_aot/fa4_siglip_fwd.h" + +#include + +namespace ggml_cuda_flashrt { + +namespace { + +__global__ void kernel_f32_to_f16_dense(const float * __restrict__ src, + __half * __restrict__ dst, int64_t n) { + const int64_t i = (int64_t) blockIdx.x * blockDim.x + threadIdx.x; + if (i < n) { + dst[i] = __float2half(src[i]); + } +} + +__global__ void kernel_f16_to_f32_dense(const __half * __restrict__ src, + float * __restrict__ dst, int64_t n) { + const int64_t i = (int64_t) blockIdx.x * blockDim.x + threadIdx.x; + if (i < n) { + dst[i] = __half2float(src[i]); + } +} + +fa4_siglip_fwd_Kernel_Module_t g_fa4_module; +bool g_fa4_loaded = false; + +} // namespace + +// Loads the AOT module once; must not run during CUDA graph capture (the +// caller checks and falls back to the unfused path on the first capture). +int fa4_vit_ensure_loaded(cudaStream_t stream) { + if (g_fa4_loaded) { + return 0; + } + cudaStreamCaptureStatus cap = cudaStreamCaptureStatusNone; + cudaStreamIsCapturing(stream, &cap); + if (cap != cudaStreamCaptureStatusNone) { + return -1; + } + fa4_siglip_fwd_Kernel_Module_Load(&g_fa4_module); + const cudaError_t e = cudaGetLastError(); + if (e != cudaSuccess) { + return -static_cast(e); + } + g_fa4_loaded = true; + return 0; +} + +// q_f32: dense (B,S,H,D); k16/v16: dense f16 same layout; dst_f32 dense. +// q16_ws / o16_ws: workspaces of B*S*H*D halves. +int fa4_vit_attention(const float * q_f32, const void * k16, const void * v16, + float * dst_f32, void * q16_ws, void * o16_ws, + int B, int S, int H, int D, float scale, + cudaStream_t stream) { + if (!g_fa4_loaded) { + return -1; + } + const int64_t n = (int64_t) B * S * H * D; + const int threads = 256; + const int64_t blocks = (n + threads - 1) / threads; + + kernel_f32_to_f16_dense<<<(unsigned) blocks, threads, 0, stream>>>( + q_f32, (__half *) q16_ws, n); + + auto fill = [&](void * data, auto * t) { + t->data = data; + t->dynamic_shapes[0] = B; + t->dynamic_shapes[1] = S; + t->dynamic_shapes[2] = H; + t->dynamic_shapes[3] = D; + t->dynamic_strides[0] = (int64_t) S * H * D; + t->dynamic_strides[1] = (int64_t) H * D; + t->dynamic_strides[2] = D; + }; + fa4_siglip_fwd_Tensor_mQ_t tq; fill(q16_ws, &tq); + fa4_siglip_fwd_Tensor_mK_t tk; fill(const_cast(k16), &tk); + fa4_siglip_fwd_Tensor_mV_t tv; fill(const_cast(v16), &tv); + fa4_siglip_fwd_Tensor_mO_t to; fill(o16_ws, &to); + + const int32_t rc = cute_dsl_fa4_siglip_fwd_wrapper( + &g_fa4_module, &tq, &tk, &tv, &to, scale, stream); + if (rc != 0) { + return -1000 - rc; + } + + kernel_f16_to_f32_dense<<<(unsigned) blocks, threads, 0, stream>>>( + (const __half *) o16_ws, dst_f32, n); + + const cudaError_t e = cudaGetLastError(); + return (e == cudaSuccess) ? 0 : -static_cast(e); +} + +} // namespace ggml_cuda_flashrt diff --git a/flash_rt/structures/adapters/ggml/fr_ggml.cuh b/flash_rt/structures/adapters/ggml/fr_ggml.cuh index c97091d4..ac65866a 100644 --- a/flash_rt/structures/adapters/ggml/fr_ggml.cuh +++ b/flash_rt/structures/adapters/ggml/fr_ggml.cuh @@ -111,6 +111,13 @@ void ggml_cuda_flashrt_gated_residual(ggml_backend_cuda_context & ctx, const ggm bool ggml_cuda_flashrt_should_fuse_dec_attn(const ggml_tensor * fa); void ggml_cuda_flashrt_dec_attn(ggml_backend_cuda_context & ctx, ggml_tensor * fa); +// SigLIP vision attention through the AOT FlashAttention-4 module +// (head_dim 80, no mask). Only available when the adapter was built with +// the fa4_aot artifacts; the first use loads the module (falls back if +// that first use happens during CUDA graph capture). +bool ggml_cuda_flashrt_should_fuse_vit_fa4(const ggml_tensor * fa, ggml_backend_cuda_context & ctx); +void ggml_cuda_flashrt_vit_fa4(ggml_backend_cuda_context & ctx, ggml_tensor * fa); + // Prefill fused QKV window: q mm->reshape->rope->scale, k mm->reshape->rope-> // pad, v mm->reshape->pad, each pad permuted+copied into a padded f16 tensor // of token rows. One fused GEMM + qkv_post + pad-row zeroing. diff --git a/flash_rt/structures/adapters/ggml/fr_kernels.h b/flash_rt/structures/adapters/ggml/fr_kernels.h index 9ce384ae..7951f12b 100644 --- a/flash_rt/structures/adapters/ggml/fr_kernels.h +++ b/flash_rt/structures/adapters/ggml/fr_kernels.h @@ -156,6 +156,16 @@ int decode_attn_decomposed(void * cublas_handle, // out[i] = a[i] + b[i] for n fp32 elements. int vec_add_f32(const float * a, const float * b, float * out, int n, cudaStream_t stream); +// AOT FlashAttention-4 for the SigLIP vision attention (head_dim 80). +// ensure_loaded loads the vendored module once (fails during CUDA graph +// capture: fall back). attention runs f32->f16 Q convert + FA4 + f16->f32 +// output convert; all tensors dense (B,S,H,D) as one linear buffer. +int fa4_vit_ensure_loaded(cudaStream_t stream); +int fa4_vit_attention(const float * q_f32, const void * k16, const void * v16, + float * dst_f32, void * q16_ws, void * o16_ws, + int B, int S, int H, int D, float scale, + cudaStream_t stream); + // Batched f32->f16 row copies: for each pair p, dst[p][r*hd + i] = // (half) src[p][r*hd + i] over n_rows rows of hd elements. One launch // replaces up to FR_CPY_ROWS_MAX individual copy kernels (the persistent diff --git a/flash_rt/structures/adapters/ggml/qualification/goldens/pi05_thor_action.json b/flash_rt/structures/adapters/ggml/qualification/goldens/pi05_thor_action.json index 813182c6..bfb31079 100644 --- a/flash_rt/structures/adapters/ggml/qualification/goldens/pi05_thor_action.json +++ b/flash_rt/structures/adapters/ggml/qualification/goldens/pi05_thor_action.json @@ -1 +1 @@ -{"action_final_raw": [[0.12940391898155212, -0.2528587281703949, 0.10582581162452698, -0.4627326428890228, -0.1517871916294098, 0.05789315328001976, -0.009118168614804745, 0.01561721134930849, 0.019083378836512566, 0.0080350236967206, 0.006184188649058342, 0.00844589713960886, 0.004920070059597492, 0.02140013314783573, 0.008404633030295372, 0.003924117889255285, -0.0030497461557388306, -0.01774791069328785, 0.016703812405467033, 0.009317993186414242, 0.003162568435072899, 0.0025065350346267223, 0.001067449222318828, -0.006151682231575251, 0.00018592315609566867, -0.0020070478785783052, -0.0008290940895676613, -0.012632769532501698, 0.0042471857741475105, 0.0028852762188762426, -0.0028384749311953783, -0.0012326251016929746], [0.15184731781482697, -0.2700265944004059, 0.09386720508337021, -0.4975416958332062, -0.1869109719991684, 0.05920719355344772, -0.009828323498368263, -0.0064353132620453835, 0.00014671108510810882, 0.0005270597757771611, 0.0031039936002343893, -0.002291979966685176, -0.004778217524290085, 0.023754512891173363, -0.002179354429244995, 0.006903328467160463, -0.018551316112279892, 0.006453586742281914, 0.006781680043786764, 0.005654975771903992, 0.006293158978223801, 0.005439461208879948, 0.004844403360038996, -0.009044076316058636, -0.008247747085988522, 0.00579757010564208, 0.0027987658977508545, -0.011956475675106049, 0.0031155601609498262, -0.0010544508695602417, 0.0010488464031368494, -0.007074669003486633], [0.13989382982254028, -0.25814589858055115, 0.09946267306804657, -0.5187717080116272, -0.17015187442302704, 0.05254214629530907, 0.004989681299775839, 0.010219192132353783, 0.01351972110569477, -0.001688823220320046, 0.002372527029365301, 0.00768582196906209, 0.020504146814346313, 0.023498935624957085, 0.0003212030860595405, -0.009923301637172699, -0.007970130071043968, -0.011813816614449024, 0.015340851619839668, 0.007163303904235363, -0.009145798161625862, -0.007841194979846478, 0.008479283191263676, -0.00021931514493189752, -0.012552785687148571, -0.004921156447380781, -0.002375485608354211, -0.0034394313115626574, 0.0035447371192276478, -0.0021622469648718834, 0.005108300130814314, 0.009533166885375977], [0.16775454580783844, -0.2645603120326996, 0.10399901866912842, -0.557577908039093, -0.18896692991256714, 0.07578115910291672, 0.02140253223478794, 0.03042648918926716, 0.0021330579183995724, 0.0029577657114714384, 0.012866067700088024, 0.004185037687420845, 0.0030482769943773746, 0.013907969929277897, -0.004347839392721653, -0.010749413631856441, -0.006680286023765802, -2.668274100869894e-05, 0.004398548509925604, 0.005133638624101877, -0.001514833071269095, -0.005666330456733704, 0.00192269217222929, -0.0006473756511695683, -0.0019386905478313565, 0.006693375762552023, -0.010833384469151497, -0.005241188686341047, -0.00850680097937584, 7.693461520830169e-05, -0.014442027546465397, 0.01909816265106201], [0.1837024986743927, -0.2717337906360626, 0.11260585486888885, -0.5920315980911255, -0.19844241440296173, 0.05850118026137352, 0.0022641720715910196, 0.00940608698874712, 0.003187427995726466, 0.009219805710017681, -0.007168120238929987, -0.0005367595585994422, 0.00547138461843133, 0.03524341061711311, 0.000594788754824549, 0.005399557761847973, -0.023198217153549194, -0.012830831110477448, 0.020483607426285744, 0.013120101764798164, 0.0053434851579368114, 0.005363209638744593, -0.0069306110963225365, 0.001020975410938263, -0.0036603708285838366, 0.002235849853605032, 0.0006648871931247413, -0.00819794274866581, 0.005948537494987249, -0.005503628868609667, -0.004608611110597849, -0.005888366140425205], [0.20225247740745544, -0.2638978958129883, 0.10064870864152908, -0.6103572845458984, -0.1930299997329712, 0.0759415552020073, 0.002225512871518731, 0.015738461166620255, 0.01841391809284687, 0.017062587663531303, -0.006852346938103437, 0.016573432832956314, 0.008369849994778633, 0.03159790113568306, -0.0018215854652225971, -0.0007803556509315968, -0.01254053320735693, -0.009736414067447186, 0.012862961739301682, 0.007550361100584269, -0.006668137852102518, 0.004641189239919186, -0.0041647846810519695, 0.003983424045145512, 0.005212467163801193, -0.0008021246176213026, -0.011270538903772831, -0.00967323500663042, -0.0006975189317017794, -0.0034966417588293552, 0.00386984390206635, 0.0034399370197206736], [0.2047695815563202, -0.2553747296333313, 0.10427432507276535, -0.6205598711967468, -0.2059750258922577, 0.09672776609659195, 0.0007960622315295041, 0.009596768766641617, 0.007772690616548061, 0.0006036338163539767, -0.0026294041890650988, -0.003348570317029953, 0.011335442773997784, 0.022261248901486397, -0.008794681169092655, -5.653189418808324e-06, -0.027853652834892273, -0.005915062502026558, 0.016558898612856865, 0.009591544046998024, 0.005161453504115343, 0.008785799145698547, -0.007114224601536989, 0.007564001716673374, 0.005259434226900339, 0.013112467713654041, -0.008463529869914055, -0.009648582898080349, -0.009051376022398472, 0.005528939887881279, -0.002176202367991209, -0.006872260011732578], [0.2194412648677826, -0.2765859365463257, 0.10265745222568512, -0.6340638995170593, -0.20541149377822876, 0.08428670465946198, 0.0097963847219944, 0.014086037874221802, 0.00032647105399519205, -0.0003711430181283504, -0.003853543195873499, 0.0008552426006644964, 0.015407024882733822, 0.02043493464589119, 0.004803684540092945, -0.0014393915189430118, -0.01593821682035923, -0.002983001060783863, 0.018109437078237534, 0.0057259202003479, -0.0007894366281107068, 0.0020676464773714542, -0.0024445049930363894, -0.010146920569241047, -0.005904162302613258, 0.008207222446799278, -0.0024562610778957605, -0.000604762404691428, -0.0026838593184947968, 0.004686345346271992, 0.003601022297516465, 0.00964265689253807], [0.22591130435466766, -0.2531126141548157, 0.10270681232213974, -0.6457228660583496, -0.21248739957809448, 0.08996982872486115, -0.00910878460854292, 0.013310388661921024, -0.005985552910715342, 0.0021230243146419525, 0.0012873277300968766, 0.0060186865739524364, 0.018974991515278816, 0.018281321972608566, 0.0002817067434079945, 0.012477329932153225, -0.022212624549865723, -0.0030888498295098543, 0.02746962569653988, 0.004553196020424366, -0.0009546522051095963, -0.012151896953582764, -0.008990723639726639, -0.001741186250001192, -0.002104580169543624, 0.011146661825478077, -0.0034014740958809853, 0.0027611013501882553, 0.0014544056029990315, 0.005902513395994902, 0.0029447246342897415, -0.0002835096383932978], [0.2338383048772812, -0.25216224789619446, 0.12483970820903778, -0.6578122973442078, -0.20657934248447418, 0.0806683748960495, -0.004676615819334984, -0.0006798385875299573, 0.015721520408988, 0.01621541567146778, -0.009323552250862122, -0.00032941167592070997, -0.0019576391205191612, 0.025488749146461487, 0.005375501234084368, 0.001192403258755803, 0.00319452746771276, -0.009350409731268883, 0.0034876212012022734, 0.0032979880925267935, 0.00373227265663445, 0.001819843309931457, -0.007980229333043098, 0.011167601682245731, -0.0028672132175415754, -0.007579560857266188, -0.015309120528399944, -0.027653951197862625, -0.012417344376444817, 0.006930120754987001, -0.004635754972696304, -0.006454023998230696]], "action_steps": 10, "action_dim": 32} \ No newline at end of file +{"action_final_raw": [[0.07204130291938782, -0.11350921541452408, 0.08429346978664398, -0.3656696081161499, -0.07903286069631577, 0.05143028125166893, -0.027163250371813774, 0.010509899817407131, 0.016304291784763336, 0.007965324446558952, -0.002198867965489626, 0.009940089657902718, 0.004432466812431812, 0.014802207238972187, 0.005763123743236065, 0.008698852732777596, -0.004932646173983812, -0.00758181419223547, 0.007238580379635096, 0.012351863086223602, 0.0005422856193035841, -0.004117514938116074, 0.004125963430851698, -0.009748348034918308, -0.004717582371085882, -0.004324060864746571, -0.003718649037182331, -0.008062406443059444, 0.0022765814792364836, 0.003958307206630707, -0.005197297316044569, -0.0011443446855992079], [0.10125378519296646, -0.12873898446559906, 0.0758739486336708, -0.42113062739372253, -0.10985685139894485, 0.05189714580774307, -0.026942923665046692, -0.004111709538847208, 0.006781688425689936, 0.005005198530852795, -0.00799341220408678, 0.0010191804030910134, -0.011189727112650871, 0.0263466015458107, 0.00395270437002182, -9.96968665276654e-05, -0.020618770271539688, 0.008116556331515312, 0.016291717067360878, 0.010052606463432312, 0.003730165073648095, 0.007512654177844524, 0.006219280883669853, -0.009574771858751774, -0.007001390680670738, 0.005158360581845045, -0.0023672597017139196, -0.007642071228474379, 0.006794589105993509, -0.005577349103987217, 0.000534163205884397, 0.003758528968319297], [0.09115591645240784, -0.11993689090013504, 0.08690856397151947, -0.46369683742523193, -0.11741956323385239, 0.05149669945240021, -0.017574412748217583, 0.0056210667826235294, 0.006649073213338852, 0.0009659163770265877, 0.0002748323604464531, 0.013001701794564724, 0.020357340574264526, 0.02157861739397049, 0.0017587818438187242, -0.009329949505627155, -0.01415527518838644, -0.004996980540454388, 0.014225887134671211, 0.00044403059291653335, -0.008308613672852516, -0.00733127212151885, 0.0013179633533582091, -0.002029938856139779, -0.004320304840803146, -0.004838001914322376, -0.003330306615680456, 0.00011348118277965114, 0.00475540105253458, 0.011434276588261127, 0.0026987316086888313, 0.006854772102087736], [0.11330023407936096, -0.11783881485462189, 0.0926673635840416, -0.5157063603401184, -0.127365842461586, 0.056598976254463196, -0.01100153662264347, 0.03379752114415169, 0.001299548428505659, 0.014514102600514889, 0.005873256362974644, 0.0022408769000321627, 0.011131459847092628, 0.018787482753396034, 0.0014650344382971525, -0.006809557788074017, -0.007111363112926483, -0.011379110626876354, 0.006421738304197788, 0.010308551602065563, -0.005321051925420761, -0.004222462885081768, 0.004251003731042147, -0.007255891337990761, 0.0031188130378723145, 0.010692982003092766, 0.004688961431384087, 0.0009694810723885894, 0.0032925044652074575, 0.008236529305577278, -0.0013498385669663548, 0.012010093778371811], [0.13430054485797882, -0.1269943118095398, 0.10221230983734131, -0.5629380941390991, -0.13885940611362457, 0.05582033470273018, -0.017639677971601486, 0.012164678424596786, 0.002125837840139866, 0.01052339281886816, -0.0034445286728441715, -0.002667462918907404, 0.012460893951356411, 0.02905707061290741, 0.0017453432083129883, 0.002192025538533926, -0.019024087116122246, -0.012002582661807537, 0.015994884073734283, 0.014112966135144234, 0.005932504311203957, 0.0005032385233789682, -0.0015208425465971231, -0.0006949722883291543, -0.00525035010650754, -0.004735289141535759, -0.0011483444832265377, -0.009948208928108215, 0.008242184296250343, -0.0022513302974402905, -0.0005639559822157025, -0.00038196842069737613], [0.1572314202785492, -0.11822830140590668, 0.09592665731906891, -0.5975432991981506, -0.1441642940044403, 0.06988880783319473, -0.01921856217086315, 0.015498095192015171, 0.01608978398144245, 0.016082610934972763, -0.006816709414124489, 0.014671353623270988, 0.009462973102927208, 0.020529944449663162, 0.003971511032432318, -0.008311562240123749, -0.017955025658011436, -0.0019077907782047987, 0.014017481356859207, 0.018251800909638405, -0.008570718578994274, 0.004647897556424141, -0.001230914844200015, 0.005506758578121662, -0.0036311494186520576, 0.0026508404407650232, -0.003659192007035017, -0.004755430854856968, -0.006892980542033911, -0.008419613353908062, -0.002728990977630019, 0.003199429716914892], [0.1634657084941864, -0.10547913610935211, 0.09950520843267441, -0.6124517321586609, -0.15696431696414948, 0.08715098351240158, -0.03141110762953758, 0.013191832229495049, 0.004589242395013571, 0.000438198127085343, 0.0008151709917001426, 0.008371998555958271, 0.011076774448156357, 0.021108262240886688, -0.005683267489075661, 1.3056736634098343e-06, -0.020556550472974777, 0.002646013628691435, 0.013472237624228, 0.012461056001484394, 0.006658147554844618, 0.008077976293861866, -0.0040149628184735775, 0.00951587688177824, 0.0076101962476968765, 0.004690102767199278, -0.008010146208107471, -0.01499431487172842, -0.014799260534346104, 0.010328761301934719, -0.008833601139485836, -0.013493570499122143], [0.17962393164634705, -0.12388500571250916, 0.09577580541372299, -0.6380575299263, -0.1597168743610382, 0.07016991823911667, -0.015582311898469925, 0.006572971120476723, 0.008294407278299332, -0.0011156095424667, 0.002037588506937027, 0.015060609206557274, 0.009366026148200035, 0.012784049846231937, 0.006767261307686567, -0.007399061694741249, -0.009957707487046719, -0.008602148853242397, 0.015372874215245247, 0.0023656415287405252, -0.0018184289801865816, 0.0012300567468628287, 0.007301063276827335, -0.005210488103330135, -0.005356528330594301, 0.01048555038869381, -0.009502486325800419, -0.01623319461941719, -0.0026356331072747707, -0.007185563910752535, -0.0012311009922996163, 0.006323504261672497], [0.1977672427892685, -0.11398372054100037, 0.09678585082292557, -0.668890655040741, -0.1637994796037674, 0.08524077385663986, -0.015174043364822865, 0.006234307307749987, 0.004931071773171425, 0.010312230326235294, 0.0005076512461528182, 0.00603244174271822, 0.02166261337697506, 0.023806912824511528, -0.005028861574828625, 0.006848940160125494, -0.0202220156788826, -6.371434574248269e-05, 0.006742647383362055, 0.011141316033899784, -0.003976040054112673, -0.0022567883133888245, -0.013189264573156834, -0.008469528518617153, -0.013478616252541542, 0.01167785469442606, -0.00862216018140316, 0.010740133933722973, 0.0010758911957964301, 0.002641480416059494, 0.008903530426323414, -0.002815911313518882], [0.19058360159397125, -0.11881575733423233, 0.1211477667093277, -0.6842043399810791, -0.1674933135509491, 0.0675734281539917, -0.02280472218990326, 0.011704090982675552, 0.014619171619415283, 0.009397161193192005, -0.002172892913222313, 0.011309277266263962, -0.00787555705755949, 0.023229902610182762, 0.0019613588228821754, 0.0010314203100278974, 0.007905149832367897, -0.004807105287909508, 0.01077890582382679, 0.016402581706643105, 0.0011862809769809246, 0.0012321020476520061, -0.009161261841654778, 0.0018737574573606253, -0.002514938125386834, -0.004140396602451801, -0.011619646102190018, -0.018781667575240135, -0.009584026411175728, 0.004854136612266302, -0.0069736698642373085, -0.005881227552890778]], "action_steps": 10, "action_dim": 32} \ No newline at end of file From f81a6c45d69896442dd0595f79ce798c3e10041e Mon Sep 17 00:00:00 2001 From: LiangSu8899 Date: Tue, 25 Aug 2026 04:04:03 -0400 Subject: [PATCH 17/32] perf(ggml-adapter): FA4 vision output converts straight past the head padding The vision FA node is followed by a head-de-pad view and a CONT whose strided copy costs more than the attention itself. Absorb the pair: the FA4 f16 output converts directly into the CONT's packed f32 destination, skipping the padded f32 store and the strided copy. The same elements go through the same f16->f32 conversion, so results are bit-identical. --- .../structures/adapters/ggml/fr_dispatch.cu | 63 ++++++++++++++++++- .../structures/adapters/ggml/fr_fa4_vit.cu | 30 +++++++-- flash_rt/structures/adapters/ggml/fr_ggml.cuh | 9 +++ .../structures/adapters/ggml/fr_kernels.h | 4 +- 4 files changed, 100 insertions(+), 6 deletions(-) diff --git a/flash_rt/structures/adapters/ggml/fr_dispatch.cu b/flash_rt/structures/adapters/ggml/fr_dispatch.cu index a00b0112..07b04eb0 100644 --- a/flash_rt/structures/adapters/ggml/fr_dispatch.cu +++ b/flash_rt/structures/adapters/ggml/fr_dispatch.cu @@ -1761,7 +1761,7 @@ void ggml_cuda_flashrt_vit_fa4(ggml_backend_cuda_context & ctx, ggml_tensor * fa const int rc = ggml_cuda_flashrt::fa4_vit_attention( (const float *) q->data, fa->src[1]->data, fa->src[2]->data, - (float *) fa->data, q16.get(), o16.get(), + (float *) fa->data, D, q16.get(), o16.get(), B, S, H, D, scale, ctx.stream()); if (rc != 0) { GGML_ABORT("flashrt: FA4 vision attention failed (B=%d S=%d H=%d rc=%d)", B, S, H, rc); @@ -1769,6 +1769,67 @@ void ggml_cuda_flashrt_vit_fa4(ggml_backend_cuda_context & ctx, ggml_tensor * fa #endif // GGML_CUDA_FLASHRT_FA4 } +// Variant absorbing the {VIEW (head de-pad), CONT} pair after the FA node: +// the FA4 output converts directly into the CONT's packed destination. +bool ggml_cuda_flashrt_should_fuse_vit_fa4_depad(const ggml_tensor * fa, const ggml_tensor * view, + const ggml_tensor * cont, ggml_backend_cuda_context & ctx) { +#ifndef GGML_CUDA_FLASHRT_FA4 + GGML_UNUSED(fa); GGML_UNUSED(view); GGML_UNUSED(cont); GGML_UNUSED(ctx); + return false; +#else + if (!ggml_cuda_flashrt_should_fuse_vit_fa4(fa, ctx)) { + return false; + } + const int64_t D = fa->ne[0]; + const int64_t H = fa->ne[1]; + const int64_t S = fa->ne[2]; + const int64_t B = fa->ne[3]; + // view: leading d2 <= D slice of the FA output, no offset + if (view->src[0] != fa || view->type != GGML_TYPE_F32 || + view->data != fa->data || + view->ne[0] > D || view->ne[1] != H || view->ne[2] != S || view->ne[3] != B) { + return false; + } + const int64_t D2 = view->ne[0]; + // cont: packed [(H*D2), S, B] contiguous f32 + if (cont->src[0] != view || cont->type != GGML_TYPE_F32 || + cont->ne[0] != H * D2 || cont->ne[1] != S || cont->ne[2] != B || cont->ne[3] != 1 || + !ggml_is_contiguous(cont)) { + return false; + } + return true; +#endif // GGML_CUDA_FLASHRT_FA4 +} + +void ggml_cuda_flashrt_vit_fa4_depad(ggml_backend_cuda_context & ctx, ggml_tensor * fa, + const ggml_tensor * view, ggml_tensor * cont) { +#ifndef GGML_CUDA_FLASHRT_FA4 + GGML_UNUSED(ctx); GGML_UNUSED(fa); GGML_UNUSED(view); GGML_UNUSED(cont); + GGML_ABORT("flashrt: FA4 vision attention not built"); +#else + const ggml_tensor * q = fa->src[0]; + const int B = (int) q->ne[3]; + const int S = (int) q->ne[1]; + const int H = (int) q->ne[2]; + const int D = (int) q->ne[0]; + const int D2 = (int) view->ne[0]; + float scale; + memcpy(&scale, (const float *) fa->op_params + 0, sizeof(float)); + + const int64_t n = (int64_t) B * S * H * D; + ggml_cuda_pool_alloc q16(ctx.pool(), n * sizeof(uint16_t)); + ggml_cuda_pool_alloc o16(ctx.pool(), n * sizeof(uint16_t)); + + const int rc = ggml_cuda_flashrt::fa4_vit_attention( + (const float *) q->data, fa->src[1]->data, fa->src[2]->data, + (float *) cont->data, D2, q16.get(), o16.get(), + B, S, H, D, scale, ctx.stream()); + if (rc != 0) { + GGML_ABORT("flashrt: FA4 vision attention (depad) failed (B=%d S=%d H=%d rc=%d)", B, S, H, rc); + } +#endif // GGML_CUDA_FLASHRT_FA4 +} + // ── Gemma-style norm fold: {RMS_NORM, MUL(w), ADD(mul, norm)} ──────────────── // out = rms_norm(x)*w + rms_norm(x) == rms_norm(x)*(1 + w): the adaLN // modulate kernel with scale = w and shift = 0. ggml's own fused rms_norm diff --git a/flash_rt/structures/adapters/ggml/fr_fa4_vit.cu b/flash_rt/structures/adapters/ggml/fr_fa4_vit.cu index 2424afcc..6e4bcdf4 100644 --- a/flash_rt/structures/adapters/ggml/fr_fa4_vit.cu +++ b/flash_rt/structures/adapters/ggml/fr_fa4_vit.cu @@ -38,6 +38,21 @@ __global__ void kernel_f16_to_f32_dense(const __half * __restrict__ src, } } +// convert dropping the head padding: one block per (b, s) token; src rows +// are H heads of D elements, dst rows are H packed slices of D2 (< D) +__global__ void kernel_f16_to_f32_depad(const __half * __restrict__ src, + float * __restrict__ dst, + int H, int D, int D2) { + const int64_t row = blockIdx.x; + const __half * s = src + row * (int64_t) H * D; + float * d = dst + row * (int64_t) H * D2; + for (int t = threadIdx.x; t < H * D2; t += blockDim.x) { + const int h = t / D2; + const int e = t - h * D2; + d[t] = __half2float(s[(int64_t) h * D + e]); + } +} + fa4_siglip_fwd_Kernel_Module_t g_fa4_module; bool g_fa4_loaded = false; @@ -63,10 +78,12 @@ int fa4_vit_ensure_loaded(cudaStream_t stream) { return 0; } -// q_f32: dense (B,S,H,D); k16/v16: dense f16 same layout; dst_f32 dense. +// q_f32: dense (B,S,H,D); k16/v16: dense f16 same layout. When d_out == D +// dst_f32 is the dense padded layout; when d_out < D the head padding is +// dropped and dst_f32 is the packed [(H*d_out), S, B] contiguous tensor. // q16_ws / o16_ws: workspaces of B*S*H*D halves. int fa4_vit_attention(const float * q_f32, const void * k16, const void * v16, - float * dst_f32, void * q16_ws, void * o16_ws, + float * dst_f32, int d_out, void * q16_ws, void * o16_ws, int B, int S, int H, int D, float scale, cudaStream_t stream) { if (!g_fa4_loaded) { @@ -100,8 +117,13 @@ int fa4_vit_attention(const float * q_f32, const void * k16, const void * v16, return -1000 - rc; } - kernel_f16_to_f32_dense<<<(unsigned) blocks, threads, 0, stream>>>( - (const __half *) o16_ws, dst_f32, n); + if (d_out == D) { + kernel_f16_to_f32_dense<<<(unsigned) blocks, threads, 0, stream>>>( + (const __half *) o16_ws, dst_f32, n); + } else { + kernel_f16_to_f32_depad<<<(unsigned) ((int64_t) B * S), threads, 0, stream>>>( + (const __half *) o16_ws, dst_f32, H, D, d_out); + } const cudaError_t e = cudaGetLastError(); return (e == cudaSuccess) ? 0 : -static_cast(e); diff --git a/flash_rt/structures/adapters/ggml/fr_ggml.cuh b/flash_rt/structures/adapters/ggml/fr_ggml.cuh index ac65866a..9b015bba 100644 --- a/flash_rt/structures/adapters/ggml/fr_ggml.cuh +++ b/flash_rt/structures/adapters/ggml/fr_ggml.cuh @@ -118,6 +118,15 @@ void ggml_cuda_flashrt_dec_attn(ggml_backend_cuda_context & ctx, ggml_tensor * f bool ggml_cuda_flashrt_should_fuse_vit_fa4(const ggml_tensor * fa, ggml_backend_cuda_context & ctx); void ggml_cuda_flashrt_vit_fa4(ggml_backend_cuda_context & ctx, ggml_tensor * fa); +// Variant that also absorbs the {VIEW (head de-pad), CONT} pair that +// follows the vision FA node: the FA4 output converts straight into the +// CONT's packed [(H*d), S, B] f32 destination, skipping the padded f32 +// store and the strided copy. +bool ggml_cuda_flashrt_should_fuse_vit_fa4_depad(const ggml_tensor * fa, const ggml_tensor * view, + const ggml_tensor * cont, ggml_backend_cuda_context & ctx); +void ggml_cuda_flashrt_vit_fa4_depad(ggml_backend_cuda_context & ctx, ggml_tensor * fa, + const ggml_tensor * view, ggml_tensor * cont); + // Prefill fused QKV window: q mm->reshape->rope->scale, k mm->reshape->rope-> // pad, v mm->reshape->pad, each pad permuted+copied into a padded f16 tensor // of token rows. One fused GEMM + qkv_post + pad-row zeroing. diff --git a/flash_rt/structures/adapters/ggml/fr_kernels.h b/flash_rt/structures/adapters/ggml/fr_kernels.h index 7951f12b..2fbb1460 100644 --- a/flash_rt/structures/adapters/ggml/fr_kernels.h +++ b/flash_rt/structures/adapters/ggml/fr_kernels.h @@ -161,8 +161,10 @@ int vec_add_f32(const float * a, const float * b, float * out, int n, cudaStream // capture: fall back). attention runs f32->f16 Q convert + FA4 + f16->f32 // output convert; all tensors dense (B,S,H,D) as one linear buffer. int fa4_vit_ensure_loaded(cudaStream_t stream); +// d_out == D: dst is the dense padded layout; d_out < D: the head padding +// is dropped and dst is the packed [(H*d_out), S, B] contiguous tensor. int fa4_vit_attention(const float * q_f32, const void * k16, const void * v16, - float * dst_f32, void * q16_ws, void * o16_ws, + float * dst_f32, int d_out, void * q16_ws, void * o16_ws, int B, int S, int H, int D, float scale, cudaStream_t stream); From 132754c649435f104254acc91a514b65748a7f48 Mon Sep 17 00:00:00 2001 From: LiangSu8899 Date: Tue, 25 Aug 2026 04:11:55 -0400 Subject: [PATCH 18/32] feat(ggml-adapter): AOT FA4 for the pi0.5 prefill self-attention Second AOT export of the vendored FA4 forward at the prefill shape (head_dim 256, GQA with one KV head, full attention). The prefill is a prefix-LM: its mask is row-uniform pad-only and the real KV length equals the query count, so passing that length as the KV dynamic shape reproduces the mask exactly and the padded tail is never read. The f32 graph boundaries again reduce to dense elementwise converts. Refreshes the e2e action golden. On the real-frame judge the overall distance to the f16 reference moves -0.0085 (within the swing this metric shows for any attention change; the gripper dimension improves +0.024); GGML_FLASHRT_NO_PREFILL_FA4=1 falls back per site. --- .../adapters/ggml/fa4_aot/README.md | 9 +- .../ggml/fa4_aot/export_fa4_siglip.py | 18 +++- .../adapters/ggml/fa4_aot/fa4_prefill_fwd.h | 101 ++++++++++++++++++ .../adapters/ggml/fa4_aot/fa4_prefill_fwd.o | Bin 0 -> 170032 bytes .../structures/adapters/ggml/fr_dispatch.cu | 96 +++++++++++++++++ .../structures/adapters/ggml/fr_fa4_vit.cu | 59 +++++++++- flash_rt/structures/adapters/ggml/fr_ggml.cuh | 7 ++ .../structures/adapters/ggml/fr_kernels.h | 9 ++ .../goldens/pi05_thor_action.json | 2 +- 9 files changed, 295 insertions(+), 6 deletions(-) create mode 100644 flash_rt/structures/adapters/ggml/fa4_aot/fa4_prefill_fwd.h create mode 100644 flash_rt/structures/adapters/ggml/fa4_aot/fa4_prefill_fwd.o diff --git a/flash_rt/structures/adapters/ggml/fa4_aot/README.md b/flash_rt/structures/adapters/ggml/fa4_aot/README.md index 0ba5f2fa..13d5b6ab 100644 --- a/flash_rt/structures/adapters/ggml/fa4_aot/README.md +++ b/flash_rt/structures/adapters/ggml/fa4_aot/README.md @@ -4,7 +4,10 @@ export of the vendored FA4 SM100-compatible forward (`csrc/attention/flash_attn_4_src/flashrt_fa4`) compiled for `sm_110a` at head_dim 80 — the padded-head layout the ggml adapter's vision path -uses. Sequence length, head count and batch stay dynamic; the softmax +uses. `fa4_prefill_fwd.h` / `fa4_prefill_fwd.o` are the same export at +the pi0.5 prefill shape (head_dim 256, GQA with one KV head, full +attention): the prefill's row-uniform pad mask is reproduced by passing +the real sequence length as the KV dynamic shape. Sequence length, head count and batch stay dynamic; the softmax scale is a runtime argument. The `.o` contains the embedded cubin plus the host launch entry; `fr_fa4_shims.c` provides the small `_cuda*` runtime aliases the object expects, so no CuTe-DSL runtime library is @@ -13,8 +16,8 @@ needed at build or run time. The ggml adapter build enables the FA4 vision-attention window automatically when these files are present (see the host build's `GGML_CUDA_FLASHRT` integration); delete them or set -`GGML_FLASHRT_NO_VIT_FA4=1` to fall back to the host's own flash -attention. +`GGML_FLASHRT_NO_VIT_FA4=1` / `GGML_FLASHRT_NO_PREFILL_FA4=1` to fall +back to the host's own flash attention per site. ## Regeneration diff --git a/flash_rt/structures/adapters/ggml/fa4_aot/export_fa4_siglip.py b/flash_rt/structures/adapters/ggml/fa4_aot/export_fa4_siglip.py index 22659a47..0bb1e3e0 100644 --- a/flash_rt/structures/adapters/ggml/fa4_aot/export_fa4_siglip.py +++ b/flash_rt/structures/adapters/ggml/fa4_aot/export_fa4_siglip.py @@ -51,6 +51,7 @@ def _compile_no_ffi(*args, **kwargs): ifw.cute.compile = _compile_no_ffi ifw._flash_attn_fwd.compile_cache.clear() +# vision attention: padded head_dim 80, MHA NV, SQ, NH, HD = 2, 256, 16, 80 q = torch.zeros(NV, SQ, NH, HD, dtype=torch.float16, device="cuda") k = torch.zeros_like(q) @@ -61,4 +62,19 @@ def _compile_no_ffi(*args, **kwargs): assert _holder, "FA4 compile did not run" _holder[0].export_to_c(str(_HERE), "fa4_siglip_fwd") -print("exported:", sorted(p.name for p in _HERE.glob("fa4_siglip_fwd.*"))) + +# prefill self-attention: head_dim 256, GQA with one KV head +_holder.clear() +ifw._flash_attn_fwd.compile_cache.clear() +B, SQ2, HQ, HK, HD2 = 1, 559, 8, 1, 256 +q2 = torch.zeros(B, SQ2, HQ, HD2, dtype=torch.float16, device="cuda") +k2 = torch.zeros(B, SQ2, HK, HD2, dtype=torch.float16, device="cuda") +v2 = torch.zeros_like(k2) +out2 = torch.empty_like(q2) +fwd(q2, k2, v2, softmax_scale=HD2 ** -0.5, causal=False, + num_splits=1, pack_gqa=True, out=out2) +torch.cuda.synchronize() + +assert _holder, "FA4 prefill compile did not run" +_holder[0].export_to_c(str(_HERE), "fa4_prefill_fwd") +print("exported:", sorted(p.name for p in _HERE.glob("fa4_*_fwd.*"))) diff --git a/flash_rt/structures/adapters/ggml/fa4_aot/fa4_prefill_fwd.h b/flash_rt/structures/adapters/ggml/fa4_aot/fa4_prefill_fwd.h new file mode 100644 index 00000000..6dd1bc77 --- /dev/null +++ b/flash_rt/structures/adapters/ggml/fa4_aot/fa4_prefill_fwd.h @@ -0,0 +1,101 @@ + +#pragma once + +#include +#include +#include +#include + + +// Macro to check for cuda errors. +#ifndef CUTE_DSL_CUDA_ERROR_CHECK +#define CUTE_DSL_CUDA_ERROR_CHECK(err) { \ + if ((err) != cudaSuccess) { \ + printf("Got Cuda Error %s: %s\n", cudaGetErrorName(err), cudaGetErrorString(err)); \ + } \ +} + +#endif + +typedef struct { + cudaLibrary_t module; +} fa4_prefill_fwd_Kernel_Module_t; + +#ifdef __cplusplus +extern "C" { +#endif +void _mlir_fa4_prefill_fwd_cuda_init(void **); +void _mlir_fa4_prefill_fwd_cuda_load_to_device(void **); +static inline void fa4_prefill_fwd_Kernel_Module_Load(fa4_prefill_fwd_Kernel_Module_t *module) { + cudaLibrary_t *libraryPtr = &(module->module); + cudaError_t ret; + struct { + cudaLibrary_t **libraryPtr; + cudaError_t *ret; + } initArgs = {&libraryPtr, &ret}; + _mlir_fa4_prefill_fwd_cuda_init((void **)(&initArgs)); + CUTE_DSL_CUDA_ERROR_CHECK(ret); + int32_t device_id = 0; + struct { + cudaLibrary_t **library; + int32_t *device_id; + cudaError_t *ret; + } loadArgs = {&libraryPtr, &device_id, &ret}; + int32_t device_count; + CUTE_DSL_CUDA_ERROR_CHECK(cudaGetDeviceCount(&device_count)); + for (int32_t i = 0; i < device_count; i++) { + device_id = i; + _mlir_fa4_prefill_fwd_cuda_load_to_device((void **)(&loadArgs)); + CUTE_DSL_CUDA_ERROR_CHECK(ret); + } +} + +static inline void fa4_prefill_fwd_Kernel_Module_Unload(fa4_prefill_fwd_Kernel_Module_t *module) { + CUTE_DSL_CUDA_ERROR_CHECK(cudaLibraryUnload(module->module)); +} + +#ifdef __cplusplus +} +#endif + +typedef struct { + void *data; + int32_t dynamic_shapes[4]; + int64_t dynamic_strides[3]; +} fa4_prefill_fwd_Tensor_mQ_t; + + +typedef struct { + void *data; + int32_t dynamic_shapes[4]; + int64_t dynamic_strides[3]; +} fa4_prefill_fwd_Tensor_mK_t; + + +typedef struct { + void *data; + int32_t dynamic_shapes[4]; + int64_t dynamic_strides[3]; +} fa4_prefill_fwd_Tensor_mV_t; + + +typedef struct { + void *data; + int32_t dynamic_shapes[4]; + int64_t dynamic_strides[3]; +} fa4_prefill_fwd_Tensor_mO_t; + +#ifdef __cplusplus +extern "C" +#endif +void _mlir_fa4_prefill_fwd__mlir_ciface_cutlass___call___flashrt_fa4cutesm100_hd256_2cta_fmha_forwardBlackwellFusedMultiHeadAttentionForward_object_at__Tensorgmemoi64i64i641_Tensorgmemoi64i64i641_Tensorgmemoi64i64i641_Te(void **args, int32_t num_args); + +static inline int32_t cute_dsl_fa4_prefill_fwd_wrapper(fa4_prefill_fwd_Kernel_Module_t *module, fa4_prefill_fwd_Tensor_mQ_t *mQ, fa4_prefill_fwd_Tensor_mK_t *mK, fa4_prefill_fwd_Tensor_mV_t *mV, fa4_prefill_fwd_Tensor_mO_t *mO, float softmax_scale, cudaStream_t stream) { + int32_t ret; + void *args[7] = { + mQ, mK, mV, mO, &softmax_scale, &stream, + &ret + }; + _mlir_fa4_prefill_fwd__mlir_ciface_cutlass___call___flashrt_fa4cutesm100_hd256_2cta_fmha_forwardBlackwellFusedMultiHeadAttentionForward_object_at__Tensorgmemoi64i64i641_Tensorgmemoi64i64i641_Tensorgmemoi64i64i641_Te(args, 7); + return ret; +} diff --git a/flash_rt/structures/adapters/ggml/fa4_aot/fa4_prefill_fwd.o b/flash_rt/structures/adapters/ggml/fa4_aot/fa4_prefill_fwd.o new file mode 100644 index 0000000000000000000000000000000000000000..fc9ee97d1a4fd00980fe8a9555ca215ad5993675 GIT binary patch literal 170032 zcmeFa3w%^XmOoxey3^fB$ctCPE8R(>0TBqIM#a#87*HUHXncq85|l^0-QbM`XoCtU zBLQ_3@e$E+2!exyvtyiHm)&t)hjIKN&d$0!``;hK_?s19!)BZr*&Qdp?>V>X-l`iP zamV%V&)LhTNq_IDI#qS*)TvXa>fT;Ib;gC685u%d8RGlGL+Y*&P4{KGAi>`fA{Qy6 zB0c9|R>RNAqFMhyALMO|39;>1R@t6k;`3%!*~tHczrVug-|!t-`>C|__oFp`+41{a zaq@TdnH?Y0Wp#X%CAR%GOYCV+ifu@H@4RAh=Cfsq&fiTDul?^#u`QODo^v#7z@B4S z{fO7S^NPfo|CpKRd<*$9GqPT5K-rPgw;Xx<;yaE!mYFz-G;R27JO1OnT{moby6gDN zSGz{O`{S(2>)Lbo(_NcqywtU4*{fY0Fa4@( z&*{HHnzK4aEW7*2o*8!^iOhVu>(IWvU7O$kao3&i{kZE#qrhhi@j0`j(b0A1`%iU^c-avTIpw9U&9ANXRg<=asIJ@BO4}`qS@07RQn1{Eiw}W zWzR*j({mz!Q)W}Ywr7en!{|He_x>4)lic^vHK*_UL$_rAXNWUjM*nAI5Uu~7W0~QL zIlmBR{tfPxbDsWt=-yvLM?R>V(($|c$9;#8Umtf|D}nO z&mWefg}#%C%%`!)>y@P@pX2Q8Z~ zI``g_IJq=OoLn$pY+D+K|4<_9JKRy7{foq z{eMQ9>5I3_X%wdp%UVBp=j~adbLt-!yuKN7d2U^wosX;=xbvmJu)7`!h)Z%0w`@;r zTJZYM*JbXk4wc+><+?#Te;yco*OSNBy#CJ{2JQLwx_Jj4%@v&);n=W}Q0ZOGZ`}O) zKkZE%SllvTXGtjgu9r3p+;emHC9e+)w9grQ^xuwL`QVfT-I?OwOK|V!>k!;v%IlebKcwy>siJ4_$r*=_A8GmEYhV{NfqIo~ycb*oPxyH6vn}cc`p)ZXFDs8C>47oW#k> z;CUm+uE$55J7V0VnvT=^OzaqS4(3A0X(H;Bkx}-SM1N=i56Io$oxJ=zQv`-dNc2;d*iM_rDR~Cpv$7wdnlK>v<^W0+creMCUIyY@G1^Qvzj#JKlL*piI$u{FkEht<%KGk5>NOZnV?z7sDcP zas+HdBtvx4-2KP8zwV%Wu#0;#KFjx%a$AxO`62%w>;A@-lk!P*exkSOu}J zihjBn{qV!Aj$dxL6FPCngkNlEo$y|T=zMpu=zOOy`Y9KElqFaXmkdgr{Ge_Q+Hpz8 zM>j&38fK$ybsc|PIje(YfHrSae%7CJ`=J+N_IsB-pCJmx!IZO}$`lzHO>|#kN;3_g zFa);(e-rqG@F^07Q171Cjf<=mir7IS%^zedJ60aWZep#?`c~fIbj7}}GQ=^?GH7r<_TGp6MHb}S_ zV2PD#Sll=sltszqRb$6CELk}2%(EKCEl4#sEMC3@|E;ucZnPFoUe>ta#+w%{TXx~9 zL&Myq%aC&Bq=vamtQn2X zD_5m1pE>K|aaFj+qM+G}tmUgxjg%R;8*m}*q}0mglz#dJ9p%> z(2V4gMr+Z+uTMj)Maf0hYP1NwWG!0uwa_-Sqw#CekcErpue!crvDLVIk<&Z=8;zrl zI~ZmQ8ka4*-fC=G!h^2KTA5mzYHmVyqODw!Of{}Zjr|&^?}Emr<%_HZUyE)=*Kk|> zeaSba>EsU7Z+s2Kyg$?tY zQES{cm{;|VdDD`{~o95LKYNDB5L_&S99S)xE9?+^BdM6|uY zO|K{!S;WV?pO%D}ynj&QsQf#X?*qsJRk>H;e!1u)(nU^GW~@yAeR+Ss?|u*&Dg9C3 z{cGg?!eWO{69tiQl|wXHs!-<^|38QC=V*7FuPboBxX_bZy42w}3jDGV$MRp@xZ?6A zeWwfMwb0*!yj#=hN)hh9HC-9zuR_rMQGO;8D;Wfq;f9kp@a7bUNckqnbo8ArDkt5V zuJjiZZbh*Jh*AhCI@fl(2gy-Bj+^235=m3#D%sO*O4sZ1Zu^+>=$@~QMHenwymZB)=-j%x8J9L(K6TEe)9Yq6Oub_6)L9oyy+E8^ z7d<0-dZV>qNi?~<0inOfD2k@KQoI!LZeLwjhz+kxi(o6@Y&qYXslOt_2K%0sCv%sScae0)JxCHl6)?s`^J?QuxyaL~^ z0_|$xYk+Bj!6s{B9zKmi6wXK9g~*4`(d&U>Tu$I~bSe1Ufb=(_%w;HZInp4|pI!-i z6L{PN8b0Z6%rlu8=!p#UW`;P7Phdm|+aa<={wq*fA`W7}v_U22K>-i`5oJN_ZP)_H zn2T+Ji_#)+&e70`OxQ9M!&e68Pl%c@>9+tT3rU!D5;!VxiKGj3Cefo37fM_yaa7{5 z5|5R*TH=Jn35jP*JWb+x5<~1lNh39db`9VN=m<9Vq4;)XfoA5 z%{U}ZNW4j6f%%T;Z4wts{D8z!iQ6R}EAeiL6B6&2c(%mPNjy*DgAzAM{IbMpi4RNM zCh;38J^GmHca(8Zv_%{o2F}Tdi2XjCfYG7!iayVWqtF+k)0L5tEl%`za3L`9n^)rC z1ml=EI#|cpl^UkwQZYNKV^pR>2sA-B`MbqDV04K9Tk!7|`-eNYNn&xT5c&8{7t?{M zeo@8|ad@=T7L~qjj5AL-^ls69hLb+c>2t+FiAjDr8DVjFyv{Ewj#iuH7s7s^*0Q{A zari>odvo-K>LY4}fXWDmAL>8NEdLeV4Z*TdNX}A z>ChGb1TdAS>eIg3q0hrTuCFb`efUmS2I_xshY%0o`+j_ge)tgww@doiM}@c_-`gZT z@e?6V$9K9YfAm#{Kj|ywZ%6}MmGq5Ue>}x{I1a8NN@<(5HFwu$s z;h#F~RrIERaIm7c0aO0NGJpClA?om*F3NubnB;K~AL8GB+`&{|qPM;6V3q&qM~;4i z%=G=RnWV3DQTl_x#P29R#P1;NDP3WSqX84AfvNurCH*Kc`h5bJ^k4By?IJva)iryrZ=;Iyny8d&FlSke{$ zXdj2J_%{LT?K>#x^CZ8*{0w8f%mdcP$9`ZckMxV>Rghum|7>7g|J#9e{XYRr?P-_! z6WHe0^}7w2>UU7mk4k)4;;~rw5&fvdS=B>DU`SX*(RDa4Bm#wbh z`dnrpC+zf17QUYqaC`vJ$hV+0f0Tbel7Ihzzy07p&_#a==_A*dPf!84)P1!c9q+rZ z);cdb$=&p7&2@qAzJzcOoo|Z2x=(GR%YPqY`q=yJ&g~xQVZn zZlVVVVDbdsVu>(mP}7_fx#C!XD77=YN&mA3Al0!kqRlP{Qugkn-6+pVPs-F(+!Utt zc{Wa{aQ^?u0uF4jPwjRz673_pB{DhvoD)Pxdjj22)gH(P^@YWw`fj26Wp1!f!I4V7 z!s6uu(bkIciB9FCzT54xVNQB(PsR?r03#0cY?R0JZnogv2k$Hp$I3+Qm7^SPBAq2Z zaLN};aVrPutM8-%hJFNTzH$0rwhtq{`xlTWM&>U{XNB$1)vUC~JQ9j zzW#9KhxF86N}QxG+pJKS=0ENq;30OJn2Dxe;IuL@Z|lQA`T)uU*K=1@6Ot0MZz?nDg9-8K?0)-^4ewvc)nyh%}>y;w?I=4ZB7!3as1%X=05 zwd!6@4|(O%{NR`0RLm!1c|t||-$44!w{hiIzTCgr=pA_-DiDK^KK%`e-zpHlT$Sxi zSmAPRPi9N5gA2&sqdluiGN^uB{<{U@TI65oV4kmNe)`k+zZD^WwYz%*x9_&*V0VNV zhVc*Mv(yd-%K$O|!)Y%Q-(DrUgAh=p$f4J=w5USV{~tr&Y>N+xEa_D#!Tu5d2mSb_ z9SY9{Zx|h(596sY>Oy^bcZl@!o9I8HLq#M#oBbcAqrc*IBsU7~Dt%^4kk<#;T4H}C zzAnI32Oa(j{4XmUmiCFpSNPk_-AoUWI}iV$)CzIjnQtQfCekpn9sQc-qZ|76&!k@np7Q17 zH~VYY7tmjEqrYxz4mtiNwIBXuJQXndC(|ZVfvW%?t-svWduR zTwmx2Dynzb|N>CU^ovB!l$Z-#ur14u?`SEkRNqP@0Ly{0Ja*AcQ8 zj(x*;j4EiU+$vlT2@!yqC{PG-r7?053 zZB}I^=~s6(#Sh?rSj7XmepFw=c3x$g3=)kOlov*Q$u#n$K7TaIuP7so`H1_wwWUOL zIOFW3obRaoWBJGrol^OC%@T!nZZ_f*1sJa(@yZwxOy=_ZCQigy9%1Asd&}h=yBq2A z^WF4s)`?6j&nZvZ_qdhK_BJ<%=I1em!DN9GZzO(;(VteIZ0?Ut(9yoqW^RoTr0-AU zi&L$f+&=i4Bj@)*yAQ=rocg_l{>J*AX1LL59wc?%#Zrgf{OG> z^3QeF`!0W{J~_l6?N#+>`7q9p(1^~K@&NxlmmlUE_^(WFN_@cnIONCvKBq_h3Ly{l zXPWyD?Lqo5$*(EVCDUhzsJ)yX^9Sf5;@^~T;z9A<`G~l}|KR=tzuj`ZP5R*Q%cuTo zn)ji@?}0vu=)$HCnpo96wF*EfZ&+1Ze%Q$Nh-g`4F1kkX_5s1MPR zUbZh4a{ND%AGL3qS)W5zoQ4Z7XZ|UyHp@R`4WaJw=#^S|XwM<5534`t2miKRaXMc^ z_2coj#S+EN_{4Ze{kCt9JND-cXFP(Q=gbEf-=Md(4x#fddj2ryr+D6rJpbm<@NJsh z#r|g7u0Gx9A3Z$`B-%g6DQ}0J1HlMU>DUvo&91Cu`5uz?p|y`=zo<# zyZ2ztqkSl~H|2 zN#RdmJy^o_q6GYp6!7{YoBU;7Ukt;1{eXCv`O$pUK=Yxu9>Va# zj&yUp5M{tW#DmCIxP-^Q?X+jR9f^F{-Tg=9ue904ISFL-)Sab)H4q>j92^bm>KE{$l#GtHdBys6wp&;IFpXB_(paar_YeT?s8O;h*q) zL+LSt!d@f&uYzKuUCQGLcesByS|ue!R`?-`$Kh&v%ZW!6w}e8az}<4bY_)Slni9~3 zcy#TmEt&3m@O_GhO82s$K0hl%|Hg`_ z#@(b($dC1FwySSi{R&ARUv=~?n85adt8e}M`bO2m<@nP*^$nFxgRJRWAj;bjtZ!|0 zEC%T66ZEf$^{+|kUo2+oUlh&L{o|bSgqFVz@e2`^be89EpMK@i5dylT{^g0df&2K< zM|kT*rH6hMN2!C_Vp1QAuwy3ckMRV3ETZmdlKu~+XZ@x0tdISqKBlEU78N=D?O;zI zp&zhE#Vy&{xP;d;QlC_Rv;L6&w?s63X@I_zmeYuG^kod}b?@}0)YX?a#G70_ z(ie(<@bJN%&!;bFv(%T;(kRu()t8cTU0>R)K@~jy-2KwhQd56QO5At`w|A(a4{i4F za<~839w0J^^7w+hVc&+k{Rxcz9LW>9?Ej5+Sk3pcKZg(htp4QjPxC$ZXE|?uqdeBf z+Eou3{a3r{gN&2p(8_qQG5=$IY1l*9OaFZB^|xc+x7oSQ`j+&8J zALe_Qv|JyM_L0A+ z=I3CL>+AZ9MV`K*e$XFdem3-l;vvvyI-$Y-E9MXEPY$Q~+wrHznEC_#+-N)NK_ubx z3CK?y$a;QfzgWu8uKbKD3{}39wL4L8IOSd<9HLVk01~1w`7xlibWjn z$JB0qMI1$XXMXIBy+d>$jpUwxnjMUSpVU|I4}@T`nEyh=8#lkK@u&5`XW3Kmrx}|0 zWB(`89iV`W!#~3DO=_PPUp*>j06X>ntoZ7>3^RV1kMW*c$m5;P<6!>7`ZpKrUnhPB z+-CJDM1Ua8`sL6$p27Yl?ydm-~%!eZRxXi?Dr@>+Ky@ZiE6F9Nz=~w$`w*-bZ{h zm)74LzefF0UfA`2v0koS^>X-ht_;n0tdCASQ?3_jeMV&EA7|oabbLelx8D4Z_z(6Y zYF9ns!~^C0>Gy}X+ogFFP~-I%>TB)~vc3j_O`yB_I!)83GK{ZtyG4R4%=1B2s=%lp z$+s{Xr7mWFn*4kEs_KvNiS0?Er*mjN3EEClP5|tWR|c`Z<%<|_deTN*E~Y?)huf_4 z&UNQ^j9*Vaj(=p#Z{XM3GQjg^7*Fuwb3J@`t0=Pa3&@_q-{JU|;#WxV800Ut`xKTD zovQT~%QK(##g%7RlV`UXZ-IW4+WAFpe&9lNev9I9;Fo8M0@fcopYT>eAY}Jp(V{e# z59ZSW+&M|?jXv5}{|Gf#qmQbg1uC}> z^2GX&4^VixvL@4@$0yK%5|3V96ENt}33LL-qYtUcHt2&U&J*@cby}mu*%K&!5hGQHJ~xAy4eGe!(BinFm&^ zkC@L;Kc4?0T)&>@HQo77&TnyRAn)(+{tT6G^*83XcxnJw&86G@jQX{>)Kcd>=zQMaJC*d8HCm9%d`iW?X%BD`j|VPw%RbK zZvTue2gvxW|3 z|3=1Zx7#_M|L`L1Uqn32_HgiE_UEL3Pw`TeDC1pqu$M!J9@h9n-iE(p^e5N1!tmEx zTjE}S6TRgu55G$O&k`A*r~b!!(l`&C&*w$B{iNThFZ}fZWmF%U6o8AcA4%eczK6u2 zGWhr5Y#Lrcxj%{dF&t_F-JQ=9{HzG%&GCyvRxWYE#q(9b^FI(TLwyU?`C*#Rx&D2< z`ZGNo27Brc!@uBoJofjE{T`fGE3!g^cs}mokHFsnzfwCe&~2aNkAwnIRe#!_iEH)m z&T6wp_BH3nqCwOeZV%*R#Ls~X?La6Bs;Y0BofwJf%hM-&!bp~<)Q2{E{79ES^0(Pz zM^b;d`e%2$;Q&=%EALz{zRmn5N8R$#{zI7@?nVyfKh#9c`cy~F z`izU3^Lyn;bAIkO(wvV+M>A2Nsz2<*xx`=1udTL!zR~6f)OVv@?9H#JFU>zum=Me- zJRfhg&c$p%*T>Aa$nVY9j(!!l3>?Vg&0X(LGUl6Bdx{Th{()0|VzN`doR9tfM34D< zLXDd~2>arn|9i~e{%lbYjH%;kjd|~}p7!HR| z(1Moi2iCubGO<6$7xW+J4^cnfKQEH=Gx_@=Fvul)#_@yk;-{RR?AaLSyyzPc0nssk z?yyTrL!k3|m&-5ZH)JUNZ?gxKaQ)r!-oFHskvl%x?0ytr)o@?WpM8<+L2*f?OQ-%Q zDjDlyqUV-WD-3znZjKuD^~QtTzqHFayn!r2V1FC-?8Q8{2XuO_6><#p{niRszUs;Ww({Suu^5>Y&`s^Q{+IU8NV83Fz{d&7ulvKk-DHl;*uLBPl&zSsE z8XxBP-Dt&PuDt>s>HCk<(r>h*p1nhP-u$Tg%d{8%@eY3d#%la}`1{SG?`VT=+F#@^ z9;MOAUUL7q_R?ohnnlq_Ej{w*R=PNY#?ugLujB7#AznDBJ4W&*`-1U)$U4`XKUrSb zAEgeV3;udr7se{Ss{X58HP^EjINyl#B7@xVgZz0o?@sd@$K#k^u4fN`(SM~5-SJ;h zzoEoGSC03Pe7+60*ne00-gLq_KU8e@>&N;c;{!T`R-kjD?}6RZb$epczN(o&-H ze1-HOE87WR(XAILzS_UP8;{0%0sdbF9a=+uYjNg)d zllvQn|H$#QCUFiYC;!j)ydnGFSPzkc({*g3>Hnd=6i=iT6R&q(sY8DTy?F5<^0xxn z++OmxU|%`@3Wv?}2S@R<+GxM@2m6tGfy?m+$$snp|I}u@vw|ZiNRRSqe@G-a1IIUp z=;wWyUWD@zUi{2^K82qz#Q8Oxzejep9`XB|aXgP_(gOihpY)IY%{IhS-2Dyo*AD1| zZtrnke-PPE_J{F2W2v2E*iU+%Vjk$+e$*G|7YE7yOAeD-pASV6=2zH~!{_nT`1IuM*sD8iN83flZ5h;m9={We*)Ul(s(){;i|>3b~@>@!*vYs{;RI3~wHjz*_lyv>{Kl|4?y` zjMv?b_Lc$jaZaQsOmq&XYna2^E*?^n#&?X@U!36dhCI4p@`5;e2oD4C9K_Qza9;Th z;p{KpgYlq0PvyujKEB9ke^2=p81zEBFpun;(pRJXtZ(@_E*&AtsHSV^v@v=oJYj-3}3W7stfaI{CuK3s`GjOjp<)GdHgr%2jn5wb3DF! z=m+HC(!X-@`0vpV=|6s=JZL`rbJCAY7>G|BFEaiR5`QN8A=kg3s2?)^@fDOuZ|sLT z|Mr|;%=x#_#`*qF(+|YEW&EeN^3dj6o?mGD4VOE=$o(&c|I71jujZHSm0G+O@qfQO zdfH3PU%`6hv*%M8e&_a;31fale*E?ndnjsu!0;b1e;fV^aBbwM=RY8SX0_bEkoyy* z=a6@bYZE?*Pk8*C{aAl{ipB*)?Eg`G7wh?Wb%@^&!u~A$lfvpyh#Ug85AT=k0dy6a41uIzTjVd`h@h*4}RVvOzYJ*icJ0Sm;W=; zk9R%&K>r!>e&_=|AIj@J(ig1PjQBqEpUwl7g2d}}(BpOy9$j_UHxxhj;&aY=uPBg4 zdN*G7hB%1y&U$YQrT5}>PWoT~@iQ;|Q6qhi{ibyVUVR`BI-lq5|M2+`xql`1pVk!` z>9KwcyZb-LAIJHDG-{{xh0aHb1a8RvAX#|a`VIgE_By( z#NRkiMeDtyqBNNE_<50VegVmoG4!#pB@p2CwTutq`LP0byb;~bE0FO)TAu;u7DV;@ zc1}U1!puKXFjiqsA1;vbL0;ea_eW@bnm~HBJ}tG2z4hx`yq*sgM^Rt7KCRucN$ank z>(4^F*jV3s_Z#;jTy(|tRpiY-#FR>n_=15HIt6Un6E;i;5|M-W0_BI|TrS&Ers>pHHoBv|i z1IP!0p^M~${k-vmjrJSzHRR{CUmFijJis2rD`1uf+SiNu(BEDe52W^Udnx>d%fY>G zFXFGfzV6Xp>0f+7?X4bE!sDO%|Et+v>0fAv)3NSzuI1+ zugA)G{Sig)tNGTLk3YbC^Z)aD#9x2%57}PI=fUZGGMzu*^W>OMYo|Zd%k{-=K7YvB zk0SrqU0;BoS3a+AG=H@zgPun#9Uj$NrOxM(kZ?YN&WAZC0-gOq<9s~UYoC1{{sHl5QSP_{HgnS z{`2qZ{S<0F7nJ*h>O6cL?+>8iL?~Sck59UY6;L7J!3UOXNTt?}$@gMq^^ldi2 zs`|(AJfD^x^2yH8(_=qRo^N;d^J;f2H00y#OW2%A@ra#%Ce)Ltjr)`5@y$ z|6@J&S^O{_BF=bm)>HoVm*>A@eTDX^_n#>FXy@nIzxUghV#G__^YYM7k018;fD2pX z`#|J+_RzR6NfH;02Yo%}><<>T#n6=wmzx6ft!_56fUf&q;&d*%m z1Vv79KU&YV_iq1CzHdhLr_n#u-{bol>piU3^GRPk`I`HW=s)TnT#h~S%g@+_`ls)2{+Y_pU9a-^{`~qUpujBuKU4YX`=9>v7oTbWpnvZE=%1YeNM*EXdHGe>VKHJ~W z_a`Ts`)SB;?#E`C>uZ#6_&2~k_hS)$C5OP>kCpC|Tz`B1^e64d27(m8<>zhQMtr_H znCq@Lct186B!`IY_4{Lb6aSDExck3lKbF?t)Lh=bq4_6;_Y0xI%KyZEmG{0q_=DPd zoYxD;pX;vgv0qh;_Y?4XpvQScXFu1FAMvw->O3v;3*mi;jz6E~^+xUV+XU~wc5ld{ z-`n`q{an<)Ne4RqBT zZ`%Gc&Lj8SkJk2=p$%@t(qq55Buf3| z#ZSEc>bc+S-%s}99sc+!&YwX4u)oXu&GLB^?L5fG#7~hP>tp_fBeu7UwfP1Z@lm6;`WbKyur7hTJGCF zj`w!|80|xQJ^e;}n&tVq_m8#n$aG#i=f8LVxSU^a0{K$^Vm|&;+gtg$_m98q_L}?0 zUqE}|sk!S*V}AJf z{?zpe_G{(-f-}GOY_F1sf4%PbGsb=~Q(gf0XNUJpbvP7r^gRXzOuqkMVwV@W=E0-g#iuKaSsHB7?>I zui%gU(_nTJ`TvL_#L*u7-qHh9e`mimgP$*bL#N+n&{6+PIv>Hm7l8F{ZKP6M2tZIxp{-&oI5ejDCM>^9ez}dy404`FTD5eUNrAl)q! z_fOLM|ENFY?|IPk&Uiz-_x`?Z^8J(D^u9jQ&(<71@2=MWwf~eBS0F#j7yb(J^ZO{9 z

    I?Tudj(Y{RUUT43Q+Qa7uCWNna^7nkc82j0HACC7vf9~%s=pMX}*pW{F?=Rr@ z4XXMS+kve0;K%VVq{sV3>3x%FMXyTT%L^!1-s`^M{hkWa``=$(rQ}UY3_7j9qugNU z{7Br|?8RR|FS5Sz4H_|cf3)*FYHCyWM8coYpF#T$P$>ZlK>oOOkHIg~z752*^ZQIh zUw4=H{!HLvd&?Ff<~i+s5%O)hQ?NtIznA8J{u=9#b>H-!zsC9+hbr+O?|;+y3m^fHKm1-x+}?Ii8mYI) z_j%&{GrbPc@qduNcJp)MUSQ_;?ila9iZfpEynpUCCOiB80V;oN40q)DokLcRM@KvX z_E^WTS9je-UB~_w^zV=rHs}NFU>p6O8jS&Zo<7eKcaegq_W``$ywH&+UL(!zwF1uj zAW45&pM!!W#N)F}yx)9*$ccg`(-+#eZ~7Y=P%_@~esf$z5*|I%x=*l$mHghv?@@*4 zNjl_h*c-?TzwdP))vud=-|JWr>{CEQDY=vv+ZVFcWMAgfd64@y{ZN(vW*N@^e{<8{ zDhz(~`$VE$rbl^sE#ezwKY08iJ_dX{=||IidVg-22-^!6qAaRHfZ9*(dr9$w{5*P^ z>BxVdlm9=EAM!Q$QU3cj{hi9s@u1r`{k@05?}7{5@|geBsV)Y;uzlf$Ca$eD@w90s zo<7~g7hU9H=6~_UE@u8SW|(;9OcT$VW#YOz6VIOQV&;FzB`#+EbLN=%(o0P|cdm&q zyUfIwU+!Y&UtjNH=6?mOmDgWk`%0Kw9ba{oiLbue#MfZ@lj)iNwb#0s`CoUPi5nVB zJa3+f8yihLf4+;E|AGZBX8!TGiH8q2@hPX6_|#KPTv_R2=09SDi<$q(ktROvG!u^+ zW#ZFMH}UAvE@u8?#<-aIj~#2`swxwY8)xD(&M@)#@h)cmXP)U|=6}{%CO#Xlo74NZ zy4u9&oMYm1&vh~LKkqyjGye$_OnmG4o%t#Kp{i=~5HlaD$0&ywSwVmYI0@au+lI6)RlK{8z3taTBIDUH&)S zWMa!QaWd&*=AVKiujgO2%EYS?h}P*h-)v&rHgPj%Oqrhf-*StKx&CX`n7F0I#B0}@ zc-=Y^uV3$C=ATZxnEBs&tBG4%O}t@)iD@@qZyz1+)9X*aZ>ZBln7VaLkDux|g6T}h zIqVLq`74b5MVHR?&&zW$*S`ClnE4kMyO{a+#d$S7 zeZPJt?%&_U0|uCQ;6N8Me|jC5&abr8#Dfs;*6D-syp)cI3^DQ0p)O|r!-lz->t9x8 z;%L;wv6zX=%S~KS;bQQIfACFoApDIFmA@eU2j};~JpTdDYodJeXI}LD1^74jZF+ zPqxo3Px5!#OZ;C^{7`@5{}Y9ozp`&>_P4^eY2Q5l&hH=el;3@uURC@g|No^h^LOo| zl$WxPuDoRZKUqG96+g-UHHDeKvad=$u6^ybe2yr7lK<-pGk;~Dm3&P5+#C7)RPmGi z|3P8q@7i}MA7$TrFQ1<&evjFuZP*VDa`54}Q!<13Q?ZxrVGyX$c&AGIFuy?pq69u&Wk{P}%}GX5R5 z)p}mZ$6e3&T0Xy1^_Bd8uQ1nN#RHUl%y>X=g^7#)R z|Nm5&>#yPwNLsrWH}EHD)Y ze-)on@^Rx+y_OFasEQx+#{yMh@K^CIB_B7w)ob}+fvfm2e=Kkn27eVFQ}Qw6W4)0N z7RZVp^Tz^NVenV+H6Y!mdl>u^3BmJ$l8+mo?X`RmkT>`v zAg?g^tN5;xj~n0ZwR{lJSNypCl^zCv6)#ls(c*<~@_Z%Z6NrC&f_=t@fWaRd0t$n_ ziZ?3xxbenb%Lf|@iXYb>8wv`8zlv8X`MB}QUdsm?5{e)5$A*N$;IHDHN7e#{>mDhh+Y ziq|UnXz|)!$_E=ViXZdGhK$1Cuj0K*K5o3X*Yd%Jj^fAsv7w_d_^Wual8+lN?zMcd zA*A>*e{2XT4E`$KtmLD`n|moAY$z#y%pV&{3WL9jS1b9r@#Y|G9tM9EFIV!>;^n=NPuN!Rbk%-0p3dzL+bW)}>hH$Wx&C2W z#nVy#ixkh+;^`>geP0B>|HJqGew+{RpLZ#>Z@-)8J96T2K7h_!w3CA`&wE6y`}1ZH z6s>&zD`Gu5C>ZxQD5Wt2Jk-0)z(8zWUn43I?7*deJh;+{+65nL5u9o4w>IL z9|L}?t;bs@gZ{WoAF=lKnF-A82R&{-u$k5)^85_y+dlMKMd$WCQgn&JNRRU|58S>V z`b6i`aNZ(f?TIxiI@j;M6=c8fbL;oSI@0HzZuw8%|D;OK{J-_kw+#IFgO3~dp@KSv z!9QZ{E^APj^FP@4QUmX*SfF9ZdpXT7d)@l9uch(U?&3#oBK!RaVXptfw@w1y*T9d4ry2OEk}C}S#C=aF%={l(N%PS|ZuvXb)A-!&;s=vt?;mvWeH*3# z?{e$Yet$c#SD%L;d01hVPe=BJ3bTB64!GRF_vc(>;BODQO2ezIT?@&-+UJ)4Sc>MG z#|U%%A6-KJ3+pTRueKhs$(}#u)^Eo{JAl3V-S^;q8iqbECjY9#<-c<^%{Qzc%>Vuy z$ba$1$AhfdB0YB;DZ$0`5uo@2$Ygce8uvh-m{V!7({GtDm=?bI%5$loC zD>b~@dg7s-z)!gRAHV-WU~l~F3Ds)pBi2I$>lJ4HyK^sAnE5|A_-YM9zKezftNvVV zJ+_MG->2R39$iZQoTsmkG}HX%>F3Vt$)8mE33=X3^Q)&%_uokVnx;={X#Vo_sh#E5 z?w0p3%g?hfJ9a*V^lVR1|JBxm+mnJLks>?aiAhxf_46Zs+r8#DVl1=Vv!(;w!&z3;Ew`Mp+3uj+Te>YIsFhk`m&g zL<{rjJlg}^?cj$u9gAZX*x$V89+yAr6SH=7-;N3(HP1&OdnX^=qgay`;k0Rb%3B1x#EW zHSvlSCJyvBaZQPiBR1{ldi9H?`lG&yi$wwzf;=L26zc2IE3C)P*Ye|haEU0Tx>L{z z{h4W%YUe9So{vw|(w9VR+JDvgJsH&L71k}cnE1idO&lCx;)w%IeEuLEM{L?p*2`P7 zMxz(TQUm*=y+5WVP`Ay|oddLIg z!E0}YwQilBzrs2@rsG(u9PRhg$5Ju0SHsqjl|wZC5&M?4raTrd)cI9dix!!9@nRES zf4zy9EHUxYr6#`N1{2?Sqlqoc#L1+IQz;X#T4mzZt4)0K%_g>O6E`=Tc>Q`4r_(0B z^;Q$Nwwidu1`}U!fr+P1HSvWPnz**s#M7pkc=~h`Uv!a)FTU8sGiI20=1ddMnq}g; zIup;HZQ@HVG4Y%^CcgAi6VIJ%;>#{G@#UABxW3-RS6pG@E3Y*1Racq#>Z?tB%{3;z z_F5BPcb$nF8caNIo{1Y9O+0_Ti5Dy|aXfC~;loXQ$|)v3^;8pAR+@Ok2osMSY2wpP zGx4ZVCLTT7#AC*ocrLU=!&75J&vVT zV!H6=g9>X$g^@nB0@H&>hkvt9x8Jc;MNFs1;7?+Lb?2X0Y8mEJeLh&Z1{11Bw^ra6 zAT^w-F#L&FY7O~ouKyCT*R40ow{E}vHk#|mNyqyPBKF!D20e8f*n9b7){I*)VY%gF zen$Ajo1bH;ThKnezBkcJx%S?`0Yh~JRGr~YS8nClZ(V6RX?mxV^%}Q(>g!vy8HL) zIMspbd-aK>?nm(+j@Wxgnfxyh7byO`{`d~HPpjY4)IJwS?B{mr`D3Y3DBj~Av-a%y zj$R+@!KaM+rk>hoq)$DA`sn<>Gg{{#PA(RU-TIOKK0QXK$E?wNpV7-p?Wgv-^^K+W zqB>svBK9*?M*h?oRL`TwtbP0T8~naY?Q{7>?EPcS{0C4Uj~-4=6;s{%()h0W&UZC_ z5&OH382PQSPan|evD9}^yjGvo(`cTDBldwujr^%cz}}#u2){f{$9ILAFKVl-`CPd>}MY{=&8S; z_PF(l*xyGuOSkvmMe!d0aPmUI@m3n2te=meIvzb{9XRk8T6wY5UywXp{^8^ju|&0Z zKDDD+VyOwQQos?u2-L^-AU$l zBJQESWhn12p3~{!WUb)%rrVz9sXcCeqt?Bh_fmb``R50yjzBxfe=Yv;fmDrLp7XNk?JhbF!De5+z)*DOMhE65shMbQ+~oIgK&WI z6NU^V{RiuGepEi#Y59kdd!zD;iqz1kmY(XjT*KgBQbOrH{+&ZKI_IbI!Bo|6xmmv- ztRsvzNqXlHv%W*4#NW#wMZ86?-}5WA@=zb%ztii}q|s6T z{Pp~Vz5Ls&%=~NNU#7MEEte4X>ObRlEj{FM@oW=sucGqdw8;8i@}NdXeP-NFbjc6; zaPe%tzTxCFF^%$4|H7ZCNL}(E(NPJ>?{;!0Res{vpz#O4znpE>|F5po=pvlFSj34J z>rasLoAr4R{#jZp@2{>S`d`WN!^zo#_NScuB;RY`@96bu*rS(MWUZZr4i??dIqypd zCufM^ZvGJ7r?74&^}V83q#E{6d0;Br%jada@^26~sQi#mk=4IH433sRzK`-Bq%_Cs zAdfxd4r=@wMr-v8hm+IAbV|$ep!US!-=rt9j97p6RT1|3vudoCAL$!L>-7yMFA=o= zq~u#v%UrPIU78^w*p zo7+S389Q(opjZCrN2z=i%H_wbfeW4aig95$d6Bq?Xfl6As%n>6-&6N{?X3&Mta}$} z`JrF4M8wTc`t>OMowQqCMQT6%qx3|U5zF_~cCG&CkC|MwQ+^iaw}FcX5{;VgDaMBVMrIr1O`dzQpFQdq+Dvg5d$>Y>^YLBW<7>{G|7M7cT;1aF;%$ODF zW`CQ?I97-9M?bF1Gn`C`6w%!L_b&Cyhd*J3yV<|w{9}MiFJyT!#`%uqDzQq-Ut~25 z9s;cDi~P03nlH!$`KOg?IGkK9Ruh)xSENpD*X3Pg4H!VafyZyi4O;t9{>|cMBmd*@ z@4Ws)f86M`XDjq;^dOpx6~A!O7PgkZ$ZDbhxJ$1{Z9ky*Uy-%4e5FPYC!0mHk$**O zg+{0RByTkyi>%ING*@}$r$ySx|9l1cK8jBHsXbo)n{K&Dqlc5Xid&8REo(3#X!#F3 zXV$;2?k_cZIN2&%jr_OM0$j`gy$8(vFRXt-qlc3l#0Hfg{$Y`I^;I-?dG$Z=9JL>f z;PFsowYAY)rs(1195IJzvj2;)9==ASSERo80OgndCHU{51*ew(QgNw~|9;-zR{Sec z2c9GOsq$;A$n~_o>!v)=$8d74n5*TFS=V2r>t~HMcsbb@m7eND{8W9y$;-rLDnIOd zjWuZcG%bD1n%2#8C#6AoHCAaYtuGY6aPo3-IniYMYpjx~G+q@wW?eWy*Y_f8)F_&d zRQtlodQq?P?;`m>^%TtoijMpZyq=*nD8I-WGlu3vMGq&h5LXb*mH#7;kZn?SCYKE_Ad|%RDN#%&^jJ}ZvK`QU4BK@-o2QxUHOKS-@@4n zTvR^y-+o$10qsOd;YpFcG zIDh|}*uK*ZV2U>t(s$0Eh5SR-vVSS*F>Cr*_V*;Nv2vC!!FO5zOp4bP5smZbqW|jZ z%<^(?yunSs74+$&b$;RG4fG}}t^Z$mp^h+JT;DzoWAW7~uMYVex_PZ4=@cI*pzpH% z;BV>tYplXmDK|gagBLEP`GYTR|3%L43Ci{(f6B~XbTjz}s=dg6nVG*<^d)9)`=_rq z^XFHQ@95;G_PRero@fSDN`NMWxD*@r(RTX8z)4+s#ky zzxrynx0Hw5KSGR9`R7yqwl=n>l3ru=UD<^1Ui+u7(D{dxBgIHJKaHPfpQX9U)Bmfl zA$!Lc_h-LO_D|>r#zQ!HnmCQVv;4UHYugCZh4g68^Llx7eu|Pt_(zVo^3lHgb^c+Tjy%K7&*i1{@?bx1_3@jh^9v`(i}6POwnm+Qo-H`urhab` z?P=Bdhm&WDGmZQgZZON6KVL5|W-VBt^QZW0z|D`}ap%k#nu zdA^W1oIFdMMc-L|e12?WCLsU*1mb^%_NGl~V8t(-+=;W5xMcc}br3xtyo<7}Wu4=2wT=Nou}m|)=Z#CZljSDb6$bHq6YegbDXamn_F zlhvZyz-NoI4LnIq(r^s#U;Gwfx+q?1<#s>mAlW~W*$oB`Uoy?WIp5~_hVF5H=I+il z=#k2Zfx{!Hs(j)6Ud-w<)6AbUOV1xp4j0*!m@+|euznk<@xywq&sBYx8UEw)bJpnm z!pXSv?m)`V_9!w+%TMKB=gYrd&tGE|Jo5;$N_{|oPZpD@JmSaY^|`^9f2oewAmM`G*hJ_>(*yGWnIoX*@doC>~O|)69SV`8?jKjojXNd^mll zi}53m==_lXVLg91nGgv|8^#CzV9Yvy0*`;X=kPl(PT%Qb{LC|E{vFQQRm#uv;UkZe zKJvxs&p+4X_sp(m40`+=lmCGOzWmi@{%4={aA=dZE)@p){?pYwbCsQmuB$=~+qQ>J+Qh(2J} zOoKjoI*(tfBiBE4+DL;wv4tgzGL#20#OJgwLx+Gro%&l@PC@^PE$S7Zf)^p)RtjQ2m| zyn8V4GBCgYIv>vq3~}rOpI0wK{_Lzwl^^_y@qFta;6QQyLj__}v&gem~fK5c%o+{+m-pDc)bp*Czab4DXY*h4im*J|Fvw^t=I| z9g@$JnePv;gZ!P}w_+aDAD;ij?Jc&lw{m;s`@_LsK5yWh|3~|>b@_Pb{lSstrN3`{ ztJ(kj`%fGBc?tD<&c#;nZnyu@-*G(uK&F!E;0Jkc%!Y%iet#ML6LiZb{n&BX@xQ3O zAM?Bh@^216q1E$X{5%FX9_f*Ph@EEC^b5S79rdf-tnvGx=kp(U|8*w6FXbTp{yO`& z^nkSUK4X5L`4U0)05LUwAGz^FPWGephz4D{u5iUn}JJW0e2RI_jU9+3h6XY+tUp!ZorzNcE@ToOeN zweJg`OH{1_AF+K`+&zXa7&UyvqAwU2AUDG`0_tJ{2xfsfzS8wI#HH%w90g7+J zo)xftvx9Tx`{DWdG~^Fad-1+!{=EpQ59R0gVLSYx0CK%V*cK2Fzb~H0SK&0RKBWJh z)t>%;Fb4Gx;zdeA%%%F1er%t^@@07d7uj>?a(fP;zC^cYWNGhv7sb2gbhG_E!;we4 zbuQ=U-)lp9*rS=5|-yBl3r@h=Kh&O?Zf->Yd24D z;vI6lx%Pexm}0!p@2gQf$EhF9522ZF5T^GVp*(uu{QiEpA?;<{a@spr>O-M@*^Gpi zKAxJzM_JVTF#z+0`+jrqi(8j^;}QKAw=VZE=$TfwGybSjFA@$p#sA0bXKcI+kW|K)D^;9qFZn#KA=@^tLsEbiYq=wINFHER|t0Pz4n ze*Zf2!}DseZ_uZZr%zeV`_f-T0-7Ipfll`2Fx4NwHwXI_bAHbv%`9LqqAm4!@Qkhm z=rq5%{)PNKj7u+NeWv*gBVLX$3Fh1UI&O<{fMV#gFe)A|1E|eJUk6L;4d0c)1@+%8Otv>b6`j2n{ zaDF|FAJziK1@)vl9xkjWQ|#fQdXBG>)^Pgb`eK9Lx4y4|`_=RCaMSm%?{Clt)DJN5 zzETDH&}nVi>dwD^`kXR{%IcM86kPH zz6}xKs3+g36{;WOrXPd+;rg*ICVr9nDg)=#b3D*(e{TI527O5V5EoPb46PsPV&XTf zewcyF>S_FV{SmE?8aP%TGjMr*xq&O{D-0a3j~jS+{cr=HQh$npPpx;>+X>k}mGzYd zeMG&p9#-^`^&<`XY4sdmb>)#)PwN}6J$>rWG;n@B#{=E;1@#=ya&cij$GcoyRDX_< zzqp>`XD+>OJ;&Dw!@lr(1mh*g?e9?{GS=&Fq7Om;@ij`*FY3S3 zefl-p(66X9M$;ea-?2XZsq*R1IG_HU;nSb-KKY&LliyiB`JL^PU$swu=lJAzu1|jF zdGh1&uiFo0Ut@gD_T(M4zDD*(>Q^50#nYehB4?hiKLOwP5BkP`wr~80 zeB(dt8~+jC_|Nf;|6Jer*Y$_`@5|Po3H20@@W%i7^-g>TY~U&NQw*HvvwwYj_AlRO{|bEeuh3`zihTC3*k}LxdiGD*ALzd~ z9{P!JlQ$k{yo`swDEkz(^!b$R&6ho&>hl%ZuP<@FlJe>Y`FQe*i*VYL7yE~8`uNi4 zHlo*x4B!trJ`9#XoB0);s$<37KA>KS(~GK7V|n^Me|H&=)nn-fx~D zBKvjy`JeID5B=}WU!z3ML0up6H2Y2Z)yHSQ^L_Taz-PY;efGP^XTOVm_Pejoe(U}{ zjqgwQ?@j-m^zBde-}}{HVCZ-M`l$vUP=BF;2iDgbxTJoXflKSB8+cItMFt*Rf3d>K zzkt4a;}z?b!?J%6&x%^Qy(am5iT0Z3W8MCeyuWPw`xRJUV7wUP=cqS+czuEX^7;?I zFL%PUKi>M)neWZ@>j2;SVxVt*QQ}))l={{egM90Y!M^py=bJyiR`Ubv?-1YmYN&60 zHO#lZD)X(cqQ3Q2%(uQO_pPrgy!Dmxcc3qheq=c7Rp^JKPanH~R!8wNwSQI{`Ger! zSCGGNYW_Yj;???oQ28+I4`6mDhk=4Xh|f_xnBooIepVnr^S`#A6$l=r_!?D-_Op&U z&!=#F(6^rjdGhZW{v8;|-&dX5oGHeVy35~-EVc{1-v>qc#{L)L_qR1?XQTWq&X4$( zx&IY1_rH$ajo-fvAt^3|?&y3A;)Q|kmr-AiKi`UYUeJk`QjiP#WtmA4a{M9SWAyw^ z956jGNOgP}&kI=bAjM1AheH2Uv;+|tAb#LacwHD98K6@a5w_w?=k$mtWn%w?`iJEU zT+6#vbc6PLwnF|nV&2=5j`m;B{H~PeLC`7Qnn{TrdA&{jljX%z!Oz&A`W44-16dsZ zM17v1{@J4SPtW&dl{RNN{D*FR|>&W08#e8`hDJ}6WCt?zj#Y%Hj%q2 ze*flH>=#@yhwv4=UqJeV`fb7u(x+b`Uf&X$L-9xK*Wh7(#FueCk@Z!^>xbJTv;J>; z-vb`ib=`U2o6%_W2Qgqwu)$^k*~Sh5Asg3pN&H4&WMlrw{1JcZR1%Uv*b=Bh7(r-w z$ccmf`K#5&Nu4xK)U4aMDZ3?&H+8?b>!J0wZa3Ri^7YHIjrWV%di$kacSE=BR_oZF z{hj+~-o1C;%zyepPeh<)mmgk4|0zLOnig9^o#&0@t+ij!| z36ht~6Z*w^q^oJYGLD$^KmIr`Iyd5$EyL zf3LfM?-WQMu|Gca5w7oY{+`7ADna@mz{{&mIFU5r6*W5iXG7QD%#%xbF<#V~`1!Y( z_@+7&Kif*5tT*x3Sm{^JF!8lk`kDq4UuUJiW2T8uTIp+3Ccef>KXaCj$NFcs2{&J3 z!u4}Zc*a~4ZkT7n@wf?Jd#wr2`H%@;cby4ef4vFMpKrpc1txsM4JJJ6MiZX5(1d5F zO?c5F6TW7#3C~@k!SJ7{ok{(Z8<|(gpC)w!@1LY3=x-kznl+EkBRHPJvlnrDk;m6x zOXnW}e#RU+{|MlQ>*)L=;GbM6?YZP%a|iX8fIj|_xB*u!C)*z26Dtx1T-{WyU@m`M zGuexPe&&2Szu++J%VWpbQsRI84XJwa?|^^RT(V~zkMkbFCs6^j3I8L9!}V#H4-HN= z!5k9&x)Yp7$@_)v-yg<#onb$?{|5;`^!R;yzMqno_5t$)vVT+`-2VOLipX9JMfO7L zuNWU9d+}4Z7hV~A(Pr6;PRm{l7Pl8+_5ml)rhV|#{NcvP9z^yanm?d>1mpXAFn{pN z*n=_49*kS|;OwQ@1E%H+n7;(;w?T(}U(O#or(++MUE%e`^7nXs(R_v17fsRn0{t;q zuU>WQi{>)(g=J;d7t5{nMN`@JMYfE6n6T`_q-7s26t@p6hpTcvYFq3FE!6y@#ePm} zO_uTBOQrottS|EWk#c=<-7#7{A<&)<_J`aNOg6~A(0(NBOMbtSoUr!M`_+jA-%lZb zYV3D3eliyLYht~Wg1>p*UlaRrb4Ow^^0(x<049G*O3(RY{Yd_llpTk;KV?GDQvRUF z{$YY_JKtaZTZjBnU0R=Ve^m0{Hum#ovM~ScH0Zhiwtpkv|M3R^)Be@#4eUwy>yot` zOyl7P0#hC0N$)IK)98$O}vQ7XKDvFYy=O#4gJo_c%V z$X1N-krPCp;{MI_{y~#yZ{!%+CjaZyb)nTNNtrhF>KY(Lj=j!(}48YtHoC>J%@wz z5AxK|3f{h0|8RMv|E{K=z}x!4<%@Cu`5gJbLmz(y?fJ#C8$)98*f3aWAUOBAF13dO~9>aM956&;~^8CS2E#I0)d6TqX z{R-ai$cL}ueW4T5-jRL6`@4;wbT$49ao!TgSrr1Ew|svKdoq@*iVOdEm-ed&cBws> z4}0OjUI_1CRp0K2%Q|Dw1*y;Lj-AHQ0fj8zk8V9iwMjv=57Cbq^iLb~hW&-UZOYp} znf?U5yN^KfQh$KGqz!X*p7rs^k<;g&0l^s0|7)L?_!Q`Azx?qx7=NFKpRX-Mc?KDF z;1Mvs!M?edo&PHq|EfTK=o{p(N~B84FYtN!FIe)ov6LZyoGnuIBlyRwypsF{z9|2{ zDB#ci$*KM&|LQXQIld_WFR?k6{MT9XPmh0!&9nID_2;t4f7;-W_DqziZ-FmbA8EEg zlb7s!HToadZ`eaqzgIBmx4;uW;9DQp_2X{ncVhv+szJ%`b(`N)oFB&*mG3#5UjyHt zm*>f*d^o@TG!yuu{9ZTsxhG}>{o{Z0{xPE=T~dDqz9|1GgFossS*^+sdk`Y8z!&9r zhBcb)QyniYpTHO8_o~5<>@lqexP4Rl>6a{jyqfSu`M_X zpOg5Mz_)%u?_V^2;C$EQH!JR!EU&;9m2VDPWAdwtmu??{FUqgp=2tztba@57XnA{V ze(~wqlkMzYv%J!t6m1V_Pl~n2X`5elqD*-?zG!(1`ITwkYK|}39_!dHv%J%6-!(SB z>Um|#%kd@3Ys;4?+TU_y->W#jXnBWOk6GS$by0oh`~L_XV@1S zzwcmUCO_jo0<*k)ycPIR`H0^Hd&%S{-?vBogqK&~i}G8*vL?Ua{rhl!0)O%Rg7?jd z-|t)Y`9H|_(Nl2a0>1V4ZTtL!EuZmzY)JhHe5ib6@0YPx%<{_jRZ)M>+4{GI`iH<1 zKj2OMI|P0Y+WZXvyb$>WK2$zx-+Ff5l+W-dGx>4(1U^*0li=4`$Zt~ixA5|c{fMIc z&IkL6SZ_UeOl-ek6lJvjT8aG!hQK!+u)b?M_O$y3E)>Y+{)n&7YrulW66|9e1917G|o2b(|x9f+Y_#DXul=T&~+7l2Ul8xPba8+0!H~#r)z3F@E87rbRR>k z8u+Zb-;n%wI%&s>^yhEM|FI!P>XiE{Y)eu4k;IHs)Zp^{a=LGW{Ga$HAM|t4P%H=7 zXAble^o=9+)PA`-x~`waReL)D^uSw#Q!%mqt^Y6R>nxL&{> zhf)7Ynr@2v<@?F77rB$7eZv6_n8bO z#r-6tL}>rIAy&iNVgmgO>8o-6!rKyP$&c@_<9xM5jX@8T2n6sDD8}+@VIs`vQ#@JbhZ=^Z6yH z_?tjkacuCLs@!BR9JijFNU$7UN<9qJtul9Kh?o)$fF9p;J*8BGP z+REWN-dw?WI8pn0emtc4$XmgFVAeVhssNZ5J^B~bKvV-!4Ma5%)j(7OmsA5Z@5am_ zJ9PuTOdUOW_ahgo*xio*(HBl0y`}Yk&EI!?=2yP|=F6M6&-%;UJ11!(o%-F5KGw=M~v95plnXfHuKl#m$mw)%nFZ}RdpY3=Z3xYKJ>SEVt<6q~Y z_K7FHy7Y5K1ltEftkdMe(|MB`y{`22`U$`G;T0{rD?yuNeKmTv^`rzn7nQGneBNRbugpHE+M1 z`^n4nb-Dq{XN85|ynfAp-1GfgZh!mjpS=9ePd0txGe7yyx!nH))rH*2ckU+YsobAX z%v-sC%H{rRE_e4^i2I|xzw`2Meet{R{P@R2k;`?wAfCCumSX=WVuA3XpL^^Zw>e)t z^-hkx`L*_MfBV}%zztVvzGjBzK+zJU3T!_5q7Dx*YM=>53I}TZ^X<;t{;2$A^zs^Z9fTxH+qM1_{ubtv* znBf9Bb&1+HIqbO0h*utejyWXh;$-4Y5}9ROk3SVB_)OqyfIo*0N_0U$dBIQdKUEz! z_?;@Cr_?Q}JLd#EodOyvh*Q@9nJM>}^Rvh#({N+0`ULr994gOu9cTWXpxy@RN6{;? zxd$9~{?DUK88GmiS~i-m8<6l&3QuOG6dpakBp%1~mP|fmGNg3NOPMZ{%hEFVrZV`u z%HX{+_}^c2`p<}{%XO_^k@XwdQpDlxbu?#+22LEay z9_BF!69ss)e$NAMx1)p!Q`oM&%`!c#wzQR)q}2D-Lq9xx6TxZimzGt)?ydE(GTa7E z!IFyMTG8Sua&fxJKBhKcE|Wav_zvv#tGMWz+^1*|Bm13|FmJgBoRAbR>J*vzch$P{ zH#ASN%&nBfp`9fU4$You`Xu7ObpmPVrCJ_Yjy|(`ZY_P>`8R^E_v=)uY%WVzEct>f zA`n{ciYMI7p8`!!CpFZ$oXlH}474v6iU%tS^YZv%D?LYJ4uX1XJy=V1cQNI>m~w^6 zBdUdO$c5r{dFMBxylqRV1lb&_LpomyiB)+^mFJP#VAOKrd=zfoGgpk#7`su$8s4c! z5Wyx$^Y*|{Lpw9%$EMjxT|A6PYsw!Be06rB<0EVWul5S@57Rw2w;`Jl= zh|#(Qd7pDVYXY3h-^<9pIe;{CB`Lz@I9>qqgeTFylGp@Ot=gvbIk3+nTJ~lDZ?|ATB23GBM`1+?GhVsrZ~XAs=!? zUKk;0kl?WRM`^B6e5yTHqfaS%jhvA_x7^e+Csb#tpa@+5q5JOqbTdHS5A%J@6EsA} z`7F8q<~PyJw^Jh5bFpw5O`tof;Oh9e7iXvACSH^;U9^0+CgY@|&eS<4>v)Mkf20lv zvox35MJs<&m0zzfl9~7EpkDL!L^ex>!um<(ZJP%ovJHjx0~)jwg)}~T{cNN9NqR~4 zSaLKTQa>C|_4k--;)D7LEsvA0AF}JXJSIOAc9LSIt!A;0^7_$cwK1;8dHPdTM9*ao zshcu&I8W6z6;p-6I*iw{tf(IkMB#M^no@O0I`mlFtt(xJCO)Xcd|T_~3FpSXzfgQisZ6JCNnmKd$o%YRAbBU_|aO9g$1>x5SRP@M}Yf4-hd_aV#= z%C;IZ@w>aI+yH3;rQ>VvoJMV@Rqxe%MPbQJi3Y*T zsiWB`UnBD$I|$%MU3!@=fMXW?s6%*7hP(#pGL+Lq%rFO`&zQoVgfx7{iMQ$u_%vWD zHV>djrY^zIGO@1#n7(qw1s+vu(4V*9yBrhmS#X;L?_%+M_B10lV6A-l(OvIBHa)e8 z9cOBm9~0UcEMP!Ch%F4Wj!gJ8@OdFYk1Y?Yj`%MNX?0}MpSRNA<(PQSg4-reSSlw^=!uENnR%=T;_XMh+Z_yjVGULr;5pz_C9pHv+2vh+V~kdQ!AIg z96l)D=nS_31HCMt8l%glxA7XkdUr-Pei`}Ix%K|#0Z9hjssj#<4CX?$ndi%Dw7shEmqNL#gq{{ zXZiChasESNeg<<&V~mAS&}aEHU)K6-e#NNz63_~dZ%(oP{C=75=I!6~=DS(3lETu2 zLK>e+(?2e61*JACbwRe@S-^J+VT%@Ar9%L(N?x1#5sSM^E0v$RV?3()M*9CaGR>c$J z-+9sRU&N_vCz0Yqc0j>crx-B2g)VB&L8rJ~L`wAFr(nfT!JHrP6*Wf*8#}`Ga5VdO zQF9l&e-yS`aH5nJ^;%K7`+jI=yNxOhcDvWm{sV0{B-{o+$^04F888{7PZaF(1oVED zJ0qYkW>f9-HXQKtsz&QCo!_O-LG1i(IFJW-p}iY<^7WaGkJi15^5oO|IHD|{-kc+` zq+K2x4$9}nM{CRT59qV5dlB?S*7L?Z(3S~sMcFN!qR}WDoN2Xw6KOSLNyFjACa2BfU8afR%O*NIL|mO&YEH@RjGN z`H^wg4;!z+d^Fbl(3X;`h=%7=1PxZBxynzSfGJp=kX6)Zj`Ho&Pu#*ra+}76jW$Yp zVi(633Xm3UWaNSL71aYEFU{BvC>_&aeMU!kr8A0N>4<{4j!cl&UhP^^X)ScM*QebB z4ESR9fCB5u_Xk|)kikQ&d{=vHep&Cp(B(6ApbVpq}or`D5w~!lYl&aD+i_2RmAg>}jI zJkxt;#PaTmym?G7j!W{;x(G{wYby&{Hg8E_DN5pDDN0%rSc)+G6|xlCy1$~vpvY2` zo(n~mf{%Wl<-2)V{kbAbfq{(X7}5A+jyKct5RT>)T&Lbw9>S5Oh%5!y>dQL*M3y45 z6x6S zB1=*7P3Fi_d>}1F*x7`>`Y4INx=u->vkz^xWX=h$>V1mnY=c{#_w;PTalLop&k*Ka z8ZwSl)JbGT_`Vr){DQMlfjD{b>=wd!&N-Wjtsr1=Gb^mEc=#>rP2H%p>2Z=VVsYq0gPsSz6UWWqG}k zhx}s@ohu94RKco^6#urs;URmb533Bht+)xPA}QMcsEE^U9JQukfu;l<}Ai!&@V^t!?%QZsY0A zNBG9$%~+3o<6Y$&PhEpTq&4h{?jYFhVZ-Ix?;@>9)bp?Ep6`eCY;C_d8-~9^JwMv- z{psxY+CANdqn^)4pAq<`>v!^ARreH6f6R}1K5yj@^i9Z7&#%1Qo2ci@)dBg5(cX=^ z1tansGiRFR*5n308uOzu|C}4m_PG|kuikC4$9Nl#dVbXNqn#2Ai!tS}pmv5&kzBEQe8_8jJY-`@hiO;W}*-P5U+nfoP4SSV`> zc=)TQ82JV|++UsgY=-nEl@EB|OZo?PCr+2(1_GYateIe`QJ(Zb(X*7CoSzRkw?Xv9g)reJWo1mf&;o&|8A@yD#tZKJ#hs&ZzsyN8Qmle6ATL*U^3CFiux&)@5i_ zIg4iZ{OMXSo1YC!%TiI6r1IVky$D-U^8F0hl9G7Xl9Kpy;WwUNAzKo8aYwc!mpe%| zl~z4h_1)8vE#b|i-6a;;5~;TzD8KM%UcznFwAwtfC6O(Wwj;XB7zZWz3w>+dUV}ty zoWkAV$d*L5B(f#>)7HqA#2nOt2(X`?W5Ol9mLppt>Mi&UhG>mLH3B`F)_fP~E6S50 zTH{Q=%_Cb9*%E0xqBTym#+lam@xHVrC2v>9IZjDD<|QTZIL8UYN9Q=k8S4jqUQ&r? zx|P;up3<*?aYF| z^-%;*S`%s$YIW!0%KX?Bl~Ok^k7G~fC`J-moFbk3p2)#0PwwxN3DZyo_pZ}9vf zPy&xUr!?D~3B46QYuxzmBXQxIKIr?}&GAKUg|Ddnt`HqF?ICW@w`oG(V!&&}6w>&B zzM}M$%WK}^c(r-eNA-CHy|``t4x0mLwmkGbTbqY%_ln@rTVS^EHl&TeR5)z@X!8i8 ziPnMKUR}|(LYO}3ZwWZzp11m20AY79S5Ec^c)esMgrM*M`ekn6Tci zSutgGqLJkajx5vf`F@$6pAEy8*+D9&ImLc+A@bS(>Fhz+J>7=OwO`cp`It`oMwnb_ zH~(H;`$#3<#yRCiJ)gJYv`!DvbKY00@u=r>ow(Xo=jM(HcD-zieQC4(sOLvL-`1e0 z=SMw1x_=q7Ia+s@^kk04{QQXsYAe6H7_IZ8b-v}35UumkLEghPqCLjja5UyeV}3N| zM`M2B>0q?O$1THqYQ3kwx6i`%?%$(Ry=ZU$r?a=Ojp^@CpD*0rTxlz?@;y0=WmMSg zOf0Y*{5pAvjTU!F8+(HxtzS{=vdViK$8I$n4rvji#cX;T4tzk*Rb{JK0{Dt*QPO>_ z4up>;&X2`-Aa)NzZ)#V!wIe;fkw84)8G;1(1D7H^1s*q|(+sBE?d&9*;;r{IjW@TU z8qED{N!}^V$5d-0$FIsd8R`F7)W6e*mQRy^P(QjTH&b|| z2BkaI_z}>V5_*P*Y9~@=R?n$z>E4=n0G9*Qf?Ec0F~@WKMvA3RhBa^sr+Xjp%D;y= zH#SePwxv{B?HhhI)4dmUQN!cupF4jU>cZo7aV{s* zN$fbDfsUP8;xEbGY_v)$dlS#cznP~P!+&>iLt=_+gmbf9H{)nkYBi(95?;b@6fO1h zXtgj2wbrUDY%NW~MgAgB`>lj?OR`HCd9PFL|1*SBwW(lPo2pldH(aQuGU!Q0Npb0f z?jijun8{ZA7mY8q62Y7=N_ijYrH2R+%(_+_1ulz6&;!ITM`$H;vP*nNNfyLs9DfOZ z8f*Rrv|;8}QXVInT_oCss}iF&Dbx>dv0fU|58KvI|7UZx)^zOF97byf5@)WVz8%2l zoGQQ~{Z{ag&9&1%13cApfpqh{YxEL~Bef=nYxD6@Q0LZUdQYgH=hOOM9wJ}?G+(DyPm0sUUkSGzX7&*}4H zaLsBcjkExu$Gjh7eNc<13i8W>O5l0lF5uq@{=5g=)d0)XISb^>t;H}qim{&n8QcPV zJb|P4|a7!70?3&jUAjECVh8O{Q)wLSNrEhkepjHf2kL?}LxX)+A2$#emDZA9Yw z80hyY{<}&6>P zwNjF=UBbc^1s!DR98 z&|sSVtx~XE{_kQA(%vFwX`n%MK_dE9Q@$?oKmQE7R0>aMu}?fw;26R~Q1D#5QWRO@ zapfpvf4fLIsP;>jWl&O}jVj6u(rjTrCUhW%x~D*8<|T?5ZyI0KH;7k23slsuRaBqs z1AViWX(2peA$+nB>BvH0_~+xY$1bnOQ$<+_Q$v``#}Gb#zduIBtel(Ej3mS0|hH+HEe5t0v2_&)S+S)w%e8BN)pIcT`8&<%~W*3W# znRsapN$VOC&DHZ&h8F%n{YI}f5^GlJeK8UzJu$cN$&onH`Di3Aw2EF_?4#0L&6ifu z>=km4iV;NxjjY8-f~1_E`thh&kU~UjGE{=8^GusfzCT8SXs43zJ78_W7Z;JvL+2}c zKA*SZ(Jn)z(KL)vFs-=059I?}cp~+_o!hWPG`Aszi0nKn!R!@M=Y6u9YHnjv$k`9y zol`S@lS1z|oyeiEXR({b$><{FayE=2nH65U{>`8m3!}i%dayQmHIyQ|h*R*g6ydQr z;1MbZJQV{T(*KTv+Y&j1 zMiK>qzEg-X=L9`x)}QkIr7W2{kE zWJ=0|0_EqA`b+KXlB39eE@4v21(5v;#j(X))&-K!pnR(Q$0SS-;(yWkeJUhBinH>Y zl>8@Td3_1f0fun ze&d#9i<L*}pUUT#ATRI>j;ruh%m>_+$Kw#nz{kjrYj~G=fP?f7y{d3wMm{~N zL%>_&tQueGRi9WJo^Zd7&3~S`8}Z{w9guJ3Yi#Tn80$QYw|tI5M@}%-bP6t=k0JRJ zc&q0rc%DCvxG&)IEI$7hac4mL??CgrjE#K@&u`=TGM=yCiJ>#~Ul{Yg2amPy1680qADdBr1>`NHU%kvwT@Gc2wCEO|D zNeTB!m=uKM9h7id!XpZgf)d{6a5YOy__TyQ36Dv*Q^L|hL?{h|Ez@55`I;}o`feP+$rJn5^h5qQ~F5`SFu4D9fFY!UiqtGSqt`3Cf>K; zEWFbxJq5=}Fn^(dLB{!cO9bq3co7?0s>8EbYMBmS$Hrloh@YUZVUxErh9;75j`2qulFjSuJ=aYPG&zt#Q`0pma zvDeJ_&l4%lyZ-2)ru)1YoL9QNA2Y?GX4*NzZmV>~r`| z!GXNlUv}6t_~)8I2H#cs3xFR7tn!=qTZf?u**HFwKKpk9COqf=hA6MXd;cI{h4%qd`U#nS!gbje ze5XL^J7chu_@2dw_z&V`5Q0fw&L3}B5UkRB3tXM;m%2v( zNy8z225@PK_gA=Fc11tdYwmH1A? zQG2H`U4I7O&q~+_toM&ez?7fr4<6SSseG1PD5?&Jvr(XkmLBY=V zN!XpR_yw>}Vezj$6&~NeEj<3U&EfGcVSX0I|4G>Ku=tIb|AocBG^sfxeg*kr7~<}8w zyZKs9Z#L0HP74>SH{X!yreyzDEbR1HinFkioTb4J;jBzRG*nwkoOy&q*oFGFc~X3* ztXfR>j9}A}{z?%I%F}a1vNJ|%v3Aj;NY5#~XNhsgwSp=w9=wN;>`Jd?G?T-`pn}zN zyG)O*2_B$!JlN^CBK9%4oR=j0@w`mGK##50xxPxgqa z0Nu+7{vrYT`T2=hv-k!TV?RP-;>i~j`HUdbPa2Uz20e=G zI_1=&N78qh{w}Rxl)okpPcevbc&=za+2eRL^RVYeq47iFN%c(x1`*)9!C#itK=tfK z!4R-~GJ>Sd$8AUYM?>;|`sYMKrT?nEZTWj;de0K=M>2h%)L%+Z*3(XZ2RiR4^0(8s z?GyQr$%awsZT+c2L|FRC10wx-M#3{NI#A=dN7E9_$bnU zI;8$O9~SAKkrhr5rM2@X^`JVr>#(55S0kwTjC!i{#P;2$qp6?jUq)sWFZ-K%>iLnZ zO0Vs73Z9e=tf%8qsy`}1`BeQwJyp8>nQiSy_urQhVeO`t=Kdqyx3~1~zJ1${hcd&h zJv&DZKd@`V!5tL0YTNeywS7JPYY%M746NMRf8Wuydv_cG-G<@z!EF@RhPZq7?iks= zbvnB^WY58^!@GC&?dxvd+_$%NC-EG9;P&?Idv>iqyl3aC<2yjNwt3~cp7zcC z-8)D6_a5B55j-~RCAhu)_@13xDIXms{QY~|Tl)4ix9`}!bHm|1;Bn8rJ%f7(wm#Cc za`o}9U4#9*nn(Kg97Gvf*5BT>bJfFpZ|{K&{dewZUwy1=XV0OY4UlhOq<{CW4M#YS zcBI?6Y6!gB_O!Mi-`x!P+E+J`jC*h2a2)a5cMps_vb%Zscz5giW8KZ|`+8d2k90S$ zKGwa8%CLTA?|Os2wfpw1$M)=8KiuEizGeG?zAf7yUZsDU`qww_*nY5i$NjDQ?|GxPwGJ4c$37nSoK$`^IIz5kv&*Y~f#e@EZowpMy}Wd=5_ZzkDB`nqr5 zbY%Cy`s1jhhr64P>ayOumFj8N1IITFf=BEA`@7cmA6nbIf3Pbva_Ih@`!^H3Zh+tq zZ(7^GdZ1_Ls)u@7S3k1%-Yw9NJ*cy-D|a{Vgxp&X-Fx5Bb*TS+T-LR}u&HH7|B;?` zn~n+i{!JO^E83Fs8R%Oj$xl zV?FCv5A5Flz>(dzw>RzCwrXH+>-ytei0f(I3Y}>h*|csgEj&%}qkquGaqhx_0as+`qs50UH3qH-(G45v_HN#^*~oq&f`71w(Z!z=Yb>CwL1p-4-9qe z>$;<3aHx0Rf&TuEeMfsbL~QqgeO=wX9o~?gGA}V;{i2p-CAn5BClo z?CsB&LV{Pi3k0ZM{PSM~1to0uYTV?Er0N z_|O2+-?y%LWyiL@uEE}pro}&-{=_f-B1*Gz`AS|UHLf|wx8j8VMrT3&LOy2FxT!*; z`hw>&d~j5f51&%Q*Y^4wywAE%$3K_&eDYW8zufSw;7s&Ad1}T+%Fq+d^`y&{eKJn>6I0pyHLi@vL}dZ#u1bWpC8_UGY)& zhKc#!;f}or4t5P4W0(i_bZzW9a>KJD z=q(UvAGp&Lwc{XRH*Rd-wC+GxKc?xM4)q-AM`odG2VPVChq`(?h7Wc0@JxlU$XduC zKq03tJ=Ow>Ae3|EuS$y5SOqSyiWLRjgc_eU_7vz-*UdyV>%?!$ZSed+|Ir zz@I~hdb)UQDf+D z0#Er1YMB@h@;{`Y)+quC(kpl?z7S8l4!lvpQRS!TZTR2I{EMYm=j|%JUH(4>!md-I zJ@Xm6M1q4&yBHB+C7l*s>r$d*z4Q(Bc?)&G}K@Gwb79v88l NvSLUKnp)WD|3B2cWFP= 64, padded KV) is +// specific to that graph; GGML_FLASHRT_NO_PREFILL_FA4 disables the window. +bool ggml_cuda_flashrt_should_fuse_prefill_fa4(const ggml_tensor * fa, ggml_backend_cuda_context & ctx) { +#ifndef GGML_CUDA_FLASHRT_FA4 + GGML_UNUSED(fa); GGML_UNUSED(ctx); + return false; +#else + static const bool disabled = getenv("GGML_FLASHRT_NO_PREFILL_FA4") != nullptr; + if (disabled) { + return false; + } + const ggml_tensor * q = fa->src[0]; + const ggml_tensor * k = fa->src[1]; + const ggml_tensor * v = fa->src[2]; + const ggml_tensor * mask = fa->src[3]; + if (q == nullptr || k == nullptr || v == nullptr || mask == nullptr || + fa->src[4] != nullptr) { + return false; + } + const int64_t hd = q->ne[0]; + const int64_t S = q->ne[1]; + const int64_t H = q->ne[2]; + if (hd != 256 || S < 64 || H < 2 || q->ne[3] != 1 || q->type != GGML_TYPE_F32) { + return false; + } + // Q: permuted view over the dense (S, H, D) f32 buffer + if (q->nb[0] != sizeof(float) || + (int64_t) q->nb[2] != hd * (int64_t) sizeof(float) || + (int64_t) q->nb[1] != hd * H * (int64_t) sizeof(float)) { + return false; + } + // K/V: one head of contiguous f16 token rows, padded to a multiple of + // 256 covering exactly S (the pi0.5 prefill padding scheme) + const int64_t SK = k->ne[1]; + if (SK < S || SK % 256 != 0 || SK - S >= 256) { + return false; + } + for (const ggml_tensor * kv : { k, v }) { + if (kv->type != GGML_TYPE_F16 || kv->ne[0] != hd || kv->ne[1] != SK || + kv->ne[2] != 1 || kv->ne[3] != 1 || + kv->nb[0] != sizeof(uint16_t) || + (int64_t) kv->nb[1] != hd * (int64_t) sizeof(uint16_t)) { + return false; + } + } + if (mask->type != GGML_TYPE_F16 || mask->ne[0] < SK || mask->ne[1] < S) { + return false; + } + // dst: contiguous [hd, H, S] f32 — the same dense linear layout + if (fa->type != GGML_TYPE_F32 || fa->ne[0] != hd || fa->ne[1] != H || + fa->ne[2] != S || fa->ne[3] != 1 || fa->nb[0] != sizeof(float) || + (int64_t) fa->nb[1] != hd * (int64_t) sizeof(float) || + (int64_t) fa->nb[2] != hd * H * (int64_t) sizeof(float)) { + return false; + } + float max_bias, softcap; + memcpy(&max_bias, (const float *) fa->op_params + 1, sizeof(float)); + memcpy(&softcap, (const float *) fa->op_params + 2, sizeof(float)); + if (max_bias != 0.0f || softcap != 0.0f) { + return false; + } + return ggml_cuda_flashrt::fa4_vit_ensure_loaded(ctx.stream()) == 0; +#endif // GGML_CUDA_FLASHRT_FA4 +} + +void ggml_cuda_flashrt_prefill_fa4(ggml_backend_cuda_context & ctx, ggml_tensor * fa) { +#ifndef GGML_CUDA_FLASHRT_FA4 + GGML_UNUSED(ctx); GGML_UNUSED(fa); + GGML_ABORT("flashrt: FA4 prefill attention not built"); +#else + const ggml_tensor * q = fa->src[0]; + const int S = (int) q->ne[1]; + const int H = (int) q->ne[2]; + const int D = (int) q->ne[0]; + float scale; + memcpy(&scale, (const float *) fa->op_params + 0, sizeof(float)); + + const int64_t n = (int64_t) S * H * D; + ggml_cuda_pool_alloc q16(ctx.pool(), n * sizeof(uint16_t)); + ggml_cuda_pool_alloc o16(ctx.pool(), n * sizeof(uint16_t)); + + const int rc = ggml_cuda_flashrt::fa4_prefill_attention( + (const float *) q->data, fa->src[1]->data, fa->src[2]->data, + (float *) fa->data, q16.get(), o16.get(), + S, H, D, scale, ctx.stream()); + if (rc != 0) { + GGML_ABORT("flashrt: FA4 prefill attention failed (S=%d H=%d rc=%d)", S, H, rc); + } +#endif // GGML_CUDA_FLASHRT_FA4 +} + // ── Gemma-style norm fold: {RMS_NORM, MUL(w), ADD(mul, norm)} ──────────────── // out = rms_norm(x)*w + rms_norm(x) == rms_norm(x)*(1 + w): the adaLN // modulate kernel with scale = w and shift = 0. ggml's own fused rms_norm diff --git a/flash_rt/structures/adapters/ggml/fr_fa4_vit.cu b/flash_rt/structures/adapters/ggml/fr_fa4_vit.cu index 6e4bcdf4..9318b61d 100644 --- a/flash_rt/structures/adapters/ggml/fr_fa4_vit.cu +++ b/flash_rt/structures/adapters/ggml/fr_fa4_vit.cu @@ -15,6 +15,7 @@ #include "fr_kernels.h" #include "fa4_aot/fa4_siglip_fwd.h" +#include "fa4_aot/fa4_prefill_fwd.h" #include @@ -56,6 +57,9 @@ __global__ void kernel_f16_to_f32_depad(const __half * __restrict__ src, fa4_siglip_fwd_Kernel_Module_t g_fa4_module; bool g_fa4_loaded = false; +fa4_prefill_fwd_Kernel_Module_t g_fa4p_module; +bool g_fa4p_loaded = false; + } // namespace // Loads the AOT module once; must not run during CUDA graph capture (the @@ -70,14 +74,67 @@ int fa4_vit_ensure_loaded(cudaStream_t stream) { return -1; } fa4_siglip_fwd_Kernel_Module_Load(&g_fa4_module); - const cudaError_t e = cudaGetLastError(); + cudaError_t e = cudaGetLastError(); if (e != cudaSuccess) { return -static_cast(e); } g_fa4_loaded = true; + fa4_prefill_fwd_Kernel_Module_Load(&g_fa4p_module); + e = cudaGetLastError(); + if (e != cudaSuccess) { + return -static_cast(e); + } + g_fa4p_loaded = true; return 0; } +// Full (non-causal) self-attention for the pi0.5 prefill: hd-256 GQA FA4. +// q_f32/dst_f32 dense (1, S, H, D); k16/v16 are the first S contiguous +// [D]-rows of the (possibly padded) f16 KV buffers, one KV head. +// The padded tail rows are simply outside the dynamic shape, which is +// exactly the graph's row-uniform pad mask. +int fa4_prefill_attention(const float * q_f32, const void * k16, const void * v16, + float * dst_f32, void * q16_ws, void * o16_ws, + int S, int H, int D, float scale, + cudaStream_t stream) { + if (!g_fa4p_loaded) { + return -1; + } + const int64_t n = (int64_t) S * H * D; + const int threads = 256; + const int64_t blocks = (n + threads - 1) / threads; + + kernel_f32_to_f16_dense<<<(unsigned) blocks, threads, 0, stream>>>( + q_f32, (__half *) q16_ws, n); + + auto fill_q = [&](void * data, auto * t, int heads) { + t->data = data; + t->dynamic_shapes[0] = 1; + t->dynamic_shapes[1] = S; + t->dynamic_shapes[2] = heads; + t->dynamic_shapes[3] = D; + t->dynamic_strides[0] = (int64_t) S * heads * D; + t->dynamic_strides[1] = (int64_t) heads * D; + t->dynamic_strides[2] = D; + }; + fa4_prefill_fwd_Tensor_mQ_t tq; fill_q(q16_ws, &tq, H); + fa4_prefill_fwd_Tensor_mK_t tk; fill_q(const_cast(k16), &tk, 1); + fa4_prefill_fwd_Tensor_mV_t tv; fill_q(const_cast(v16), &tv, 1); + fa4_prefill_fwd_Tensor_mO_t to; fill_q(o16_ws, &to, H); + + const int32_t rc = cute_dsl_fa4_prefill_fwd_wrapper( + &g_fa4p_module, &tq, &tk, &tv, &to, scale, stream); + if (rc != 0) { + return -1000 - rc; + } + + kernel_f16_to_f32_dense<<<(unsigned) blocks, threads, 0, stream>>>( + (const __half *) o16_ws, dst_f32, n); + + const cudaError_t e2 = cudaGetLastError(); + return (e2 == cudaSuccess) ? 0 : -static_cast(e2); +} + // q_f32: dense (B,S,H,D); k16/v16: dense f16 same layout. When d_out == D // dst_f32 is the dense padded layout; when d_out < D the head padding is // dropped and dst_f32 is the packed [(H*d_out), S, B] contiguous tensor. diff --git a/flash_rt/structures/adapters/ggml/fr_ggml.cuh b/flash_rt/structures/adapters/ggml/fr_ggml.cuh index 9b015bba..38b19630 100644 --- a/flash_rt/structures/adapters/ggml/fr_ggml.cuh +++ b/flash_rt/structures/adapters/ggml/fr_ggml.cuh @@ -127,6 +127,13 @@ bool ggml_cuda_flashrt_should_fuse_vit_fa4_depad(const ggml_tensor * fa, const g void ggml_cuda_flashrt_vit_fa4_depad(ggml_backend_cuda_context & ctx, ggml_tensor * fa, const ggml_tensor * view, ggml_tensor * cont); +// pi0.5 prefill self-attention (head_dim 256, GQA 1 KV head, full +// attention with a row-uniform pad mask) through the second AOT FA4 +// module. The window assumes the prefix-LM mask semantics of the pi0.5 +// prefill graph (see the dispatch-side checks). +bool ggml_cuda_flashrt_should_fuse_prefill_fa4(const ggml_tensor * fa, ggml_backend_cuda_context & ctx); +void ggml_cuda_flashrt_prefill_fa4(ggml_backend_cuda_context & ctx, ggml_tensor * fa); + // Prefill fused QKV window: q mm->reshape->rope->scale, k mm->reshape->rope-> // pad, v mm->reshape->pad, each pad permuted+copied into a padded f16 tensor // of token rows. One fused GEMM + qkv_post + pad-row zeroing. diff --git a/flash_rt/structures/adapters/ggml/fr_kernels.h b/flash_rt/structures/adapters/ggml/fr_kernels.h index 2fbb1460..0ed45a11 100644 --- a/flash_rt/structures/adapters/ggml/fr_kernels.h +++ b/flash_rt/structures/adapters/ggml/fr_kernels.h @@ -168,6 +168,15 @@ int fa4_vit_attention(const float * q_f32, const void * k16, const void * v16, int B, int S, int H, int D, float scale, cudaStream_t stream); +// Full (non-causal) self-attention for the pi0.5 prefill via a second AOT +// FA4 module (head_dim 256, GQA with one KV head). k16/v16 are the first S +// contiguous rows of the padded f16 KV buffers; the pad rows lie outside +// the dynamic shape, which reproduces the row-uniform pad mask exactly. +int fa4_prefill_attention(const float * q_f32, const void * k16, const void * v16, + float * dst_f32, void * q16_ws, void * o16_ws, + int S, int H, int D, float scale, + cudaStream_t stream); + // Batched f32->f16 row copies: for each pair p, dst[p][r*hd + i] = // (half) src[p][r*hd + i] over n_rows rows of hd elements. One launch // replaces up to FR_CPY_ROWS_MAX individual copy kernels (the persistent diff --git a/flash_rt/structures/adapters/ggml/qualification/goldens/pi05_thor_action.json b/flash_rt/structures/adapters/ggml/qualification/goldens/pi05_thor_action.json index bfb31079..87e37c05 100644 --- a/flash_rt/structures/adapters/ggml/qualification/goldens/pi05_thor_action.json +++ b/flash_rt/structures/adapters/ggml/qualification/goldens/pi05_thor_action.json @@ -1 +1 @@ -{"action_final_raw": [[0.07204130291938782, -0.11350921541452408, 0.08429346978664398, -0.3656696081161499, -0.07903286069631577, 0.05143028125166893, -0.027163250371813774, 0.010509899817407131, 0.016304291784763336, 0.007965324446558952, -0.002198867965489626, 0.009940089657902718, 0.004432466812431812, 0.014802207238972187, 0.005763123743236065, 0.008698852732777596, -0.004932646173983812, -0.00758181419223547, 0.007238580379635096, 0.012351863086223602, 0.0005422856193035841, -0.004117514938116074, 0.004125963430851698, -0.009748348034918308, -0.004717582371085882, -0.004324060864746571, -0.003718649037182331, -0.008062406443059444, 0.0022765814792364836, 0.003958307206630707, -0.005197297316044569, -0.0011443446855992079], [0.10125378519296646, -0.12873898446559906, 0.0758739486336708, -0.42113062739372253, -0.10985685139894485, 0.05189714580774307, -0.026942923665046692, -0.004111709538847208, 0.006781688425689936, 0.005005198530852795, -0.00799341220408678, 0.0010191804030910134, -0.011189727112650871, 0.0263466015458107, 0.00395270437002182, -9.96968665276654e-05, -0.020618770271539688, 0.008116556331515312, 0.016291717067360878, 0.010052606463432312, 0.003730165073648095, 0.007512654177844524, 0.006219280883669853, -0.009574771858751774, -0.007001390680670738, 0.005158360581845045, -0.0023672597017139196, -0.007642071228474379, 0.006794589105993509, -0.005577349103987217, 0.000534163205884397, 0.003758528968319297], [0.09115591645240784, -0.11993689090013504, 0.08690856397151947, -0.46369683742523193, -0.11741956323385239, 0.05149669945240021, -0.017574412748217583, 0.0056210667826235294, 0.006649073213338852, 0.0009659163770265877, 0.0002748323604464531, 0.013001701794564724, 0.020357340574264526, 0.02157861739397049, 0.0017587818438187242, -0.009329949505627155, -0.01415527518838644, -0.004996980540454388, 0.014225887134671211, 0.00044403059291653335, -0.008308613672852516, -0.00733127212151885, 0.0013179633533582091, -0.002029938856139779, -0.004320304840803146, -0.004838001914322376, -0.003330306615680456, 0.00011348118277965114, 0.00475540105253458, 0.011434276588261127, 0.0026987316086888313, 0.006854772102087736], [0.11330023407936096, -0.11783881485462189, 0.0926673635840416, -0.5157063603401184, -0.127365842461586, 0.056598976254463196, -0.01100153662264347, 0.03379752114415169, 0.001299548428505659, 0.014514102600514889, 0.005873256362974644, 0.0022408769000321627, 0.011131459847092628, 0.018787482753396034, 0.0014650344382971525, -0.006809557788074017, -0.007111363112926483, -0.011379110626876354, 0.006421738304197788, 0.010308551602065563, -0.005321051925420761, -0.004222462885081768, 0.004251003731042147, -0.007255891337990761, 0.0031188130378723145, 0.010692982003092766, 0.004688961431384087, 0.0009694810723885894, 0.0032925044652074575, 0.008236529305577278, -0.0013498385669663548, 0.012010093778371811], [0.13430054485797882, -0.1269943118095398, 0.10221230983734131, -0.5629380941390991, -0.13885940611362457, 0.05582033470273018, -0.017639677971601486, 0.012164678424596786, 0.002125837840139866, 0.01052339281886816, -0.0034445286728441715, -0.002667462918907404, 0.012460893951356411, 0.02905707061290741, 0.0017453432083129883, 0.002192025538533926, -0.019024087116122246, -0.012002582661807537, 0.015994884073734283, 0.014112966135144234, 0.005932504311203957, 0.0005032385233789682, -0.0015208425465971231, -0.0006949722883291543, -0.00525035010650754, -0.004735289141535759, -0.0011483444832265377, -0.009948208928108215, 0.008242184296250343, -0.0022513302974402905, -0.0005639559822157025, -0.00038196842069737613], [0.1572314202785492, -0.11822830140590668, 0.09592665731906891, -0.5975432991981506, -0.1441642940044403, 0.06988880783319473, -0.01921856217086315, 0.015498095192015171, 0.01608978398144245, 0.016082610934972763, -0.006816709414124489, 0.014671353623270988, 0.009462973102927208, 0.020529944449663162, 0.003971511032432318, -0.008311562240123749, -0.017955025658011436, -0.0019077907782047987, 0.014017481356859207, 0.018251800909638405, -0.008570718578994274, 0.004647897556424141, -0.001230914844200015, 0.005506758578121662, -0.0036311494186520576, 0.0026508404407650232, -0.003659192007035017, -0.004755430854856968, -0.006892980542033911, -0.008419613353908062, -0.002728990977630019, 0.003199429716914892], [0.1634657084941864, -0.10547913610935211, 0.09950520843267441, -0.6124517321586609, -0.15696431696414948, 0.08715098351240158, -0.03141110762953758, 0.013191832229495049, 0.004589242395013571, 0.000438198127085343, 0.0008151709917001426, 0.008371998555958271, 0.011076774448156357, 0.021108262240886688, -0.005683267489075661, 1.3056736634098343e-06, -0.020556550472974777, 0.002646013628691435, 0.013472237624228, 0.012461056001484394, 0.006658147554844618, 0.008077976293861866, -0.0040149628184735775, 0.00951587688177824, 0.0076101962476968765, 0.004690102767199278, -0.008010146208107471, -0.01499431487172842, -0.014799260534346104, 0.010328761301934719, -0.008833601139485836, -0.013493570499122143], [0.17962393164634705, -0.12388500571250916, 0.09577580541372299, -0.6380575299263, -0.1597168743610382, 0.07016991823911667, -0.015582311898469925, 0.006572971120476723, 0.008294407278299332, -0.0011156095424667, 0.002037588506937027, 0.015060609206557274, 0.009366026148200035, 0.012784049846231937, 0.006767261307686567, -0.007399061694741249, -0.009957707487046719, -0.008602148853242397, 0.015372874215245247, 0.0023656415287405252, -0.0018184289801865816, 0.0012300567468628287, 0.007301063276827335, -0.005210488103330135, -0.005356528330594301, 0.01048555038869381, -0.009502486325800419, -0.01623319461941719, -0.0026356331072747707, -0.007185563910752535, -0.0012311009922996163, 0.006323504261672497], [0.1977672427892685, -0.11398372054100037, 0.09678585082292557, -0.668890655040741, -0.1637994796037674, 0.08524077385663986, -0.015174043364822865, 0.006234307307749987, 0.004931071773171425, 0.010312230326235294, 0.0005076512461528182, 0.00603244174271822, 0.02166261337697506, 0.023806912824511528, -0.005028861574828625, 0.006848940160125494, -0.0202220156788826, -6.371434574248269e-05, 0.006742647383362055, 0.011141316033899784, -0.003976040054112673, -0.0022567883133888245, -0.013189264573156834, -0.008469528518617153, -0.013478616252541542, 0.01167785469442606, -0.00862216018140316, 0.010740133933722973, 0.0010758911957964301, 0.002641480416059494, 0.008903530426323414, -0.002815911313518882], [0.19058360159397125, -0.11881575733423233, 0.1211477667093277, -0.6842043399810791, -0.1674933135509491, 0.0675734281539917, -0.02280472218990326, 0.011704090982675552, 0.014619171619415283, 0.009397161193192005, -0.002172892913222313, 0.011309277266263962, -0.00787555705755949, 0.023229902610182762, 0.0019613588228821754, 0.0010314203100278974, 0.007905149832367897, -0.004807105287909508, 0.01077890582382679, 0.016402581706643105, 0.0011862809769809246, 0.0012321020476520061, -0.009161261841654778, 0.0018737574573606253, -0.002514938125386834, -0.004140396602451801, -0.011619646102190018, -0.018781667575240135, -0.009584026411175728, 0.004854136612266302, -0.0069736698642373085, -0.005881227552890778]], "action_steps": 10, "action_dim": 32} \ No newline at end of file +{"action_final_raw": [[0.09624627977609634, -0.1961977779865265, 0.06703978776931763, -0.5476640462875366, -0.14697180688381195, 0.07153154909610748, -0.002638055244460702, 0.013559071347117424, 0.016760675236582756, 0.007358239963650703, -0.002592590870335698, 0.002914540935307741, 0.006588217802345753, 0.02381635643541813, 0.000132353205117397, 0.004709492437541485, -0.00813357625156641, -0.013270657509565353, 0.009339207783341408, 0.020445771515369415, 0.003282379824668169, -0.002165404614061117, 0.002376250457018614, -0.0039012939669191837, 0.0014441026141867042, 0.0002516416134312749, 0.003512145485728979, -0.007321126293390989, 0.0017400443321093917, 0.004314625635743141, -0.00405048206448555, -0.005579052492976189], [0.12092496454715729, -0.1998528391122818, 0.054504960775375366, -0.5915311574935913, -0.1741236299276352, 0.06810212880373001, -0.003277003997936845, -0.0034880663733929396, 0.003501307452097535, 0.005087920930236578, -0.0022917466703802347, 0.007821531035006046, -0.006648858077824116, 0.027163516730070114, -0.0011810704600065947, -0.00011856241326313466, -0.01714256964623928, 0.004868022631853819, 0.007669864222407341, 0.0008107393514364958, 0.0073386593721807, -0.0016696092206984758, -0.00048284963122569025, -0.009979461319744587, -0.01471013855189085, 0.0012931901728734374, -0.002198341768234968, -0.01144750788807869, 0.010050974786281586, 0.005830400623381138, -0.007881347090005875, 0.0015526266070082784], [0.1221553236246109, -0.20009484887123108, 0.07306937873363495, -0.6259077787399292, -0.17243315279483795, 0.07382355630397797, 0.01399582251906395, 0.008802780881524086, 0.007673643529415131, -0.003587680170312524, -0.00198595249094069, 0.009418884292244911, 0.01879180409014225, 0.02367384359240532, -0.002254959661513567, -0.0034092110581696033, -0.015890508890151978, -0.012552782893180847, 0.010030963458120823, 0.004080003593116999, -0.006269955076277256, -0.004506299272179604, 0.005475796293467283, -0.003492268966510892, -0.005690732505172491, -0.0047112093307077885, -0.007371499668806791, -0.0032012038864195347, 0.00254984968341887, 0.0003990198310930282, 0.006445983424782753, 0.009539195336401463], [0.14189547300338745, -0.20173774659633636, 0.06596016138792038, -0.6811384558677673, -0.1881892830133438, 0.08414757251739502, 0.0160377100110054, 0.02549123391509056, 0.005759424064308405, 0.005022574216127396, 0.014015900902450085, -0.0007282908773049712, 0.010810497216880322, 0.017419779673218727, -0.0012818204704672098, -0.013671827502548695, -0.009128696285188198, -0.002643367275595665, 0.01019663829356432, 0.010841459967195988, -0.002366367494687438, -0.0019233895000070333, 0.0014648950891569257, 0.004127361811697483, 0.0019628419540822506, 0.005564515478909016, -0.004168566316366196, 0.003304382786154747, -0.0038799545727670193, 0.0023837818298488855, -0.008190134540200233, 0.009892459958791733], [0.1522434502840042, -0.20274382829666138, 0.08215358853340149, -0.7039123177528381, -0.19095152616500854, 0.07766478508710861, -0.01269338559359312, 0.00959023181349039, -0.002931674476712942, 0.016207940876483917, -0.009432277642190456, 0.0004770905652549118, 0.004761648364365101, 0.034024689346551895, -0.008643164299428463, 0.0033901091665029526, -0.020978260785341263, -0.010628817602992058, 0.02480119839310646, 0.00730505958199501, 0.0014040278038010001, -0.0018820121185854077, 0.003079682355746627, 0.0030281723011285067, 0.0009533020784147084, 0.0014496227959170938, -0.004266194999217987, -0.007991557009518147, 0.007314159069210291, -0.005737284664064646, -0.0032646663021296263, -0.0032309989910572767], [0.18085116147994995, -0.20060132443904877, 0.06863894313573837, -0.7330130338668823, -0.1915489137172699, 0.09672202914953232, 0.01540116872638464, 0.007338434923440218, 0.017650838941335678, 0.01925886608660221, -0.008773401379585266, 0.013958729803562164, 0.01029625441879034, 0.02788020297884941, -0.0022923294454813004, -0.013784453272819519, -0.016574576497077942, -0.004523152951151133, 0.01866416074335575, 0.013041609898209572, -0.001756405341438949, 0.007274901028722525, -0.0009395781089551747, 0.0024610029067844152, 0.0031260414980351925, 0.003031071275472641, -0.006058946251869202, -0.006632425356656313, -0.0009360113763250411, -0.00719192810356617, -0.001834192662499845, 0.0030427579768002033], [0.18593062460422516, -0.19550056755542755, 0.07160288095474243, -0.7510550022125244, -0.19466954469680786, 0.10721323639154434, -0.013059914112091064, 0.008966893889009953, 0.0030186132062226534, 0.0019175108755007386, 0.0020664960611611605, -0.00020583205332513899, 0.013402078300714493, 0.02108699642121792, -0.002505762968212366, 0.007308184169232845, -0.028859110549092293, -0.005017835181206465, 0.01019445899873972, 0.012598940171301365, -0.004889514297246933, -0.002243961440399289, -0.007915007881820202, 0.002988256746903062, 0.008123243227601051, 0.0008216212736442685, -0.008317344821989536, -0.013078191317617893, -0.01282085757702589, 0.007809976581484079, -0.005311104469001293, -0.0201072059571743], [0.207765594124794, -0.21467824280261993, 0.06326164305210114, -0.7796165943145752, -0.19674517214298248, 0.10123662650585175, 0.00892910547554493, 0.009866893291473389, 0.0011979155242443085, -0.005581721663475037, -0.00986845325678587, -0.00014581959112547338, 0.0097585991024971, 0.0157171580940485, 0.00469200499355793, -0.009907061234116554, -0.01478598453104496, -0.0044782706536352634, 0.015776481479406357, 0.004031747113913298, 0.0027923069428652525, 0.00022482436907012016, 0.0036739581264555454, -0.014940707944333553, -0.009240969084203243, 0.012544935569167137, 0.000710038875695318, -0.0036881084088236094, -0.0074166469275951385, -0.002954066963866353, 0.004561707377433777, 0.00133541040122509], [0.21232953667640686, -0.1888059377670288, 0.06611032038927078, -0.7755351066589355, -0.2044234275817871, 0.10427963733673096, -0.014994261786341667, 0.018358377739787102, -0.0003517218283377588, 0.0059860870242118835, -0.0068831369280815125, 0.003228644607588649, 0.014275294728577137, 0.018018964678049088, 0.0024007963947951794, -0.004222281742841005, -0.025340044870972633, 0.006304721813648939, 0.017566384747624397, 0.0018504681065678596, -0.005991443060338497, -0.0029909336008131504, -0.0013041617348790169, 8.168922795448452e-05, -0.008946222253143787, 0.018412627279758453, -0.003031445201486349, 0.0029337876476347446, 0.0039048572070896626, 0.0065581281669437885, 0.01261038240045309, -0.007260486483573914], [0.21789924800395966, -0.1944425106048584, 0.0784466341137886, -0.8004544973373413, -0.20648646354675293, 0.10525409877300262, -0.012714697048068047, 0.01368754357099533, 0.010657607577741146, 0.009618283249437809, -0.008845558390021324, 0.007256275042891502, -0.011666398495435715, 0.020875461399555206, 0.014044287614524364, 0.0017091069603338838, 0.003284646663814783, -0.004290799144655466, 0.016678009182214737, 0.004360348451882601, 0.006027561146765947, -0.009836606681346893, -0.009942407719790936, 0.001068001496605575, -0.002853973535820842, -0.006787698715925217, -0.012501326389610767, -0.022064579650759697, -0.004612510558217764, 0.007948911748826504, -0.0033561980817466974, -0.008420849218964577]], "action_steps": 10, "action_dim": 32} \ No newline at end of file From 0de6c4afc2eb57938ece8471bdc17c689c4741b2 Mon Sep 17 00:00:00 2001 From: LiangSu8899 Date: Tue, 25 Aug 2026 04:34:05 -0400 Subject: [PATCH 19/32] docs(ggml-adapter): usage, testing, and development guides Documents building a host against the adapter, the runtime switch matrix, the three validation layers (operator tests, qualification gates, benchmark/parity methodology), the layer architecture with its capture-safety and fusion-window invariants, and the AOT FA4 module mechanics. Adds measured Thor numbers to the overview. --- .../structures/adapters/ggml/DEVELOPMENT.md | 113 ++++++++++++++++++ flash_rt/structures/adapters/ggml/README.md | 84 +++++++++---- flash_rt/structures/adapters/ggml/TESTING.md | 76 ++++++++++++ flash_rt/structures/adapters/ggml/USAGE.md | 77 ++++++++++++ 4 files changed, 328 insertions(+), 22 deletions(-) create mode 100644 flash_rt/structures/adapters/ggml/DEVELOPMENT.md create mode 100644 flash_rt/structures/adapters/ggml/TESTING.md create mode 100644 flash_rt/structures/adapters/ggml/USAGE.md diff --git a/flash_rt/structures/adapters/ggml/DEVELOPMENT.md b/flash_rt/structures/adapters/ggml/DEVELOPMENT.md new file mode 100644 index 00000000..0c5f4742 --- /dev/null +++ b/flash_rt/structures/adapters/ggml/DEVELOPMENT.md @@ -0,0 +1,113 @@ +# Development guide + +## Architecture + +The adapter has two halves with a hard boundary: + +- **Framework-free half** (`fr_repack.cu`, `fr_quant_act.cu`, + `fr_qkv_post.cu`, `fr_ada.cu`, `fr_decode_attn.cu`, `fr_fa4_vit.cu`): + plain CUDA translation units, entry points declared in `fr_kernels.h` + (raw pointers + `cudaStream_t`, no ggml or CUTLASS types in the + header). GEMMs with fused epilogues live in `csrc/gemm/fp4/` and are + compiled alongside. +- **ggml-facing half** (`fr_ggml.cuh`, `fr_dispatch.cu`): window + predicates (`ggml_cuda_flashrt_should_fuse_*`) and executors that speak + `ggml_tensor`, plus the caches. The host's `ggml-cuda.cu` calls the + predicates from its fuse hook; that call-site code lives in the host + tree, not here. + +Keep the boundary: nothing under `csrc/` or in the framework-free half +may include ggml headers, and `fr_kernels.h` must stay consumable from a +plain C++ translation unit. + +## Caches (all capture-safe) + +- **Weight repack cache** — keyed by weight data pointer, never evicted + (weights are immortal in a loaded model). ggml's split-nibble NVFP4 + blocks are repacked to the CUTLASS wire format (adjacent-pair nibbles, + scale bytes in the Sm1xx atom layout) on first use. +- **Per-evaluation activation cache** — a producer (e.g. the fused adaLN) + can register its already-quantized output; later GEMMs in the same + evaluation reuse it. Keyed by tensor pointer + an evaluation counter so + recycled addresses can never alias. Slots are grow-only so device + addresses stay stable for captured CUDA graphs. +- **One-shot handoffs** (f16 Q from the QKV window to the decode + attention) — single grow-only slot, key cleared on consumption. +- **CUTLASS workspaces** — shape-keyed, grown only outside capture. + +Rule for all of them: no allocation while a CUDA graph is being captured. +Check `cudaStreamIsCapturing` and fall back to the unfused path (or pool +memory) when growth would be needed mid-capture. + +## Adding a fusion window + +1. Express the executor in the framework-free half with a C entry point + in `fr_kernels.h`; consume existing `csrc` GEMMs where possible. +2. Add the predicate/executor pair to `fr_ggml.cuh` / `fr_dispatch.cu`. + The predicate must pin every assumption the kernel makes: dtypes, + shapes, strides (element-exact, not just "contiguous"), op params + (`max_bias`, softcap), and use counts where the window elides + intermediates. +3. Add the call site to the host's fuse hook, and a + `GGML_FLASHRT_NO_` switch in the predicate. +4. Validate per TESTING.md (trigger proof, A/B/A, parity or judge). + +Invariants and pitfalls learned the hard way: + +- **The fuse hook's return contract**: returning 0 means "not fused" and + the anchor node executes normally afterwards — a window that replaces a + single node must also consume the pure-view node that follows it and + return ≥1, or its work is silently overwritten (symptom: identical + results, slower). +- **Overlap checks are allocator-sensitive.** The generic fusion memory + range check vetoes a window when the destination aliases an + outside-window source. The allocator legitimately hands a window's + output the block of an input that dies inside the window; whether that + alias is safe depends on the fused implementation's read-before-write + order, so exemptions are per-window and must be argued in a comment + (see the GeGLU window: the activation is fully consumed by the quantize + kernel before the down GEMM writes). Any change that shifts allocation + (new nodes, another sched) can re-trigger vetoes elsewhere — symptom is + a silent GPU-time regression; diagnose with a kernel census diff. +- **Numeric equivalences must be argued or measured**, e.g. an epilogue + that converts the fp32 accumulator to f16 once is bit-equal to f32 + output plus a separate cast; a fused kernel writing the same values + through the same conversion is bit-identical to the copy chain it + replaces. Anything weaker goes through the real-observation judge. +- **RoPE in `fr_qkv_post.cu` mirrors ggml's `rope_neox`** (yarn + corrections included) and must stay bit-exact with it; the predicate + rejects non-NEOX modes. +- Windows only ever fire on `cc == 1100` (checked at the call site). + +## AOT FlashAttention-4 modules + +`fa4_aot/` holds ahead-of-time exports of the vendored FA4 forward +(vision shape: padded head_dim 80, MHA; prefill shape: head_dim 256, GQA +with one KV head). Regeneration and the export mechanics are documented +in `fa4_aot/README.md`; the short version: + +- CuTe-DSL's `export_to_c` emits a C header (host launch entry, tensor + argument structs, embedded cubin) plus a host object. The tvm-ffi + compile variant only exports a TVM ABI, so the export script strips + `--enable-tvm-ffi` and never executes the resulting object in-process + (its calling convention differs). +- `fr_fa4_shims.c` supplies the small `_cuda*` runtime aliases the object + expects, so neither the build nor the runtime depends on any CuTe-DSL + library. +- Module loading must happen outside CUDA graph capture; the adapter + preloads from `ggml_cuda_flashrt_begin_eval`, which always runs before + a capture can begin. +- The wrapper takes dynamic shapes/strides per tensor, so one export per + (head_dim, GQA config) covers all sequence lengths. The prefill + window's mask handling relies on the pi0.5 prefix-LM property that the + mask is row-uniform pad-only and the real KV length equals the query + count; the padded tail is excluded by the dynamic shape instead of by + a mask. + +## Single-source rule + +Structure changes (GEMM tiles, epilogues, attention decompositions) +belong in `csrc/` or the structures catalog so every host adapter +inherits them; this directory only translates. Nothing here may be +copy-pasted into a host tree, and the host integration must stay behind +its own opt-in build flag so stock builds are unaffected. diff --git a/flash_rt/structures/adapters/ggml/README.md b/flash_rt/structures/adapters/ggml/README.md index 84aff171..e9dd53a1 100644 --- a/flash_rt/structures/adapters/ggml/README.md +++ b/flash_rt/structures/adapters/ggml/README.md @@ -3,30 +3,70 @@ Native C++/CUDA host adapter that maps FlashRT structures onto ggml's CUDA backend (llama.cpp family), targeting Jetson AGX Thor (SM110). Unlike the Python runtime adapters (`vllm_engine.py`, `sglang_engine.py`), this adapter -is consumed at build time: the host's CMake compiles these translation units -inside its own build tree. +is consumed at **build time**: the host's CMake compiles these translation +units inside its own build tree and only C symbols cross the boundary. The +host never links against Python, PyTorch, or any FlashRT runtime. -Layout: +Documentation: + +- [USAGE.md](USAGE.md) — building a host against this adapter, runtime + switches, deployment notes. +- [TESTING.md](TESTING.md) — operator tests, the qualification gates, and + the benchmarking / parity methodology every change must pass. +- [DEVELOPMENT.md](DEVELOPMENT.md) — layer architecture, how to add a + fusion window, invariants and known pitfalls, AOT FlashAttention-4 + regeneration. + +## What it is + +The adapter is the third host of the `flash_rt/structures` catalog. The +same structures that the torch frontend and the vllm/sglang adapters +consume — block-scaled NVFP4 GEMMs with fused epilogues, fused +norm/modulation producers, the decomposed tiny-M decode attention, the +FlashAttention-4 forward — are mapped here onto ggml's graph executor +through pattern-matched subgraph windows. Heavy math is single-source: + +- **NVFP4 GEMMs** come from `csrc/gemm/fp4/` in this repository + (GeGLU-interleaved, SigLIP-FFN pair, bias/f16-out variants). Nothing is + vendored into the host. +- **FlashAttention-4** is the vendored CuTe-DSL forward under + `csrc/attention/flash_attn_4_src`, consumed as ahead-of-time compiled + modules (see `fa4_aot/`), so the host build needs no CuTe-DSL toolchain. +- The `fr_*.cu` files here are the translation layer only: wire-format + repack (ggml split-nibble NVFP4 → CUTLASS atom layout), activation + quantize, fused RoPE/norm producers, and the dispatch/caching half that + speaks `ggml_tensor`. + +## Layout - `fr_kernels.h` — pure C entry points (no ggml, no CUTLASS in the header). - `fr_gemm_f32out.cu`, `fr_ada.cu`, `fr_qkv_post.cu`, `fr_quant_act.cu`, - `fr_repack.cu` — framework-free CUDA kernels (NVFP4 wire format, repack, - fused norm/modulation/rope, activation quantize). + `fr_repack.cu`, `fr_decode_attn.cu`, `fr_fa4_vit.cu`, `fr_fa4_shims.c` — + framework-free CUDA translation units. - `fr_dispatch.cu`, `fr_ggml.cuh` — the ggml-facing half: subgraph window - matchers over `ggml_tensor` chains, weight/activation caches, and kernel - dispatch. Requires ggml-cuda's internal headers on the include path. -- GEMMs with fused epilogues are consumed from `csrc/gemm/fp4/` in this - repository (GeGLU interleaved, SigLIP FFN f32-boundary pair); nothing is - vendored. - -Host-side integration (fuse-hook call sites, graph construction changes, -build wiring) lives in the host tree and points its build at this directory. - -Qualification (`qualification/`): the pipeline binding -`bindings/jetson_pi_edge_pi05.yaml` maps the host's hot path onto catalog -structures under the complete-hot-path contract, and -`qualification/run_qualification.py` gates it — manifest validation and -structure-version pins offline, plus an opt-in on-device gate comparing the -steady-state action chunk against a stored golden (exact by default; the -adapter is bitwise deterministic across processes after warmup). A catalog -version bump or any numeric change in the fused windows turns a gate red. + predicates and executors over `ggml_tensor` chains, weight/activation + caches. Requires ggml-cuda's internal headers on the include path. +- `fa4_aot/` — AOT FlashAttention-4 modules (vision and prefill shapes) + plus their regeneration script and provenance notes. +- `qualification/` — the release gates (see TESTING.md). +- `../../bindings/jetson_pi_edge_pi05.yaml` — the pipeline binding that + maps the host's hot path onto catalog structures under the + complete-hot-path contract. + +## Measured performance (Jetson AGX Thor, pi0.5, 2 camera views) + +| metric | stock llama.cpp (BF16) | with this adapter (NVFP4) | +|---|---|---| +| `llama_encode` + `llama_decode` (host `total_ms`, P50 warm) | 202.7 ms | 35.5 ms (**5.7×**) | +| end-to-end action chunk (ViT + prefill + 10 denoise steps) | — | **42.5 ms** | +| phase split | — | ViT 6.7 + prefill 15.6 + decode 19.8 | + +For context, the FlashRT torch frontend runs the same checkpoint at +36.4 ms end-to-end on the same device; the remaining gap is dominated by +the host graph's fp32 activation dtype (the torch pipeline holds +activations in fp16). + +Numerics: the adapter is bitwise deterministic across processes after +warmup; changes are gated by an exact e2e action golden plus a +real-observation parity protocol against an f16 reference (see +TESTING.md). diff --git a/flash_rt/structures/adapters/ggml/TESTING.md b/flash_rt/structures/adapters/ggml/TESTING.md new file mode 100644 index 00000000..341a20ce --- /dev/null +++ b/flash_rt/structures/adapters/ggml/TESTING.md @@ -0,0 +1,76 @@ +# Testing + +Three layers of validation, from operator level to release gate. + +## 1. Operator tests (`test-backend-ops`) + +The host's `test-backend-ops` exercises the NVFP4 mul_mat path against the +CPU reference: + +```bash +GGML_CUDA_FLASHRT_NO_CACHE=1 ./build/bin/test-backend-ops test -o MUL_MAT +``` + +`GGML_CUDA_FLASHRT_NO_CACHE=1` is **required**: the weight repack cache is +keyed by tensor data pointer under the assumption that weights are +immortal, which holds for models but not for the test harness's rapidly +recycled tensors. NVFP4 mismatches up to ~2e-2 are inherent W4A4 +activation-quantization noise (upstream applies the same tolerance to +native FP4 backends), not failures. + +## 2. Qualification gates (`qualification/`) + +`qualification/run_qualification.py` gates a build the way a release +would: + +- **manifest** — the pipeline binding + (`bindings/jetson_pi_edge_pi05.yaml`) must map the host's complete hot + path onto catalog structures. +- **pins** — the structure versions the binding names must match the + catalog (`qualification/pins.yaml`). +- **e2e golden** (`--e2e`, on-device, needs a running server) — drives the + fixed synthetic-input protocol and compares the steady-state action + chunk against `qualification/goldens/pi05_thor_action.json` **exactly**. + The adapter is bitwise deterministic across processes after warmup, so + any bit difference is a real change. + +```bash +python qualification/run_qualification.py # offline gates +python qualification/run_qualification.py --e2e # + on-device golden +python qualification/run_qualification.py --e2e --update-golden +``` + +Take the golden only after at least two warm-up inferences (the first +inference after cold start differs from steady state) and only for +changes whose numerics were judged (below). + +## 3. Benchmark and parity methodology + +Every performance change must pass this protocol on device: + +- **Hot-regime A/B/A sandwich** — run the candidate, the fallback (via its + runtime switch), and the candidate again as three separate server + processes, ≥15 warm-up + ~20 measured inferences each, comparing P50. + Thor drifts ±1–3 ms across long sessions, so only same-session + back-to-back numbers are comparable; single measurements and + cross-session comparisons are not accepted. +- **Bitwise parity** — save the action chunk from each leg. A change that + claims numeric neutrality must be bit-identical to the previous + accepted state. Note the converse trap: bit-identical output *plus* + zero performance delta usually means the window never fired — verify + the window triggers (kernel census, `GGML_FLASHRT_DEBUG`) before + interpreting the A/B. +- **Real-observation judge** — for changes that move numerics, run a set + of real robot observations (gripper-active frames) through the NVFP4 + build and an f16-weights build of the same tree, and compare per-dim + cosine of the action chunks against the f16 reference. The distance to + the reference must not systematically regress. Any bit-level change in + the action path amplifies to ~2e-2 absolute wobble on final actions + through the 10 denoise steps, so raw action diffs are meaningless — + only the distance-to-reference comparison judges accuracy. +- **Kernel-level accounting** — attribute wins with an nsys census + (`GGML_CUDA_DISABLE_GRAPHS=1`, full-lifetime `-t cuda` trace with a + graceful server exit so buffers flush). CUDA-graph replays hide kernels + from the profiler, and profiling on Tegra inflates kernel times, so the + census attributes *where* time went while the non-profiled A/B decides + *whether* the change lands. diff --git a/flash_rt/structures/adapters/ggml/USAGE.md b/flash_rt/structures/adapters/ggml/USAGE.md new file mode 100644 index 00000000..c309ca58 --- /dev/null +++ b/flash_rt/structures/adapters/ggml/USAGE.md @@ -0,0 +1,77 @@ +# Usage + +## Building a host against the adapter + +The reference host is the Jetson-PI-Edge llama.cpp tree, which carries the +integration side (CMake wiring, fuse-hook call sites, pi0 graph changes) +on its FlashRT branch and consumes this repository as a submodule at +`ggml/src/ggml-cuda/flashrt/flashrt-public`: + +```bash +git clone --recursive -b feat/flashrt-thor-kernels +cd Jetson-PI-Edge +cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_FLASHRT=ON -DCMAKE_BUILD_TYPE=Release +cmake --build build --target llama-server -j +``` + +Options: + +- `GGML_CUDA_FLASHRT` (OFF by default) — enables the layer. Without it the + build is stock llama.cpp; every integration point is compiled out. +- `GGML_CUDA_FLASHRT_PUBLIC_DIR` — path to a FlashRT checkout, overriding + the submodule location. +- `GGML_CUDA_FLASHRT_CUTLASS_DIR` — CUTLASS override; defaults to + `third_party/cutlass` inside this repository. + +The adapter is built as a separate CMake OBJECT library with +`-arch=sm_110a` and CUTLASS headers; the rest of ggml-cuda compiles +unchanged. The AOT FlashAttention-4 windows enable automatically when the +`fa4_aot/*.o` artifacts are present (they are checked in; see +`fa4_aot/README.md` to regenerate). + +## Model preparation + +- LLM weights: quantize with the host's `llama-quantize` to the `NVFP4` + target (exposed by the FlashRT branch). Setting `GGML_NVFP4_MSE=1` + during quantization selects per-block scales by reconstruction-MSE + search instead of plain absmax (slower to quantize, more accurate). +- The mmproj (vision tower) is quantized to NVFP4 the same way. + +## Running + +```bash +PI_MODEL=pi05 ./build/bin/llama-server -m --mmproj \ + -ngl 99 --flash-attn on --port +``` + +The server exposes the host's action-chunk HTTP protocol (reset → images → +state → infer). Warm-up matters on Thor: latency reaches its steady state +after roughly 15 inferences. + +## Runtime switches + +All switches are environment variables; unset means enabled/default. + +| variable | effect | +|---|---| +| `GGML_CUDA_FLASHRT_DISABLE=1` | disable the whole layer at runtime (stock kernels) | +| `GGML_FLASHRT_NO_RMS_GEMMA=1` | disable the Gemma norm-chain window | +| `GGML_FLASHRT_NO_QKV_PREFILL=1` | disable the fused prefill QKV window | +| `GGML_FLASHRT_NO_DEC_ATTN=1` | disable the decomposed decode attention | +| `GGML_FLASHRT_NO_VIT_FA4=1` | disable the AOT FA4 vision attention | +| `GGML_FLASHRT_NO_PREFILL_FA4=1` | disable the AOT FA4 prefill attention | +| `GGML_FLASHRT_NO_KV_TAIL=1` | disable the batched persistent-KV tail copies | +| `GGML_FLASHRT_NO_VIS_F16=1` | disable the vision QKV window's direct f16 K/V outputs | +| `GGML_CUDA_FLASHRT_NO_CACHE=1` | disable the pointer-keyed weight repack cache (required for `test-backend-ops`, see TESTING.md) | +| `GGML_FLASHRT_DEBUG=1` | print window-match failure diagnostics | +| `GGML_FLASHRT_DUMP=` / `GGML_FLASHRT_DUMP_MAX=` | dump the first n evaluated graphs' node sequences | + +Host-side switches on the FlashRT branch (outside this repository): +`GGML_PI05_MOD_PRECOMP=0` disables the denoise-schedule modulation +precompute, `LLAMA_GRAPH_REUSE_DISABLE=1` disables graph reuse, +`GGML_CUDA_DISABLE_GRAPHS=1` disables CUDA graphs (useful for profiling: +kernels are invisible to nsys while CUDA graphs replay). + +Every window degrades gracefully: when its predicate does not match (or +its switch is set) the nodes run on stock ggml kernels, so the switches +bisect regressions window by window. From 34cf7c26045bf5eca898a096c32afe7a9475c4ed Mon Sep 17 00:00:00 2001 From: LiangSu8899 Date: Tue, 25 Aug 2026 04:37:43 -0400 Subject: [PATCH 20/32] build: track CUTLASS as a submodule The ggml adapter's build resolves CUTLASS from third_party/cutlass by default; pin it (NVIDIA/cutlass v4.4.2) as a submodule so a recursive clone is self-contained instead of relying on a local checkout. --- .gitignore | 1 - .gitmodules | 3 +++ third_party/cutlass | 1 + 3 files changed, 4 insertions(+), 1 deletion(-) create mode 100644 .gitmodules create mode 160000 third_party/cutlass diff --git a/.gitignore b/.gitignore index 3b475834..f4e696dc 100644 --- a/.gitignore +++ b/.gitignore @@ -43,4 +43,3 @@ training/_runs/** internal-tests/ internal-docs/ notes/ -third_party/cutlass diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 00000000..281cb2d8 --- /dev/null +++ b/.gitmodules @@ -0,0 +1,3 @@ +[submodule "third_party/cutlass"] + path = third_party/cutlass + url = https://github.com/NVIDIA/cutlass.git diff --git a/third_party/cutlass b/third_party/cutlass new file mode 160000 index 00000000..da5e086d --- /dev/null +++ b/third_party/cutlass @@ -0,0 +1 @@ +Subproject commit da5e086dab31d63815acafdac9a9c5893b1c69e2 From 3be46d8145606016b4e782a5f17e047f25bf6c7e Mon Sep 17 00:00:00 2001 From: LiangSu8899 <7thuniversels@gmail.com> Date: Tue, 25 Aug 2026 11:39:31 -0400 Subject: [PATCH 21/32] ggml adapter: add SM120/Qwen3.6-35B target (LLM decode window set) Second (arch, model-family) target for the native ggml adapter: fused-region NVFP4 W4A4 GEMVs (GDN in_proj, attn qkv, lm head), GDN cell span with M<=4 per-token state snapshots and checkpoint replay, format-native MoE expert span over ggml K-quant blocks (bit-exact q8_1 activation clone), out-proj / router / spec-draft-head windows. All launches join the host's PDL chain; verify batches up to M=4 are first-class (MTP speculative decode). Host integration is a CMake option (GGML_CUDA_FLASHRT_SM120 + GGML_CUDA_FLASHRT_PUBLIC_DIR) compiling this translation unit inside the host's ggml-cuda build, with all call sites ifdef-guarded. Binding and qualification gates recorded for the migration baseline. --- flash_rt/structures/adapters/ggml/README.md | 16 + flash_rt/structures/adapters/ggml/USAGE.md | 22 + .../adapters/ggml/fr_win_qwen36_sm120.cu | 2355 +++++++++++++++++ .../ggml/qualification/pins_qwen36_sm120.yaml | 23 + .../bindings/llamacpp_qwen36_35b_sm120.yaml | 86 + 5 files changed, 2502 insertions(+) create mode 100644 flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu create mode 100644 flash_rt/structures/adapters/ggml/qualification/pins_qwen36_sm120.yaml create mode 100644 flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml diff --git a/flash_rt/structures/adapters/ggml/README.md b/flash_rt/structures/adapters/ggml/README.md index e9dd53a1..1502c562 100644 --- a/flash_rt/structures/adapters/ggml/README.md +++ b/flash_rt/structures/adapters/ggml/README.md @@ -70,3 +70,19 @@ Numerics: the adapter is bitwise deterministic across processes after warmup; changes are gated by an exact e2e action golden plus a real-observation parity protocol against an f16 reference (see TESTING.md). + +## Second target: RTX 5090 (SM120) + Qwen3.6-35B-A3B + +`fr_win_qwen36_sm120.cu` carries the adapter's second (arch, model-family) +target: an LLM decode window set for the Qwen3.6 hybrid (GDN + attention, +256-expert MoE, MTP speculative decode) on SM120, consuming ggml's native +K-quant weights in place plus NVFP4 W4A4 fused regions. It follows the same +two-half discipline with one deliberate difference: its MoE/out-proj/router +kernels reproduce ggml's mmvq numerics through ggml's own `vec_dot_*_q8_1` +device functions (bit-exact q8_1 activation clone), so those kernels live in +the ggml-facing half by construction. Windows are M<=4 aware (speculative +verify batches) and carry the recurrent-state snapshot/checkpoint discipline +documented in DEVELOPMENT.md. + +Binding: `flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml`; +gates: `qualification/pins_qwen36_sm120.yaml`. diff --git a/flash_rt/structures/adapters/ggml/USAGE.md b/flash_rt/structures/adapters/ggml/USAGE.md index c309ca58..83ed2ec2 100644 --- a/flash_rt/structures/adapters/ggml/USAGE.md +++ b/flash_rt/structures/adapters/ggml/USAGE.md @@ -75,3 +75,25 @@ kernels are invisible to nsys while CUDA graphs replay). Every window degrades gracefully: when its predicate does not match (or its switch is set) the nodes run on stock ggml kernels, so the switches bisect regressions window by window. + +## SM120 / Qwen3.6 target + +Build a llama.cpp tree against this checkout: + +```bash +cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 \ + -DGGML_CUDA_FLASHRT_SM120=ON -DGGML_CUDA_FLASHRT_PUBLIC_DIR= \ + -DCMAKE_CUDA_FLAGS="-I/include -gencode=arch=compute_120a,code=sm_120a" +``` + +The target's windows are opt-in per mechanism (historic `FRT_*` switches; +unset = off). Safe-tier set: `FRT_INPROJ_SWAP=1 FRT_ATTNQKV_SWAP=1 +FRT_GDN_SWAP=1 FRT_MOEGLUE_SWAP=1 FRT_MOEFUSE_SWAP=1 FRT_MOEFUSE_SHEXP=1 +FRT_OUTNATIVE_SWAP=1 FRT_REGIONS_PACK=`; full tier adds +`FRT_HEAD_SWAP=1 FRT_HEAD_PACK=`. Speculative serving adds +`FRT_HEAD_DRAFT=1` (safe tier: FP4-serve only the draft's head copy) and the +host-side `LLAMA_GRAPH_SLOTS=6` + `--backend-sampling`. FP4 weights currently +come from side-band packs; replacing them with the in-process repack cache is +the next step for this target. Diagnostics: `FRT_STATS=1`, +`FRT_MOEFUSE_DBG=`, `FRT_MOEFUSE_SELFTEST=1`, `FRT_DUMP_GRAPH=1` + +`FRT_DUMP_M=` (+ `FRT_DUMP_PATH`). diff --git a/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu b/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu new file mode 100644 index 00000000..e6c4505d --- /dev/null +++ b/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu @@ -0,0 +1,2355 @@ +// SPDX-License-Identifier: Apache-2.0 +// +// FlashRT ggml adapter — second target: RTX 5090 (SM120) + Qwen3.6-35B-A3B +// window set. Pattern-matched subgraph windows over ggml-cuda's fuse hook: +// +// - fused-region GEMVs (GDN in_proj / attn qkv) through the NVFP4 W4A4 +// warp-split-K blockscale MMA GEMV (f32 act -> NVFP4 quant + swizzled +// SFA -> GEMV -> staging served to all region members) +// - GDN cell span (conv + gated delta net + epilogue, M<=4 with +// per-token state/conv snapshots and checkpoint replay) +// - MoE expert span K0/K1/K2 consuming ggml's native K-quant blocks via +// its own vec_dot device functions (bit-exact q8_1 activation clone), +// shared-expert folded, M<=4 +// - out-proj / lm-head / router windows, spec-draft head serving +// +// All launches join the host's PDL chain (ggml_cuda_kernel_launch). Weights +// for the FP4 windows come from side-band packs (FRT_REGIONS_PACK / +// FRT_HEAD_PACK) until the in-process repack cache lands (see DEVELOPMENT). +// Runtime switches: FRT_* per window (see the target section in USAGE.md). +// Kernels ported from FlashRT csrc (Apache-2.0, same authorship). + +#include "common.cuh" +#include "vecdotq.cuh" + +#include +#include +#include +#include +#include +#include +#include + +#include "cute/arch/mma_sm120.hpp" +#include "cutlass/numeric_types.h" + +namespace frt { + +// ---------------- activation quantize (f32 row -> NVFP4 + swizzled SFA) --- + +__device__ __forceinline__ int sfa_offset_128x64(int row, int k, int dim) { + const int row_block = row >> 7; + const int row_in_block = row & 127; + const int k_block = k >> 6; + const int k_in_block = k & 63; + const int k_blocks = (dim + 63) >> 6; + return row_block * k_blocks * 512 + k_block * 512 + + (row_in_block & 31) * 16 + (row_in_block >> 5) * 4 + + (k_in_block >> 4); +} + +__device__ __forceinline__ uint8_t fp32_to_e2m1(float x) { + uint8_t sign = (x < 0.f) ? 0x8u : 0x0u; + float ax = fabsf(x); + uint8_t mant; + if (ax <= 0.25f) mant = 0u; + else if (ax <= 0.75f) mant = 1u; + else if (ax <= 1.25f) mant = 2u; + else if (ax <= 1.75f) mant = 3u; + else if (ax <= 2.5f) mant = 4u; + else if (ax <= 3.5f) mant = 5u; + else if (ax <= 5.0f) mant = 6u; + else mant = 7u; + return sign | mant; +} + +// quantize a f32 row of length D into packed e2m1 + SFA (device body, callable +// from any single participating block). row selects the SFA atom-layout row and +// the packed output row (row-major, D/16 uint2 per row). +template +__device__ __forceinline__ void quant_act_fp4_f32_body( + const float * __restrict__ x, + uint2 * __restrict__ dst_packed, + uint8_t * __restrict__ dst_sfa, + int D, int row = 0) { + const int n_blocks = D / 16; + uint2 * dst_row = dst_packed + (size_t) row * n_blocks; + for (int b = threadIdx.x; b < n_blocks; b += THREADS) { + float vals[16]; + float amax = 0.f; +#pragma unroll + for (int i = 0; i < 16; ++i) { + vals[i] = x[b * 16 + i]; + const float a = fabsf(vals[i]); + if (a > amax) amax = a; + } + float desired = amax / 6.f; + if (desired < 1e-12f) desired = 1e-12f; + __nv_fp8_e4m3 bs_q = __nv_fp8_e4m3(fmaxf(desired, 0.f)); + const float bs_dq = static_cast(bs_q); + dst_sfa[sfa_offset_128x64(row, b * 16, D)] = *reinterpret_cast(&bs_q); + const float inv_bs = 1.f / bs_dq; + uint2 out; + uint8_t * ob = reinterpret_cast(&out); +#pragma unroll + for (int p = 0; p < 8; ++p) { + const uint8_t lo = fp32_to_e2m1(vals[2 * p] * inv_bs); + const uint8_t hi = fp32_to_e2m1(vals[2 * p + 1] * inv_bs); + ob[p] = static_cast(lo | (hi << 4)); + } + dst_row[b] = out; + } +} + +// MT is compile-time: a runtime token count in the hot path costs measurable +// time even at M=1 (handoff §5.2), so callers dispatch through FRT_M_DISPATCH. +template +__global__ void quant_act_fp4_f32( + const float * __restrict__ x, + uint2 * __restrict__ dst_packed, + uint8_t * __restrict__ dst_sfa, + int D, int64_t x_srow = 0) { + ggml_cuda_pdl_lc(); ggml_cuda_pdl_sync(); +#pragma unroll + for (int r = blockIdx.x; r < MT; r += gridDim.x) // launch with grid = MT + quant_act_fp4_f32_body(x + (size_t) r * x_srow, dst_packed, dst_sfa, D, r); +} + +// switch a runtime M in [1,4] onto a compile-time MT inside __VA_ARGS__. +#define FRT_M_DISPATCH(M, ...) do { switch (M) { \ + case 1: { constexpr int MT = 1; __VA_ARGS__; } break; \ + case 2: { constexpr int MT = 2; __VA_ARGS__; } break; \ + case 3: { constexpr int MT = 3; __VA_ARGS__; } break; \ + default:{ constexpr int MT = 4; __VA_ARGS__; } break; } } while (0) + +// ---------------- warp-split-K NVFP4 W4A4 M=1 GEMV (f32 out) --------------- + +#if defined(__CUDA_ARCH_FEAT_SM120_ALL) || !defined(__CUDA_ARCH__) +#define FRT_SM120A_OK 1 +#endif + +using AtomType = cute::SM120::BLOCKSCALED::SM120_16x8x64_TN_VS< + cutlass::float_e2m1_t, cutlass::float_e2m1_t, float, + cutlass::float_ue4m3_t, 16>; + +__device__ __forceinline__ uint32_t fa(const uint8_t * s, int t0, int t1, int r) { + int ro = ((r & 1) ? (t1 + 8) : t1) * 32; + return *reinterpret_cast(s + ro + t0 * 4 + ((r >> 1) & 1) * 16); +} +__device__ __forceinline__ uint32_t fb(const uint8_t * s, int t0, int t1, int r) { + return *reinterpret_cast(s + t1 * 32 + t0 * 4 + r * 16); +} +__device__ __forceinline__ uint32_t fsa(const uint8_t * p, int u) { + return *reinterpret_cast(p + u * 4); +} +__device__ __forceinline__ void cpa(uint8_t * d, const uint8_t * s) { + uint32_t i = __cvta_generic_to_shared(d); + asm volatile("cp.async.ca.shared.global.L2::128B [%0], [%1], 4;\n" :: "r"(i), "l"(s)); +} +__device__ __forceinline__ void commit() { asm volatile("cp.async.commit_group;\n" ::); } +template __device__ __forceinline__ void waitg() { + asm volatile("cp.async.wait_group %0;\n" :: "n"(N)); +} + +template +__global__ void warpsplit_kernel_f32out( + const uint8_t * __restrict__ A, const uint8_t * __restrict__ B, + const uint8_t * __restrict__ SFA, const uint8_t * __restrict__ SFB, + float * __restrict__ D, float alpha, int N, int K) { + constexpr int M = MT; +#if defined(FRT_SM120A_OK) + ggml_cuda_pdl_lc(); ggml_cuda_pdl_sync(); + __shared__ uint8_t sA[WARPS][STAGES][16 * 32]; + __shared__ uint8_t sSFA[WARPS][STAGES][16 * 4]; + __shared__ uint8_t sB[WARPS][STAGES][8 * 32]; + __shared__ uint8_t sSFB[WARPS][STAGES][8 * 4]; + __shared__ float s_red[WARPS][4 * 8]; + + int tid = threadIdx.x, warp = tid >> 5, lane = tid & 31; + int my_n = blockIdx.x * 8; + const int KI = K / 64, KIw = KI / WARPS; + const int kt0 = warp * KIw; + const int KH = K / 2, ncs = (K / 16 + 3) / 4; + int t0 = lane & 3, t1 = lane >> 2, sau = (lane & 1) * 8 + (lane >> 2), sbu = lane >> 2; + float c0 = 0, c1 = 0, c2 = 0, c3 = 0; + + uint8_t (*mA)[16 * 32] = sA[warp]; + uint8_t (*mSFA)[16 * 4] = sSFA[warp]; + uint8_t (*mB)[8 * 32] = sB[warp]; + uint8_t (*mSFB)[8 * 4] = sSFB[warp]; + + if (lane >= 1 && lane < 16) { +#pragma unroll + for (int st = 0; st < STAGES; ++st) { + int4 * av = reinterpret_cast(mA[st]); int4 z{0, 0, 0, 0}; + av[lane * 2] = z; av[lane * 2 + 1] = z; + } + if (lane < 4) for (int st = 0; st < STAGES; ++st) + for (int i = 4 + lane; i < 64; i += 4) mSFA[st][i] = 0; + } + __syncwarp(); // M>1: row-1 cp.async below must not race the zero-init + auto ld = [&](int bf, int kt) { + int bo = kt * 32; + if (lane < 8) cpa(mA[bf] + lane * 4, A + bo + lane * 4); + if (lane == 0) cpa(mSFA[bf], SFA + kt * 512); +#pragma unroll + for (int rr = 1; rr < MT; ++rr) { // extra token rows: act tile + atom-layout scales (row r -> +r*16) + if (lane < 8) cpa(mA[bf] + rr * 32 + lane * 4, A + (size_t) rr * KH + bo + lane * 4); + if (lane == 0) cpa(mSFA[bf] + rr * 4, SFA + kt * 512 + rr * 16); + } + for (int c = 0; c < 2; ++c) { int ch = lane + c * 32, col = ch >> 3, off = ch & 7; + cpa(mB[bf] + ch * 4, B + (size_t)(my_n + col) * KH + bo + off * 4); } + if (lane < 8) { int col = my_n + lane, rb = col >> 7, ri = col & 127; + int si = rb * ncs + kt, ib = (ri & 31) * 16 + ((ri >> 5) & 3) * 4; + cpa(mSFB[bf] + lane * 4, SFB + (size_t)si * 512 + ib); } + }; +#pragma unroll + for (int st = 0; st < STAGES - 1; ++st) { if (st < KIw) ld(st, kt0 + st); commit(); } + for (int j = 0; j < KIw; ++j) { + int cb = j % STAGES, jp = j + STAGES - 1; + if (jp < KIw) ld(jp % STAGES, kt0 + jp); + commit(); waitg(); __syncwarp(); + uint32_t a0 = fa(mA[cb], t0, t1, 0), a1 = fa(mA[cb], t0, t1, 1); + uint32_t a2 = fa(mA[cb], t0, t1, 2), a3 = fa(mA[cb], t0, t1, 3); + uint32_t b0 = fb(mB[cb], t0, t1, 0), b1 = fb(mB[cb], t0, t1, 1); + uint32_t sfa_v = fsa(mSFA[cb], sau), sfb_v = fsa(mSFB[cb], sbu); + float d0, d1, d2, d3; + AtomType::fma(d0, d1, d2, d3, a0, a1, a2, a3, b0, b1, c0, c1, c2, c3, sfa_v, sfb_v); + c0 = d0; c1 = d1; c2 = d2; c3 = d3; + } + // m16n8 C fragment: {c0,c1} hold row (lane>>2) -> token t lives in lanes 4t..4t+3. + int q = lane >> 2, r = lane & 3; + if (q < M) { s_red[warp][q * 8 + r * 2] = c0; s_red[warp][q * 8 + r * 2 + 1] = c1; } + __syncthreads(); + if (warp == 0 && lane < 8) { + int col = my_n + lane; + if (col < N) { +#pragma unroll + for (int t = 0; t < MT; ++t) { + float acc = 0.f; +#pragma unroll + for (int w = 0; w < WARPS; ++w) acc += s_red[w][t * 8 + lane]; + D[(size_t) t * N + col] = acc * alpha; + } + } + } +#endif // FRT_SM120A_OK +} + +// runtime (STAGES, WARPS) selection for the region GEMVs: FRT_WS_CFG=sw +// (default s4w2). K/64 must be divisible by W. +static void frt_ws_launch(const uint8_t * A, const uint8_t * B, + const uint8_t * SFA, const uint8_t * SFB, float * D, + float alpha, int N, int K, int M, cudaStream_t stream, int def_cfg = 0) { + static int env_cfg = -1; + if (env_cfg < 0) { + const char * e = getenv("FRT_WS_CFG"); + env_cfg = 0; + if (e) { + int s = 0, w = 0; + if (sscanf(e, "s%dw%d", &s, &w) == 2) env_cfg = s * 10 + w; + } + } + const int cfg = env_cfg ? env_cfg : def_cfg; + dim3 grid(N / 8); +#define FRT_WS_CASE(S, W) do { \ + auto kf = M == 1 ? warpsplit_kernel_f32out : M == 2 ? warpsplit_kernel_f32out : \ + M == 3 ? warpsplit_kernel_f32out : warpsplit_kernel_f32out; \ + ggml_cuda_kernel_launch(kf, ggml_cuda_kernel_launch_params(grid, dim3(W * 32), 0, stream), A, B, SFA, SFB, D, alpha, N, K); \ + } while (0) + switch (cfg) { + case 34: FRT_WS_CASE(3, 4); break; + case 44: FRT_WS_CASE(4, 4); break; + case 64: FRT_WS_CASE(6, 4); break; + case 38: FRT_WS_CASE(3, 8); break; + case 48: FRT_WS_CASE(4, 8); break; + case 32: FRT_WS_CASE(3, 2); break; + case 62: FRT_WS_CASE(6, 2); break; + default: FRT_WS_CASE(4, 2); break; + } +#undef FRT_WS_CASE +} + +// ---------------- W4A16 matvec (NVFP4 weight x f32 act, f32 out) ---------- +// Ported from FlashRT w4a16_matvec_sm120 (same swizzled NVFP4 weight layout), +// modified: f32 activation staged in smem, f32 output. No activation quant. + +__device__ __constant__ float c_ue4m3[256]; // CUTLASS UE4M3 (FlashRT packs) +__device__ __constant__ float c_e4m3_half[256]; // ggml NVFP4 scale: E4M3 / 2, NaN->0 + +static void frt_init_ue4m3_lut(void) { + static bool inited = false; + if (inited) return; + inited = true; + float lut[256]; + for (int i = 0; i < 256; ++i) { + const int e = (i >> 3) & 0xF; + const int m = i & 0x7; + lut[i] = (e == 0) ? (float) m * ldexpf(1.0f, -9) + : (1.0f + (float) m / 8.0f) * ldexpf(1.0f, e - 7); + } + CUDA_CHECK(cudaMemcpyToSymbol(c_ue4m3, lut, sizeof(lut))); + float lut2[256]; + for (int i = 0; i < 256; ++i) { + const int lo = i & 0x7F; + if (lo == 0x7F) { lut2[i] = 0.0f; continue; } // E4M3 NaN -> 0 (ggml CPU semantics) + const int e = (lo >> 3) & 0xF; + const int m = lo & 0x7; + // their stored scale = e4m3; their nibble table = 2x e2m1; we use true + // e2m1 via the cvt intrinsic, so the plain e4m3 value pairs correctly. + float v = (e == 0) ? (float) m / 8.0f * ldexpf(1.0f, -6) + : (1.0f + (float) m / 8.0f) * ldexpf(1.0f, e - 7); + lut2[i] = (i & 0x80) ? -v : v; + } + CUDA_CHECK(cudaMemcpyToSymbol(c_e4m3_half, lut2, sizeof(lut2))); +} + +__device__ __forceinline__ int frt_sf_off(int rb_ncs, int row_inner, int k_block) { + return (rb_ncs + (k_block >> 2)) * 512 + row_inner + (k_block & 3); +} + +__device__ __forceinline__ float frt_blockdot_f32(uint64_t b_pack, const float2 * xb2) { + float acc = 0.0f; +#pragma unroll + for (int j = 0; j < 8; ++j) { + const __nv_fp4x2_storage_t bb = static_cast<__nv_fp4x2_storage_t>(b_pack >> (j * 8)); + const __half2_raw wr = __nv_cvt_fp4x2_to_halfraw2(bb, __NV_E2M1); + const float2 wf = __half22float2(*reinterpret_cast(&wr)); + const float2 xf = xb2[j]; + acc = fmaf(wf.x, xf.x, acc); + acc = fmaf(wf.y, xf.y, acc); + } + return acc; +} + +// 8 rows / block, 1 warp / row; x (f32) staged in smem shared by the warps. +__global__ void w4a16_matvec_f32( + const float * __restrict__ x, + const uint8_t * __restrict__ W, + const uint8_t * __restrict__ SFB, + float * __restrict__ out, + float alpha, int N, int K, int n_col_super) { + extern __shared__ float x_shf[]; + const int K_int4 = K >> 2; // 4 f32 per int4 + const int4 * x_i4 = reinterpret_cast(x); + int4 * x_sh_i4 = reinterpret_cast(x_shf); + for (int j = threadIdx.x; j < K_int4; j += 256) x_sh_i4[j] = x_i4[j]; + __syncthreads(); + + const int lane = threadIdx.x & 31; + const int row = blockIdx.x * 8 + (threadIdx.x >> 5); + if (row >= N) return; + + const int K_BLOCKS = K >> 4; + const uint64_t * w_blk = reinterpret_cast(W + (size_t) row * (K >> 1)); + const float2 * x_blk = reinterpret_cast(x_shf); + + const int rb = row >> 7; + const int ri = row & 127; + const int rb_ncs = rb * n_col_super; + const int row_inner = (ri & 31) * 16 + ((ri >> 5) & 3) * 4; + + float acc = 0.0f; + int kb = lane; + constexpr int UNROLL = 4; + const int step = 32 * UNROLL; + for (; kb + 32 * (UNROLL - 1) < K_BLOCKS; kb += step) { + uint64_t wv[UNROLL]; + float sf[UNROLL]; +#pragma unroll + for (int u = 0; u < UNROLL; ++u) wv[u] = w_blk[kb + 32 * u]; +#pragma unroll + for (int u = 0; u < UNROLL; ++u) + sf[u] = c_ue4m3[__ldg(SFB + frt_sf_off(rb_ncs, row_inner, kb + 32 * u))]; +#pragma unroll + for (int u = 0; u < UNROLL; ++u) + acc += frt_blockdot_f32(wv[u], x_blk + (size_t)(kb + 32 * u) * 8) * sf[u]; + } + for (; kb < K_BLOCKS; kb += 32) { + const float s = c_ue4m3[__ldg(SFB + frt_sf_off(rb_ncs, row_inner, kb))]; + acc += frt_blockdot_f32(w_blk[kb], x_blk + (size_t) kb * 8) * s; + } +#pragma unroll + for (int off = 16; off > 0; off >>= 1) + acc += __shfl_xor_sync(0xffffffff, acc, off); + if (lane == 0) out[row] = acc * alpha; +} + +// ---------------- MoE expert GEMV on ggml-native NVFP4 blocks -------------- +// Reads llama.cpp's own block_nvfp4 layout in place (no repack, no extra VRAM): +// 64 elems = d[4] UE4M3 sub-scales (16 elems each) + qs[32] +// qs[sub*8 + j] holds elem (sub*16 + j) in low nibble, (sub*16 + j + 8) high. +// f32 activations read directly (no q8_1 activation quant at all). + +// one warp computes one output row; lane l handles K-block (64 elems) l, l+32, ... +__device__ __forceinline__ float frt_nvfp4_rowdot( + const uint8_t * __restrict__ row, // K/64 blocks * 36 B + const float * __restrict__ x, int K) { + const int lane = threadIdx.x & 31; + const int kb_n = K >> 6; + float acc = 0.0f; + for (int kb = lane; kb < kb_n; kb += 32) { + const uint8_t * blk = row + (size_t) kb * 36; + const float * xb = x + (size_t) kb * 64; +#pragma unroll + for (int sub = 0; sub < 4; ++sub) { + const float d = c_e4m3_half[blk[sub]]; + // block stride is 36 B: qs is only 4-byte aligned, build the u64 from two u32 loads + const uint32_t q_lo = *reinterpret_cast(blk + 4 + sub * 8); + const uint32_t q_hi = *reinterpret_cast(blk + 8 + sub * 8); + const uint64_t q = ((uint64_t) q_hi << 32) | q_lo; + const float * xs = xb + sub * 16; + float sacc = 0.0f; +#pragma unroll + for (int j = 0; j < 8; ++j) { + const __nv_fp4x2_storage_t bb = static_cast<__nv_fp4x2_storage_t>(q >> (j * 8)); + const __half2_raw wr = __nv_cvt_fp4x2_to_halfraw2(bb, __NV_E2M1); + const float2 wf = __half22float2(*reinterpret_cast(&wr)); + sacc = fmaf(wf.x, xs[j], sacc); // low nibble -> elem j + sacc = fmaf(wf.y, xs[j + 8], sacc); // high nibble -> elem j+8 + } + acc = fmaf(d, sacc, acc); + } + } +#pragma unroll + for (int off = 16; off > 0; off >>= 1) + acc += __shfl_xor_sync(0xffffffff, acc, off); + return acc; +} + +// 8 warps / block, one row each. rows_total = n_used * n_per_expert. +// BROADCAST: all experts share x (gate/up). Else x per expert slot (down). +template +__global__ void frt_moe_mmid_f32( + const float * __restrict__ x, // (K) or (K, n_used) + const uint8_t * __restrict__ w, // expert-major NVFP4 + const int32_t * __restrict__ ids, // (n_used) + float * __restrict__ out, // (n_per, n_used) + int K, int n_per, int n_used, + int64_t expert_stride, int64_t x_stride) { + const int r = blockIdx.x * 8 + (threadIdx.x >> 5); + if (r >= n_used * n_per) return; + const int e = r / n_per; + const int n = r % n_per; + const uint8_t * row = w + (size_t) ids[e] * expert_stride + (size_t) n * ((K >> 6) * 36); + const float * xe = BROADCAST ? x : x + (size_t) e * x_stride; + const float acc = frt_nvfp4_rowdot(row, xe, K); + if ((threadIdx.x & 31) == 0) out[(size_t) e * n_per + n] = acc; +} + +// ---------------- fused-region packs (GDN in_proj / attn qkv) ------------- + +struct frt_region { + int64_t N = 0, K = 0; + float alpha = 0.f; + uint8_t * d_packed = nullptr; + uint8_t * d_sf = nullptr; +}; + +struct frt_region_state { + bool tried = false; + bool ok = false; + bool inproj_on = false; // FRT_INPROJ_SWAP (GDN kind 0) + bool attn_on = false; // FRT_ATTNQKV_SWAP (attn kind 1) + bool shexp_on = false; // FRT_SHEXP_SWAP (shared expert span, kinds 2+3) + bool outproj_on = false; // FRT_OUTPROJ_SWAP (ssm_out / attn_output, kind 4) + frt_region regions[5][64]; // [kind][layer]: 0=gdn in_proj 1=attn qkv 2=shexp gate|up 3=shexp down 4=out_proj + float * d_staging = nullptr; // 12352 f32 + float * d_conv_out = nullptr; // 8192 f32 (GDN cell) + float * d_attn_buf = nullptr; // 4096 f32 (GDN cell) + float * d_scalar = nullptr; // 1 f32 (shexp sigmoid gate) + block_q8_1 * d_outq8 = nullptr; // gdn epilogue q8 output (128 blocks) + const void * outq8_node = nullptr; // graph node whose q8 is staged in d_outq8 + uint8_t * d_apack = nullptr; // K/2 + uint8_t * d_sfa = nullptr; // 128 * K/16 + // capture-time leader tracking + const void * leader_src = nullptr; + int64_t leader_key = -1; +}; + +static frt_region_state g_reg; + +static bool frt_regions_load(void) { + if (g_reg.tried) return g_reg.ok; + g_reg.tried = true; + const char * a = getenv("FRT_INPROJ_SWAP"); + const char * b = getenv("FRT_ATTNQKV_SWAP"); + const char * c = getenv("FRT_SHEXP_SWAP"); + const char * d = getenv("FRT_OUTPROJ_SWAP"); + g_reg.inproj_on = a && a[0] == '1'; + g_reg.attn_on = b && b[0] == '1'; + g_reg.shexp_on = c && c[0] == '1'; + g_reg.outproj_on = d && d[0] == '1'; + if (!g_reg.inproj_on && !g_reg.attn_on && !g_reg.shexp_on && !g_reg.outproj_on) return false; + const char * path = getenv("FRT_REGIONS_PACK"); + if (!path) { fprintf(stderr, "frt-regions: FRT_REGIONS_PACK missing\n"); return false; } + FILE * f = fopen(path, "rb"); + if (!f) { fprintf(stderr, "frt-regions: cannot open %s\n", path); return false; } + int64_t hdr[2]; + if (fread(hdr, 8, 2, f) != 2 || hdr[0] != 0x46525452) { fclose(f); return false; } + const int64_t count = hdr[1]; + int64_t maxN = 0, maxK = 0; + for (int64_t e = 0; e < count; ++e) { + int64_t layer, kind, N, K, pkb, sfb; double alpha; + if (fread(&layer, 8, 1, f) != 1) break; + fread(&kind, 8, 1, f); fread(&N, 8, 1, f); fread(&K, 8, 1, f); + fread(&alpha, 8, 1, f); fread(&pkb, 8, 1, f); fread(&sfb, 8, 1, f); + if (kind < 0 || kind > 4 || layer < 0 || layer >= 64) { fclose(f); return false; } + frt_region & r = g_reg.regions[kind][layer]; + r.N = N; r.K = K; r.alpha = (float) alpha; + uint8_t * h = (uint8_t *) malloc((size_t)(pkb > sfb ? pkb : sfb)); + CUDA_CHECK(cudaMalloc(&r.d_packed, pkb)); + fread(h, 1, pkb, f); + CUDA_CHECK(cudaMemcpy(r.d_packed, h, pkb, cudaMemcpyHostToDevice)); + CUDA_CHECK(cudaMalloc(&r.d_sf, sfb)); + fread(h, 1, sfb, f); + CUDA_CHECK(cudaMemcpy(r.d_sf, h, sfb, cudaMemcpyHostToDevice)); + free(h); + if (N > maxN) maxN = N; + if (K > maxK) maxK = K; + } + fclose(f); + CUDA_CHECK(cudaMalloc(&g_reg.d_staging, 4 * maxN * sizeof(float))); + CUDA_CHECK(cudaMalloc(&g_reg.d_conv_out, 4 * 8192 * sizeof(float))); + CUDA_CHECK(cudaMalloc(&g_reg.d_attn_buf, 4 * 4096 * sizeof(float))); + CUDA_CHECK(cudaMalloc(&g_reg.d_scalar, sizeof(float))); + CUDA_CHECK(cudaMalloc(&g_reg.d_outq8, 4 * 128 * sizeof(block_q8_1))); + CUDA_CHECK(cudaMalloc(&g_reg.d_apack, 2 * maxK)); + CUDA_CHECK(cudaMalloc(&g_reg.d_sfa, 128 * (maxK / 16))); + CUDA_CHECK(cudaMemset(g_reg.d_sfa, 0, 128 * (maxK / 16))); + fprintf(stderr, "frt-regions: loaded %lld regions (inproj=%d attnqkv=%d)\n", + (long long) count, (int) g_reg.inproj_on, (int) g_reg.attn_on); + g_reg.ok = true; + return true; +} + +// serve one member of a fused region. leader==true runs quant+GEMV into staging. +static bool frt_region_serve(ggml_backend_cuda_context & ctx, int kind, int layer, + bool leader, int64_t row_off, int64_t rows, + const ggml_tensor * src1, ggml_tensor * dst) { + frt_region & r = g_reg.regions[kind][layer]; + if (r.N == 0) return false; + if (src1->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) return false; + const int M = (int) src1->ne[1]; // token-batch width (spec verify runs M = 1 + n_draft) + if (M < 1 || M > 4 || src1->ne[2] != 1 || src1->ne[3] != 1) return false; + if (dst->ne[1] != M || !ggml_is_contiguous(dst)) return false; + if (!ggml_is_contiguous(src1)) return false; + const int64_t key = ((int64_t) kind << 32) | layer; + cudaStream_t stream = ctx.stream(); + if (leader) { + FRT_M_DISPATCH(M, ggml_cuda_kernel_launch((frt::quant_act_fp4_f32<256, MT>), ggml_cuda_kernel_launch_params(dim3(M), dim3(256), 0, stream), + (const float *) src1->data, (uint2 *) g_reg.d_apack, g_reg.d_sfa, (int) r.K, (int64_t) r.K)); + frt::frt_ws_launch(g_reg.d_apack, r.d_packed, g_reg.d_sfa, r.d_sf, + g_reg.d_staging, r.alpha, (int) r.N, (int) r.K, M, stream); + g_reg.leader_src = src1->data; + g_reg.leader_key = (key << 2) | M; + } else { + // follower: only valid if the leader ran with the same activation + if (g_reg.leader_key != ((key << 2) | M) || g_reg.leader_src != src1->data) return false; + } + if (M == 1) { + CUDA_CHECK(cudaMemcpyAsync(dst->data, g_reg.d_staging + row_off, + rows * sizeof(float), cudaMemcpyDeviceToDevice, stream)); + } else { // one strided copy for all token rows + CUDA_CHECK(cudaMemcpy2DAsync(dst->data, rows * sizeof(float), + g_reg.d_staging + row_off, r.N * sizeof(float), + rows * sizeof(float), M, cudaMemcpyDeviceToDevice, stream)); + } + return true; +} + +// dims: GDN in_proj = [qkv 8192 | z 4096 | a 32 | b 32]; attn = [q 8192 | k 512 | v 512] +static bool frt_regions_mul_mat(ggml_backend_cuda_context & ctx, + const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) { + if (!frt_regions_load()) return false; + int layer = -1; char rest[64] = {0}; + if (sscanf(src0->name, "blk.%d.%63s", &layer, rest) != 2 || layer < 0 || layer >= 64) return false; + if (g_reg.inproj_on) { + if (strcmp(rest, "attn_qkv.weight") == 0) return frt_region_serve(ctx, 0, layer, true, 0, 8192, src1, dst); + if (strcmp(rest, "attn_gate.weight") == 0) return frt_region_serve(ctx, 0, layer, false, 8192, 4096, src1, dst); + if (strcmp(rest, "ssm_alpha.weight") == 0) return frt_region_serve(ctx, 0, layer, false, 12288, 32, src1, dst); + if (strcmp(rest, "ssm_beta.weight") == 0) return frt_region_serve(ctx, 0, layer, false, 12288 + 32, 32, src1, dst); + } + if (g_reg.attn_on) { + if (strcmp(rest, "attn_q.weight") == 0) return frt_region_serve(ctx, 1, layer, true, 0, 8192, src1, dst); + if (strcmp(rest, "attn_k.weight") == 0) return frt_region_serve(ctx, 1, layer, false, 8192, 512, src1, dst); + if (strcmp(rest, "attn_v.weight") == 0) return frt_region_serve(ctx, 1, layer, false, 8704, 512, src1, dst); + } + if (g_reg.shexp_on) { + if (strcmp(rest, "ffn_gate_shexp.weight") == 0) return frt_region_serve(ctx, 2, layer, true, 0, 512, src1, dst); + if (strcmp(rest, "ffn_up_shexp.weight") == 0) return frt_region_serve(ctx, 2, layer, false, 512, 512, src1, dst); + } + if (g_reg.outproj_on) { + if (strcmp(rest, "ssm_out.weight") == 0 || strcmp(rest, "attn_output.weight") == 0) + return frt_region_serve(ctx, 4, layer, true, 0, 2048, src1, dst); + } + return false; +} + +// ---------------- side-band pack + hook ------------------------------------ + +struct frt_head_state { + bool tried = false; + bool ok = false; + bool draft_only = false; // FRT_HEAD_DRAFT without FRT_HEAD_SWAP + int64_t N = 0; + int64_t K = 0; + float alpha = 0.f; + uint8_t * d_packed = nullptr; + uint8_t * d_sf = nullptr; + uint8_t * d_apack = nullptr; // K/2 bytes + uint8_t * d_sfa = nullptr; // 128 * K/16 bytes +}; + +static frt_head_state g_head; + +static bool frt_head_load(void) { + if (g_head.tried) return g_head.ok; + g_head.tried = true; + // FRT_HEAD_SWAP serves every output.weight head (full tier). FRT_HEAD_DRAFT + // alone serves only the spec-draft copy of the head (identified by its Q8_0 + // storage; the target head stays Q6_K/stock) — draft logits only steer + // acceptance, never the verified output, so this is quality-free. + const char * sw = getenv("FRT_HEAD_SWAP"); + const char * dr = getenv("FRT_HEAD_DRAFT"); + const bool sw_on = sw && sw[0] == '1'; + const bool dr_on = dr && dr[0] == '1'; + if (!sw_on && !dr_on) return false; + g_head.draft_only = !sw_on; + const char * path = getenv("FRT_HEAD_PACK"); + if (!path) { fprintf(stderr, "frt-head: FRT_HEAD_SWAP/FRT_HEAD_DRAFT set but FRT_HEAD_PACK missing\n"); return false; } + FILE * f = fopen(path, "rb"); + if (!f) { fprintf(stderr, "frt-head: cannot open %s\n", path); return false; } + int64_t hdr[4] = {0, 0, 0, 0}; // magic, N, K, alpha bits (f64) + if (fread(hdr, 8, 4, f) != 4 || hdr[0] != 0x46525448) { fclose(f); fprintf(stderr, "frt-head: bad header\n"); return false; } + g_head.N = hdr[1]; g_head.K = hdr[2]; + double alpha_d; memcpy(&alpha_d, &hdr[3], 8); + g_head.alpha = (float) alpha_d; + const size_t packed_bytes = (size_t) g_head.N * (size_t) g_head.K / 2; + const size_t nrb = ((size_t) g_head.N + 127) / 128; + const size_t sf_bytes = nrb * (size_t)((g_head.K + 63) / 64) * 512; + uint8_t * h = (uint8_t *) malloc(packed_bytes > sf_bytes ? packed_bytes : sf_bytes); + CUDA_CHECK(cudaMalloc(&g_head.d_packed, packed_bytes)); + if (fread(h, 1, packed_bytes, f) != packed_bytes) { fclose(f); free(h); fprintf(stderr, "frt-head: short packed\n"); return false; } + CUDA_CHECK(cudaMemcpy(g_head.d_packed, h, packed_bytes, cudaMemcpyHostToDevice)); + CUDA_CHECK(cudaMalloc(&g_head.d_sf, sf_bytes)); + if (fread(h, 1, sf_bytes, f) != sf_bytes) { fclose(f); free(h); fprintf(stderr, "frt-head: short sf\n"); return false; } + CUDA_CHECK(cudaMemcpy(g_head.d_sf, h, sf_bytes, cudaMemcpyHostToDevice)); + fclose(f); free(h); + CUDA_CHECK(cudaMalloc(&g_head.d_apack, 4 * (g_head.K / 2))); + CUDA_CHECK(cudaMalloc(&g_head.d_sfa, 128 * (g_head.K / 16))); + CUDA_CHECK(cudaMemset(g_head.d_sfa, 0, 128 * (g_head.K / 16))); + frt_init_ue4m3_lut(); // eager-time upload; never during graph capture + fprintf(stderr, "frt-head: loaded pack N=%lld K=%lld alpha=%g\n", + (long long) g_head.N, (long long) g_head.K, (double) g_head.alpha); + g_head.ok = true; + return true; +} + +} // namespace frt + +// ---- GDN cell fusion (FRT_GDN_SWAP) --------------------------------------- +// Replaces the whole per-layer GDN cell span (conv-cache dance + SSM_CONV + +// silu + l2norms + gate prep + GATED_DELTA_NET + state copies + gated norm) +// with: leader GEMV (staging) -> K1 conv -> K2 cell. State/conv caches are +// updated in place: 1R+1W instead of the graph's multi-copy dance. + +namespace gdn { + +__device__ __forceinline__ float frt_silu(float x) { return x / (1.0f + expf(-x)); } + +// same semantics as quantize_q8_1 (d=amax/127, s=raw sum), one warp = one block +__device__ __forceinline__ void frt_q8_1_block_g(float xi, int lane, block_q8_1 * dst) { + float amax = fabsf(xi), sum = xi; +#pragma unroll + for (int o = 16; o > 0; o >>= 1) { + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, o)); + sum += __shfl_xor_sync(0xffffffffu, sum, o); + } + const float d = amax / 127.0f; + const int8_t q = amax == 0.0f ? 0 : (int8_t) roundf(xi / d); + dst->qs[lane] = q; + if (lane == 0) dst->ds = make_half2(d, sum); +} +__device__ __forceinline__ float frt_softplus(float x) { return (x > 20.0f) ? x : logf(1.0f + expf(x)); } + +// K1: 4-tap causal conv over [conv_state | x_new(M tokens)] + silu; shifts conv +// state in place (state after = last 3 inputs of the 3+M window). staging is +// token-major (t*12288 + ch); conv_out is token-major (t*8192 + ch). +// MT==1: window shifted in place (the M=1 graph's snapshot CPY targets the same +// slot row). MT>=2 (spec verify): the source slot must stay pristine for +// rollback; instead a per-token window snapshot is written to the cache rows +// the graph's M conv-state CPY nodes target (snap0..snap3, token order). +template +__global__ void frt_gdn_conv_silu( + const float * __restrict__ staging, // qkv rows, token stride 12288 + const float * __restrict__ conv_w, // (4, 8192): w(j,ch) = conv_w[ch*4+j] + float * __restrict__ r_base, // conv cache base, slot stride 24576 floats + const int32_t * __restrict__ r_slot, + float * __restrict__ conv_out, + float * __restrict__ snap0, float * __restrict__ snap1, + float * __restrict__ snap2, float * __restrict__ snap3) { + constexpr int M = MT; + ggml_cuda_pdl_lc(); ggml_cuda_pdl_sync(); + const int ch = blockIdx.x * 256 + threadIdx.x; + if (ch >= 8192) return; + float * cs = r_base + (size_t) (*r_slot) * 24576 + (size_t) ch * 3; + const float * w = conv_w + (size_t) ch * 4; + float s0 = cs[0], s1 = cs[1], s2 = cs[2]; + float * const snaps[4] = { snap0, snap1, snap2, snap3 }; + for (int t = 0; t < M; ++t) { + const float x = staging[(size_t) t * 12288 + ch]; + const float o = s0 * w[0] + s1 * w[1] + s2 * w[2] + x * w[3]; + conv_out[(size_t) t * 8192 + ch] = frt_silu(o); + s0 = s1; s1 = s2; s2 = x; + if (MT >= 2) { + float * sp = snaps[t] + (size_t) ch * 3; + sp[0] = s0; sp[1] = s1; sp[2] = s2; + } + } + if (MT == 1) { cs[0] = s0; cs[1] = s1; cs[2] = s2; } +} + +// checkpoint save: dst[i] = base[idx[0]*row + i] (the graph's pre-update +// snapshot of the current cache slot into a checkpoint slot). +__global__ void frt_gdn_ckpt_copy( + const float * __restrict__ base, const int32_t * __restrict__ idx, + int64_t row, float * __restrict__ dst, int n) { + ggml_cuda_pdl_lc(); ggml_cuda_pdl_sync(); + const int i = blockIdx.x * 256 + threadIdx.x; + if (i < n) dst[i] = base[(size_t) idx[0] * row + i]; +} + +// K0n: norm-fused variant: each block redundantly computes the RMS norm of the +// raw hidden state (8KB read, latency-free) into smem, then proceeds like K0. +__global__ void frt_gdn_norm_quant_ab( + const float * __restrict__ raw, // 2048 f32 pre-norm hidden + const float * __restrict__ normw, // attn_norm weight + float eps, + uint2 * __restrict__ dst_packed, + uint8_t * __restrict__ dst_sfa, + const float * __restrict__ w_alpha, + const float * __restrict__ w_beta, + float * __restrict__ staging) { + ggml_cuda_pdl_lc(); ggml_cuda_pdl_sync(); + __shared__ float act[2048]; + __shared__ float red[8]; + const int tid = threadIdx.x, lane = tid & 31, warp = tid >> 5; + const float4 * r4 = (const float4 *) raw; + float s2 = 0.0f; + for (int k = tid; k < 512; k += 256) { + const float4 v = r4[k]; + s2 += v.x * v.x + v.y * v.y + v.z * v.z + v.w * v.w; + } +#pragma unroll + for (int o = 16; o > 0; o >>= 1) s2 += __shfl_xor_sync(0xffffffffu, s2, o); + if (lane == 0) red[warp] = s2; + __syncthreads(); + float tot = 0.0f; +#pragma unroll + for (int q = 0; q < 8; ++q) tot += red[q]; + const float rrms = rsqrtf(tot / 2048.0f + eps); + for (int k = tid; k < 2048; k += 256) act[k] = raw[k] * rrms * normw[k]; + __syncthreads(); + if (blockIdx.x == 0) { + frt::quant_act_fp4_f32_body<256>(act, dst_packed, dst_sfa, 2048); + return; + } + const int row = (blockIdx.x - 1) * 8 + warp; + const float * w = (row < 32 ? w_alpha + (size_t) row * 2048 + : w_beta + (size_t) (row - 32) * 2048); + const float4 * w4 = (const float4 *) w; + float acc = 0.0f; +#pragma unroll 4 + for (int k = lane; k < 512; k += 32) { + const float4 wv = w4[k]; + acc += wv.x * act[k * 4] + wv.y * act[k * 4 + 1] + wv.z * act[k * 4 + 2] + wv.w * act[k * 4 + 3]; + } +#pragma unroll + for (int o = 16; o > 0; o >>= 1) acc += __shfl_xor_sync(0xffffffffu, acc, o); + if (lane == 0) staging[12288 + row] = acc; +} + +// K0: fused act-quant + F32 a/b gate rows. Block 0 quantizes the f32 act for +// the W4A4 GEMV (which is then launched with N=12288 so it never touches the +// gate rows); blocks 1..8 compute the 64 a/b rows in F32 (weights are stored +// F32 in the GGUF; W4A4 staging values for these rows cost ~1% PPL). The ab +// blocks run concurrently with the single quant block, so they are ~free. +// a/b outputs land token-major after the GEMV rows: staging[M*12288 + t*64 + row]. +template +__global__ void frt_gdn_quant_ab( + const float * __restrict__ act, // M x 2048 f32 rows (attn_norm) + uint2 * __restrict__ dst_packed, + uint8_t * __restrict__ dst_sfa, + const float * __restrict__ w_alpha, // (2048, 32) row-major K-contig + const float * __restrict__ w_beta, + float * __restrict__ staging) { + constexpr int M = MT; + ggml_cuda_pdl_lc(); ggml_cuda_pdl_sync(); + if (blockIdx.x == 0) { + for (int t = 0; t < M; ++t) + frt::quant_act_fp4_f32_body<256>(act + (size_t) t * 2048, dst_packed, dst_sfa, 2048, t); + return; + } + const int warp = threadIdx.x >> 5, lane = threadIdx.x & 31; + const int row = (blockIdx.x - 1) * 8 + warp; // 0..63: alpha rows then beta + const float * w = (row < 32 ? w_alpha + (size_t) row * 2048 + : w_beta + (size_t) (row - 32) * 2048); + const float4 * w4 = (const float4 *) w; + float acc[MT]; +#pragma unroll + for (int t = 0; t < MT; ++t) acc[t] = 0.0f; +#pragma unroll 4 + for (int k = lane; k < 512; k += 32) { + const float4 wv = w4[k]; +#pragma unroll + for (int t = 0; t < MT; ++t) { + const float4 av = ((const float4 *) (act + (size_t) t * 2048))[k]; + acc[t] += wv.x * av.x + wv.y * av.y + wv.z * av.z + wv.w * av.w; + } + } +#pragma unroll + for (int t = 0; t < MT; ++t) { +#pragma unroll + for (int o = 16; o > 0; o >>= 1) acc[t] += __shfl_xor_sync(0xffffffffu, acc[t], o); + if (lane == 0) staging[(size_t) M * 12288 + t * 64 + row] = acc[t]; + } +} + +// K2a: grid (32 heads, 8 col-groups) x 128 thr; each block updates 16 state cols +// in place and writes raw attn cols to attn_buf. l2norm/gates recomputed per block. +// M tokens: sequential recurrence per state column, state kept in registers +// across the token loop (1R + 1W per column regardless of M). conv_out and +// attn_buf are token-major (t*8192 / t*4096); a/b live at staging[M*12288 + t*64 + h]. +template +__global__ void frt_gdn_cell_part( + const float * __restrict__ conv_out, + const float * __restrict__ staging, + const float * __restrict__ dtb, + const float * __restrict__ A, + float * __restrict__ s_base, + const int32_t * __restrict__ s_slot, + float * __restrict__ attn_buf, // (M x 4096) + float l2eps, + float * __restrict__ s_snap, int64_t s_snap_stride) { + constexpr int M = MT; + ggml_cuda_pdl_lc(); ggml_cuda_pdl_sync(); + const int h = blockIdx.x; + const int cg = blockIdx.y; // 0..7 -> cols [cg*16, cg*16+16) + const int tid = threadIdx.x, lane = tid & 31, warp = tid >> 5; + __shared__ float qh[MT][128], kh[MT][128]; + __shared__ float q2s[4], k2s[4]; + const int qk = h & 15; + float g_val[MT], beta[MT]; + +#pragma unroll + for (int t = 0; t < M; ++t) { + const float qv = conv_out[(size_t) t * 8192 + qk * 128 + tid]; + const float kv = conv_out[(size_t) t * 8192 + 2048 + qk * 128 + tid]; + float q2 = qv * qv, k2 = kv * kv; +#pragma unroll + for (int o = 16; o > 0; o >>= 1) { q2 += __shfl_xor_sync(0xffffffffu, q2, o); k2 += __shfl_xor_sync(0xffffffffu, k2, o); } + if (lane == 0) { q2s[warp] = q2; k2s[warp] = k2; } + __syncthreads(); + const float q2t = q2s[0] + q2s[1] + q2s[2] + q2s[3]; + const float k2t = k2s[0] + k2s[1] + k2s[2] + k2s[3]; + qh[t][tid] = qv * rsqrtf(fmaxf(q2t, l2eps * l2eps)); + kh[t][tid] = kv * rsqrtf(fmaxf(k2t, l2eps * l2eps)); + __syncthreads(); + const float * ab = staging + (size_t) M * 12288 + (size_t) t * 64; + g_val[t] = expf(frt_softplus(ab[h] + dtb[h]) * A[h]); + beta[t] = 1.0f / (1.0f + expf(-ab[32 + h])); + } + const float scale = 0.088388347648318447f; + + float * S = s_base + (size_t) (*s_slot) * 524288 + (size_t) h * 16384; + + for (int cc = 0; cc < 4; ++cc) { + const int c = cg * 16 + warp * 4 + cc; + float * Sc = S + (size_t) c * 128; + float s_sh[4]; +#pragma unroll + for (int r = 0; r < 4; ++r) s_sh[r] = Sc[r * 32 + lane]; +#pragma unroll + for (int t = 0; t < M; ++t) { + const float vc = conv_out[(size_t) t * 8192 + 4096 + h * 128 + c]; + float kvr = 0.0f; +#pragma unroll + for (int r = 0; r < 4; ++r) kvr += s_sh[r] * kh[t][r * 32 + lane]; +#pragma unroll + for (int o = 16; o > 0; o >>= 1) kvr += __shfl_xor_sync(0xffffffffu, kvr, o); + const float delta = (vc - g_val[t] * kvr) * beta[t]; + float ap = 0.0f; +#pragma unroll + for (int r = 0; r < 4; ++r) { + const int i = r * 32 + lane; + s_sh[r] = g_val[t] * s_sh[r] + kh[t][i] * delta; + ap += s_sh[r] * qh[t][i]; + } +#pragma unroll + for (int o = 16; o > 0; o >>= 1) ap += __shfl_xor_sync(0xffffffffu, ap, o); + if (lane == 0) attn_buf[(size_t) t * 4096 + h * 128 + c] = ap * scale; + if (MT >= 2) { // spec verify: per-token state snapshot; source slot untouched + float * Dc = s_snap + (size_t) t * s_snap_stride + (size_t) h * 16384 + (size_t) c * 128; +#pragma unroll + for (int r = 0; r < 4; ++r) Dc[r * 32 + lane] = s_sh[r]; + } + } + if (MT == 1) { +#pragma unroll + for (int r = 0; r < 4; ++r) Sc[r * 32 + lane] = s_sh[r]; + } + } +} + +// K2b: per-head gated RMS norm x silu(z) epilogue. Optionally also emits the +// q8_1 quantization of the output (one 32-elem block per warp) so the +// out-proj span can skip its quant launch. +// grid (32 heads, M tokens); z sits per token at staging[t*12288 + 8192 + ...], +// out/out_q8 are token-major (t*4096 floats / t*128 q8 blocks). +__global__ void frt_gdn_epilogue( + const float * __restrict__ attn_buf, + const float * __restrict__ staging, + const float * __restrict__ normw, + float * __restrict__ out, + float rmseps, block_q8_1 * __restrict__ out_q8) { + ggml_cuda_pdl_lc(); ggml_cuda_pdl_sync(); + const int h = blockIdx.x; + const int t = blockIdx.y; + const int tid = threadIdx.x, lane = tid & 31, warp = tid >> 5; + __shared__ float red[4]; + const float xa = attn_buf[(size_t) t * 4096 + h * 128 + tid]; + float s2 = xa * xa; +#pragma unroll + for (int o = 16; o > 0; o >>= 1) s2 += __shfl_xor_sync(0xffffffffu, s2, o); + if (lane == 0) red[warp] = s2; + __syncthreads(); + const float mean2 = (red[0] + red[1] + red[2] + red[3]) / 128.0f; + const float rrms = rsqrtf(mean2 + rmseps); + const float z = staging[(size_t) t * 12288 + 8192 + h * 128 + tid]; + const float v = xa * rrms * normw[tid] * frt_silu(z); + out[(size_t) t * 4096 + h * 128 + tid] = v; + if (out_q8) frt_q8_1_block_g(v, lane, &out_q8[(size_t) t * 128 + h * 4 + warp]); +} + +} // namespace gdn + +// ---- GDN cell surgery: detect span in eval loop and execute our kernels --- + +// Returns number of nodes consumed starting at i (0 = not ours). +bool ggml_cuda_frt_gdn_try_impl(ggml_backend_cuda_context & ctx, ggml_cgraph * cgraph, int i, int * skip_count) { + static int mode = -1; + if (mode < 0) { const char * m = getenv("FRT_GDN_SWAP"); mode = (m && m[0] == '1') ? 1 : 0; } + if (!mode) return false; + if (!frt::frt_regions_load() || !frt::g_reg.inproj_on) return false; + + // anchor: preferred = the layer's leading RMS_NORM (lets us fold the attn + // norm into the quant+ab kernel); fallback = the conv-state GET_ROWS. + ggml_tensor * n0 = cgraph->nodes[i]; + static int nf_mode = -1; + if (nf_mode < 0) { const char * m = getenv("FRT_GDN_NORMFOLD"); nf_mode = (m && m[0] == '1') ? 1 : 0; } + bool norm_anchor = false; + if (nf_mode && n0->op == GGML_OP_RMS_NORM && n0->ne[0] == 2048 && n0->ne[1] == 1) { + norm_anchor = true; // must find attn_norm MUL + GDN members below + } else if (n0->op == GGML_OP_RMS_NORM) { return false; + } else if (n0->op != GGML_OP_GET_ROWS || n0->ne[0] != 24576 || + strncmp(n0->name, "conv_states", 11) != 0) return false; + + // scan forward for the span members + const int LIM = i + 70 < cgraph->n_nodes ? i + 70 : cgraph->n_nodes; + int layer = -1; + const ggml_tensor * qkv_mm = nullptr, * ssm_conv = nullptr, * gdn = nullptr; + const ggml_tensor * l2n = nullptr, * rmsn = nullptr, * normw_mul = nullptr; + const ggml_tensor * add_dtb = nullptr, * mul_A = nullptr; + const ggml_tensor * alpha_mm = nullptr, * beta_mm = nullptr; + const ggml_tensor * norm_mul = nullptr; + const ggml_tensor * gr_r = nullptr, * gr_s = nullptr; + const ggml_tensor * conv_cpy[4] = {nullptr, nullptr, nullptr, nullptr}; + const ggml_tensor * state_cpy = nullptr; + const ggml_tensor * ck_r_gr = nullptr, * ck_r_cpy = nullptr; + const ggml_tensor * ck_s_gr = nullptr, * ck_s_cpy = nullptr; + int n_conv_cpy = 0; + ggml_tensor * final_rs = nullptr; + int end_idx = -1; + for (int j = i; j < LIM; ++j) { + ggml_tensor * n = cgraph->nodes[j]; + if (j == i && norm_anchor) continue; + switch (n->op) { + case GGML_OP_MUL_MAT: + if (n->src[0] && strstr(n->src[0]->name, ".attn_qkv.weight")) { + qkv_mm = n; + sscanf(n->src[0]->name, "blk.%d.", &layer); + } + if (n->src[0] && strstr(n->src[0]->name, ".ssm_alpha.weight")) alpha_mm = n; + if (n->src[0] && strstr(n->src[0]->name, ".ssm_beta.weight")) beta_mm = n; + break; + case GGML_OP_SSM_CONV: ssm_conv = n; break; + case GGML_OP_GATED_DELTA_NET: gdn = n; break; + case GGML_OP_L2_NORM: if (!l2n) l2n = n; break; + case GGML_OP_RMS_NORM: rmsn = n; break; + case GGML_OP_ADD: + if (n->src[1] && strstr(n->src[1]->name, ".ssm_dt.bias")) add_dtb = n; + break; + case GGML_OP_MUL: + if (n->src[1] && strstr(n->src[1]->name, ".ssm_a")) mul_A = n; + if (n->src[1] && strstr(n->src[1]->name, ".ssm_norm.weight")) normw_mul = n; + if (n->src[1] && strstr(n->src[1]->name, ".attn_norm.weight")) norm_mul = n; + break; + case GGML_OP_GET_ROWS: + // first hit = the slot read; a second single-row one is the + // checkpoint save (we replay it ourselves); anything else -> stock. + if (n->ne[1] == 1 && n->ne[0] == 24576) { if (!gr_r) gr_r = n; else if (!ck_r_gr) ck_r_gr = n; else return false; } + else if (n->ne[1] == 1 && n->ne[0] == 524288) { if (!gr_s) gr_s = n; else if (!ck_s_gr) ck_s_gr = n; else return false; } + else if (ggml_nelements(n) != 0 && (n->ne[0] == 24576 || n->ne[0] == 524288)) return false; + break; + case GGML_OP_SCALE: + if (ggml_nelements(n) != 0) return false; // reset machinery active -> fall back + break; + case GGML_OP_RESHAPE: + if (strncmp(n->name, "final_output", 12) == 0) { final_rs = n; end_idx = j; } + break; + case GGML_OP_CPY: + if (ggml_nelements(n) == 0) break; + if (n->src[0] && n->src[0]->op == GGML_OP_VIEW && + n->src[0]->ne[0] == 3 && n->src[0]->ne[1] == 8192) { + if (n_conv_cpy >= 4) return false; + conv_cpy[n_conv_cpy++] = n; + } else if (n->ne[0] == 524288 && n->src[0] && n->src[0]->op == GGML_OP_VIEW && + gdn && n->src[0]->src[0] == gdn) { + if (state_cpy) return false; + state_cpy = n; + } else if (ck_r_gr && n->src[0] == ck_r_gr) { + if (ck_r_cpy) return false; + ck_r_cpy = n; + } else if (ck_s_gr && n->src[0] == ck_s_gr) { + if (ck_s_cpy) return false; + ck_s_cpy = n; + } else { + return false; // unknown non-empty copy -> stock runs this eval + } + break; + case GGML_OP_VIEW: case GGML_OP_TRANSPOSE: + case GGML_OP_CONCAT: case GGML_OP_UNARY: + break; + default: + return false; // unexpected op inside span -> not ours + } + if (end_idx >= 0) break; + } + if (layer < 0 || layer >= 64 || !qkv_mm || !ssm_conv || !gdn || !l2n || !rmsn || + !normw_mul || !add_dtb || !mul_A || !gr_r || !gr_s || !final_rs) return false; + if (norm_anchor && (!norm_mul || qkv_mm->src[1] != norm_mul)) return false; + frt::frt_region & reg = frt::g_reg.regions[0][layer]; + if (reg.N != 12352) return false; + if (gdn->src[0]->ne[0] != 128 || gdn->src[0]->ne[1] != 16 || gdn->src[2]->ne[1] != 32) return false; + const int M = (int) qkv_mm->src[1]->ne[1]; // decode M=1; spec verify M = 1 + n_draft + if (M < 1 || M > 4 || !ggml_is_contiguous(qkv_mm->src[1])) return false; + if (norm_anchor && M != 1) return false; + // spec verify (M>1): the graph stores per-token conv/state snapshots for + // rollback; we must reproduce them (and leave the source slots pristine). + float * conv_snap[4] = {nullptr, nullptr, nullptr, nullptr}; + float * state_snap = nullptr; + int64_t state_snap_stride = 0; // floats between token snapshots + if (M > 1) { + if (n_conv_cpy != M || !state_cpy || state_cpy->ne[2] != M) return false; + if (!state_cpy->src[1] || state_cpy->src[1]->nb[2] % sizeof(float) != 0) return false; + // conv snapshot CPYs in token order = ascending src view offset into conv_input + const ggml_tensor * cc_sorted[4]; + for (int t = 0; t < M; ++t) cc_sorted[t] = conv_cpy[t]; + for (int x = 0; x < M; ++x) + for (int y = x + 1; y < M; ++y) + if ((const char *) cc_sorted[y]->src[0]->data < (const char *) cc_sorted[x]->src[0]->data) { + const ggml_tensor * tmp = cc_sorted[x]; cc_sorted[x] = cc_sorted[y]; cc_sorted[y] = tmp; + } + for (int t = 0; t < M; ++t) conv_snap[t] = (float *) cc_sorted[t]->src[1]->data; + state_snap = (float *) state_cpy->src[1]->data; + state_snap_stride = (int64_t) (state_cpy->src[1]->nb[2] / sizeof(float)); + } + + const float l2eps = *(const float *) l2n->op_params; + const float rmseps = *(const float *) rmsn->op_params; + + if ((ck_r_gr != nullptr) != (ck_r_cpy != nullptr)) return false; + if ((ck_s_gr != nullptr) != (ck_s_cpy != nullptr)) return false; + + cudaStream_t stream = ctx.stream(); + // 0) checkpoint saves (pre-update snapshot of the current slot), if due + if (ck_r_cpy) + ggml_cuda_kernel_launch(gdn::frt_gdn_ckpt_copy, ggml_cuda_kernel_launch_params(dim3(24576 / 256), dim3(256), 0, stream), + (const float *) ck_r_gr->src[0]->data, (const int32_t *) ck_r_gr->src[1]->data, + (int64_t) 24576, (float *) ck_r_cpy->src[1]->data, 24576); + if (ck_s_cpy) + ggml_cuda_kernel_launch(gdn::frt_gdn_ckpt_copy, ggml_cuda_kernel_launch_params(dim3(524288 / 256), dim3(256), 0, stream), + (const float *) ck_s_gr->src[0]->data, (const int32_t *) ck_s_gr->src[1]->data, + (int64_t) 524288, (float *) ck_s_cpy->src[1]->data, 524288); + // 1) act quant (+ F32 a/b gate rows; FRT_GDN_AB=0 falls back to W4A4 rows), + // then fused in_proj GEMV into staging + static int ab_f32 = -1; + if (ab_f32 < 0) { const char * e = getenv("FRT_GDN_AB"); ab_f32 = (e && e[0] == '0') ? 0 : 1; } + const bool ab_ok = ab_f32 && alpha_mm && beta_mm && + alpha_mm->src[0]->type == GGML_TYPE_F32 && beta_mm->src[0]->type == GGML_TYPE_F32 && + alpha_mm->src[1]->data == qkv_mm->src[1]->data && + beta_mm->src[1]->data == qkv_mm->src[1]->data; + if (norm_anchor && !ab_ok) return false; // fallback: plain anchor triggers later + if (norm_anchor) { + const float eps = ((const float *) n0->op_params)[0]; + ggml_cuda_kernel_launch(gdn::frt_gdn_norm_quant_ab, ggml_cuda_kernel_launch_params(dim3(9), dim3(256), 0, stream), + (const float *) n0->src[0]->data, (const float *) norm_mul->src[1]->data, eps, + (uint2 *) frt::g_reg.d_apack, frt::g_reg.d_sfa, + (const float *) alpha_mm->src[0]->data, (const float *) beta_mm->src[0]->data, + frt::g_reg.d_staging); + } else if (ab_ok) { + auto qab = M == 1 ? gdn::frt_gdn_quant_ab<1> : M == 2 ? gdn::frt_gdn_quant_ab<2> : + M == 3 ? gdn::frt_gdn_quant_ab<3> : gdn::frt_gdn_quant_ab<4>; + ggml_cuda_kernel_launch(qab, ggml_cuda_kernel_launch_params(dim3(9), dim3(256), 0, stream), + (const float *) qkv_mm->src[1]->data, (uint2 *) frt::g_reg.d_apack, frt::g_reg.d_sfa, + (const float *) alpha_mm->src[0]->data, (const float *) beta_mm->src[0]->data, + frt::g_reg.d_staging); + } else { + if (M != 1) return false; // W4A4 a/b staging fallback stays M=1 + ggml_cuda_kernel_launch((frt::quant_act_fp4_f32<256, 1>), ggml_cuda_kernel_launch_params(dim3(1), dim3(256), 0, stream), + (const float *) qkv_mm->src[1]->data, (uint2 *) frt::g_reg.d_apack, frt::g_reg.d_sfa, (int) reg.K, (int64_t) 0); + } + const int gemv_n = (ab_ok || norm_anchor) ? 12288 : (int) reg.N; // ab rows owned by K0 when on + frt::frt_ws_launch(frt::g_reg.d_apack, reg.d_packed, frt::g_reg.d_sfa, reg.d_sf, + frt::g_reg.d_staging, reg.alpha, gemv_n, (int) reg.K, M, stream); + // 2) conv + shift + silu + auto conv = M == 1 ? gdn::frt_gdn_conv_silu<1> : M == 2 ? gdn::frt_gdn_conv_silu<2> : + M == 3 ? gdn::frt_gdn_conv_silu<3> : gdn::frt_gdn_conv_silu<4>; + ggml_cuda_kernel_launch(conv, ggml_cuda_kernel_launch_params(dim3(32), dim3(256), 0, stream), + frt::g_reg.d_staging, (const float *) ssm_conv->src[1]->data, + (float *) gr_r->src[0]->data, (const int32_t *) gr_r->src[1]->data, frt::g_reg.d_conv_out, + conv_snap[0], conv_snap[1], conv_snap[2], conv_snap[3]); + // 3) cell: state update in place (256 blocks) + gated-norm epilogue + { + dim3 cg(32, 8); + auto cell = M == 1 ? gdn::frt_gdn_cell_part<1> : M == 2 ? gdn::frt_gdn_cell_part<2> : + M == 3 ? gdn::frt_gdn_cell_part<3> : gdn::frt_gdn_cell_part<4>; + ggml_cuda_kernel_launch(cell, ggml_cuda_kernel_launch_params(cg, dim3(128), 0, stream), + frt::g_reg.d_conv_out, frt::g_reg.d_staging, + (const float *) add_dtb->src[1]->data, (const float *) mul_A->src[1]->data, + (float *) gr_s->src[0]->data, (const int32_t *) gr_s->src[1]->data, + frt::g_reg.d_attn_buf, l2eps, state_snap, state_snap_stride); + ggml_cuda_kernel_launch(gdn::frt_gdn_epilogue, ggml_cuda_kernel_launch_params(dim3(32, (unsigned) M), dim3(128), 0, stream), + frt::g_reg.d_attn_buf, frt::g_reg.d_staging, + (const float *) normw_mul->src[1]->data, + (float *) final_rs->data, rmseps, frt::g_reg.d_outq8); + frt::g_reg.outq8_node = (const void *) final_rs; + } + *skip_count = end_idx - i + 1; + return true; +} + +// ---- MoE glue span takeover (FRT_MOEGLUE_SWAP) ---------------------------- +// combine span: MUL(weights) + chained ADDs over 8 expert outputs +// (stock fuses the ADDs only partially: MUL + ~2 bcast-adds) -> 1 kernel. +// (router span not taken: stock already fuses it into one topk_moe_cuda.) + +namespace moeglue { + +// out[i] = sum_e down[e*hidden + i] * w[e] +__global__ void frt_moe_combine( + const float * __restrict__ down, const float * __restrict__ w, + float * __restrict__ out, int hidden, int nexp) { + ggml_cuda_pdl_lc(); ggml_cuda_pdl_sync(); + const int i = blockIdx.x * 256 + threadIdx.x; + if (i >= hidden) return; + float acc = 0.0f; + for (int e = 0; e < nexp; ++e) acc += down[(size_t) e * hidden + i] * w[e]; + out[i] = acc; +} + +// ---- shared-expert span kernels (FRT_SHEXP_SWAP) ---- +// span: gate/up GEMV + swiglu + down GEMV + gate_inp dot + sigmoid + mul + add +// replaced by: pre (quant act + sigmoid gate) -> gate|up GEMV (kind2) -> +// glu+quant -> down GEMV (kind3) -> finish. + +// block 0: FP4-quantize the f32 act (K=2048); block 1: sigmoid(ginp . act). +__global__ void frt_shexp_pre( + const float * __restrict__ act, + uint2 * __restrict__ dst_packed, uint8_t * __restrict__ dst_sfa, + const float * __restrict__ ginp, float * __restrict__ s_out) { + if (blockIdx.x == 0) { + frt::quant_act_fp4_f32_body<256>(act, dst_packed, dst_sfa, 2048); + return; + } + const int tid = threadIdx.x, lane = tid & 31, warp = tid >> 5; + __shared__ float red[8]; + const float4 * g4 = (const float4 *) ginp; + const float4 * a4 = (const float4 *) act; + float acc = 0.0f; +#pragma unroll 2 + for (int k = tid; k < 512; k += 256) { + const float4 gv = g4[k], av = a4[k]; + acc += gv.x * av.x + gv.y * av.y + gv.z * av.z + gv.w * av.w; + } +#pragma unroll + for (int o = 16; o > 0; o >>= 1) acc += __shfl_xor_sync(0xffffffffu, acc, o); + if (lane == 0) red[warp] = acc; + __syncthreads(); + if (tid == 0) { + float s = 0.0f; +#pragma unroll + for (int w = 0; w < 8; ++w) s += red[w]; + s_out[0] = 1.0f / (1.0f + expf(-s)); + } +} + +// one block: swiglu over staging [gate 512 | up 512] then FP4-quantize (K=512). +__global__ void frt_shexp_glu_quant( + const float * __restrict__ staging, + uint2 * __restrict__ dst_packed, uint8_t * __restrict__ dst_sfa) { + __shared__ float tmp[512]; + for (int i = threadIdx.x; i < 512; i += 256) { + const float g = staging[i]; + tmp[i] = (g / (1.0f + expf(-g))) * staging[512 + i]; + } + __syncthreads(); + frt::quant_act_fp4_f32_body<256>(tmp, dst_packed, dst_sfa, 512); +} + +// out[i] = moe_out[i] + s * down_out[i] +__global__ void frt_shexp_finish( + const float * __restrict__ moe_out, const float * __restrict__ down_out, + const float * __restrict__ s, float * __restrict__ out) { + const int i = blockIdx.x * 256 + threadIdx.x; + if (i < 2048) out[i] = moe_out[i] + s[0] * down_out[i]; +} + +// ---- fused MoE expert segment (FRT_MOEFUSE_SWAP) -------------------------- +// Consumes the whole expert sub-span (gate/up MUL_MAT_ID + GLU + down +// MUL_MAT_ID + weighted combine, ~7 launches) with 2 kernels that read the +// GGUF-native quant blocks via llama.cpp's own vec_dot device functions and +// replicate its q8_1 activation quantization: same math, no repacking, the +// win is launch-count and intermediate-tensor elimination. + +// per-32-elem q8_1 quantization identical to quantize_q8_1 (d=amax/127, s=raw sum) +__device__ __forceinline__ void frt_q8_1_block(float xi, int lane, block_q8_1 * dst) { + float amax = fabsf(xi), sum = xi; +#pragma unroll + for (int o = 16; o > 0; o >>= 1) { + amax = fmaxf(amax, __shfl_xor_sync(0xffffffffu, amax, o)); + sum += __shfl_xor_sync(0xffffffffu, sum, o); + } + const float d = amax / 127.0f; + const int8_t q = amax == 0.0f ? 0 : (int8_t) roundf(xi / d); + dst->qs[lane] = q; + if (lane == 0) dst->ds = make_half2(d, sum); +} + +// one warp computes one full row dot against q8_1 blocks, mmvq iteration order. +template +__device__ __forceinline__ float frt_row_dot(const void * row_base, const block_q8_1 * y, + int blocks_per_row, int lane) { + constexpr int qi = ggml_cuda_type_traits::qi; + constexpr int qk = ggml_cuda_type_traits::qk; + constexpr int vdr = T == GGML_TYPE_Q8_0 ? VDR_Q8_0_Q8_1_MMVQ : + T == GGML_TYPE_Q4_K ? VDR_Q4_K_Q8_1_MMVQ : + T == GGML_TYPE_Q6_K ? VDR_Q6_K_Q8_1_MMVQ : VDR_Q5_K_Q8_1_MMVQ; + float acc = 0.0f; + for (int kbx = lane / (qi / vdr); kbx < blocks_per_row; kbx += vdr * 32 / qi) { + const int kqs = vdr * (lane % (qi / vdr)); + if constexpr (T == GGML_TYPE_Q8_0) acc += vec_dot_q8_0_q8_1(row_base, &y[kbx * (qk / QK8_1)], kbx, kqs); + if constexpr (T == GGML_TYPE_Q4_K) acc += vec_dot_q4_K_q8_1(row_base, &y[kbx * (qk / QK8_1)], kbx, kqs); + if constexpr (T == GGML_TYPE_Q5_K) acc += vec_dot_q5_K_q8_1(row_base, &y[kbx * (qk / QK8_1)], kbx, kqs); + if constexpr (T == GGML_TYPE_Q6_K) acc += vec_dot_q6_K_q8_1(row_base, &y[kbx * (qk / QK8_1)], kbx, kqs); + } +#pragma unroll + for (int o = 16; o > 0; o >>= 1) acc += __shfl_xor_sync(0xffffffffu, acc, o); + return acc; +} + +// K1: grid (n_used, N/32) x 256. Block = 32 rows of one selected expert: +// quantize act (redundant per block, latency-hidden), gate+up row dots, +// swiglu, requantize the 32 glu outputs into one q8_1 block. +// meta = private snapshot of ids + weights: the graph allocator may alias the +// out tensor onto the (dead-after-us) ids/weights buffers, so K2 must not read +// them while it writes out. K0 snapshots them first (stream order protects K0). +struct frt_moe_meta { int32_t ids[4][8]; float w[4][8]; float sig[4]; }; + +// K0: grid 8 (+1 when ginp): q8_1-quantize the shared act (M token rows) + +// snapshot per-token ids/weights (+ shexp gate: sigmoid(ginp . act_t) -> meta->sig[t]). +// ids rows are strided (topk is a view of the argsort output), hence ids_srow. +template +__global__ void frt_moe_quant_meta( + const float * __restrict__ act, int K, int64_t act_srow, + const int32_t * __restrict__ ids, int64_t ids_srow, + const float * __restrict__ wnorm, int64_t w_srow, int n_used, + const float * __restrict__ ginp, + block_q8_1 * __restrict__ act_q8, frt_moe_meta * __restrict__ meta) { + ggml_cuda_pdl_lc(); ggml_cuda_pdl_sync(); + const int lane = threadIdx.x & 31, warp = threadIdx.x >> 5; + const int nb = K / 32; + if (blockIdx.x == 8) { // shexp gate dots (launched only when ginp != null) + __shared__ float red[8]; + const float4 * g4 = (const float4 *) ginp; +#pragma unroll + for (int t = 0; t < MT; ++t) { + const float4 * a4 = (const float4 *) (act + (size_t) t * act_srow); + float acc = 0.0f; + for (int k = threadIdx.x; k < K / 4; k += 256) { + const float4 gv = g4[k], av = a4[k]; + acc += gv.x * av.x + gv.y * av.y + gv.z * av.z + gv.w * av.w; + } +#pragma unroll + for (int o = 16; o > 0; o >>= 1) acc += __shfl_xor_sync(0xffffffffu, acc, o); + if (lane == 0) red[warp] = acc; + __syncthreads(); + if (threadIdx.x == 0) { + float s = 0.0f; +#pragma unroll + for (int q = 0; q < 8; ++q) s += red[q]; + meta->sig[t] = 1.0f / (1.0f + expf(-s)); + } + __syncthreads(); + } + return; + } + if (blockIdx.x == 0 && threadIdx.x < (unsigned) (n_used * MT)) { + const int t = MT == 1 ? 0 : (int) threadIdx.x / n_used; + const int j = MT == 1 ? (int) threadIdx.x : (int) threadIdx.x % n_used; + meta->ids[t][j] = ids[t * ids_srow + j]; + meta->w[t][j] = wnorm[t * w_srow + j]; + } + const int per = (nb + 7) / 8; +#pragma unroll + for (int t = 0; t < MT; ++t) + for (int b = blockIdx.x * per + warp; b < (blockIdx.x + 1) * per && b < nb; b += 8) + frt_q8_1_block(act[(size_t) t * act_srow + b * 32 + lane], lane, &act_q8[t * nb + b]); +} + +// K1: grid (n_used, N/32) x 512 (16 warps, 2 rows each): gate+up row dots +// against the pre-quantized act, swiglu, requantize 32 outputs per block. +template +__global__ void frt_moe_k1( + const char * __restrict__ gate_w, const char * __restrict__ up_w, + size_t expert_stride, size_t row_stride, + const block_q8_1 * __restrict__ act_q8, int K, + const frt_moe_meta * __restrict__ meta, + block_q8_1 * __restrict__ glu_q8, int N, + int n_used, const char * __restrict__ shg_w, const char * __restrict__ shu_w, + size_t sh_row_stride) { + ggml_cuda_pdl_lc(); ggml_cuda_pdl_sync(); + const int lane = threadIdx.x & 31, warp = threadIdx.x >> 5; + __shared__ float glu_f32[32]; + const int row0 = blockIdx.y * 32; + // grid.x covers M x slots pairs: slot in [0,n_used) = routed expert, slot n_used = shexp + const int slots = shg_w ? n_used + 1 : n_used; + const int t = MT == 1 ? 0 : (int) blockIdx.x / slots; + const int slot = MT == 1 ? (int) blockIdx.x : (int) blockIdx.x % slots; + const block_q8_1 * aq = act_q8 + (size_t) t * (K / 32); + block_q8_1 * gout = glu_q8 + ((size_t) t * slots + slot) * (N / 32); + if (shg_w && slot == n_used) { // shared-expert gate|up rows, Q8_0 + const int bpr8 = K / 32; + const int ra = 2 * warp, rb = ra + 1; + const char * ga = shg_w + (size_t)(row0 + ra) * sh_row_stride; + const char * gb = shg_w + (size_t)(row0 + rb) * sh_row_stride; + const char * ua = shu_w + (size_t)(row0 + ra) * sh_row_stride; + const char * ub = shu_w + (size_t)(row0 + rb) * sh_row_stride; + float a0 = 0, a1 = 0, a2 = 0, a3 = 0; + for (int kbx = lane / 4; kbx < bpr8; kbx += 8) { + const int kqs = 2 * (lane % 4); + const block_q8_1 * y = &aq[kbx]; + a0 += vec_dot_q8_0_q8_1(ga, y, kbx, kqs); + a1 += vec_dot_q8_0_q8_1(ua, y, kbx, kqs); + a2 += vec_dot_q8_0_q8_1(gb, y, kbx, kqs); + a3 += vec_dot_q8_0_q8_1(ub, y, kbx, kqs); + } +#pragma unroll + for (int o = 16; o > 0; o >>= 1) { + a0 += __shfl_xor_sync(0xffffffffu, a0, o); + a1 += __shfl_xor_sync(0xffffffffu, a1, o); + a2 += __shfl_xor_sync(0xffffffffu, a2, o); + a3 += __shfl_xor_sync(0xffffffffu, a3, o); + } + if (lane == 0) { + glu_f32[ra] = (a0 / (1.0f + expf(-a0))) * a1; + glu_f32[rb] = (a2 / (1.0f + expf(-a2))) * a3; + } + __syncthreads(); + if (warp == 0) + frt_q8_1_block(glu_f32[lane], lane, &gout[blockIdx.y]); + return; + } + const int e = meta->ids[t][slot]; + const int bpr = K / ggml_cuda_type_traits::qk; + constexpr int qi = ggml_cuda_type_traits::qi; + constexpr int qk = ggml_cuda_type_traits::qk; + constexpr int vdr = TW == GGML_TYPE_Q8_0 ? VDR_Q8_0_Q8_1_MMVQ : + TW == GGML_TYPE_Q4_K ? VDR_Q4_K_Q8_1_MMVQ : VDR_Q5_K_Q8_1_MMVQ; + // warp owns rows {2*warp, 2*warp+1}; 4 independent dot accumulators per + // kbx iteration (2 rows x gate/up) so the weight loads overlap 4-wide. + { + const int ra = 2 * warp, rb = ra + 1; + const size_t ebase = (size_t) e * expert_stride; + const char * ga = gate_w + ebase + (size_t)(row0 + ra) * row_stride; + const char * gb = gate_w + ebase + (size_t)(row0 + rb) * row_stride; + const char * ua = up_w + ebase + (size_t)(row0 + ra) * row_stride; + const char * ub = up_w + ebase + (size_t)(row0 + rb) * row_stride; + float a0 = 0, a1 = 0, a2 = 0, a3 = 0; + for (int kbx = lane / (qi / vdr); kbx < bpr; kbx += vdr * 32 / qi) { + const int kqs = vdr * (lane % (qi / vdr)); + const block_q8_1 * y = &aq[kbx * (qk / QK8_1)]; + if constexpr (TW == GGML_TYPE_Q8_0) { + a0 += vec_dot_q8_0_q8_1(ga, y, kbx, kqs); + a1 += vec_dot_q8_0_q8_1(ua, y, kbx, kqs); + a2 += vec_dot_q8_0_q8_1(gb, y, kbx, kqs); + a3 += vec_dot_q8_0_q8_1(ub, y, kbx, kqs); + } else if constexpr (TW == GGML_TYPE_Q4_K) { + a0 += vec_dot_q4_K_q8_1(ga, y, kbx, kqs); + a1 += vec_dot_q4_K_q8_1(ua, y, kbx, kqs); + a2 += vec_dot_q4_K_q8_1(gb, y, kbx, kqs); + a3 += vec_dot_q4_K_q8_1(ub, y, kbx, kqs); + } else { + a0 += vec_dot_q5_K_q8_1(ga, y, kbx, kqs); + a1 += vec_dot_q5_K_q8_1(ua, y, kbx, kqs); + a2 += vec_dot_q5_K_q8_1(gb, y, kbx, kqs); + a3 += vec_dot_q5_K_q8_1(ub, y, kbx, kqs); + } + } +#pragma unroll + for (int o = 16; o > 0; o >>= 1) { + a0 += __shfl_xor_sync(0xffffffffu, a0, o); + a1 += __shfl_xor_sync(0xffffffffu, a1, o); + a2 += __shfl_xor_sync(0xffffffffu, a2, o); + a3 += __shfl_xor_sync(0xffffffffu, a3, o); + } + if (lane == 0) { + glu_f32[ra] = (a0 / (1.0f + expf(-a0))) * a1; + glu_f32[rb] = (a2 / (1.0f + expf(-a2))) * a3; + } + } + __syncthreads(); + if (warp == 0) + frt_q8_1_block(glu_f32[lane], lane, &gout[blockIdx.y]); +} + +// K2: warp per output column: 8 expert row-dots over the staged glu q8_1 +// vectors, weighted sum, single write. grid hidden/8 x 256. +template +__global__ void frt_moe_k2( + const char * __restrict__ down_w, size_t expert_stride, size_t row_stride, + const block_q8_1 * __restrict__ glu_q8, + const frt_moe_meta * __restrict__ meta, + float * __restrict__ out, int Kd, int n_used, int hidden, + const char * __restrict__ shd_w, size_t shd_row_stride, + const float * __restrict__ resid) { + ggml_cuda_pdl_lc(); ggml_cuda_pdl_sync(); + const int lane = threadIdx.x & 31, warp = threadIdx.x >> 5; + const int col = blockIdx.x * 8 + warp; + if (col >= hidden) return; + const int t = MT == 1 ? 0 : (int) blockIdx.y; // token row (grid.y = M) + const int slots = shd_w ? n_used + 1 : n_used; // must mirror K1's glu layout + const block_q8_1 * gq = glu_q8 + (size_t) t * slots * (Kd / 32); + const int bpr = Kd / ggml_cuda_type_traits::qk; + const int kdb = Kd / 32; + float res = 0.0f; + for (int e = 0; e < n_used; ++e) { + const char * row = down_w + (size_t) meta->ids[t][e] * expert_stride + (size_t) col * row_stride; + res += frt_row_dot(row, &gq[e * kdb], bpr, lane) * meta->w[t][e]; + } + if (shd_w) { // shared expert: sigmoid-gated Q8_0 down + const char * row = shd_w + (size_t) col * shd_row_stride; + res += frt_row_dot(row, &gq[n_used * kdb], Kd / 32, lane) * meta->sig[t]; + } + const size_t oi = (size_t) t * hidden + col; + if (lane == 0) out[oi] = resid ? res + resid[oi] : res; +} + +// attn output gate: out[i] = fa[i] * sigmoid(gate[i]) (replaces CONT+UNARY+MUL) +__global__ void frt_attn_gate( + const float * __restrict__ fa, const float * __restrict__ gate, + float * __restrict__ out, int n) { + ggml_cuda_pdl_lc(); ggml_cuda_pdl_sync(); + const int i = blockIdx.x * 256 + threadIdx.x; + if (i < n) out[i] = fa[i] * (1.0f / (1.0f + expf(-gate[i]))); +} + +// fused router: gate_inp logits GEMV + exact replication of their topk_moe +// (softmax over n_exp, iterative top-k with lower-index tie-break, clamp-norm). +// grid 8 x 256; last finishing block runs the warp top-k (self-resetting counter). +__global__ void frt_router_fused( + const float * __restrict__ gate_w, // [256, 2048] f32 K-contig + const float * __restrict__ act, int K, + int n_used, float clamp_val, + float * __restrict__ logits_buf, unsigned int * __restrict__ counter, + int32_t * __restrict__ ids_out, float * __restrict__ w_out, + int64_t ids_srow, int64_t w_srow) { + ggml_cuda_pdl_lc(); ggml_cuda_pdl_sync(); + const int lane = threadIdx.x & 31, warp = threadIdx.x >> 5; + const int t = blockIdx.y; // token row (grid.y = M) + act += (size_t) t * K; + logits_buf += (size_t) t * 256; + counter += t; + ids_out += (size_t) t * ids_srow; + w_out += (size_t) t * w_srow; + { // phase 1: warp per row, grid 32 x 8 warps = 256 rows + const int row = blockIdx.x * 8 + warp; + const float4 * A = (const float4 *) act; + const float4 * W = (const float4 *) (gate_w + (size_t) row * K); + float a0 = 0, a1 = 0; + for (int k = lane; k < K / 4; k += 64) { + const float4 av0 = A[k], w0 = W[k]; + const float4 av1 = A[k + 32], w1 = W[k + 32]; + a0 += w0.x * av0.x + w0.y * av0.y + w0.z * av0.z + w0.w * av0.w; + a1 += w1.x * av1.x + w1.y * av1.y + w1.z * av1.z + w1.w * av1.w; + } + float acc = a0 + a1; +#pragma unroll + for (int o = 16; o > 0; o >>= 1) acc += __shfl_xor_sync(0xffffffffu, acc, o); + if (lane == 0) logits_buf[row] = acc; + } + __shared__ bool amlast; + __threadfence(); + __syncthreads(); + if (threadIdx.x == 0) amlast = (atomicInc(counter, gridDim.x - 1) == gridDim.x - 1); + __syncthreads(); + if (!amlast || warp != 0) return; + + // phase 2: single warp, exact topk_moe semantics (no bias, softmax, with_norm) + float wt[8]; +#pragma unroll + for (int i = 0; i < 8; ++i) wt[i] = logits_buf[lane + i * 32]; + float mx = -INFINITY; +#pragma unroll + for (int i = 0; i < 8; ++i) mx = fmaxf(mx, wt[i]); + mx = warp_reduce_max(mx); + float sum = 0.0f; +#pragma unroll + for (int i = 0; i < 8; ++i) { const float v = expf(wt[i] - mx); wt[i] = v; sum += v; } + sum = warp_reduce_sum(sum); + const float inv = 1.0f / sum; +#pragma unroll + for (int i = 0; i < 8; ++i) { + wt[i] *= inv; + if (__isnanf(wt[i])) wt[i] = -FLT_MAX; + } + float wt_sum = 0.0f, outw = 0.0f; + for (int k = 0; k < n_used; ++k) { + float max_val = wt[0]; + int max_expert = lane; +#pragma unroll + for (int i = 1; i < 8; ++i) { + const int e = lane + i * 32; + if (wt[i] > max_val) { max_val = wt[i]; max_expert = e; } + } +#pragma unroll + for (int mask = 16; mask > 0; mask >>= 1) { + const float val = __shfl_xor_sync(0xffffffffu, max_val, mask, 32); + const int e = __shfl_xor_sync(0xffffffffu, max_expert, mask, 32); + if (val > max_val || (val == max_val && e < max_expert)) { max_val = val; max_expert = e; } + } + if ((max_expert & 31) == lane) { + wt[max_expert / 32] = -INFINITY; + ids_out[k] = max_expert; + wt_sum += max_val; + } + if (k == lane) outw = max_val; + } + wt_sum = warp_reduce_sum(wt_sum); + wt_sum = fmaxf(wt_sum, clamp_val); + const float invs = 1.0f / wt_sum; + if (lane < n_used) w_out[lane] = outw * invs; +} + +// out-proj (ssm_out / attn_output) native-format GEMV with fused residual add. +// warp per output column, grid.y = token row; reads the q8_1-staged act +// (K/32 blocks per token), writes out = dot + residual. +template +__global__ void frt_outproj_gemv( + const char * __restrict__ w, size_t row_stride, + const block_q8_1 * __restrict__ y, + const float * __restrict__ residual, + float * __restrict__ out, int N, int K) { + ggml_cuda_pdl_lc(); ggml_cuda_pdl_sync(); + const int lane = threadIdx.x & 31, warp = threadIdx.x >> 5; + const int col = blockIdx.x * 8 + warp; + if (col >= N) return; + const int t = blockIdx.y; + const float acc = frt_row_dot(w + (size_t) col * row_stride, y + (size_t) t * (K / 32), + K / ggml_cuda_type_traits::qk, lane); + const size_t oi = (size_t) t * N + col; + if (lane == 0) out[oi] = acc + residual[oi]; +} + +} // namespace moeglue + +// attn-gate glue span: CONT(gate view) -> UNARY sigmoid -> MUL -> 1 kernel. +static bool frt_attn_gate_try(ggml_backend_cuda_context & ctx, ggml_cgraph * cgraph, int i, int * skip_count) { + static int mode = -1; + if (mode < 0) { const char * m = getenv("FRT_ATTNGATE_SWAP"); mode = (m && m[0] == '1') ? 1 : 0; } + if (!mode) return false; + ggml_tensor * n0 = cgraph->nodes[i]; + if (n0->op != GGML_OP_CONT || strncmp(n0->name, "gate_reshaped", 13) != 0) return false; + if (!ggml_is_contiguous(n0->src[0])) return false; // gate view must be flat + const ggml_tensor * sig = nullptr; + ggml_tensor * gmul = nullptr; + int end_idx = -1; + const int LIM = i + 4 < cgraph->n_nodes ? i + 4 : cgraph->n_nodes; + for (int j = i + 1; j < LIM; ++j) { + ggml_tensor * n = cgraph->nodes[j]; + if (n->op == GGML_OP_UNARY) { + if (ggml_get_unary_op(n) != GGML_UNARY_OP_SIGMOID || n->src[0] != n0) return false; + sig = n; + } else if (n->op == GGML_OP_MUL && sig) { + if (n->src[1] == sig) { gmul = n; end_idx = j; } + else if (n->src[0] == sig) { gmul = n; end_idx = j; } + break; + } else if (n->op == GGML_OP_RESHAPE || n->op == GGML_OP_VIEW) { + continue; + } else return false; + } + if (!sig || !gmul || end_idx < 0) return false; + const ggml_tensor * fa = gmul->src[0] == sig ? gmul->src[1] : gmul->src[0]; + const int64_t n = gmul->ne[0]; + if (n % 256 != 0 || gmul->ne[1] != 1) return false; + if (fa->type != GGML_TYPE_F32 || !ggml_is_contiguous(fa)) return false; + // out may alias the (dead-after-us) gate source at a different index: check overlap + const char * gsrc = (const char *) n0->src[0]->data; + const char * outp = (const char *) gmul->data; + if (outp < gsrc + n * 4 && gsrc < outp + n * 4 && outp != (const char *) fa->data) { + if (outp != gsrc) return false; // partial overlap -> unsafe, fall back + } + ggml_cuda_kernel_launch(moeglue::frt_attn_gate, ggml_cuda_kernel_launch_params(dim3((unsigned) (n / 256)), dim3(256), 0, ctx.stream()), + (const float *) fa->data, (const float *) n0->src[0]->data, + (float *) gmul->data, (int) n); + *skip_count = end_idx - i + 1; + return true; +} + +// fused router span: [MUL_MAT gate_inp logits] -> SOFT_MAX -> ARGSORT -> ... -> DIV +// (their path: mmv_f + fused topk_moe = 2 launches) -> 1 kernel. +static bool frt_router_span_try(ggml_backend_cuda_context & ctx, ggml_cgraph * cgraph, int i, int * skip_count) { + // mode: unset -> M=1 off (their fused topk_moe wins there, judged -3), M>=2 on + // (their fusion is M=1-only; the M=2 fallback is the unfused argsort chain). + // "1" -> on for all M; "0" -> off entirely. + static int mode = -1; + if (mode < 0) { const char * m = getenv("FRT_ROUTER_SWAP"); mode = m ? ((m[0] == '1') ? 2 : 0) : 1; } + if (!mode) return false; + ggml_tensor * n0 = cgraph->nodes[i]; + if (n0->op != GGML_OP_MUL_MAT || !n0->src[0] || + !strstr(n0->src[0]->name, ".ffn_gate_inp.weight")) return false; + const ggml_tensor * w = n0->src[0]; + const ggml_tensor * act = n0->src[1]; + if (w->type != GGML_TYPE_F32 || w->ne[0] != 2048 || w->ne[1] != 256) return false; + if (act->type != GGML_TYPE_F32 || !ggml_is_contiguous(act)) return false; + const int M = (int) act->ne[1]; + if (M < 1 || M > 4 || act->ne[2] != 1) return false; + + const ggml_tensor * sm = nullptr, * argsort = nullptr, * clampn = nullptr; + ggml_tensor * divn = nullptr; + int end_idx = -1; + const int LIM = i + 12 < cgraph->n_nodes ? i + 12 : cgraph->n_nodes; + for (int j = i + 1; j < LIM; ++j) { + ggml_tensor * n = cgraph->nodes[j]; + switch (n->op) { + case GGML_OP_SOFT_MAX: + if (n->src[0] != n0 || n->src[1] != nullptr) return false; + if (((const float *) n->op_params)[0] != 1.0f || + ((const float *) n->op_params)[1] != 0.0f) return false; + sm = n; + break; + case GGML_OP_ARGSORT: argsort = n; break; + case GGML_OP_CLAMP: clampn = n; break; + case GGML_OP_DIV: + if (strstr(n->name, "ffn_moe_weights_norm")) { divn = n; end_idx = j; } + break; + case GGML_OP_RESHAPE: case GGML_OP_VIEW: + case GGML_OP_GET_ROWS: case GGML_OP_SUM_ROWS: + break; + default: return false; + } + if (end_idx >= 0) break; + } + if (!sm || !argsort || !clampn || !divn || argsort->ne[0] != 256 || divn->ne[0] > 8) return false; + if (mode == 1 && M == 1) return false; // default policy: M=1 stays on their fused path + if (argsort->ne[1] != M || divn->ne[1] != M) return false; + const int n_used = (int) divn->ne[0]; + const float cmin = ((const float *) clampn->op_params)[0]; + + static float * d_logits = nullptr; + static unsigned int * d_counter = nullptr; + if (!d_logits) { + CUDA_CHECK(cudaMalloc(&d_logits, 4 * 256 * sizeof(float))); + CUDA_CHECK(cudaMalloc(&d_counter, 4 * sizeof(unsigned int))); + CUDA_CHECK(cudaMemset(d_counter, 0, 4 * sizeof(unsigned int))); + } + ggml_cuda_kernel_launch(moeglue::frt_router_fused, ggml_cuda_kernel_launch_params(dim3(32, (unsigned) M), dim3(256), 0, ctx.stream()), + (const float *) w->data, (const float *) act->data, 2048, + n_used, cmin, d_logits, d_counter, + (int32_t *) argsort->data, (float *) divn->data, + (int64_t) (argsort->nb[1] / sizeof(int32_t)), (int64_t) (divn->nb[1] / sizeof(float))); + *skip_count = end_idx - i + 1; + return true; +} + +// fused MoE expert segment: anchor = MUL_MAT_ID on ffn_gate_exps.weight. +static bool frt_moefuse_try(ggml_backend_cuda_context & ctx, ggml_cgraph * cgraph, int i, int * skip_count) { + static int mode = -1; + if (mode < 0) { const char * m = getenv("FRT_MOEFUSE_SWAP"); mode = (m && m[0] == '1') ? 1 : 0; } + if (!mode) return false; + ggml_tensor * gate_id = cgraph->nodes[i]; + if (gate_id->op != GGML_OP_MUL_MAT_ID || !gate_id->src[0] || + !strstr(gate_id->src[0]->name, ".ffn_gate_exps.weight")) return false; + static int stats_on = -1; + if (stats_on < 0) { const char * e = getenv("FRT_STATS"); stats_on = (e && e[0] == '1') ? 1 : 0; } + static long anchors = 0; static long fires_ok = 0; + if (stats_on && (++anchors & 0x3FF) == 1) fprintf(stderr, "frt-stats moefuse anchors=%ld fires=%ld\n", anchors, fires_ok); + static int dbg = -1; + if (dbg < 0) { const char * d = getenv("FRT_MOEFUSE_DBG"); dbg = d ? atoi(d) : 0; } + struct frt_dbg_guard { + bool armed; ggml_cgraph * g; int i; + ~frt_dbg_guard() { + if (!armed) return; + fprintf(stderr, "frt-moefuse-dbg REJECT at anchor %d:\n", i); + for (int j = i; j < i + 30 && j < g->n_nodes; ++j) { + const ggml_tensor * n = g->nodes[j]; + fprintf(stderr, " %4d %-14s %-26s ne=[%lld,%lld,%lld]\n", j, ggml_op_name(n->op), n->name, + (long long) n->ne[0], (long long) n->ne[1], (long long) n->ne[2]); + } + } + } dbg_guard{false, cgraph, i}; + if (dbg > 0) { dbg_guard.armed = true; --dbg; } + + const ggml_tensor * gw = gate_id->src[0]; + const ggml_tensor * act = gate_id->src[1]; + const ggml_tensor * ids = gate_id->src[2]; + const int64_t K = gw->ne[0], N = gw->ne[1]; + const int n_used = (int) gate_id->ne[1]; + if (gw->type != GGML_TYPE_Q4_K && gw->type != GGML_TYPE_Q8_0) return false; + if (K != 2048 || N % 32 != 0 || N > 2048) return false; + if (act->type != GGML_TYPE_F32 || act->ne[1] != 1 || !ggml_is_contiguous(act)) return false; + const int M = (int) act->ne[2]; // token-batch width (spec verify runs M = 1 + n_draft, default 4) + if (M < 1 || M > 4) return false; + if (n_used < 1 || n_used > 8 || gate_id->ne[2] != M) return false; + if (ids->ne[1] != M) return false; + + static int sh_mode = -1; + if (sh_mode < 0) { const char * m = getenv("FRT_MOEFUSE_SHEXP"); sh_mode = (m && m[0] == '1') ? 1 : 0; } + const ggml_tensor * up_id = nullptr, * glu = nullptr, * down_id = nullptr; + const ggml_tensor * wmul = nullptr; + const ggml_tensor * shg = nullptr, * shu = nullptr, * shd = nullptr, * ginp = nullptr; + const ggml_tensor * sh_glu = nullptr, * sig = nullptr, * gmul = nullptr; + ggml_tensor * out_add = nullptr, * moe_add = nullptr; + int end_idx = -1; + const int LIM = i + 2 * n_used + 22 < cgraph->n_nodes ? i + 2 * n_used + 22 : cgraph->n_nodes; + for (int j = i + 1; j < LIM; ++j) { + ggml_tensor * n = cgraph->nodes[j]; + switch (n->op) { + case GGML_OP_MUL_MAT_ID: + if (n->src[0] && strstr(n->src[0]->name, ".ffn_up_exps.weight")) up_id = n; + else if (n->src[0] && strstr(n->src[0]->name, ".ffn_down_exps.weight")) down_id = n; + else return false; + break; + case GGML_OP_MUL_MAT: + if (!sh_mode || !moe_add) return false; // shexp mul_mats only after moe_out + if (n->src[0] && strstr(n->src[0]->name, ".ffn_gate_shexp.weight")) shg = n; + else if (n->src[0] && strstr(n->src[0]->name, ".ffn_up_shexp.weight")) shu = n; + else if (n->src[0] && strstr(n->src[0]->name, ".ffn_down_shexp.weight")) shd = n; + else if (n->src[0] && strstr(n->src[0]->name, ".ffn_gate_inp_shexp.weight")) ginp = n; + else return false; + break; + case GGML_OP_GLU: + if (ggml_get_glu_op(n) != GGML_GLU_OP_SWIGLU) return false; + if (n->src[0] == gate_id) glu = n; + else if (shg && n->src[0] == shg) sh_glu = n; + else return false; + break; + case GGML_OP_UNARY: + if (ggml_get_unary_op(n) != GGML_UNARY_OP_SIGMOID || !ginp || n->src[0] != ginp) return false; + sig = n; + break; + case GGML_OP_MUL: + if (!moe_add) wmul = n; + else gmul = n; + break; + case GGML_OP_ADD: + if (strstr(n->name, "ffn_moe_out")) { // also matches mtp_ffn_moe_out (draft) + moe_add = n; + if (!sh_mode) { out_add = n; end_idx = j; } + } else if (sh_mode && strstr(n->name, "ffn_out")) { // also mtp_ffn_out + out_add = n; end_idx = j; + } + break; + case GGML_OP_VIEW: case GGML_OP_RESHAPE: + break; + default: + return false; + } + if (end_idx >= 0) break; + } + if (!up_id || !glu || !down_id || !wmul || !out_add || !moe_add) return false; + // shexp wiring (only in sh_mode) + const bool sh_ok = sh_mode && shg && shu && shd && ginp && sh_glu && sig && gmul && + shg->src[0]->type == GGML_TYPE_Q8_0 && shu->src[0]->type == GGML_TYPE_Q8_0 && + shd->src[0]->type == GGML_TYPE_Q8_0 && ginp->src[0]->type == GGML_TYPE_F32 && + shg->src[0]->ne[0] == K && shg->src[0]->ne[1] == N && + shu->src[0]->ne[0] == K && shu->src[0]->ne[1] == N && + shd->src[0]->ne[0] == N && shd->src[0]->ne[1] == K && + shg->src[1]->data == act->data && shu->src[1]->data == act->data && + ginp->src[1]->data == act->data && + sh_glu->src[1] == shu && shd->src[1] == sh_glu && + ((gmul->src[0] == shd && gmul->src[1] == sig) || (gmul->src[0] == sig && gmul->src[1] == shd)) && + ((out_add->src[0] == moe_add && out_add->src[1] == gmul) || + (out_add->src[0] == gmul && out_add->src[1] == moe_add)); + if (sh_mode && !sh_ok) return false; + // fold the layer residual add too when the very next real node is l_out = ffn_out + resid + const ggml_tensor * lresid = nullptr; + if (sh_ok && end_idx + 1 < cgraph->n_nodes) { + ggml_tensor * nl = cgraph->nodes[end_idx + 1]; + if (nl->op == GGML_OP_ADD && + (strncmp(nl->name, "l_out", 5) == 0 || strncmp(nl->name, "mtp_post_ffn", 12) == 0)) { + if (nl->src[0] == out_add && nl->src[1] != out_add) { lresid = nl->src[1]; out_add = nl; end_idx += 1; } + else if (nl->src[1] == out_add && nl->src[0] != out_add) { lresid = nl->src[0]; out_add = nl; end_idx += 1; } + } + } + // wiring + if (up_id->src[0]->type != gw->type || up_id->src[0]->ne[0] != K || up_id->src[0]->ne[1] != N) return false; + if (up_id->src[1]->data != act->data || up_id->src[2]->data != ids->data) return false; + if (glu->src[0] != gate_id || glu->src[1] != up_id) return false; + if (down_id->src[1] != glu || down_id->src[2]->data != ids->data) return false; + const ggml_tensor * dw = down_id->src[0]; + if (dw->ne[0] != N || dw->ne[1] != K) return false; + if (dw->type != GGML_TYPE_Q5_K && dw->type != GGML_TYPE_Q4_K && dw->type != GGML_TYPE_Q8_0 && + dw->type != GGML_TYPE_Q6_K) return false; + if (wmul->src[0] != down_id || wmul->src[1]->type != GGML_TYPE_F32) return false; + const ggml_tensor * wnorm = wmul->src[1]; + + static block_q8_1 * d_gluq8 = nullptr; + static block_q8_1 * d_actq8 = nullptr; + static moeglue::frt_moe_meta * d_meta = nullptr; + if (!d_gluq8) { + CUDA_CHECK(cudaMalloc(&d_gluq8, 4 * 9 * 64 * sizeof(block_q8_1))); + CUDA_CHECK(cudaMalloc(&d_actq8, 4 * 64 * sizeof(block_q8_1))); + CUDA_CHECK(cudaMalloc(&d_meta, sizeof(moeglue::frt_moe_meta))); + } + + + cudaStream_t stream = ctx.stream(); + + // FRT_MOEFUSE_SELFTEST=1: before the real launches (inputs still pristine — + // the out write may alias them), replay this span's input as both an M=1 run + // and a duplicated-token M=2 run on scratch buffers; all three out rows must + // be bit-identical (kills any (t, stride) indexing bug in the M=2 path). + static int selftest = -1; + if (selftest < 0) { const char * s = getenv("FRT_MOEFUSE_SELFTEST"); selftest = (s && s[0]=='1') ? 1 : 0; } + if (selftest == 1 && M == 1 && dw->type == GGML_TYPE_Q5_K) { + selftest = 2; // once + float * s_act; int32_t * s_ids; float * s_w; float * s_out1; float * s_out2; float * s_resid; + block_q8_1 * s_actq8; block_q8_1 * s_gluq8; moeglue::frt_moe_meta * s_meta1; moeglue::frt_moe_meta * s_meta2; + CUDA_CHECK(cudaMalloc(&s_act, 2 * K * sizeof(float))); + CUDA_CHECK(cudaMalloc(&s_ids, 2 * 8 * sizeof(int32_t))); + CUDA_CHECK(cudaMalloc(&s_w, 2 * 8 * sizeof(float))); + CUDA_CHECK(cudaMalloc(&s_out1, K * sizeof(float))); + CUDA_CHECK(cudaMalloc(&s_out2, 2 * K * sizeof(float))); + CUDA_CHECK(cudaMalloc(&s_resid, 2 * K * sizeof(float))); + CUDA_CHECK(cudaMalloc(&s_actq8, 2 * 64 * sizeof(block_q8_1))); + CUDA_CHECK(cudaMalloc(&s_gluq8, 2 * 9 * 64 * sizeof(block_q8_1))); + CUDA_CHECK(cudaMalloc(&s_meta1, sizeof(moeglue::frt_moe_meta))); + CUDA_CHECK(cudaMalloc(&s_meta2, sizeof(moeglue::frt_moe_meta))); + for (int t = 0; t < 2; ++t) { + cudaMemcpyAsync(s_act + t * K, act->data, K * 4, cudaMemcpyDeviceToDevice, stream); + cudaMemcpyAsync(s_ids + t * 8, ids->data, n_used * 4, cudaMemcpyDeviceToDevice, stream); + cudaMemcpyAsync(s_w + t * 8, wnorm->data, n_used * 4, cudaMemcpyDeviceToDevice, stream); + if (lresid) cudaMemcpyAsync(s_resid + t * K, lresid->data, K * 4, cudaMemcpyDeviceToDevice, stream); + } + const float * s_ginp = sh_ok ? (const float *) ginp->src[0]->data : (const float *) nullptr; + const char * s_shg = sh_ok ? (const char *) shg->src[0]->data : (const char *) nullptr; + const char * s_shu = sh_ok ? (const char *) shu->src[0]->data : (const char *) nullptr; + const char * s_shd = sh_ok ? (const char *) shd->src[0]->data : (const char *) nullptr; + const size_t s_shs = sh_ok ? shg->src[0]->nb[1] : (size_t) 0; + const size_t s_shds = sh_ok ? shd->src[0]->nb[1] : (size_t) 0; + for (int m = 1; m <= 2; ++m) { + block_q8_1 * aq = s_actq8; moeglue::frt_moe_meta * mt = (m == 1) ? s_meta1 : s_meta2; + float * so = (m == 1) ? s_out1 : s_out2; + dim3 tg0(sh_ok ? 9 : 8), tg1((unsigned) (m * (sh_ok ? n_used + 1 : n_used)), (unsigned) (N / 32)), tg2((unsigned) ((K + 7) / 8), (unsigned) m); + if (m == 1) { + moeglue::frt_moe_quant_meta<1><<>>( + s_act, (int) K, (int64_t) K, s_ids, (int64_t) 8, s_w, (int64_t) 8, n_used, + s_ginp, aq, mt); + moeglue::frt_moe_k1<<>>( + (const char *) gw->data, (const char *) up_id->src[0]->data, gw->nb[2], gw->nb[1], + aq, (int) K, mt, s_gluq8, (int) N, n_used, s_shg, s_shu, s_shs); + moeglue::frt_moe_k2<<>>( + (const char *) dw->data, dw->nb[2], dw->nb[1], s_gluq8, mt, + so, (int) N, n_used, (int) K, s_shd, s_shds, + lresid ? s_resid : (const float *) nullptr); + } else { + moeglue::frt_moe_quant_meta<2><<>>( + s_act, (int) K, (int64_t) K, s_ids, (int64_t) 8, s_w, (int64_t) 8, n_used, + s_ginp, aq, mt); + moeglue::frt_moe_k1<<>>( + (const char *) gw->data, (const char *) up_id->src[0]->data, gw->nb[2], gw->nb[1], + aq, (int) K, mt, s_gluq8, (int) N, n_used, s_shg, s_shu, s_shs); + moeglue::frt_moe_k2<<>>( + (const char *) dw->data, dw->nb[2], dw->nb[1], s_gluq8, mt, + so, (int) N, n_used, (int) K, s_shd, s_shds, + lresid ? s_resid : (const float *) nullptr); + } + } + cudaStreamSynchronize(stream); + std::vector h_ref(K), h0(K), h1(K); + cudaMemcpy(h_ref.data(), s_out1, K * 4, cudaMemcpyDeviceToHost); + cudaMemcpy(h0.data(), s_out2, K * 4, cudaMemcpyDeviceToHost); + cudaMemcpy(h1.data(), s_out2 + K, K * 4, cudaMemcpyDeviceToHost); + int bad0 = 0, bad1 = 0; + for (int c = 0; c < (int) K; ++c) { + if (h0[c] != h_ref[c]) ++bad0; + if (h1[c] != h_ref[c]) ++bad1; + } + fprintf(stderr, "frt-moefuse-selftest (%s, sh=%d, lresid=%d): m2row0 vs m1 mismatch %d/%d, m2row1 vs m1 mismatch %d/%d %s\n", + dw->name, (int) sh_ok, (int) (lresid != nullptr), bad0, (int) K, bad1, (int) K, + (bad0 == 0 && bad1 == 0) ? "PASS" : "FAIL"); + cudaFree(s_act); cudaFree(s_ids); cudaFree(s_w); cudaFree(s_out1); cudaFree(s_out2); cudaFree(s_resid); + cudaFree(s_actq8); cudaFree(s_gluq8); cudaFree(s_meta1); cudaFree(s_meta2); + } + { + FRT_M_DISPATCH(M, ggml_cuda_kernel_launch(moeglue::frt_moe_quant_meta, ggml_cuda_kernel_launch_params(dim3(sh_ok ? 9 : 8), dim3(256), 0, stream), + (const float *) act->data, (int) K, (int64_t) (act->nb[2] / sizeof(float)), + (const int32_t *) ids->data, (int64_t) (ids->nb[1] / sizeof(int32_t)), + (const float *) wnorm->data, (int64_t) (wnorm->nb[2] / sizeof(float)), n_used, + sh_ok ? (const float *) ginp->src[0]->data : (const float *) nullptr, + d_actq8, d_meta)); + dim3 g1((unsigned) (M * (sh_ok ? n_used + 1 : n_used)), (unsigned) (N / 32)); + if (gw->type == GGML_TYPE_Q4_K) { + FRT_M_DISPATCH(M, ggml_cuda_kernel_launch((moeglue::frt_moe_k1), ggml_cuda_kernel_launch_params(g1, dim3(512), 0, stream), + (const char *) gw->data, (const char *) up_id->src[0]->data, + gw->nb[2], gw->nb[1], + d_actq8, (int) K, d_meta, d_gluq8, (int) N, + n_used, + sh_ok ? (const char *) shg->src[0]->data : (const char *) nullptr, + sh_ok ? (const char *) shu->src[0]->data : (const char *) nullptr, + sh_ok ? shg->src[0]->nb[1] : (size_t) 0)); + } else { // Q8_0 experts (draft MTP layer) + FRT_M_DISPATCH(M, ggml_cuda_kernel_launch((moeglue::frt_moe_k1), ggml_cuda_kernel_launch_params(g1, dim3(512), 0, stream), + (const char *) gw->data, (const char *) up_id->src[0]->data, + gw->nb[2], gw->nb[1], + d_actq8, (int) K, d_meta, d_gluq8, (int) N, + n_used, + sh_ok ? (const char *) shg->src[0]->data : (const char *) nullptr, + sh_ok ? (const char *) shu->src[0]->data : (const char *) nullptr, + sh_ok ? shg->src[0]->nb[1] : (size_t) 0)); + } + dim3 g2((unsigned) ((K + 7) / 8), (unsigned) M); +#define FRT_K2_LAUNCH(TT) FRT_M_DISPATCH(M, ggml_cuda_kernel_launch((moeglue::frt_moe_k2), ggml_cuda_kernel_launch_params(g2, dim3(256), 0, stream), \ + (const char *) dw->data, dw->nb[2], dw->nb[1], d_gluq8, \ + d_meta, (float *) out_add->data, (int) N, n_used, (int) K, \ + sh_ok ? (const char *) shd->src[0]->data : (const char *) nullptr, \ + sh_ok ? shd->src[0]->nb[1] : (size_t) 0, \ + lresid ? (const float *) lresid->data : (const float *) nullptr)) + switch (dw->type) { + case GGML_TYPE_Q6_K: FRT_K2_LAUNCH(GGML_TYPE_Q6_K); break; + case GGML_TYPE_Q5_K: FRT_K2_LAUNCH(GGML_TYPE_Q5_K); break; + case GGML_TYPE_Q4_K: FRT_K2_LAUNCH(GGML_TYPE_Q4_K); break; + default: FRT_K2_LAUNCH(GGML_TYPE_Q8_0); break; + } +#undef FRT_K2_LAUNCH + } + + ++fires_ok; + dbg_guard.armed = false; + *skip_count = end_idx - i + 1; + return true; +} + +// out-proj native span: [MUL_MAT ssm_out|attn_output] -> RESHAPE -> ADD residual +// (their path: quantize_q8_1 + mmvq + add = 3 launches) -> quant + gemv/add = 2. +static bool frt_outproj_native_try(ggml_backend_cuda_context & ctx, ggml_cgraph * cgraph, int i, int * skip_count) { + static int mode = -1; + if (mode < 0) { const char * m = getenv("FRT_OUTNATIVE_SWAP"); mode = (m && m[0] == '1') ? 1 : 0; } + if (!mode) return false; + ggml_tensor * n0 = cgraph->nodes[i]; + if (n0->op != GGML_OP_MUL_MAT || !n0->src[0]) return false; + const char * wn = n0->src[0]->name; + if (!strstr(wn, ".ssm_out.weight") && !strstr(wn, ".attn_output.weight")) return false; + const ggml_tensor * w = n0->src[0]; + const ggml_tensor * act = n0->src[1]; + if (w->type != GGML_TYPE_Q8_0) return false; + const int64_t K = w->ne[0], N = w->ne[1]; + if (K % 32 != 0 || K > 8192 || N % 8 != 0) return false; + if (act->type != GGML_TYPE_F32 || !ggml_is_contiguous(act)) return false; + const int M = (int) act->ne[1]; // token-batch width (spec verify runs M = 1 + n_draft) + if (M < 1 || M > 4 || act->ne[2] != 1) return false; + + ggml_tensor * out_add = nullptr; + const ggml_tensor * residual = nullptr; + int end_idx = -1; + const int LIM = i + 4 < cgraph->n_nodes ? i + 4 : cgraph->n_nodes; + for (int j = i + 1; j < LIM; ++j) { + ggml_tensor * n = cgraph->nodes[j]; + if (n->op == GGML_OP_RESHAPE || n->op == GGML_OP_VIEW) continue; + if (n->op == GGML_OP_ADD) { + const ggml_tensor * a = n->src[0], * b = n->src[1]; + auto is_mm = [&](const ggml_tensor * t) { + return t == n0 || ((t->op == GGML_OP_RESHAPE || t->op == GGML_OP_VIEW) && t->src[0] == n0); + }; + if (is_mm(a) && !is_mm(b)) { residual = b; out_add = n; end_idx = j; } + else if (is_mm(b) && !is_mm(a)) { residual = a; out_add = n; end_idx = j; } + } + break; + } + if (!out_add || end_idx < 0) return false; + if (out_add->ne[0] != N || out_add->ne[1] != M || residual->type != GGML_TYPE_F32) return false; + if (!ggml_is_contiguous(out_add) || !ggml_is_contiguous(residual)) return false; + + static block_q8_1 * d_actq8b = nullptr; + if (!d_actq8b) CUDA_CHECK(cudaMalloc(&d_actq8b, 4 * 256 * sizeof(block_q8_1))); + static moeglue::frt_moe_meta * d_meta_dummy = nullptr; + if (!d_meta_dummy) CUDA_CHECK(cudaMalloc(&d_meta_dummy, sizeof(moeglue::frt_moe_meta))); + + cudaStream_t stream = ctx.stream(); + const block_q8_1 * y_q8 = d_actq8b; + if (frt::g_reg.ok && frt::g_reg.outq8_node == (const void *) act && K == 4096) { + y_q8 = frt::g_reg.d_outq8; // GDN epilogue already staged the q8 act + frt::g_reg.outq8_node = nullptr; + } else { + FRT_M_DISPATCH(M, ggml_cuda_kernel_launch(moeglue::frt_moe_quant_meta, ggml_cuda_kernel_launch_params(dim3(8), dim3(256), 0, stream), + (const float *) act->data, (int) K, (int64_t) K, + (const int32_t *) nullptr, (int64_t) 0, + (const float *) nullptr, (int64_t) 0, 0, + (const float *) nullptr, + d_actq8b, d_meta_dummy)); + } + ggml_cuda_kernel_launch(moeglue::frt_outproj_gemv, ggml_cuda_kernel_launch_params(dim3((unsigned) (N / 8), (unsigned) M), dim3(256), 0, stream), + (const char *) w->data, w->nb[1], y_q8, + (const float *) residual->data, (float *) out_add->data, (int) N, (int) K); + *skip_count = end_idx - i + 1; + return true; +} + +// shared-expert span: anchor = MUL_MAT on ffn_gate_shexp.weight. +static bool frt_shexp_span_try(ggml_backend_cuda_context & ctx, ggml_cgraph * cgraph, int i, int * skip_count) { + ggml_tensor * n0 = cgraph->nodes[i]; + if (n0->op != GGML_OP_MUL_MAT || !n0->src[0] || + !strstr(n0->src[0]->name, ".ffn_gate_shexp.weight")) return false; + if (!frt::frt_regions_load() || !frt::g_reg.shexp_on) return false; + int layer = -1; + sscanf(n0->src[0]->name, "blk.%d.", &layer); + if (layer < 0 || layer >= 64) return false; + frt::frt_region & r2 = frt::g_reg.regions[2][layer]; + frt::frt_region & r3 = frt::g_reg.regions[3][layer]; + if (r2.N != 1024 || r2.K != 2048 || r3.N != 2048 || r3.K != 512) return false; + if (n0->src[1]->ne[1] != 1 || !ggml_is_contiguous(n0->src[1])) return false; + + const ggml_tensor * up_mm = nullptr, * glu = nullptr, * down_mm = nullptr; + const ggml_tensor * ginp_mm = nullptr, * sig = nullptr, * gmul = nullptr; + ggml_tensor * out_add = nullptr; + int end_idx = -1; + const int LIM = i + 10 < cgraph->n_nodes ? i + 10 : cgraph->n_nodes; + for (int j = i + 1; j < LIM; ++j) { + ggml_tensor * n = cgraph->nodes[j]; + switch (n->op) { + case GGML_OP_MUL_MAT: + if (n->src[0] && strstr(n->src[0]->name, ".ffn_up_shexp.weight")) up_mm = n; + else if (n->src[0] && strstr(n->src[0]->name, ".ffn_down_shexp.weight")) down_mm = n; + else if (n->src[0] && strstr(n->src[0]->name, ".ffn_gate_inp_shexp.weight")) ginp_mm = n; + else return false; + break; + case GGML_OP_GLU: + if (ggml_get_glu_op(n) != GGML_GLU_OP_SWIGLU) return false; + glu = n; + break; + case GGML_OP_UNARY: + if (ggml_get_unary_op(n) != GGML_UNARY_OP_SIGMOID) return false; + sig = n; + break; + case GGML_OP_MUL: + gmul = n; + break; + case GGML_OP_ADD: + if (gmul && (n->src[1] == gmul || n->src[0] == gmul)) { out_add = n; end_idx = j; } + else return false; + break; + case GGML_OP_RESHAPE: case GGML_OP_VIEW: + break; + default: + return false; + } + if (end_idx >= 0) break; + } + if (!up_mm || !glu || !down_mm || !ginp_mm || !sig || !gmul || !out_add) return false; + // wiring guards + if (glu->src[0] != n0 || glu->src[1] != up_mm) return false; // silu(gate)*up + if (down_mm->src[1] != glu) return false; + if (sig->src[0] != ginp_mm) return false; + if (!(gmul->src[0] == down_mm && gmul->src[1] == sig) && + !(gmul->src[0] == sig && gmul->src[1] == down_mm)) return false; + if (up_mm->src[1]->data != n0->src[1]->data || + ginp_mm->src[1]->data != n0->src[1]->data) return false; // same activation + if (ginp_mm->src[0]->type != GGML_TYPE_F32) return false; + const ggml_tensor * moe_out = out_add->src[0] == gmul ? out_add->src[1] : out_add->src[0]; + + cudaStream_t stream = ctx.stream(); + moeglue::frt_shexp_pre<<<2, 256, 0, stream>>>( + (const float *) n0->src[1]->data, + (uint2 *) frt::g_reg.d_apack, frt::g_reg.d_sfa, + (const float *) ginp_mm->src[0]->data, frt::g_reg.d_scalar); + frt::frt_ws_launch(frt::g_reg.d_apack, r2.d_packed, frt::g_reg.d_sfa, r2.d_sf, + frt::g_reg.d_staging, r2.alpha, 1024, 2048, 1, stream); + moeglue::frt_shexp_glu_quant<<<1, 256, 0, stream>>>( + frt::g_reg.d_staging, (uint2 *) frt::g_reg.d_apack, frt::g_reg.d_sfa); + frt::frt_ws_launch(frt::g_reg.d_apack, r3.d_packed, frt::g_reg.d_sfa, r3.d_sf, + frt::g_reg.d_staging + 4096, r3.alpha, 2048, 512, 1, stream); + moeglue::frt_shexp_finish<<<8, 256, 0, stream>>>( + (const float *) moe_out->data, frt::g_reg.d_staging + 4096, + frt::g_reg.d_scalar, (float *) out_add->data); + *skip_count = end_idx - i + 1; + return true; +} + +// Returns true and sets *skip_count when either span matched at node i. +bool ggml_cuda_frt_moeglue_try_impl(ggml_backend_cuda_context & ctx, ggml_cgraph * cgraph, int i, int * skip_count) { + if (frt_moefuse_try(ctx, cgraph, i, skip_count)) return true; + if (frt_router_span_try(ctx, cgraph, i, skip_count)) return true; + if (frt_attn_gate_try(ctx, cgraph, i, skip_count)) return true; + if (frt_outproj_native_try(ctx, cgraph, i, skip_count)) return true; + if (frt_shexp_span_try(ctx, cgraph, i, skip_count)) return true; + static int mode = -1; + if (mode < 0) { const char * m = getenv("FRT_MOEGLUE_SWAP"); mode = (m && m[0] == '1') ? 1 : 0; } + if (!mode) return false; + ggml_tensor * n0 = cgraph->nodes[i]; + + // ---- combine span ---- + if (n0->op == GGML_OP_MUL && strncmp(n0->name, "ffn_moe_weighted", 16) == 0 && + n0->ne[1] > 1 && n0->ne[1] <= 32 && n0->ne[2] == 1 && n0->ne[3] == 1 && + n0->src[1]->ne[0] == 1 && ggml_is_contiguous(n0->src[0])) { + const int hidden = (int) n0->ne[0]; + const int nexp = (int) n0->ne[1]; + // expect nexp VIEWs of n0 then nexp-1 chained ADDs ending at ffn_moe_out + ggml_tensor * out_add = nullptr; + int end_idx = -1; + uint32_t seen = 0; // bitmask of expert slices consumed by the ADD chain + const ggml_tensor * chain = nullptr; + const int LIM = i + 2 * nexp + 2 < cgraph->n_nodes ? i + 2 * nexp + 2 : cgraph->n_nodes; + for (int j = i + 1; j < LIM; ++j) { + ggml_tensor * n = cgraph->nodes[j]; + if (n->op == GGML_OP_VIEW) continue; + if (n->op != GGML_OP_ADD) return false; + auto slice_of = [&](const ggml_tensor * t) -> int { + if (t->op != GGML_OP_VIEW || t->src[0] != n0 || t->ne[0] != hidden) return -1; + const ptrdiff_t off = (const char *) t->data - (const char *) n0->data; + if (off < 0 || off % ((ptrdiff_t) hidden * 4) != 0) return -1; + const ptrdiff_t e = off / ((ptrdiff_t) hidden * 4); + return e < nexp ? (int) e : -1; + }; + int e0 = slice_of(n->src[0]); + int e1 = slice_of(n->src[1]); + if (chain == nullptr) { + if (e0 < 0 || e1 < 0) return false; + seen |= 1u << e0; + } else { + if (n->src[0] != chain || e1 < 0) return false; + } + if (seen & (1u << e1)) return false; + seen |= 1u << e1; + chain = n; + if (strncmp(n->name, "ffn_moe_out", 11) == 0) { out_add = n; end_idx = j; break; } + } + if (!out_add || seen != (nexp >= 32 ? 0xffffffffu : ((1u << nexp) - 1))) return false; + // write via private staging: the graph allocator may alias out_add onto + // the (dead-after-us) weights/down buffers this kernel still reads. + if (hidden > 4096) return false; + static float * d_comb = nullptr; + if (!d_comb) CUDA_CHECK(cudaMalloc(&d_comb, 4096 * sizeof(float))); + ggml_cuda_kernel_launch(moeglue::frt_moe_combine, ggml_cuda_kernel_launch_params(dim3((hidden + 255) / 256), dim3(256), 0, ctx.stream()), + (const float *) n0->src[0]->data, (const float *) n0->src[1]->data, + d_comb, hidden, nexp); + CUDA_CHECK(cudaMemcpyAsync(out_add->data, d_comb, + (size_t) hidden * sizeof(float), cudaMemcpyDeviceToDevice, ctx.stream())); + *skip_count = end_idx - i + 1; + return true; + } + + return false; +} + +// ---- MoE expert takeover (ggml-native NVFP4 blocks, FRT_MOE_SWAP) -------- + +static int frt_moe_mode(void) { + static int mode = -1; + if (mode < 0) { + const char * m = getenv("FRT_MOE_SWAP"); + mode = (m && m[0] == '1') ? 1 : 0; + } + return mode; +} + +// blocks llama.cpp's own mmvq/mmf GLU fusion for expert tensors we take over +bool ggml_cuda_frt_moe_blocks_fusion(const ggml_tensor * mm) { + if (!frt_moe_mode()) return false; + if (!mm || mm->op != GGML_OP_MUL_MAT_ID) return false; + const ggml_tensor * w = mm->src[0]; + return w && w->type == GGML_TYPE_NVFP4 && strstr(w->name, "_exps.weight") != nullptr; +} + +bool ggml_cuda_frt_moe_mul_mat_id(ggml_backend_cuda_context & ctx, ggml_tensor * dst) { + if (!frt_moe_mode()) return false; + const ggml_tensor * w = dst->src[0]; + const ggml_tensor * x = dst->src[1]; + const ggml_tensor * ids = dst->src[2]; + if (!w || !x || !ids) return false; + if (w->type != GGML_TYPE_NVFP4 || x->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) return false; + if (strstr(w->name, "_exps.weight") == nullptr) return false; + if (ids->type != GGML_TYPE_I32) return false; + { // FRT_MOE_KIND=gate|up|down|gu|all (isolation aid) + static const char * kind = getenv("FRT_MOE_KIND"); + if (kind && strcmp(kind, "all") != 0) { + const bool g = strstr(w->name, "ffn_gate_exps") != nullptr; + const bool u = strstr(w->name, "ffn_up_exps") != nullptr; + const bool d = strstr(w->name, "ffn_down_exps") != nullptr; + if (strcmp(kind, "gate") == 0 && !g) return false; + if (strcmp(kind, "up") == 0 && !u) return false; + if (strcmp(kind, "down") == 0 && !d) return false; + if (strcmp(kind, "gu") == 0 && d) return false; + } + } + + const int64_t K = w->ne[0]; + const int64_t n_per = w->ne[1]; + if ((K % 64) != 0) return false; + const int64_t n_used = ids->ne[0]; + const int64_t n_tokens = ids->ne[1]; + if (n_tokens != 1 || n_used <= 0 || n_used > 64) return false; + if (!ggml_is_contiguous(dst) || !ggml_is_contiguous(ids)) return false; + if (dst->ne[0] != n_per || dst->ne[1] * dst->ne[2] != n_used) return false; + + // activation layout: broadcast (one row for all experts) or per-expert-slot + bool broadcast; + int64_t x_stride_f = 0; + const int64_t x_rows = x->ne[1] * x->ne[2]; + if (x->ne[0] == K && x_rows == 1) { + broadcast = true; + } else if (x->ne[0] == K && x_rows == n_used) { + broadcast = false; + x_stride_f = (x->ne[1] == n_used ? x->nb[1] : x->nb[2]) / sizeof(float); + } else { + return false; + } + + // one-time UE4M3 LUT upload; never during graph capture + static bool lut_done = false; + if (!lut_done) { + cudaStreamCaptureStatus cap = cudaStreamCaptureStatusNone; + cudaStreamIsCapturing(ctx.stream(), &cap); + if (cap != cudaStreamCaptureStatusNone) return false; + frt::frt_init_ue4m3_lut(); + lut_done = true; + } + + const int64_t expert_stride = w->nb[2]; + const int64_t rows_total = n_used * n_per; + dim3 grid((unsigned)((rows_total + 7) / 8)); + cudaStream_t stream = ctx.stream(); + + static int check = -1; + if (check < 0) { const char * c = getenv("FRT_MOE_CHECK"); check = (c && c[0]=='1') ? 1 : 0; } + if (check == 1) { + check = 2; // once + cudaStreamSynchronize(stream); + int32_t h_ids[64]; cudaMemcpy(h_ids, ids->data, n_used * 4, cudaMemcpyDeviceToHost); + std::vector h_x(K); + const float * xsrc = (const float *) x->data; // slot 0 row + cudaMemcpy(h_x.data(), xsrc, K * 4, cudaMemcpyDeviceToHost); + std::vector h_row((K / 64) * 36); + float lut[256]; + for (int i = 0; i < 256; ++i) { + const int lo = i & 0x7F; const int e = (lo >> 3) & 0xF; const int m = lo & 7; + float v = (lo == 0x7F) ? 0.f : (e == 0 ? (float) m / 8.f * ldexpf(1.f, -6) + : (1.f + (float) m / 8.f) * ldexpf(1.f, e - 7)); + lut[i] = (i & 0x80) ? -v : v; + } + const float e2m1v[16] = {0,.5f,1,1.5f,2,3,4,6,-0.f,-.5f,-1,-1.5f,-2,-3,-4,-6}; + fprintf(stderr, "frt-moe-check %s: n_used=%lld n_per=%lld K=%lld estride=%lld ids0=%d bcast=%d xne=[%lld,%lld,%lld]\n", + w->name, (long long)n_used, (long long)n_per, (long long)K, (long long)expert_stride, + h_ids[0], (int)broadcast, (long long)x->ne[0], (long long)x->ne[1], (long long)x->ne[2]); + for (int n = 0; n < 3; ++n) { + cudaMemcpy(h_row.data(), (const uint8_t *) w->data + (size_t) h_ids[0] * expert_stride + + (size_t) n * ((K / 64) * 36), h_row.size(), cudaMemcpyDeviceToHost); + double ref = 0; + for (int kb = 0; kb < K / 64; ++kb) { + const uint8_t * blk = h_row.data() + (size_t) kb * 36; + for (int sub = 0; sub < 4; ++sub) { + const float d = lut[blk[sub]]; + for (int j = 0; j < 8; ++j) { + const uint8_t q = blk[4 + sub * 8 + j]; + ref += (double) d * e2m1v[q & 0xF] * h_x[kb * 64 + sub * 16 + j]; + ref += (double) d * e2m1v[q >> 4] * h_x[kb * 64 + sub * 16 + j + 8]; + } + } + } + fprintf(stderr, "frt-moe-check ref out[0][%d] = %g\n", n, ref); + } + } + if (broadcast) { + frt::frt_moe_mmid_f32<<>>( + (const float *) x->data, (const uint8_t *) w->data, (const int32_t *) ids->data, + (float *) dst->data, (int) K, (int) n_per, (int) n_used, expert_stride, 0); + } else { + frt::frt_moe_mmid_f32<<>>( + (const float *) x->data, (const uint8_t *) w->data, (const int32_t *) ids->data, + (float *) dst->data, (int) K, (int) n_per, (int) n_used, expert_stride, x_stride_f); + } + if (check == 2) { + check = 3; + const cudaError_t le = cudaGetLastError(); + const cudaError_t se = cudaStreamSynchronize(stream); + float h_out[3]; cudaMemcpy(h_out, dst->data, 3 * 4, cudaMemcpyDeviceToHost); + fprintf(stderr, "frt-moe-check OUR out[0][0..2] = %g %g %g (launch=%s sync=%s grid=%u)\n", + h_out[0], h_out[1], h_out[2], cudaGetErrorString(le), cudaGetErrorString(se), grid.x); + } + return true; +} + +// Returns true if it handled the mul_mat. +bool ggml_cuda_frt_head_mul_mat(ggml_backend_cuda_context & ctx, + const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) { + if (!frt::g_reg.ok) { + cudaStreamCaptureStatus rcap = cudaStreamCaptureStatusNone; + cudaStreamIsCapturing(ctx.stream(), &rcap); + if (rcap == cudaStreamCaptureStatusNone && frt::frt_regions_load()) { /* loaded */ } + } + if (frt::g_reg.ok && frt::frt_regions_mul_mat(ctx, src0, src1, dst)) { + return true; + } + if (!frt::g_head.ok) { + // never allocate while a CUDA graph capture is in flight + cudaStreamCaptureStatus cap = cudaStreamCaptureStatusNone; + cudaStreamIsCapturing(ctx.stream(), &cap); + if (cap != cudaStreamCaptureStatusNone) return false; + } + if (!frt::frt_head_load()) return false; + if (strcmp(src0->name, "output.weight") != 0) return false; + if (frt::g_head.draft_only && src0->type != GGML_TYPE_Q8_0) return false; + if (src1->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) return false; + const int M = (int) src1->ne[1]; // spec verify asks for logits at M = 1 + n_draft rows + if (M < 1 || M > 4 || src1->ne[2] != 1 || src1->ne[3] != 1) return false; + if (src0->ne[0] != frt::g_head.K || src0->ne[1] != frt::g_head.N) return false; + if (!ggml_is_contiguous(src1) || !ggml_is_contiguous(dst)) return false; + + cudaStream_t stream = ctx.stream(); + const float * x = (const float *) src1->data; + float * out = (float *) dst->data; + const int N = (int) frt::g_head.N, K = (int) frt::g_head.K; + + static int head_mode = -1; // 0 = w4a4 (default), 1 = w4a16 + if (head_mode < 0) { + const char * m = getenv("FRT_HEAD_MODE"); + head_mode = (m && strcmp(m, "w4a16") == 0) ? 1 : 0; + if (head_mode == 1) frt::frt_init_ue4m3_lut(); + } + if (head_mode == 1) { + if (M != 1) return false; // w4a16 path stays M=1 + const int n_col_super = ((K >> 4) + 3) / 4; + dim3 grid((N + 7) / 8); + frt::w4a16_matvec_f32<<>>( + x, frt::g_head.d_packed, frt::g_head.d_sf, out, frt::g_head.alpha, N, K, n_col_super); + return true; + } + + FRT_M_DISPATCH(M, ggml_cuda_kernel_launch((frt::quant_act_fp4_f32<256, MT>), ggml_cuda_kernel_launch_params(dim3(M), dim3(256), 0, stream), + x, (uint2 *) frt::g_head.d_apack, frt::g_head.d_sfa, (int) frt::g_head.K, (int64_t) K)); + + frt::frt_ws_launch(frt::g_head.d_apack, frt::g_head.d_packed, frt::g_head.d_sfa, frt::g_head.d_sf, + out, frt::g_head.alpha, (int) N, (int) K, M, stream, 44); // head: s4w4 wins (+6 t/s) + return true; +} diff --git a/flash_rt/structures/adapters/ggml/qualification/pins_qwen36_sm120.yaml b/flash_rt/structures/adapters/ggml/qualification/pins_qwen36_sm120.yaml new file mode 100644 index 00000000..84eb2ac6 --- /dev/null +++ b/flash_rt/structures/adapters/ggml/qualification/pins_qwen36_sm120.yaml @@ -0,0 +1,23 @@ +# Structure versions the SM120/Qwen3.6 windows were written against, plus +# the target's shipping gates. The families marked proposed are carried by +# this target until they are formalized in the catalog; their pin is the +# adapter file itself (fr_win_qwen36_sm120.cu revision). +binding: llamacpp_qwen36_35b_sm120 +pins: + llm_decode_pipeline: 1 + format_native_gemv: 1 # proposed family (warp-split-K NVFP4 GEMV, M<=4) + moe_expert_span: 1 # proposed family (format-native K-quant MoE) + gdn_cell: 1 # proposed family (recurrent cell span, snapshot/checkpoint discipline) + +# Shipping gates (re-established after every adapter or host change; the +# numbers are the 2026-08-25 migration baseline on RTX 5090): +gates: + quality: + ppl_24ch_ub1_safe_tier: 6.1106 # must stay bit-identical (wikitext, -ub 1 -c 512 --chunks 24) + moefuse_selftest: pass # FRT_MOEFUSE_SELFTEST=1 duplicated-token bit-exact replay + spec_text_check: required # greedy spec output must be coherent text, never judged by tps alone + perf: + tg128_full_tier_bench: 377.8 # llama-bench r=5, full-tier env + spec_full_tier_median: 444.6 # 4-prompt server battery, greedy 384 tok, LLAMA_GRAPH_SLOTS=6 -bs + spec_safe_tier_median: 433.3 + stock_reference: 283.07 # same-machine stock @95b8e33 diff --git a/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml b/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml new file mode 100644 index 00000000..252f91f4 --- /dev/null +++ b/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml @@ -0,0 +1,86 @@ +binding: llamacpp_qwen36_35b_sm120 +structure: llm_decode_pipeline + +# Host: stock llama.cpp (upstream master lineage) running Qwen3.6-35B-A3B +# UD-Q4_K_M on RTX 5090 (SM120) with the native ggml adapter +# (flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu). Structure +# regions execute as fused subgraph windows matched inside ggml-cuda's +# graph evaluation. Two quality tiers share one binary: the safe tier keeps +# the target head and all matched math bit-comparable to stock (24-chunk +# PPL receipt), the full tier adds the NVFP4 lm-head swap. + +stages: + prefill: + seam: "llama_decode prompt processing (stock kernels; adapter windows are decode-shaped and stay out)" + capture: none + decode: + seam: "llama_decode token generation, M in [1,4] (speculative verify batches)" + capture: cuda_graphs_keyed + spec_draft: + seam: "MTP draft chain (single trained head, 3 chained drafts/step) + M=4 h-extraction eval" + capture: cuda_graphs_keyed + +cadences: + token: [decode] + spec_step: [decode, spec_draft] + +coverage: + contract: hot_path_takeover + hot_path: + - gdn_in_proj_region + - attn_qkv_region + - gdn_cell_span + - moe_expert_span + - moe_router + - out_proj + - lm_head + - spec_draft_serving + segments: + - name: gdn_in_proj_region + stage: decode + classification: structure + seam: "fused-region NVFP4 W4A4 GEMV serving qkv|z|a|b members from one staging pass (30 GDN layers)" + structures: [format_native_gemv] # proposed family: warp-split-K blockscale MMA GEMV, M<=4 + - name: attn_qkv_region + stage: decode + classification: structure + seam: "attention qkv fused region, same GEMV (10 attention layers)" + structures: [format_native_gemv] + - name: gdn_cell_span + stage: decode + classification: structure + seam: "conv + gated-delta-net + epilogue span -> 4 launches; M>1 writes per-token state/conv snapshots and replays checkpoint saves (rollback discipline, DEVELOPMENT.md)" + structures: [gdn_cell] # proposed family + - name: moe_expert_span + stage: decode + classification: structure + seam: "K0 quant+meta / K1 gate|up|GLU / K2 down+combine over ggml-native K-quant blocks (bit-exact q8_1 activation clone), shared expert folded, M<=4" + structures: [moe_expert_span] # proposed family: format-native MoE + - name: moe_router + stage: decode + classification: structure + seam: "fused gate GEMV + exact topk replication; default-on only for M>=2 (host's own fusion is M=1-only)" + structures: [format_native_gemv] + - name: out_proj + stage: decode + classification: structure + seam: "ssm_out/attn_output Q8_0 GEMV + residual, consumes the GDN epilogue's q8 handoff" + structures: [format_native_gemv] + - name: lm_head + stage: decode + classification: structure + seam: "full tier: NVFP4 W4A4 head swap (M<=4 in one pass); safe tier: stock Q6_K mmvq (measured ~91% BW, left alone)" + structures: [format_native_gemv] + - name: spec_draft_serving + stage: spec_draft + classification: structure + seam: "draft-side head serving: the draft GGUF's Q8_0 lm-head copy is served from the FP4 pack (spec acceptance-only, output distribution unchanged); draft MTP layer's Q8_0 experts consume the same MoE span" + structures: [format_native_gemv, moe_expert_span] + - name: host_graph_slots + stage: spec_draft + classification: host_stage + seam: "host-tree change (llama-context): per-shape graph slots keyed (n_tokens, gtype, has_embd), each slot owning its scheduler; plus target-side backend sampling (-bs)" + +hosts: + llamacpp_dev_sm120: + versions: "upstream 95b8e33 + FlashRT hook guards (GGML_CUDA_FLASHRT_SM120)" From f4ab9e17df12f3500febb46bf84c9f91ce186297 Mon Sep 17 00:00:00 2001 From: LiangSu8899 <7thuniversels@gmail.com> Date: Tue, 25 Aug 2026 11:49:45 -0400 Subject: [PATCH 22/32] ggml adapter sm120: in-process weight repack for the FP4 regions Region wire buffers (packed e2m1 + atom-layout SF + per-tensor alpha) are now built on first sight of the weight tensors in an evaluated graph, from a pre-capture hook: ggml dequant -> bf16 -> global amax -> ue4m3-ceil block scales -> e2m1 nibbles, reproducing the offline packer byte-for-byte (FRT_REPACK_CHECK memcmp gate, 40/40 regions identical; perplexity receipt unchanged). This removes the side-band region pack dependency. The lm-head keeps its pack: it is quantized from the BF16 checkpoint, which both scores and drafts measurably better than a rebuild from the GGUF's Q6_K copy; the online head build remains as a fallback. --- flash_rt/structures/adapters/ggml/USAGE.md | 18 +- .../adapters/ggml/fr_win_qwen36_sm120.cu | 362 ++++++++++++++++++ 2 files changed, 375 insertions(+), 5 deletions(-) diff --git a/flash_rt/structures/adapters/ggml/USAGE.md b/flash_rt/structures/adapters/ggml/USAGE.md index 83ed2ec2..ee6a872c 100644 --- a/flash_rt/structures/adapters/ggml/USAGE.md +++ b/flash_rt/structures/adapters/ggml/USAGE.md @@ -92,8 +92,16 @@ FRT_GDN_SWAP=1 FRT_MOEGLUE_SWAP=1 FRT_MOEFUSE_SWAP=1 FRT_MOEFUSE_SHEXP=1 FRT_OUTNATIVE_SWAP=1 FRT_REGIONS_PACK=`; full tier adds `FRT_HEAD_SWAP=1 FRT_HEAD_PACK=`. Speculative serving adds `FRT_HEAD_DRAFT=1` (safe tier: FP4-serve only the draft's head copy) and the -host-side `LLAMA_GRAPH_SLOTS=6` + `--backend-sampling`. FP4 weights currently -come from side-band packs; replacing them with the in-process repack cache is -the next step for this target. Diagnostics: `FRT_STATS=1`, -`FRT_MOEFUSE_DBG=`, `FRT_MOEFUSE_SELFTEST=1`, `FRT_DUMP_GRAPH=1` + -`FRT_DUMP_M=` (+ `FRT_DUMP_PATH`). +host-side `LLAMA_GRAPH_SLOTS=6` + `--backend-sampling`. + +FP4 region weights repack **in-process** with `FRT_ONLINE_REPACK=1` (leave +`FRT_REGIONS_PACK` unset): the pre-capture hook dequantizes the GGUF members +on device and rebuilds the wire format byte-identically to the offline packer +(validated by `FRT_REPACK_CHECK=1` with `FRT_REGIONS_PACK_REF=`). The +lm-head is the exception: the shipped head pack is quantized from the BF16 +checkpoint (the GGUF only holds Q6_K), and the BF16-sourced pack drafts and +scores measurably better than an online Q6_K-sourced rebuild — keep +`FRT_HEAD_PACK` for the head (both tiers); the online head build is a +fallback only. Diagnostics: `FRT_STATS=1`, `FRT_MOEFUSE_DBG=`, +`FRT_MOEFUSE_SELFTEST=1`, `FRT_DUMP_GRAPH=1` + `FRT_DUMP_M=` +(+ `FRT_DUMP_PATH`). diff --git a/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu b/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu index e6c4505d..46b2647b 100644 --- a/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu +++ b/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu @@ -21,6 +21,7 @@ #include "common.cuh" #include "vecdotq.cuh" +#include "convert.cuh" #include #include @@ -28,6 +29,7 @@ #include #include #include +#include #include #include "cute/arch/mma_sm120.hpp" @@ -469,6 +471,8 @@ struct frt_region_state { static frt_region_state g_reg; +static bool frt_online_on(void); // defined with the in-process repack section + static bool frt_regions_load(void) { if (g_reg.tried) return g_reg.ok; g_reg.tried = true; @@ -482,6 +486,24 @@ static bool frt_regions_load(void) { g_reg.outproj_on = d && d[0] == '1'; if (!g_reg.inproj_on && !g_reg.attn_on && !g_reg.shexp_on && !g_reg.outproj_on) return false; const char * path = getenv("FRT_REGIONS_PACK"); + if (!path && frt_online_on()) { + // online repack: per-region weight buffers arrive from the pre-capture + // hook; only the shared serve buffers are sized here (known target + // shapes: kind0 12352x2048, kind1 9216x2048). + const int64_t maxN = 12352, maxK = 2048; + CUDA_CHECK(cudaMalloc(&g_reg.d_staging, 4 * maxN * sizeof(float))); + CUDA_CHECK(cudaMalloc(&g_reg.d_conv_out, 4 * 8192 * sizeof(float))); + CUDA_CHECK(cudaMalloc(&g_reg.d_attn_buf, 4 * 4096 * sizeof(float))); + CUDA_CHECK(cudaMalloc(&g_reg.d_scalar, sizeof(float))); + CUDA_CHECK(cudaMalloc(&g_reg.d_outq8, 4 * 128 * sizeof(block_q8_1))); + CUDA_CHECK(cudaMalloc(&g_reg.d_apack, 2 * maxK)); + CUDA_CHECK(cudaMalloc(&g_reg.d_sfa, 128 * (maxK / 16))); + CUDA_CHECK(cudaMemset(g_reg.d_sfa, 0, 128 * (maxK / 16))); + fprintf(stderr, "frt-regions: online repack mode (inproj=%d attnqkv=%d)\n", + (int) g_reg.inproj_on, (int) g_reg.attn_on); + g_reg.ok = true; + return true; + } if (!path) { fprintf(stderr, "frt-regions: FRT_REGIONS_PACK missing\n"); return false; } FILE * f = fopen(path, "rb"); if (!f) { fprintf(stderr, "frt-regions: cannot open %s\n", path); return false; } @@ -617,6 +639,10 @@ static bool frt_head_load(void) { if (!sw_on && !dr_on) return false; g_head.draft_only = !sw_on; const char * path = getenv("FRT_HEAD_PACK"); + if (!path && frt_online_on()) { + g_head.tried = false; // built by the pre-capture repack hook + return g_head.ok; + } if (!path) { fprintf(stderr, "frt-head: FRT_HEAD_SWAP/FRT_HEAD_DRAFT set but FRT_HEAD_PACK missing\n"); return false; } FILE * f = fopen(path, "rb"); if (!f) { fprintf(stderr, "frt-head: cannot open %s\n", path); return false; } @@ -646,6 +672,335 @@ static bool frt_head_load(void) { return true; } +// ---- in-process weight repack (FRT_ONLINE_REPACK=1) ----------------------- +// Replaces the side-band pack files: region/head FP4 wire buffers are built +// on first sight of the weight tensors in an evaluated graph, before any +// CUDA graph capture (called from the pre-capture hook in ggml-cuda.cu). +// The pipeline reproduces the offline packer bit-for-bit: ggml dequant -> +// bf16 (RNE) -> global amax -> global_scale = amax/2688 -> per-16 ue4m3-ceil +// block scales -> e2m1 nibbles + Sm1xx atom-layout SF bytes. +// FRT_REPACK_CHECK=1 memcmp-validates against the pack files when both are +// given. + +__device__ __forceinline__ uint8_t frt_ue4m3_ceil(float v) { + if (v <= 0.0f) return 0; + if (v > 240.0f) return 0xFE; + uint32_t bits = __float_as_uint(v); + int float_exp = ((bits >> 23) & 0xFF) - 127; + uint32_t frac = bits & 0x7FFFFF; + int ue_exp = float_exp + 7; + if (ue_exp <= 0) { + float scaled = v * 512.0f; + int m = (int) ceilf(scaled); + if (m > 7) return (1 << 3) | 0; + if (m < 1) m = 1; + return (uint8_t) m; + } + if (ue_exp >= 15) return 0xFE; + int m = (int) (frac >> 20); + if (frac & 0xFFFFF) m++; + if (m >= 8) { m = 0; ue_exp++; } + if (ue_exp >= 15) return 0xFE; + return (uint8_t) ((ue_exp << 3) | m); +} + +__device__ __forceinline__ float frt_ue4m3_f32(uint8_t v) { + int e = (v >> 3) & 0xF; + int m = v & 0x7; + if (e == 0) return ldexpf((float) m / 8.0f, -6); + return ldexpf(1.0f + (float) m / 8.0f, e - 7); +} + +// e2m1 with the offline packer's strict-< boundaries (the activation +// quantizer above uses <=; at exact tie values the codes differ, so weight +// repack must use this one to stay byte-identical with the pack files). +__device__ __forceinline__ uint8_t frt_e2m1_weight(float v) { + uint8_t sign = (v < 0.0f) ? 0x8u : 0x0u; + float a = fabsf(v); + uint8_t mag; + if (a < 0.25f) mag = 0; + else if (a < 0.75f) mag = 1; + else if (a < 1.25f) mag = 2; + else if (a < 1.75f) mag = 3; + else if (a < 2.5f) mag = 4; + else if (a < 3.5f) mag = 5; + else if (a < 5.0f) mag = 6; + else mag = 7; + return sign | mag; +} + +__global__ void frt_w_amax_bf16(const __nv_bfloat16 * __restrict__ w, float * __restrict__ gmax, int N, int K) { + const int row = blockIdx.x; + if (row >= N) return; + const size_t off = (size_t) row * K; + float tm = 0.f; + for (int c = threadIdx.x; c < K; c += blockDim.x) { + const float a = fabsf(__bfloat162float(w[off + c])); + if (a > tm) tm = a; + } + __shared__ float smem[32]; + const int lane = threadIdx.x & 31, wid = threadIdx.x >> 5; +#pragma unroll + for (int o = 16; o > 0; o >>= 1) tm = fmaxf(tm, __shfl_xor_sync(0xffffffffu, tm, o)); + if (lane == 0) smem[wid] = tm; + __syncthreads(); + if (wid == 0) { + const int nw = (blockDim.x + 31) >> 5; + tm = (lane < nw) ? smem[lane] : 0.f; +#pragma unroll + for (int o = 16; o > 0; o >>= 1) tm = fmaxf(tm, __shfl_xor_sync(0xffffffffu, tm, o)); + if (lane == 0) atomicMax(reinterpret_cast(gmax), __float_as_int(tm)); + } +} + +__global__ void frt_w_gscale(const float * gmax, float * gs) { + if (threadIdx.x == 0 && blockIdx.x == 0) { + const float a = *gmax; + *gs = (a > 0.f) ? (a / 2688.f) : 1.f; + } +} + +// rows [0, n_rows) of w correspond to absolute output rows row_base + r. +__global__ void frt_w_pass2_bf16(const __nv_bfloat16 * __restrict__ w, const float * __restrict__ gs_ptr, + uint8_t * __restrict__ packed, uint8_t * __restrict__ sf_swz, + int n_rows, int K, int row_base, int n_col_super) { + const int r = blockIdx.x; + if (r >= n_rows) return; + const float gscale = *gs_ptr; + const float inv_g = (gscale > 0.f) ? (1.f / gscale) : 0.f; + const int row = row_base + r; + const size_t in_off = (size_t) r * K; + const size_t out_off = (size_t) row * (K / 2); + const int rb = row / 128, ri = row % 128; + const int nbr = K / 16; + for (int b = threadIdx.x; b < nbr; b += blockDim.x) { + const int col0 = b * 16; + float v[16]; + float bmax = 0.f; +#pragma unroll + for (int i = 0; i < 16; ++i) { + v[i] = __bfloat162float(w[in_off + col0 + i]); + const float a = fabsf(v[i]); + if (a > bmax) bmax = a; + } + const uint8_t sf_byte = frt_ue4m3_ceil((bmax / 6.f) * inv_g); + const float bs = frt_ue4m3_f32(sf_byte) * gscale; + const float inv_bs = (bs > 0.f) ? (1.f / bs) : 0.f; + uint8_t * prow = packed + out_off; +#pragma unroll + for (int i = 0; i < 16; i += 2) { + const uint8_t lo = frt_e2m1_weight(v[i] * inv_bs); + const uint8_t hi = frt_e2m1_weight(v[i + 1] * inv_bs); + prow[(col0 + i) >> 1] = (uint8_t) ((hi << 4) | (lo & 0x0F)); + } + const int cb = b / 4, ci = b % 4; + sf_swz[(rb * n_col_super + cb) * 512 + (ri % 32) * 16 + (ri / 32) * 4 + ci] = sf_byte; + } +} + +static bool frt_online_on(void) { + static int on = -1; + if (on < 0) { const char * s = getenv("FRT_ONLINE_REPACK"); on = (s && s[0] == '1') ? 1 : 0; } + return on == 1; +} + +// One source tensor contributing `rows` rows to an [N, K] concat target. +struct frt_repack_src { const ggml_tensor * t; int64_t rows; }; + +// Build packed+SF (+alpha) for a row-concatenation of ggml tensors. Eager +// only (allocates, synchronizes); chunked so even the 248320-row head needs +// a bounded bf16 staging buffer. +static bool frt_repack_build(const frt_repack_src * srcs, int n_src, int64_t N, int64_t K, + uint8_t * d_packed, uint8_t * d_sf, float * out_alpha, cudaStream_t stream) { + const int64_t CHUNK = 8192; + static __nv_bfloat16 * d_stage = nullptr; + static float * d_scr = nullptr; // [amax, gscale] + if (!d_stage) CUDA_CHECK(cudaMalloc(&d_stage, CHUNK * K * sizeof(__nv_bfloat16))); + if (!d_scr) CUDA_CHECK(cudaMalloc(&d_scr, 2 * sizeof(float))); + const int n_col_super = ((int) (K / 16) + 3) / 4; + CUDA_CHECK(cudaMemsetAsync(d_scr, 0, sizeof(float), stream)); + for (int pass = 0; pass < 2; ++pass) { // 0 = amax, 1 = quantize + int64_t row_base = 0; + for (int s = 0; s < n_src; ++s) { + const ggml_tensor * t = srcs[s].t; + const to_bf16_cuda_t conv = ggml_get_to_bf16_cuda(t->type); + if (conv == nullptr) return false; + const size_t row_bytes = ggml_row_size(t->type, K); + for (int64_t r0 = 0; r0 < srcs[s].rows; r0 += CHUNK) { + const int64_t rows = std::min(CHUNK, srcs[s].rows - r0); + conv((const char *) t->data + r0 * row_bytes, d_stage, rows * K, stream); + if (pass == 0) { + frt_w_amax_bf16<<>>(d_stage, d_scr, (int) rows, (int) K); + } else { + frt_w_pass2_bf16<<>>(d_stage, d_scr + 1, + d_packed, d_sf, (int) rows, (int) K, (int) (row_base + r0), n_col_super); + } + } + row_base += srcs[s].rows; + } + if (pass == 0) frt_w_gscale<<<1, 1, 0, stream>>>(d_scr, d_scr + 1); + } + CUDA_CHECK(cudaStreamSynchronize(stream)); + CUDA_CHECK(cudaMemcpy(out_alpha, d_scr + 1, sizeof(float), cudaMemcpyDeviceToHost)); + return *out_alpha != 0.0f; +} + +// FRT_REPACK_CHECK=1: byte-compare an online-built region against the pack +// file entry it replaces (pack path from FRT_REGIONS_PACK/FRT_HEAD_PACK). +static void frt_repack_check_region(int kind, int layer, const frt_region & r) { + static int check = -1; + if (check < 0) { const char * c = getenv("FRT_REPACK_CHECK"); check = (c && c[0] == '1') ? 1 : 0; } + if (!check) return; + const char * path = (kind == 5) ? getenv("FRT_HEAD_PACK_REF") : getenv("FRT_REGIONS_PACK_REF"); + if (!path) return; + const size_t pkb = (size_t) r.N * r.K / 2; + const size_t sfb = (size_t) ((r.N + 127) / 128) * ((r.K + 63) / 64) * 512; + std::vector ref(pkb > sfb ? pkb : sfb), got(pkb > sfb ? pkb : sfb); + FILE * f = fopen(path, "rb"); + if (!f) return; + bool found = false; + double ref_alpha = 0.0; + if (kind == 5) { // head pack: single entry + int64_t hdr[4]; + if (fread(hdr, 8, 4, f) == 4 && hdr[1] == r.N && hdr[2] == r.K) { + memcpy(&ref_alpha, &hdr[3], 8); + found = fread(ref.data(), 1, pkb, f) == pkb; + std::vector sfref(sfb); + if (found && fread(sfref.data(), 1, sfb, f) == sfb) { + CUDA_CHECK(cudaMemcpy(got.data(), r.d_packed, pkb, cudaMemcpyDeviceToHost)); + const bool pk_ok = memcmp(got.data(), ref.data(), pkb) == 0; + CUDA_CHECK(cudaMemcpy(got.data(), r.d_sf, sfb, cudaMemcpyDeviceToHost)); + const bool sf_ok = memcmp(got.data(), sfref.data(), sfb) == 0; + fprintf(stderr, "frt-repack-check head: packed=%s sf=%s alpha %.9g vs %.9g\n", + pk_ok ? "OK" : "MISMATCH", sf_ok ? "OK" : "MISMATCH", (double) r.alpha, ref_alpha); + } + } + fclose(f); + return; + } + int64_t hdr[2]; + if (fread(hdr, 8, 2, f) != 2) { fclose(f); return; } + for (int64_t e = 0; e < hdr[1]; ++e) { + int64_t el, ek, en, ekk, epkb, esfb; double ea; + if (fread(&el, 8, 1, f) != 1) break; + if (fread(&ek, 8, 1, f) != 1 || fread(&en, 8, 1, f) != 1 || fread(&ekk, 8, 1, f) != 1 || + fread(&ea, 8, 1, f) != 1 || fread(&epkb, 8, 1, f) != 1 || fread(&esfb, 8, 1, f) != 1) break; + if (el == layer && ek == kind) { + found = (epkb == (int64_t) pkb && esfb == (int64_t) sfb); + if (found) { + if (fread(ref.data(), 1, pkb, f) != pkb) break; + CUDA_CHECK(cudaMemcpy(got.data(), r.d_packed, pkb, cudaMemcpyDeviceToHost)); + const bool pk_ok = memcmp(got.data(), ref.data(), pkb) == 0; + if (fread(ref.data(), 1, sfb, f) != sfb) break; + CUDA_CHECK(cudaMemcpy(got.data(), r.d_sf, sfb, cudaMemcpyDeviceToHost)); + const bool sf_ok = memcmp(got.data(), ref.data(), sfb) == 0; + fprintf(stderr, "frt-repack-check kind%d layer%d: packed=%s sf=%s alpha %.9g vs %.9g\n", + kind, layer, pk_ok ? "OK" : "MISMATCH", sf_ok ? "OK" : "MISMATCH", (double) r.alpha, ea); + } + break; + } + fseek(f, epkb + esfb, SEEK_CUR); + } + fclose(f); + if (!found) fprintf(stderr, "frt-repack-check kind%d layer%d: no reference entry\n", kind, layer); +} + +// Pre-capture hook body: scan the graph for region/head weight tensors and +// build any missing online buffers. Eager only — the caller guarantees no +// CUDA graph capture is in flight. +static void frt_online_prepare(ggml_backend_cuda_context & ctx, const ggml_cgraph * cgraph) { + if (!frt_online_on()) return; + static bool all_done = false; + if (all_done) return; + if (!frt_regions_load() && !getenv("FRT_HEAD_SWAP") && !getenv("FRT_HEAD_DRAFT")) { all_done = true; return; } + + static const char * k0_names[4] = { "attn_qkv.weight", "attn_gate.weight", "ssm_alpha.weight", "ssm_beta.weight" }; + static const int64_t k0_rows[4] = { 8192, 4096, 32, 32 }; + static const char * k1_names[3] = { "attn_q.weight", "attn_k.weight", "attn_v.weight" }; + static const int64_t k1_rows[3] = { 8192, 512, 512 }; + + const ggml_tensor * mem[2][64][4] = {}; + const ggml_tensor * head_w = nullptr; + for (int i = 0; i < cgraph->n_nodes; ++i) { + const ggml_tensor * n = cgraph->nodes[i]; + if (n->op != GGML_OP_MUL_MAT || !n->src[0]) continue; + const ggml_tensor * w = n->src[0]; + if (strcmp(w->name, "output.weight") == 0 && w->type != GGML_TYPE_Q8_0 && w->ne[0] == 2048) { + head_w = w; + continue; + } + int layer = -1; char rest[64] = {0}; + if (sscanf(w->name, "blk.%d.%63s", &layer, rest) != 2 || layer < 0 || layer >= 64) continue; + for (int m = 0; m < 4; ++m) if (strcmp(rest, k0_names[m]) == 0 && w->ne[1] == k0_rows[m]) mem[0][layer][m] = w; + for (int m = 0; m < 3; ++m) if (strcmp(rest, k1_names[m]) == 0 && w->ne[1] == k1_rows[m]) mem[1][layer][m] = w; + } + + cudaStream_t stream = ctx.stream(); + int built_k0 = 0, built_k1 = 0; + for (int kind = 0; kind < 2; ++kind) { + if (kind == 0 && !g_reg.inproj_on) continue; + if (kind == 1 && !g_reg.attn_on) continue; + const int n_mem = kind == 0 ? 4 : 3; + const int64_t N = kind == 0 ? 12352 : 9216, K = 2048; + for (int layer = 0; layer < 64; ++layer) { + frt_region & r = g_reg.regions[kind][layer]; + if (r.N != 0) { (kind == 0 ? built_k0 : built_k1)++; continue; } + bool have = true; + for (int m = 0; m < n_mem; ++m) have = have && mem[kind][layer][m] != nullptr; + if (!have) continue; + frt_repack_src srcs[4]; + for (int m = 0; m < n_mem; ++m) srcs[m] = { mem[kind][layer][m], kind == 0 ? k0_rows[m] : k1_rows[m] }; + const size_t pkb = (size_t) N * K / 2; + const size_t sfb = (size_t) ((N + 127) / 128) * ((K + 63) / 64) * 512; + CUDA_CHECK(cudaMalloc(&r.d_packed, pkb)); + CUDA_CHECK(cudaMalloc(&r.d_sf, sfb)); + float alpha = 0.f; + if (!frt_repack_build(srcs, n_mem, N, K, r.d_packed, r.d_sf, &alpha, stream)) { + fprintf(stderr, "frt-repack: kind%d layer%d FAILED\n", kind, layer); + cudaFree(r.d_packed); cudaFree(r.d_sf); + r.d_packed = nullptr; r.d_sf = nullptr; + continue; + } + r.alpha = alpha; r.K = K; r.N = N; // N last: serve fires only on complete regions + frt_repack_check_region(kind, layer, r); + (kind == 0 ? built_k0 : built_k1)++; + } + } + + if (head_w && !g_head.ok && (getenv("FRT_HEAD_SWAP") || getenv("FRT_HEAD_DRAFT"))) { + const int64_t N = head_w->ne[1], K = head_w->ne[0]; + const size_t pkb = (size_t) N * K / 2; + const size_t sfb = (size_t) ((N + 127) / 128) * ((K + 63) / 64) * 512; + CUDA_CHECK(cudaMalloc(&g_head.d_packed, pkb)); + CUDA_CHECK(cudaMalloc(&g_head.d_sf, sfb)); + float alpha = 0.f; + frt_repack_src src = { head_w, N }; + if (frt_repack_build(&src, 1, N, K, g_head.d_packed, g_head.d_sf, &alpha, stream)) { + g_head.N = N; g_head.K = K; g_head.alpha = alpha; + CUDA_CHECK(cudaMalloc(&g_head.d_apack, 4 * (K / 2))); + CUDA_CHECK(cudaMalloc(&g_head.d_sfa, 128 * (K / 16))); + CUDA_CHECK(cudaMemset(g_head.d_sfa, 0, 128 * (K / 16))); + frt_init_ue4m3_lut(); + frt_region hr; hr.N = N; hr.K = K; hr.alpha = alpha; hr.d_packed = g_head.d_packed; hr.d_sf = g_head.d_sf; + frt_repack_check_region(5, 0, hr); + fprintf(stderr, "frt-repack: head online N=%lld K=%lld alpha=%g\n", (long long) N, (long long) K, (double) alpha); + g_head.ok = true; + } else { + cudaFree(g_head.d_packed); cudaFree(g_head.d_sf); + g_head.d_packed = nullptr; g_head.d_sf = nullptr; + } + } + + const bool head_pending = (getenv("FRT_HEAD_SWAP") || getenv("FRT_HEAD_DRAFT")) && !g_head.ok; + const bool k0_pending = g_reg.inproj_on && built_k0 < 30; + const bool k1_pending = g_reg.attn_on && built_k1 < 10; + if (!head_pending && !k0_pending && !k1_pending) { + fprintf(stderr, "frt-repack: online repack complete (kind0=%d kind1=%d head=%d)\n", + built_k0, built_k1, (int) g_head.ok); + all_done = true; + } +} + } // namespace frt // ---- GDN cell fusion (FRT_GDN_SWAP) --------------------------------------- @@ -2353,3 +2708,10 @@ bool ggml_cuda_frt_head_mul_mat(ggml_backend_cuda_context & ctx, out, frt::g_head.alpha, (int) N, (int) K, M, stream, 44); // head: s4w4 wins (+6 t/s) return true; } + +// Pre-capture hook: called at the start of every backend graph evaluation, +// before any CUDA graph capture can begin. Builds online-repacked weight +// buffers (FRT_ONLINE_REPACK=1) so no allocation ever happens mid-capture. +void ggml_cuda_frt_prepare(ggml_backend_cuda_context & ctx, const ggml_cgraph * cgraph) { + frt::frt_online_prepare(ctx, cgraph); +} From c8e1a8bc864bea4c455d6fd41b5310722fb22d8f Mon Sep 17 00:00:00 2001 From: LiangSu8899 <7thuniversels@gmail.com> Date: Tue, 25 Aug 2026 12:17:48 -0400 Subject: [PATCH 23/32] catalog: map the sm120/Qwen3.6 target onto real structure families MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - new family moe_expert_ffn v1: routed expert GLU-FFN with exact in-boundary top-k routing, optional sigmoid-gated shared expert, format-native weight consumption (host K-quant blocks in place, bit-exact activation-quant clone) and first-class token batches; reference implementation replicates the Qwen3.x softmax/top-k/clamp/renorm convention including tie-break. - gated_delta_core v2: snapshot_per_token state-update and replay_in_region checkpoint variants — the rollback discipline speculative verify batches require of stateful regions, with the failure modes documented. - autoregressive_decode_pipeline v3: optional spec_draft stage plus the speculative conformance set (output distribution matches target, explicit draft-state rollback, throughput judged on text). - rewrite the sm120 binding/pins against these families (previous ones were placeholders); qkv_pack/linear_proj variant additions are recorded as proposals instead of version bumps so the pi0.5 binding's pins stay green. Registry loads all 18 families; the existing pi0.5 pins verify green against the updated catalog. --- .../ggml/qualification/pins_qwen36_sm120.yaml | 25 +++-- .../bindings/llamacpp_qwen36_35b_sm120.yaml | 61 +++++++------ .../structure.yaml | 39 +++++++- .../catalog/gated_delta_core/structure.yaml | 22 ++++- .../catalog/moe_expert_ffn/reference.py | 84 +++++++++++++++++ .../catalog/moe_expert_ffn/structure.yaml | 91 +++++++++++++++++++ 6 files changed, 279 insertions(+), 43 deletions(-) create mode 100644 flash_rt/structures/catalog/moe_expert_ffn/reference.py create mode 100644 flash_rt/structures/catalog/moe_expert_ffn/structure.yaml diff --git a/flash_rt/structures/adapters/ggml/qualification/pins_qwen36_sm120.yaml b/flash_rt/structures/adapters/ggml/qualification/pins_qwen36_sm120.yaml index 84eb2ac6..61e8e165 100644 --- a/flash_rt/structures/adapters/ggml/qualification/pins_qwen36_sm120.yaml +++ b/flash_rt/structures/adapters/ggml/qualification/pins_qwen36_sm120.yaml @@ -1,13 +1,23 @@ # Structure versions the SM120/Qwen3.6 windows were written against, plus -# the target's shipping gates. The families marked proposed are carried by -# this target until they are formalized in the catalog; their pin is the -# adapter file itself (fr_win_qwen36_sm120.cu revision). +# the target's shipping gates. binding: llamacpp_qwen36_35b_sm120 pins: - llm_decode_pipeline: 1 - format_native_gemv: 1 # proposed family (warp-split-K NVFP4 GEMV, M<=4) - moe_expert_span: 1 # proposed family (format-native K-quant MoE) - gdn_cell: 1 # proposed family (recurrent cell span, snapshot/checkpoint discipline) + autoregressive_decode_pipeline: 3 # v3 adds the optional spec_draft stage + gated_delta_core: 2 # v2 adds snapshot_per_token / replay_in_region + moe_expert_ffn: 1 # new family introduced by this target + qkv_pack: 1 + linear_proj: 1 + +# Variant values this target uses that are NOT yet in the pinned families +# (deliberately deferred: qkv_pack and linear_proj are pinned at v1 by the +# Thor/pi0.5 binding, and a version bump turns its qualification RED; the +# additions land in a follow-up catalog change coordinated with that line): +# qkv_pack.in_dtype += fp4_block_scaled (NVFP4 W4A4 leaf pack) +# linear_proj.in_dtype += q8_native (host-convention q8_1 act, +# K-quant weights in place) +proposed_variants: + qkv_pack: {in_dtype: fp4_block_scaled} + linear_proj: {in_dtype: q8_native} # Shipping gates (re-established after every adapter or host change; the # numbers are the 2026-08-25 migration baseline on RTX 5090): @@ -15,6 +25,7 @@ gates: quality: ppl_24ch_ub1_safe_tier: 6.1106 # must stay bit-identical (wikitext, -ub 1 -c 512 --chunks 24) moefuse_selftest: pass # FRT_MOEFUSE_SELFTEST=1 duplicated-token bit-exact replay + repack_check: byte_identical # FRT_REPACK_CHECK=1 vs the offline packs (regions) spec_text_check: required # greedy spec output must be coherent text, never judged by tps alone perf: tg128_full_tier_bench: 377.8 # llama-bench r=5, full-tier env diff --git a/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml b/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml index 252f91f4..9a211951 100644 --- a/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml +++ b/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml @@ -1,5 +1,5 @@ binding: llamacpp_qwen36_35b_sm120 -structure: llm_decode_pipeline +structure: autoregressive_decode_pipeline # Host: stock llama.cpp (upstream master lineage) running Qwen3.6-35B-A3B # UD-Q4_K_M on RTX 5090 (SM120) with the native ggml adapter @@ -7,7 +7,9 @@ structure: llm_decode_pipeline # regions execute as fused subgraph windows matched inside ggml-cuda's # graph evaluation. Two quality tiers share one binary: the safe tier keeps # the target head and all matched math bit-comparable to stock (24-chunk -# PPL receipt), the full tier adds the NVFP4 lm-head swap. +# PPL receipt), the full tier adds the NVFP4 lm-head swap. The speculative +# form runs the pipeline's spec_draft stage through the model's trained MTP +# head (single head, 3 chained drafts/step, verify batches M = 4). stages: prefill: @@ -17,69 +19,68 @@ stages: seam: "llama_decode token generation, M in [1,4] (speculative verify batches)" capture: cuda_graphs_keyed spec_draft: - seam: "MTP draft chain (single trained head, 3 chained drafts/step) + M=4 h-extraction eval" + seam: "MTP draft chain evals + M=4 h-extraction eval (common/speculative draft-mtp driver)" capture: cuda_graphs_keyed cadences: - token: [decode] - spec_step: [decode, spec_draft] + token: [decode, token_select] + spec_step: [decode, token_select, spec_draft] coverage: contract: hot_path_takeover hot_path: - - gdn_in_proj_region - - attn_qkv_region + - gdn_in_proj_pack + - attn_qkv_pack - gdn_cell_span - moe_expert_span - - moe_router - out_proj - lm_head - spec_draft_serving segments: - - name: gdn_in_proj_region + - name: gdn_in_proj_pack stage: decode classification: structure - seam: "fused-region NVFP4 W4A4 GEMV serving qkv|z|a|b members from one staging pass (30 GDN layers)" - structures: [format_native_gemv] # proposed family: warp-split-K blockscale MMA GEMV, M<=4 - - name: attn_qkv_region + seam: "qkv|gate|alpha|beta sibling pack served from one NVFP4 W4A4 GEMV staging pass (30 GDN layers); leaf form — later members read the stash" + structures: [qkv_pack] + variants: {bind: leaf, in_dtype: fp4_block_scaled} # in_dtype value proposed, see pins + - name: attn_qkv_pack stage: decode classification: structure - seam: "attention qkv fused region, same GEMV (10 attention layers)" - structures: [format_native_gemv] + seam: "attention q|k|v sibling pack, same GEMV (10 attention layers)" + structures: [qkv_pack] + variants: {bind: leaf, in_dtype: fp4_block_scaled} - name: gdn_cell_span stage: decode classification: structure - seam: "conv + gated-delta-net + epilogue span -> 4 launches; M>1 writes per-token state/conv snapshots and replays checkpoint saves (rollback discipline, DEVELOPMENT.md)" - structures: [gdn_cell] # proposed family + seam: "conv + gated-delta recurrence + gated norm + epilogue as one 4-launch region; M>1 verify batches under snapshot_per_token + replay_in_region" + structures: [gated_delta_core] + variants: {phase: decode_recurrent, state_update: snapshot_per_token, qk_norm: l2_in_kernel, checkpoint: replay_in_region} - name: moe_expert_span stage: decode classification: structure - seam: "K0 quant+meta / K1 gate|up|GLU / K2 down+combine over ggml-native K-quant blocks (bit-exact q8_1 activation clone), shared expert folded, M<=4" - structures: [moe_expert_span] # proposed family: format-native MoE - - name: moe_router - stage: decode - classification: structure - seam: "fused gate GEMV + exact topk replication; default-on only for M>=2 (host's own fusion is M=1-only)" - structures: [format_native_gemv] + seam: "K0 quant+meta / K1 gate|up|GLU / K2 down+combine over ggml-native K-quant blocks; shared expert folded; router fused for M>=2 (host's own top-k fusion is M=1-only)" + structures: [moe_expert_ffn] + variants: {routing: fused, shared_expert: sigmoid_gated, weight_consumption: format_native, batch: m_le_4} - name: out_proj stage: decode classification: structure - seam: "ssm_out/attn_output Q8_0 GEMV + residual, consumes the GDN epilogue's q8 handoff" - structures: [format_native_gemv] + seam: "ssm_out/attn_output Q8_0 GEMV + residual epilogue, consuming the GDN epilogue's q8 activation handoff" + structures: [linear_proj] + variants: {epilogue: residual_add} # format_native weight consumption proposed, see pins - name: lm_head stage: decode classification: structure - seam: "full tier: NVFP4 W4A4 head swap (M<=4 in one pass); safe tier: stock Q6_K mmvq (measured ~91% BW, left alone)" - structures: [format_native_gemv] + seam: "full tier: NVFP4 W4A4 head (M<=4 one pass, weights from the BF16-checkpoint pack); safe tier: stock Q6_K mmvq (~91% BW, left alone)" + structures: [linear_proj] - name: spec_draft_serving stage: spec_draft classification: structure - seam: "draft-side head serving: the draft GGUF's Q8_0 lm-head copy is served from the FP4 pack (spec acceptance-only, output distribution unchanged); draft MTP layer's Q8_0 experts consume the same MoE span" - structures: [format_native_gemv, moe_expert_span] + seam: "draft-side lm-head serving (the draft GGUF's Q8_0 head copy from the FP4 pack — acceptance-only by spec math) + the draft MTP layer's Q8_0 experts through the same MoE span" + structures: [linear_proj, moe_expert_ffn] - name: host_graph_slots stage: spec_draft classification: host_stage - seam: "host-tree change (llama-context): per-shape graph slots keyed (n_tokens, gtype, has_embd), each slot owning its scheduler; plus target-side backend sampling (-bs)" + seam: "host-tree change (llama-context): per-shape graph slots keyed (n_tokens, gtype, has_embd), each owning its scheduler; target-side backend sampling (-bs)" hosts: llamacpp_dev_sm120: diff --git a/flash_rt/structures/catalog/autoregressive_decode_pipeline/structure.yaml b/flash_rt/structures/catalog/autoregressive_decode_pipeline/structure.yaml index 465b7adc..c1304eb6 100644 --- a/flash_rt/structures/catalog/autoregressive_decode_pipeline/structure.yaml +++ b/flash_rt/structures/catalog/autoregressive_decode_pipeline/structure.yaml @@ -1,7 +1,7 @@ structure: autoregressive_decode_pipeline kind: stage_pipeline family: autoregressive_decode -version: 2 +version: 3 description: > Schedule-layer structure for autoregressive generation: optional host input preparation and modality encoding feed a causal prefill that @@ -56,13 +56,32 @@ stages: - {name: token_id, window: swap} - {name: stop, window: host} capture: host_dependent + - name: spec_draft + optional: true + cadence: token + description: > + Speculative drafting between decode steps: a draft model (or trained + MTP head chain) proposes n_draft tokens; the next decode step becomes + a verify batch over 1 + n_draft rows whose logits feed acceptance. + Draft state (KV / recurrent / conv) advances tentatively and is rolled + back to the accepted position, which is what makes the + snapshot_per_token discipline of stateful embedded regions mandatory. + inputs: + - {name: token_id, window: swap} + - {name: target_hidden, buffer: mutable, optional: true} # MTP h rows + - {name: draft_state, buffer: mutable} + outputs: + - {name: draft_tokens, window: swap} + - {name: draft_state, buffer: mutable} + capture: bucketed_or_eager -embedded_regions: [decoder_ffn, vision_ffn, qkv_pack, qk_norm_rope, qkv_rope, - attention_core, gated_delta_core, linear_proj, - patch_projection] +embedded_regions: [decoder_ffn, vision_ffn, moe_expert_ffn, qkv_pack, + qk_norm_rope, qkv_rope, attention_core, gated_delta_core, + linear_proj, patch_projection] state_regions: - {name: kv_state, writers: [prefill, decode], reader: decode} + - {name: draft_state, writers: [spec_draft, decode], reader: spec_draft, optional: true} - {name: graph_buckets, cadence: shape_or_position} conformance: @@ -71,6 +90,18 @@ conformance: - token_selection_is_outside_decoder_blocks - modality_features_have_request_cadence - graph_bucket_fallback_is_reported + # speculative (spec_draft stage present): + - spec_output_distribution_matches_target # acceptance math guarantees the + # emitted tokens equal target-only + # decoding; draft quality may only + # change speed, never output + - draft_state_rollback_is_explicit # verify batches must leave state + # rollback-capable (snapshot + # discipline in stateful regions) + - spec_throughput_judged_on_text # tps under speculation is invalid + # evidence without verifying the + # generated text (broken rollback + # inflates acceptance and tps) gates: parity: diff --git a/flash_rt/structures/catalog/gated_delta_core/structure.yaml b/flash_rt/structures/catalog/gated_delta_core/structure.yaml index 68b69ac4..9406793d 100644 --- a/flash_rt/structures/catalog/gated_delta_core/structure.yaml +++ b/flash_rt/structures/catalog/gated_delta_core/structure.yaml @@ -1,5 +1,5 @@ structure: gated_delta_core -version: 1 +version: 2 description: > Stateful Gated DeltaNet recurrence over Q/K/V, log-decay and update strength. The structure owns the recurrent state transition and exposes the @@ -27,8 +27,21 @@ weights: [] variants: phase: [decode_recurrent, sequence_scan, wy_chunk] - state_update: [in_place, explicit_output] + state_update: [in_place, explicit_output, snapshot_per_token] + # snapshot_per_token: token batches (speculative verify, + # S = 1 + n_draft) write one state snapshot per token and + # leave the source state slot pristine, so the host can + # roll back to any accepted position; in-place update under + # a rollback-capable host is a correctness bug whose + # symptoms (degenerate text, inflated acceptance) evade + # perplexity-style gates — adjudicate on end-to-end text qk_norm: [l2_in_kernel, host] + checkpoint: [none, replay_in_region] + # replay_in_region: the host graph's normally zero-sized + # checkpoint save nodes become real on checkpoint steps and + # must be replayed inside the region (or the whole region + # must decline); silently dropping them corrupts rollback + # in ways invisible until a rollback occurs calibration: points: [] @@ -53,6 +66,11 @@ qualification: produce a final state accepted by the next invocation" evidence: + - "Qwen3.6-35B-A3B decode on RTX 5090 (ggml adapter sm120 target): whole + cell span (conv + recurrence + gated norm + epilogue) as 4 launches, + register-resident state, M<=4 verify batches under snapshot_per_token + + replay_in_region; bit-exact duplicated-token replay across batch + variants is the acceptance gate" - "Qwen3.6/Qwen3.5 hybrid decoder: 48 of 64 layers carry this state" - "Nex-N2 hybrid decoder uses the same recurrent and WY chunk boundary at H=32 while Qwen3.6 uses H=48" diff --git a/flash_rt/structures/catalog/moe_expert_ffn/reference.py b/flash_rt/structures/catalog/moe_expert_ffn/reference.py new file mode 100644 index 00000000..acb8602e --- /dev/null +++ b/flash_rt/structures/catalog/moe_expert_ffn/reference.py @@ -0,0 +1,84 @@ +"""Ground-truth reference for the sparse MoE feed-forward block. + +Plainest possible PyTorch, never executed on a serving hot path. Routing +replicates the Qwen3.x convention exactly: softmax over all experts, +iterative top-k with lower-index tie-break, clamp, renormalize. +""" + +from __future__ import annotations + +import torch +import torch.nn.functional as F + + +def _route_softmax_topk_clamp_renorm( + logits: torch.Tensor, k_top: int, clamp_min: float = 1e-20 +) -> tuple[torch.Tensor, torch.Tensor]: + """Softmax over E -> iterative top-k (lower index wins ties) -> renorm.""" + probs = torch.softmax(logits.float(), dim=-1) + m, _ = probs.shape + ids = torch.empty(m, k_top, dtype=torch.int64, device=logits.device) + vals = torch.empty(m, k_top, dtype=torch.float32, device=logits.device) + work = probs.clone() + for j in range(k_top): + # argmax returns the first (lowest-index) maximum, matching the + # host convention this structure binds to. + idx = work.argmax(dim=-1) + ids[:, j] = idx + vals[:, j] = work.gather(-1, idx[:, None]).squeeze(-1) + work.scatter_(-1, idx[:, None], float("-inf")) + vals = vals.clamp_min(clamp_min) + vals = vals / vals.sum(dim=-1, keepdim=True) + return ids, vals + + +def moe_expert_ffn_ref( + x: torch.Tensor, + w_gate_exps: torch.Tensor, + w_up_exps: torch.Tensor, + w_down_exps: torch.Tensor, + *, + w_router: torch.Tensor | None = None, + expert_ids: torch.Tensor | None = None, + expert_weights: torch.Tensor | None = None, + k_top: int = 8, + w_gate_shexp: torch.Tensor | None = None, + w_up_shexp: torch.Tensor | None = None, + w_down_shexp: torch.Tensor | None = None, + w_gate_inp_shexp: torch.Tensor | None = None, + residual: torch.Tensor | None = None, +) -> torch.Tensor: + """Routed expert GLU-FFN with optional sigmoid-gated shared expert. + + Either ``w_router`` (fused routing) or ``expert_ids``/``expert_weights`` + (external routing) must be provided. + """ + if expert_ids is None: + assert w_router is not None, "fused routing needs w_router" + logits = x.float() @ w_router.float() + expert_ids, expert_weights = _route_softmax_topk_clamp_renorm(logits, k_top) + assert expert_weights is not None + + m = x.shape[0] + y = torch.zeros(m, w_down_exps.shape[-1], dtype=torch.float32, device=x.device) + for t in range(m): + xt = x[t].float() + for j in range(expert_ids.shape[1]): + e = int(expert_ids[t, j]) + g = xt @ w_gate_exps[e].float() + u = xt @ w_up_exps[e].float() + h = F.silu(g) * u + y[t] += float(expert_weights[t, j]) * (h @ w_down_exps[e].float()) + + if w_gate_shexp is not None: + assert w_up_shexp is not None and w_down_shexp is not None + assert w_gate_inp_shexp is not None + for t in range(m): + xt = x[t].float() + sig = torch.sigmoid(xt @ w_gate_inp_shexp.float()) + h = F.silu(xt @ w_gate_shexp.float()) * (xt @ w_up_shexp.float()) + y[t] += sig * (h @ w_down_shexp.float()) + + if residual is not None: + y = y + residual.float() + return y.to(x.dtype) diff --git a/flash_rt/structures/catalog/moe_expert_ffn/structure.yaml b/flash_rt/structures/catalog/moe_expert_ffn/structure.yaml new file mode 100644 index 00000000..778b7aa5 --- /dev/null +++ b/flash_rt/structures/catalog/moe_expert_ffn/structure.yaml @@ -0,0 +1,91 @@ +structure: moe_expert_ffn +version: 1 +description: > + Sparse mixture-of-experts feed-forward block: route x[M,D] over E experts, + gather the top-k expert gate/up/down triples, activation(gate) * up per + expert, down projection, routing-weighted combine, plus an optional + always-on shared expert whose output is sigmoid-gated by its own scalar + projection. The routing decision (softmax/top-k/renorm) is part of the + boundary under the fused variant so a host whose own router fusion only + covers part of the batch space can hand the whole block over; under the + external variant the block consumes precomputed ids/weights. Per-token + expert sets are independent, so token batches (speculative verify, M > 1) + are first-class: the executable form must not assume M == 1. + +reference: + module: moe_expert_ffn.reference + entrypoint: moe_expert_ffn_ref + +boundary: + symbolic_dims: [M, D, F, E, K_top] + inputs: + - {name: x, dims: [M, D], dtype: "@binding"} + - {name: expert_ids, dims: [M, K_top], dtype: int32, optional: true} # external routing + - {name: expert_weights, dims: [M, K_top], dtype: "@binding", optional: true} + - {name: residual, dims: [M, D], dtype: "@binding", optional: true} + outputs: + - {name: y, dims: [M, D], dtype: "@binding"} + +weights: + - {slot: w_router, dims: [D, E], optional: true} # fused routing only + - {slot: w_gate_exps, dims: [E, D, F]} + - {slot: w_up_exps, dims: [E, D, F]} + - {slot: w_down_exps, dims: [E, F, D]} + - {slot: w_gate_shexp, dims: [D, F_sh], optional: true} + - {slot: w_up_shexp, dims: [D, F_sh], optional: true} + - {slot: w_down_shexp, dims: [F_sh, D], optional: true} + - {slot: w_gate_inp_shexp, dims: [D], optional: true} # sigmoid gate scalar + +variants: + routing: [external, fused] # fused = router GEMM + exact top-k in-boundary + topk_norm: [softmax_topk_clamp_renorm] # Qwen3.x convention: softmax over E, + # iterative top-k with lower-index + # tie-break, clamp, renormalize + shared_expert: [none, sigmoid_gated] + activation: [silu] + weight_consumption: [dequant_gemm, format_native] + # format_native = expert weights are + # consumed in the host's storage + # format (e.g. GGUF K-quant blocks + # through the host's own vec_dot + # device functions, activations + # cloned bit-exactly into the host's + # q8 convention) — no repack, no + # dequant materialization + batch: [m1, m_le_4] + +calibration: + points: [x] + +gates: + parity: + metrics: [cosine, max_abs, p99_abs] + data: real_distribution + latency: + baselines: [host_fused_moe, unfused_chain, vendor] + rule: net_positive_including_boundary + per_shape: true + qualification: + routing_is_exact: true # fused routing must replicate the + # host's tie-break and normalization + # bit-for-bit, adjudicated on logits + per_token_expert_sets_independent: true + +qualification: + - "expert_ids/expert_weights rows may be strided views of a sort output; + the row stride is part of the boundary, never assumed dense" + - "the format_native form must reproduce the host's activation-quant + convention exactly (adjudicated by a duplicated-token bit-exact replay + across batch variants: an M=2 batch of identical tokens must reproduce + the M=1 result bit-for-bit in every row)" + - "the combine (weighted sum + shared expert + residual) is part of the + boundary; outputs may alias dead routing inputs under the host's graph + allocator, so inputs the kernel still needs must be snapshotted before + the first output write" + +evidence: + - "Qwen3.6-35B-A3B (E=256, K_top=8, shared expert) on RTX 5090: 3-kernel + span over GGUF Q4_K/Q5_K/Q6_K expert weights, shared-expert folding + +21.6 tok/s, whole-span takeover net +2.2 at M=1 after PDL alignment; + M<=4 speculative verify batches served by the same span" + - "same span serves the Q8_0 expert weights of the model's MTP draft layer" From 87e6a30aeaddb47201ab2f64ad8df4bdf904a79d Mon Sep 17 00:00:00 2001 From: LiangSu8899 <7thuniversels@gmail.com> Date: Tue, 25 Aug 2026 13:06:24 -0400 Subject: [PATCH 24/32] =?UTF-8?q?catalog:=20charter=20=E2=80=94=20structur?= =?UTF-8?q?es=20express=20boundaries,=20never=20adjudicate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the catalog charter (catalog/README.md): a structure entry is a local boundary expression — dimensions, contracts, variant semantics, executable reference — serving context alignment against native pipelines and the torch-side distribution boundary. It adjudicates nothing: performance numbers, verdicts and case histories are conditional on model/hardware/ host/driver and expire silently, so the only arbiter is a test run against the live system; dated results live in campaign records and per-binding qualification gates that are re-established by re-running them. Scrub the recent entries accordingly: drop campaign evidence and throughput numbers from moe_expert_ffn and gated_delta_core, reduce variant comments to pure semantics, drop the judgment-phrased speculative conformance line from autoregressive_decode_pipeline, keep only the parity-vs-reference gate on the new family. --- .../bindings/llamacpp_qwen36_35b_sm120.yaml | 2 +- flash_rt/structures/catalog/README.md | 40 ++++++++++++++ .../structure.yaml | 21 +++----- .../catalog/gated_delta_core/structure.yaml | 24 +++------ .../catalog/moe_expert_ffn/structure.yaml | 52 +++++-------------- 5 files changed, 69 insertions(+), 70 deletions(-) create mode 100644 flash_rt/structures/catalog/README.md diff --git a/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml b/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml index 9a211951..fa3076e4 100644 --- a/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml +++ b/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml @@ -70,7 +70,7 @@ coverage: - name: lm_head stage: decode classification: structure - seam: "full tier: NVFP4 W4A4 head (M<=4 one pass, weights from the BF16-checkpoint pack); safe tier: stock Q6_K mmvq (~91% BW, left alone)" + seam: "full tier: NVFP4 W4A4 head (M<=4 one pass, weights from the BF16-checkpoint pack); safe tier: stock Q6_K mmvq" structures: [linear_proj] - name: spec_draft_serving stage: spec_draft diff --git a/flash_rt/structures/catalog/README.md b/flash_rt/structures/catalog/README.md new file mode 100644 index 00000000..60011398 --- /dev/null +++ b/flash_rt/structures/catalog/README.md @@ -0,0 +1,40 @@ +# Structure catalog + +## What this is + +A structure entry is a **local boundary expression**: what a computation +region is — its symbolic dimensions, inputs/outputs, weight slots, variant +semantics — plus an executable reference implementation that defines what +"the same computation" means. The catalog exists for two purposes: + +1. **Context alignment against native pipelines.** When an adapter or a + host integration is being built (or read), the catalog is the map that + says which region of the host graph corresponds to which structure and + under which variant — so N host implementations of one boundary can be + compared, ported, and reasoned about as one thing. +2. **Distribution boundary management on the torch side** — the frontend, + discovery, and swap machinery consume these boundaries to decide what + can be handed to an implementation and at what seam. + +## What this is not + +**The catalog adjudicates nothing.** No performance claims, no expected +wins, no negative results, no campaign case histories, no tuning guidance. +Every judgment of that kind is conditional on a model, a hardware +generation, a driver, and a host version, and it expires the moment any of +those move — the only arbiter of whether an implementation is correct or +faster is a test run against the live system, never a statement recorded +here. Dated results belong to campaign records and per-binding +qualification gates, which are re-established by re-running them, not by +being quoted. + +Practically, for an entry in this directory: + +- **Belongs here**: boundary math, dimension/stride contracts, variant + *semantics* (including correctness-critical ones such as state snapshot + or rollback semantics — properties of the computation itself), the + executable reference, version numbers. +- **Does not belong here**: throughput or latency numbers, "X was judged + negative/positive", hardware-specific observations, host-specific war + stories, anything phrased as a verdict. If it can go stale without this + file changing, it goes elsewhere. diff --git a/flash_rt/structures/catalog/autoregressive_decode_pipeline/structure.yaml b/flash_rt/structures/catalog/autoregressive_decode_pipeline/structure.yaml index c1304eb6..c0e9b50b 100644 --- a/flash_rt/structures/catalog/autoregressive_decode_pipeline/structure.yaml +++ b/flash_rt/structures/catalog/autoregressive_decode_pipeline/structure.yaml @@ -63,9 +63,8 @@ stages: Speculative drafting between decode steps: a draft model (or trained MTP head chain) proposes n_draft tokens; the next decode step becomes a verify batch over 1 + n_draft rows whose logits feed acceptance. - Draft state (KV / recurrent / conv) advances tentatively and is rolled - back to the accepted position, which is what makes the - snapshot_per_token discipline of stateful embedded regions mandatory. + Draft state (KV / recurrent / conv) advances tentatively and is + rolled back to the accepted position. inputs: - {name: token_id, window: swap} - {name: target_hidden, buffer: mutable, optional: true} # MTP h rows @@ -91,17 +90,11 @@ conformance: - modality_features_have_request_cadence - graph_bucket_fallback_is_reported # speculative (spec_draft stage present): - - spec_output_distribution_matches_target # acceptance math guarantees the - # emitted tokens equal target-only - # decoding; draft quality may only - # change speed, never output - - draft_state_rollback_is_explicit # verify batches must leave state - # rollback-capable (snapshot - # discipline in stateful regions) - - spec_throughput_judged_on_text # tps under speculation is invalid - # evidence without verifying the - # generated text (broken rollback - # inflates acceptance and tps) + - spec_output_distribution_matches_target # emitted tokens equal target-only + # decoding; the draft affects + # speed, never output + - draft_state_rollback_is_explicit # verify batches leave every + # tentative state rollback-capable gates: parity: diff --git a/flash_rt/structures/catalog/gated_delta_core/structure.yaml b/flash_rt/structures/catalog/gated_delta_core/structure.yaml index 9406793d..f05fbced 100644 --- a/flash_rt/structures/catalog/gated_delta_core/structure.yaml +++ b/flash_rt/structures/catalog/gated_delta_core/structure.yaml @@ -28,20 +28,15 @@ weights: [] variants: phase: [decode_recurrent, sequence_scan, wy_chunk] state_update: [in_place, explicit_output, snapshot_per_token] - # snapshot_per_token: token batches (speculative verify, - # S = 1 + n_draft) write one state snapshot per token and - # leave the source state slot pristine, so the host can - # roll back to any accepted position; in-place update under - # a rollback-capable host is a correctness bug whose - # symptoms (degenerate text, inflated acceptance) evade - # perplexity-style gates — adjudicate on end-to-end text + # snapshot_per_token: each token of a batch writes its own + # state snapshot and the source state slot stays pristine, + # so the host can roll back to any position qk_norm: [l2_in_kernel, host] checkpoint: [none, replay_in_region] - # replay_in_region: the host graph's normally zero-sized - # checkpoint save nodes become real on checkpoint steps and - # must be replayed inside the region (or the whole region - # must decline); silently dropping them corrupts rollback - # in ways invisible until a rollback occurs + # replay_in_region: checkpoint saves that the host graph + # places inside the region's span are part of the boundary + # and must be reproduced by the region (or the region must + # decline the whole span) calibration: points: [] @@ -66,11 +61,6 @@ qualification: produce a final state accepted by the next invocation" evidence: - - "Qwen3.6-35B-A3B decode on RTX 5090 (ggml adapter sm120 target): whole - cell span (conv + recurrence + gated norm + epilogue) as 4 launches, - register-resident state, M<=4 verify batches under snapshot_per_token - + replay_in_region; bit-exact duplicated-token replay across batch - variants is the acceptance gate" - "Qwen3.6/Qwen3.5 hybrid decoder: 48 of 64 layers carry this state" - "Nex-N2 hybrid decoder uses the same recurrent and WY chunk boundary at H=32 while Qwen3.6 uses H=48" diff --git a/flash_rt/structures/catalog/moe_expert_ffn/structure.yaml b/flash_rt/structures/catalog/moe_expert_ffn/structure.yaml index 778b7aa5..eb38bb04 100644 --- a/flash_rt/structures/catalog/moe_expert_ffn/structure.yaml +++ b/flash_rt/structures/catalog/moe_expert_ffn/structure.yaml @@ -6,11 +6,9 @@ description: > expert, down projection, routing-weighted combine, plus an optional always-on shared expert whose output is sigmoid-gated by its own scalar projection. The routing decision (softmax/top-k/renorm) is part of the - boundary under the fused variant so a host whose own router fusion only - covers part of the batch space can hand the whole block over; under the - external variant the block consumes precomputed ids/weights. Per-token - expert sets are independent, so token batches (speculative verify, M > 1) - are first-class: the executable form must not assume M == 1. + boundary under the fused variant; under the external variant the block + consumes precomputed ids/weights. Per-token expert sets are independent, + so token batches (M > 1) are part of the boundary, not a special case. reference: module: moe_expert_ffn.reference @@ -38,19 +36,15 @@ weights: variants: routing: [external, fused] # fused = router GEMM + exact top-k in-boundary - topk_norm: [softmax_topk_clamp_renorm] # Qwen3.x convention: softmax over E, - # iterative top-k with lower-index - # tie-break, clamp, renormalize + topk_norm: [softmax_topk_clamp_renorm] # softmax over E, iterative top-k with + # lower-index tie-break, clamp, renormalize shared_expert: [none, sigmoid_gated] activation: [silu] weight_consumption: [dequant_gemm, format_native] # format_native = expert weights are - # consumed in the host's storage - # format (e.g. GGUF K-quant blocks - # through the host's own vec_dot - # device functions, activations - # cloned bit-exactly into the host's - # q8 convention) — no repack, no + # consumed in the host's own storage + # format, activations quantized to the + # host's own convention — no repack, no # dequant materialization batch: [m1, m_le_4] @@ -61,31 +55,13 @@ gates: parity: metrics: [cosine, max_abs, p99_abs] data: real_distribution - latency: - baselines: [host_fused_moe, unfused_chain, vendor] - rule: net_positive_including_boundary - per_shape: true - qualification: - routing_is_exact: true # fused routing must replicate the - # host's tie-break and normalization - # bit-for-bit, adjudicated on logits - per_token_expert_sets_independent: true -qualification: +boundary_notes: - "expert_ids/expert_weights rows may be strided views of a sort output; the row stride is part of the boundary, never assumed dense" - - "the format_native form must reproduce the host's activation-quant - convention exactly (adjudicated by a duplicated-token bit-exact replay - across batch variants: an M=2 batch of identical tokens must reproduce - the M=1 result bit-for-bit in every row)" + - "the routing decision under the fused variant is part of the boundary, + including the tie-break and normalization semantics" + - "per-token expert sets are independent: batch variants of the same token + denote the same computation row for row" - "the combine (weighted sum + shared expert + residual) is part of the - boundary; outputs may alias dead routing inputs under the host's graph - allocator, so inputs the kernel still needs must be snapshotted before - the first output write" - -evidence: - - "Qwen3.6-35B-A3B (E=256, K_top=8, shared expert) on RTX 5090: 3-kernel - span over GGUF Q4_K/Q5_K/Q6_K expert weights, shared-expert folding - +21.6 tok/s, whole-span takeover net +2.2 at M=1 after PDL alignment; - M<=4 speculative verify batches served by the same span" - - "same span serves the Q8_0 expert weights of the model's MTP draft layer" + boundary" From 83e828f1773e1cac2dd77668aea787e39717d110 Mon Sep 17 00:00:00 2001 From: LiangSu8899 <7thuniversels@gmail.com> Date: Tue, 25 Aug 2026 14:15:21 -0400 Subject: [PATCH 25/32] ggml adapter sm120: window code consumes the binding (model facts unsoldered) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The binding yaml becomes the single source for every model-specific constant in the window translation unit: qkv_pack member tables (names, rows, offsets, leaders), region shapes and layer counts, gated-delta cell dims (conv row, state size), out-proj/head names and dims, layer scan bounds. tools/gen_binding_header.py generates a checked-in constants header from the binding's host_params section; the window logic itself is now family-level — a same-family model is a new binding file plus a header regeneration, with no window-code edit. Framework-wide ggml naming conventions (ffn_gate_exps, ffn_moe_out, ...) stay in the window code deliberately: they are host facts shared by every model on this host, not model facts. Also brings the binding up to the strict loader contract (token_select stage, complete_hot_path coverage with stock-served segments declared as host stages); it now validates through flash_rt.structures.binding with pipeline coverage required. Receipts: online repack byte-identity 40/40 vs the offline packs; tg128 and 24-chunk perplexity unchanged; speculative smoke coherent. --- .../ggml/fr_binding_qwen36_35b_sm120.h | 56 +++++++++ .../adapters/ggml/fr_win_qwen36_sm120.cu | 118 ++++++++++-------- .../adapters/ggml/tools/gen_binding_header.py | 115 +++++++++++++++++ .../bindings/llamacpp_qwen36_35b_sm120.yaml | 112 ++++++++++++++--- 4 files changed, 333 insertions(+), 68 deletions(-) create mode 100644 flash_rt/structures/adapters/ggml/fr_binding_qwen36_35b_sm120.h create mode 100644 flash_rt/structures/adapters/ggml/tools/gen_binding_header.py diff --git a/flash_rt/structures/adapters/ggml/fr_binding_qwen36_35b_sm120.h b/flash_rt/structures/adapters/ggml/fr_binding_qwen36_35b_sm120.h new file mode 100644 index 00000000..22b1566b --- /dev/null +++ b/flash_rt/structures/adapters/ggml/fr_binding_qwen36_35b_sm120.h @@ -0,0 +1,56 @@ +// Generated by tools/gen_binding_header.py from +// flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml — edit the binding, not this file. +#pragma once + +namespace frt_binding { + +constexpr int n_layer = 40; +constexpr int layer_scan_max = 64; +constexpr int d_model = 2048; + +// fused qkv_pack regions (kind-indexed) +struct region_member { const char * name; int rows; int off; bool leader; }; +constexpr int n_region_kinds = 2; +constexpr int region_max_members = 4; +constexpr int region_layers[2] = {30, 10}; +constexpr int region_k[2] = {2048, 2048}; +constexpr int region_n[2] = {12352, 9216}; // sum of member rows +constexpr int region_n_members[2] = {4, 3}; +constexpr region_member region_members[2][4] = { + {{"attn_qkv.weight", 8192, 0, true}, {"attn_gate.weight", 4096, 8192, false}, {"ssm_alpha.weight", 32, 12288, false}, {"ssm_beta.weight", 32, 12320, false}}, + {{"attn_q.weight", 8192, 0, true}, {"attn_k.weight", 512, 8192, false}, {"attn_v.weight", 512, 8704, false}, {nullptr, 0, 0, false}}, +}; + +// gated_delta_core instance dims +constexpr int gdn_qkvz_rows = 12288; +constexpr int gdn_qkv_rows = 8192; +constexpr int gdn_z_rows = 4096; +constexpr int gdn_conv_channels = 8192; +constexpr int gdn_conv_window = 3; +constexpr int gdn_conv_cache_row = 24576; +constexpr int gdn_state_size = 524288; +constexpr int gdn_out_dim = 4096; +constexpr int gdn_head_dim = 128; + +// moe_expert_ffn instance (dims are read from tensors at run time) +constexpr const char * moe_gate_exps = "ffn_gate_exps"; +constexpr const char * moe_up_exps = "ffn_up_exps"; +constexpr const char * moe_down_exps = "ffn_down_exps"; +constexpr const char * moe_gate_shexp = "ffn_gate_shexp.weight"; +constexpr const char * moe_up_shexp = "ffn_up_shexp.weight"; +constexpr const char * moe_down_shexp = "ffn_down_shexp.weight"; +constexpr const char * moe_gate_inp_shexp = "ffn_gate_inp_shexp.weight"; +constexpr int moe_n_expert = 256; +constexpr const char * moe_draft_prefix = "mtp_"; +constexpr const char * moe_out_names[2] = {"ffn_moe_out", "ffn_out"}; +constexpr const char * moe_resid_names[2] = {"l_out", "mtp_post_ffn"}; + +// out-proj linear_proj instances +constexpr const char * out_proj_names[2] = {"ssm_out.weight", "attn_output.weight"}; +constexpr int out_proj_k = 4096; + +// lm head +constexpr const char * head_name = "output.weight"; +constexpr int head_n_vocab = 248320; + +} // namespace frt_binding diff --git a/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu b/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu index 46b2647b..075e67f7 100644 --- a/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu +++ b/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu @@ -23,6 +23,11 @@ #include "vecdotq.cuh" #include "convert.cuh" +// Model-specific constants come from the binding (single source: +// flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml); regenerate +// the header with tools/gen_binding_header.py after editing the binding. +#include "fr_binding_qwen36_35b_sm120.h" + #include #include #include @@ -488,9 +493,12 @@ static bool frt_regions_load(void) { const char * path = getenv("FRT_REGIONS_PACK"); if (!path && frt_online_on()) { // online repack: per-region weight buffers arrive from the pre-capture - // hook; only the shared serve buffers are sized here (known target - // shapes: kind0 12352x2048, kind1 9216x2048). - const int64_t maxN = 12352, maxK = 2048; + // hook; only the shared serve buffers are sized here, from the binding. + int64_t maxN = 0, maxK = 0; + for (int kind = 0; kind < frt_binding::n_region_kinds; ++kind) { + maxN = std::max(maxN, frt_binding::region_n[kind]); + maxK = std::max(maxK, frt_binding::region_k[kind]); + } CUDA_CHECK(cudaMalloc(&g_reg.d_staging, 4 * maxN * sizeof(float))); CUDA_CHECK(cudaMalloc(&g_reg.d_conv_out, 4 * 8192 * sizeof(float))); CUDA_CHECK(cudaMalloc(&g_reg.d_attn_buf, 4 * 4096 * sizeof(float))); @@ -585,24 +593,22 @@ static bool frt_regions_mul_mat(ggml_backend_cuda_context & ctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) { if (!frt_regions_load()) return false; int layer = -1; char rest[64] = {0}; - if (sscanf(src0->name, "blk.%d.%63s", &layer, rest) != 2 || layer < 0 || layer >= 64) return false; - if (g_reg.inproj_on) { - if (strcmp(rest, "attn_qkv.weight") == 0) return frt_region_serve(ctx, 0, layer, true, 0, 8192, src1, dst); - if (strcmp(rest, "attn_gate.weight") == 0) return frt_region_serve(ctx, 0, layer, false, 8192, 4096, src1, dst); - if (strcmp(rest, "ssm_alpha.weight") == 0) return frt_region_serve(ctx, 0, layer, false, 12288, 32, src1, dst); - if (strcmp(rest, "ssm_beta.weight") == 0) return frt_region_serve(ctx, 0, layer, false, 12288 + 32, 32, src1, dst); - } - if (g_reg.attn_on) { - if (strcmp(rest, "attn_q.weight") == 0) return frt_region_serve(ctx, 1, layer, true, 0, 8192, src1, dst); - if (strcmp(rest, "attn_k.weight") == 0) return frt_region_serve(ctx, 1, layer, false, 8192, 512, src1, dst); - if (strcmp(rest, "attn_v.weight") == 0) return frt_region_serve(ctx, 1, layer, false, 8704, 512, src1, dst); + if (sscanf(src0->name, "blk.%d.%63s", &layer, rest) != 2 || layer < 0 || layer >= frt_binding::layer_scan_max) return false; + for (int kind = 0; kind < frt_binding::n_region_kinds; ++kind) { + if (kind == 0 && !g_reg.inproj_on) continue; + if (kind == 1 && !g_reg.attn_on) continue; + for (int m = 0; m < frt_binding::region_n_members[kind]; ++m) { + const auto & mem = frt_binding::region_members[kind][m]; + if (strcmp(rest, mem.name) == 0) + return frt_region_serve(ctx, kind, layer, mem.leader, mem.off, mem.rows, src1, dst); + } } if (g_reg.shexp_on) { if (strcmp(rest, "ffn_gate_shexp.weight") == 0) return frt_region_serve(ctx, 2, layer, true, 0, 512, src1, dst); if (strcmp(rest, "ffn_up_shexp.weight") == 0) return frt_region_serve(ctx, 2, layer, false, 512, 512, src1, dst); } if (g_reg.outproj_on) { - if (strcmp(rest, "ssm_out.weight") == 0 || strcmp(rest, "attn_output.weight") == 0) + if (strcmp(rest, frt_binding::out_proj_names[0]) == 0 || strcmp(rest, frt_binding::out_proj_names[1]) == 0) return frt_region_serve(ctx, 4, layer, true, 0, 2048, src1, dst); } return false; @@ -914,42 +920,40 @@ static void frt_online_prepare(ggml_backend_cuda_context & ctx, const ggml_cgrap if (all_done) return; if (!frt_regions_load() && !getenv("FRT_HEAD_SWAP") && !getenv("FRT_HEAD_DRAFT")) { all_done = true; return; } - static const char * k0_names[4] = { "attn_qkv.weight", "attn_gate.weight", "ssm_alpha.weight", "ssm_beta.weight" }; - static const int64_t k0_rows[4] = { 8192, 4096, 32, 32 }; - static const char * k1_names[3] = { "attn_q.weight", "attn_k.weight", "attn_v.weight" }; - static const int64_t k1_rows[3] = { 8192, 512, 512 }; - - const ggml_tensor * mem[2][64][4] = {}; + using namespace frt_binding; + const ggml_tensor * mem[n_region_kinds][64][region_max_members] = {}; const ggml_tensor * head_w = nullptr; for (int i = 0; i < cgraph->n_nodes; ++i) { const ggml_tensor * n = cgraph->nodes[i]; if (n->op != GGML_OP_MUL_MAT || !n->src[0]) continue; const ggml_tensor * w = n->src[0]; - if (strcmp(w->name, "output.weight") == 0 && w->type != GGML_TYPE_Q8_0 && w->ne[0] == 2048) { + if (strcmp(w->name, head_name) == 0 && w->type != GGML_TYPE_Q8_0 && w->ne[0] == d_model) { head_w = w; continue; } int layer = -1; char rest[64] = {0}; - if (sscanf(w->name, "blk.%d.%63s", &layer, rest) != 2 || layer < 0 || layer >= 64) continue; - for (int m = 0; m < 4; ++m) if (strcmp(rest, k0_names[m]) == 0 && w->ne[1] == k0_rows[m]) mem[0][layer][m] = w; - for (int m = 0; m < 3; ++m) if (strcmp(rest, k1_names[m]) == 0 && w->ne[1] == k1_rows[m]) mem[1][layer][m] = w; + if (sscanf(w->name, "blk.%d.%63s", &layer, rest) != 2 || layer < 0 || layer >= layer_scan_max) continue; + for (int kind = 0; kind < n_region_kinds; ++kind) + for (int m = 0; m < region_n_members[kind]; ++m) + if (strcmp(rest, region_members[kind][m].name) == 0 && w->ne[1] == region_members[kind][m].rows) + mem[kind][layer][m] = w; } cudaStream_t stream = ctx.stream(); - int built_k0 = 0, built_k1 = 0; - for (int kind = 0; kind < 2; ++kind) { + int built[n_region_kinds] = {}; + for (int kind = 0; kind < n_region_kinds; ++kind) { if (kind == 0 && !g_reg.inproj_on) continue; if (kind == 1 && !g_reg.attn_on) continue; - const int n_mem = kind == 0 ? 4 : 3; - const int64_t N = kind == 0 ? 12352 : 9216, K = 2048; - for (int layer = 0; layer < 64; ++layer) { + const int n_mem = region_n_members[kind]; + const int64_t N = region_n[kind], K = region_k[kind]; + for (int layer = 0; layer < layer_scan_max; ++layer) { frt_region & r = g_reg.regions[kind][layer]; - if (r.N != 0) { (kind == 0 ? built_k0 : built_k1)++; continue; } + if (r.N != 0) { built[kind]++; continue; } bool have = true; for (int m = 0; m < n_mem; ++m) have = have && mem[kind][layer][m] != nullptr; if (!have) continue; - frt_repack_src srcs[4]; - for (int m = 0; m < n_mem; ++m) srcs[m] = { mem[kind][layer][m], kind == 0 ? k0_rows[m] : k1_rows[m] }; + frt_repack_src srcs[region_max_members]; + for (int m = 0; m < n_mem; ++m) srcs[m] = { mem[kind][layer][m], region_members[kind][m].rows }; const size_t pkb = (size_t) N * K / 2; const size_t sfb = (size_t) ((N + 127) / 128) * ((K + 63) / 64) * 512; CUDA_CHECK(cudaMalloc(&r.d_packed, pkb)); @@ -963,7 +967,7 @@ static void frt_online_prepare(ggml_backend_cuda_context & ctx, const ggml_cgrap } r.alpha = alpha; r.K = K; r.N = N; // N last: serve fires only on complete regions frt_repack_check_region(kind, layer, r); - (kind == 0 ? built_k0 : built_k1)++; + built[kind]++; } } @@ -992,11 +996,14 @@ static void frt_online_prepare(ggml_backend_cuda_context & ctx, const ggml_cgrap } const bool head_pending = (getenv("FRT_HEAD_SWAP") || getenv("FRT_HEAD_DRAFT")) && !g_head.ok; - const bool k0_pending = g_reg.inproj_on && built_k0 < 30; - const bool k1_pending = g_reg.attn_on && built_k1 < 10; - if (!head_pending && !k0_pending && !k1_pending) { + bool region_pending = false; + for (int kind = 0; kind < n_region_kinds; ++kind) { + const bool on = (kind == 0 && g_reg.inproj_on) || (kind == 1 && g_reg.attn_on); + region_pending = region_pending || (on && built[kind] < region_layers[kind]); + } + if (!head_pending && !region_pending) { fprintf(stderr, "frt-repack: online repack complete (kind0=%d kind1=%d head=%d)\n", - built_k0, built_k1, (int) g_head.ok); + built[0], built[1], (int) g_head.ok); all_done = true; } } @@ -1011,6 +1018,11 @@ static void frt_online_prepare(ggml_backend_cuda_context & ctx, const ggml_cgrap namespace gdn { +// model dims from the binding (single source; kernels are compile-time +// specialized to these) +constexpr int CONV_ROW = frt_binding::gdn_conv_cache_row; // conv cache slot stride (floats) +constexpr int STATE_SZ = frt_binding::gdn_state_size; // recurrent state slot size (floats) + __device__ __forceinline__ float frt_silu(float x) { return x / (1.0f + expf(-x)); } // same semantics as quantize_q8_1 (d=amax/127, s=raw sum), one warp = one block @@ -1048,7 +1060,7 @@ __global__ void frt_gdn_conv_silu( ggml_cuda_pdl_lc(); ggml_cuda_pdl_sync(); const int ch = blockIdx.x * 256 + threadIdx.x; if (ch >= 8192) return; - float * cs = r_base + (size_t) (*r_slot) * 24576 + (size_t) ch * 3; + float * cs = r_base + (size_t) (*r_slot) * CONV_ROW + (size_t) ch * 3; const float * w = conv_w + (size_t) ch * 4; float s0 = cs[0], s1 = cs[1], s2 = cs[2]; float * const snaps[4] = { snap0, snap1, snap2, snap3 }; @@ -1217,7 +1229,7 @@ __global__ void frt_gdn_cell_part( } const float scale = 0.088388347648318447f; - float * S = s_base + (size_t) (*s_slot) * 524288 + (size_t) h * 16384; + float * S = s_base + (size_t) (*s_slot) * STATE_SZ + (size_t) h * 16384; for (int cc = 0; cc < 4; ++cc) { const int c = cg * 16 + warp * 4 + cc; @@ -1304,10 +1316,10 @@ bool ggml_cuda_frt_gdn_try_impl(ggml_backend_cuda_context & ctx, ggml_cgraph * c static int nf_mode = -1; if (nf_mode < 0) { const char * m = getenv("FRT_GDN_NORMFOLD"); nf_mode = (m && m[0] == '1') ? 1 : 0; } bool norm_anchor = false; - if (nf_mode && n0->op == GGML_OP_RMS_NORM && n0->ne[0] == 2048 && n0->ne[1] == 1) { + if (nf_mode && n0->op == GGML_OP_RMS_NORM && n0->ne[0] == frt_binding::d_model && n0->ne[1] == 1) { norm_anchor = true; // must find attn_norm MUL + GDN members below } else if (n0->op == GGML_OP_RMS_NORM) { return false; - } else if (n0->op != GGML_OP_GET_ROWS || n0->ne[0] != 24576 || + } else if (n0->op != GGML_OP_GET_ROWS || n0->ne[0] != gdn::CONV_ROW || strncmp(n0->name, "conv_states", 11) != 0) return false; // scan forward for the span members @@ -1353,9 +1365,9 @@ bool ggml_cuda_frt_gdn_try_impl(ggml_backend_cuda_context & ctx, ggml_cgraph * c case GGML_OP_GET_ROWS: // first hit = the slot read; a second single-row one is the // checkpoint save (we replay it ourselves); anything else -> stock. - if (n->ne[1] == 1 && n->ne[0] == 24576) { if (!gr_r) gr_r = n; else if (!ck_r_gr) ck_r_gr = n; else return false; } - else if (n->ne[1] == 1 && n->ne[0] == 524288) { if (!gr_s) gr_s = n; else if (!ck_s_gr) ck_s_gr = n; else return false; } - else if (ggml_nelements(n) != 0 && (n->ne[0] == 24576 || n->ne[0] == 524288)) return false; + if (n->ne[1] == 1 && n->ne[0] == gdn::CONV_ROW) { if (!gr_r) gr_r = n; else if (!ck_r_gr) ck_r_gr = n; else return false; } + else if (n->ne[1] == 1 && n->ne[0] == gdn::STATE_SZ) { if (!gr_s) gr_s = n; else if (!ck_s_gr) ck_s_gr = n; else return false; } + else if (ggml_nelements(n) != 0 && (n->ne[0] == gdn::CONV_ROW || n->ne[0] == gdn::STATE_SZ)) return false; break; case GGML_OP_SCALE: if (ggml_nelements(n) != 0) return false; // reset machinery active -> fall back @@ -1369,7 +1381,7 @@ bool ggml_cuda_frt_gdn_try_impl(ggml_backend_cuda_context & ctx, ggml_cgraph * c n->src[0]->ne[0] == 3 && n->src[0]->ne[1] == 8192) { if (n_conv_cpy >= 4) return false; conv_cpy[n_conv_cpy++] = n; - } else if (n->ne[0] == 524288 && n->src[0] && n->src[0]->op == GGML_OP_VIEW && + } else if (n->ne[0] == gdn::STATE_SZ && n->src[0] && n->src[0]->op == GGML_OP_VIEW && gdn && n->src[0]->src[0] == gdn) { if (state_cpy) return false; state_cpy = n; @@ -1395,7 +1407,7 @@ bool ggml_cuda_frt_gdn_try_impl(ggml_backend_cuda_context & ctx, ggml_cgraph * c !normw_mul || !add_dtb || !mul_A || !gr_r || !gr_s || !final_rs) return false; if (norm_anchor && (!norm_mul || qkv_mm->src[1] != norm_mul)) return false; frt::frt_region & reg = frt::g_reg.regions[0][layer]; - if (reg.N != 12352) return false; + if (reg.N != frt_binding::region_n[0]) return false; if (gdn->src[0]->ne[0] != 128 || gdn->src[0]->ne[1] != 16 || gdn->src[2]->ne[1] != 32) return false; const int M = (int) qkv_mm->src[1]->ne[1]; // decode M=1; spec verify M = 1 + n_draft if (M < 1 || M > 4 || !ggml_is_contiguous(qkv_mm->src[1])) return false; @@ -1430,13 +1442,13 @@ bool ggml_cuda_frt_gdn_try_impl(ggml_backend_cuda_context & ctx, ggml_cgraph * c cudaStream_t stream = ctx.stream(); // 0) checkpoint saves (pre-update snapshot of the current slot), if due if (ck_r_cpy) - ggml_cuda_kernel_launch(gdn::frt_gdn_ckpt_copy, ggml_cuda_kernel_launch_params(dim3(24576 / 256), dim3(256), 0, stream), + ggml_cuda_kernel_launch(gdn::frt_gdn_ckpt_copy, ggml_cuda_kernel_launch_params(dim3(gdn::CONV_ROW / 256), dim3(256), 0, stream), (const float *) ck_r_gr->src[0]->data, (const int32_t *) ck_r_gr->src[1]->data, - (int64_t) 24576, (float *) ck_r_cpy->src[1]->data, 24576); + (int64_t) gdn::CONV_ROW, (float *) ck_r_cpy->src[1]->data, gdn::CONV_ROW); if (ck_s_cpy) - ggml_cuda_kernel_launch(gdn::frt_gdn_ckpt_copy, ggml_cuda_kernel_launch_params(dim3(524288 / 256), dim3(256), 0, stream), + ggml_cuda_kernel_launch(gdn::frt_gdn_ckpt_copy, ggml_cuda_kernel_launch_params(dim3(gdn::STATE_SZ / 256), dim3(256), 0, stream), (const float *) ck_s_gr->src[0]->data, (const int32_t *) ck_s_gr->src[1]->data, - (int64_t) 524288, (float *) ck_s_cpy->src[1]->data, 524288); + (int64_t) gdn::STATE_SZ, (float *) ck_s_cpy->src[1]->data, gdn::STATE_SZ); // 1) act quant (+ F32 a/b gate rows; FRT_GDN_AB=0 falls back to W4A4 rows), // then fused in_proj GEMV into staging static int ab_f32 = -1; @@ -2354,7 +2366,7 @@ static bool frt_outproj_native_try(ggml_backend_cuda_context & ctx, ggml_cgraph cudaStream_t stream = ctx.stream(); const block_q8_1 * y_q8 = d_actq8b; - if (frt::g_reg.ok && frt::g_reg.outq8_node == (const void *) act && K == 4096) { + if (frt::g_reg.ok && frt::g_reg.outq8_node == (const void *) act && K == frt_binding::out_proj_k) { y_q8 = frt::g_reg.d_outq8; // GDN epilogue already staged the q8 act frt::g_reg.outq8_node = nullptr; } else { @@ -2673,7 +2685,7 @@ bool ggml_cuda_frt_head_mul_mat(ggml_backend_cuda_context & ctx, if (cap != cudaStreamCaptureStatusNone) return false; } if (!frt::frt_head_load()) return false; - if (strcmp(src0->name, "output.weight") != 0) return false; + if (strcmp(src0->name, frt_binding::head_name) != 0) return false; if (frt::g_head.draft_only && src0->type != GGML_TYPE_Q8_0) return false; if (src1->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) return false; const int M = (int) src1->ne[1]; // spec verify asks for logits at M = 1 + n_draft rows diff --git a/flash_rt/structures/adapters/ggml/tools/gen_binding_header.py b/flash_rt/structures/adapters/ggml/tools/gen_binding_header.py new file mode 100644 index 00000000..acf5ff97 --- /dev/null +++ b/flash_rt/structures/adapters/ggml/tools/gen_binding_header.py @@ -0,0 +1,115 @@ +#!/usr/bin/env python3 +"""Generate the model-constants header for a ggml adapter target from its +binding's host_params section. + +The binding yaml is the single source for every model-specific constant in +the window code; the generated header is checked in next to the window +translation unit (regenerate after editing the binding): + + python3 tools/gen_binding_header.py llamacpp_qwen36_35b_sm120 +""" + +from __future__ import annotations + +import pathlib +import sys + +import yaml + +_HERE = pathlib.Path(__file__).resolve().parent +_REPO = _HERE.parents[4] +_BINDINGS = _REPO / "flash_rt" / "structures" / "bindings" + + +def main() -> None: + name = sys.argv[1] if len(sys.argv) > 1 else "llamacpp_qwen36_35b_sm120" + data = yaml.safe_load((_BINDINGS / f"{name}.yaml").read_text()) + hp = data["host_params"] + + lines: list[str] = [] + a = lines.append + a("// Generated by tools/gen_binding_header.py from") + a(f"// flash_rt/structures/bindings/{name}.yaml — edit the binding, not this file.") + a("#pragma once") + a("") + a("namespace frt_binding {") + a("") + a(f"constexpr int n_layer = {hp['n_layer']};") + a(f"constexpr int layer_scan_max = {hp['layer_scan_max']};") + a(f"constexpr int d_model = {hp['d_model']};") + a("") + a("// fused qkv_pack regions (kind-indexed)") + a("struct region_member { const char * name; int rows; int off; bool leader; };") + regions = hp["regions"] + n_kinds = len(regions) + max_members = max(len(r["members"]) for r in regions) + a(f"constexpr int n_region_kinds = {n_kinds};") + a(f"constexpr int region_max_members = {max_members};") + layers = ", ".join(str(r["layers"]) for r in regions) + ks = ", ".join(str(r["k"]) for r in regions) + ns = ", ".join(str(sum(m["rows"] for m in r["members"])) for r in regions) + counts = ", ".join(str(len(r["members"])) for r in regions) + a(f"constexpr int region_layers[{n_kinds}] = {{{layers}}};") + a(f"constexpr int region_k[{n_kinds}] = {{{ks}}};") + a(f"constexpr int region_n[{n_kinds}] = {{{ns}}}; // sum of member rows") + a(f"constexpr int region_n_members[{n_kinds}] = {{{counts}}};") + a(f"constexpr region_member region_members[{n_kinds}][{max_members}] = {{") + for r in regions: + row = [] + for m in r["members"]: + leader = "true" if m.get("leader") else "false" + row.append(f'{{"{m["name"]}", {m["rows"]}, {m["row_off"]}, {leader}}}') + while len(row) < max_members: + row.append("{nullptr, 0, 0, false}") + a(" {" + ", ".join(row) + "},") + a("};") + a("") + g = hp["gdn"] + a("// gated_delta_core instance dims") + a(f"constexpr int gdn_qkvz_rows = {g['qkvz_rows']};") + a(f"constexpr int gdn_qkv_rows = {g['qkv_rows']};") + a(f"constexpr int gdn_z_rows = {g['z_rows']};") + a(f"constexpr int gdn_conv_channels = {g['conv_channels']};") + a(f"constexpr int gdn_conv_window = {g['conv_window']};") + a(f"constexpr int gdn_conv_cache_row = {g['conv_cache_row']};") + a(f"constexpr int gdn_state_size = {g['state_size']};") + a(f"constexpr int gdn_out_dim = {g['out_dim']};") + a(f"constexpr int gdn_head_dim = {g['head_dim']};") + a("") + m = hp["moe"] + a("// moe_expert_ffn instance (dims are read from tensors at run time)") + a(f'constexpr const char * moe_gate_exps = "{m["gate_exps"]}";') + a(f'constexpr const char * moe_up_exps = "{m["up_exps"]}";') + a(f'constexpr const char * moe_down_exps = "{m["down_exps"]}";') + a(f'constexpr const char * moe_gate_shexp = "{m["gate_shexp"]}";') + a(f'constexpr const char * moe_up_shexp = "{m["up_shexp"]}";') + a(f'constexpr const char * moe_down_shexp = "{m["down_shexp"]}";') + a(f'constexpr const char * moe_gate_inp_shexp = "{m["gate_inp_shexp"]}";') + a(f"constexpr int moe_n_expert = {m['n_expert']};") + a(f'constexpr const char * moe_draft_prefix = "{m["draft_prefix"]}";') + outs = ", ".join(f'"{n}"' for n in m["out_names"]) + resids = ", ".join(f'"{n}"' for n in m["resid_names"]) + a(f"constexpr const char * moe_out_names[{len(m['out_names'])}] = {{{outs}}};") + a(f"constexpr const char * moe_resid_names[{len(m['resid_names'])}] = {{{resids}}};") + a("") + o = hp["out_proj"] + names = ", ".join(f'"{n}"' for n in o["names"]) + a("// out-proj linear_proj instances") + a(f"constexpr const char * out_proj_names[{len(o['names'])}] = {{{names}}};") + a(f"constexpr int out_proj_k = {o['k']};") + a("") + h = hp["head"] + a("// lm head") + a(f'constexpr const char * head_name = "{h["name"]}";') + a(f"constexpr int head_n_vocab = {h['n_vocab']};") + a("") + a("} // namespace frt_binding") + a("") + + out = _HERE.parent / f"fr_binding_{name.split('llamacpp_')[-1]}.h" + out.write_text("\n".join(lines)) + print(f"wrote {out} ({len(lines)} lines)") + + +if __name__ == "__main__": + main() diff --git a/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml b/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml index fa3076e4..8b751472 100644 --- a/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml +++ b/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml @@ -18,6 +18,9 @@ stages: decode: seam: "llama_decode token generation, M in [1,4] (speculative verify batches)" capture: cuda_graphs_keyed + token_select: + seam: "server sampler chain; greedy or speculative acceptance (backend sampling via -bs)" + capture: host_dependent spec_draft: seam: "MTP draft chain evals + M=4 h-extraction eval (common/speculative draft-mtp driver)" capture: cuda_graphs_keyed @@ -26,52 +29,131 @@ cadences: token: [decode, token_select] spec_step: [decode, token_select, spec_draft] +# Machine-readable model facts consumed by the ggml adapter (build-time +# header generation: adapters/ggml/tools/gen_binding_header.py). This is the +# single source for every model-specific constant in the window code; the +# window logic itself is family-level. +host_params: + arch: qwen35moe + n_layer: 40 + layer_scan_max: 64 # matcher bound for blk.%d parsing + d_model: 2048 + regions: # qkv_pack instances served by the fused NVFP4 GEMV + - kind: 0 # GDN in_proj pack + layers: 30 + k: 2048 + members: # [tensor suffix, rows, row offset in the pack] + - {name: attn_qkv.weight, rows: 8192, row_off: 0, leader: true} + - {name: attn_gate.weight, rows: 4096, row_off: 8192} + - {name: ssm_alpha.weight, rows: 32, row_off: 12288} + - {name: ssm_beta.weight, rows: 32, row_off: 12320} + - kind: 1 # full-attention qkv pack + layers: 10 + k: 2048 + members: + - {name: attn_q.weight, rows: 8192, row_off: 0, leader: true} + - {name: attn_k.weight, rows: 512, row_off: 8192} + - {name: attn_v.weight, rows: 512, row_off: 8704} + gdn: # gated_delta_core instance dims + qkvz_rows: 12288 # qkv 8192 + z 4096 (in_proj GEMV span served to the cell) + qkv_rows: 8192 + z_rows: 4096 + conv_channels: 8192 + conv_window: 3 + conv_cache_row: 24576 # conv_channels * conv_window + state_size: 524288 # out_dim * head_dim (32 heads x 128 x 128) + out_dim: 4096 + head_dim: 128 + moe: # moe_expert_ffn instance (dims read from tensors at run time) + gate_exps: ffn_gate_exps + up_exps: ffn_up_exps + down_exps: ffn_down_exps + gate_shexp: ffn_gate_shexp.weight + up_shexp: ffn_up_shexp.weight + down_shexp: ffn_down_shexp.weight + gate_inp_shexp: ffn_gate_inp_shexp.weight + n_expert: 256 + draft_prefix: mtp_ # MTP draft graph node-name prefix + out_names: [ffn_moe_out, ffn_out] + resid_names: [l_out, mtp_post_ffn] + out_proj: # linear_proj instances (Q8_0 GEMV + residual) + names: [ssm_out.weight, attn_output.weight] + k: 4096 + head: # lm head linear_proj instance + name: output.weight + n_vocab: 248320 + draft_copy_type: q8_0 # the spec draft's head copy is stored Q8_0; + # the target head is not — this discriminates them + coverage: - contract: hot_path_takeover + contract: complete_hot_path hot_path: + - token_embed - gdn_in_proj_pack - - attn_qkv_pack - gdn_cell_span + - attn_qkv_pack + - attention_full + - norms - moe_expert_span - out_proj + - recurrent_state - lm_head - - spec_draft_serving + - token_select segments: + - name: token_embed + stage: decode + classification: host_stage + seam: "embedding row gather (stock get_rows)" - name: gdn_in_proj_pack stage: decode classification: structure - seam: "qkv|gate|alpha|beta sibling pack served from one NVFP4 W4A4 GEMV staging pass (30 GDN layers); leaf form — later members read the stash" + seam: "qkv|gate|alpha|beta sibling pack served from one NVFP4 W4A4 GEMV staging pass (30 GDN layers); qkv_pack leaf form, fp4 block-scaled input (variant value proposed, see pins)" structures: [qkv_pack] - variants: {bind: leaf, in_dtype: fp4_block_scaled} # in_dtype value proposed, see pins + - name: gdn_cell_span + stage: decode + classification: structure + seam: "conv + gated-delta recurrence + gated norm + epilogue as one 4-launch region; decode_recurrent, l2 in kernel; M>1 verify batches under snapshot_per_token + replay_in_region" + structures: [gated_delta_core] - name: attn_qkv_pack stage: decode classification: structure seam: "attention q|k|v sibling pack, same GEMV (10 attention layers)" structures: [qkv_pack] - variants: {bind: leaf, in_dtype: fp4_block_scaled} - - name: gdn_cell_span + - name: attention_full stage: decode - classification: structure - seam: "conv + gated-delta recurrence + gated norm + epilogue as one 4-launch region; M>1 verify batches under snapshot_per_token + replay_in_region" - structures: [gated_delta_core] - variants: {phase: decode_recurrent, state_update: snapshot_per_token, qk_norm: l2_in_kernel, checkpoint: replay_in_region} + classification: host_stage + seam: "flash-attention core runs on the host's own kernels (not taken over)" + - name: norms + stage: decode + classification: host_stage + seam: "rms/l2 norm chain outside the fused regions runs on stock kernels" - name: moe_expert_span stage: decode classification: structure - seam: "K0 quant+meta / K1 gate|up|GLU / K2 down+combine over ggml-native K-quant blocks; shared expert folded; router fused for M>=2 (host's own top-k fusion is M=1-only)" + seam: "K0 quant+meta / K1 gate|up|GLU / K2 down+combine over ggml-native K-quant blocks (format_native consumption); shared expert folded; routing fused for M>=2" structures: [moe_expert_ffn] - variants: {routing: fused, shared_expert: sigmoid_gated, weight_consumption: format_native, batch: m_le_4} - name: out_proj stage: decode classification: structure seam: "ssm_out/attn_output Q8_0 GEMV + residual epilogue, consuming the GDN epilogue's q8 activation handoff" structures: [linear_proj] - variants: {epilogue: residual_add} # format_native weight consumption proposed, see pins + - name: recurrent_state + stage: decode + classification: state_region + seam: "conv/state caches updated in place at M=1, per-token snapshots + checkpoint replay under speculative verify" - name: lm_head stage: decode classification: structure seam: "full tier: NVFP4 W4A4 head (M<=4 one pass, weights from the BF16-checkpoint pack); safe tier: stock Q6_K mmvq" structures: [linear_proj] + - name: token_select + stage: token_select + classification: host_stage + seam: "server sampler chain; backend sampling (-bs) keeps greedy/acceptance on device" + - name: prefill_stock + stage: prefill + classification: host_stage + seam: "prompt processing runs entirely on stock kernels (adapter windows are decode-shaped)" - name: spec_draft_serving stage: spec_draft classification: structure @@ -80,7 +162,7 @@ coverage: - name: host_graph_slots stage: spec_draft classification: host_stage - seam: "host-tree change (llama-context): per-shape graph slots keyed (n_tokens, gtype, has_embd), each owning its scheduler; target-side backend sampling (-bs)" + seam: "host-tree change (llama-context): per-shape graph slots keyed (n_tokens, gtype, has_embd), each owning its scheduler" hosts: llamacpp_dev_sm120: From 5303064bee05721ff011808058c5a88f0dc73687 Mon Sep 17 00:00:00 2001 From: LiangSu8899 <7thuniversels@gmail.com> Date: Tue, 25 Aug 2026 14:21:26 -0400 Subject: [PATCH 26/32] csrc: promote the M-rows f32out warpsplit GEMV and activation quantizer Move the sm120 heavy math out of the ggml window file into csrc as the next generation of the warp-split-K family (additive; the earlier bf16out/runtime-M entry is unchanged): compile-time row-count specialization (a runtime M in the MMA hot loop costs measurable time even at M=1), f32 output, and an optional programmatic-dependent-launch join so hosts that overlap every launch keep their chain. The M-rows activation quantizer ships alongside with its device body exposed for fused producers. The ggml adapter now only translates: it includes the csrc device header for its fused GDN producers and calls the csrc entries for standalone quantize/GEMV launches. Receipts unchanged: repack byte-identity 40/40, duplicated-token selftest bit-exact, tg128 and 24-chunk perplexity flat. --- ...4_w4a4_mma_warpsplit_mrows_f32out_sm120.cu | 200 +++++++++++++++ ..._w4a4_mma_warpsplit_mrows_f32out_sm120.cuh | 30 +++ .../f32_act_to_nvfp4_swizzled_mrows_sm120.cu | 72 ++++++ .../f32_act_to_nvfp4_swizzled_mrows_sm120.cuh | 96 +++++++ .../adapters/ggml/fr_win_qwen36_sm120.cu | 242 +++--------------- 5 files changed, 432 insertions(+), 208 deletions(-) create mode 100644 csrc/kernels/fp4_w4a4_mma_warpsplit_mrows_f32out_sm120.cu create mode 100644 csrc/kernels/fp4_w4a4_mma_warpsplit_mrows_f32out_sm120.cuh create mode 100644 csrc/quantize/f32_act_to_nvfp4_swizzled_mrows_sm120.cu create mode 100644 csrc/quantize/f32_act_to_nvfp4_swizzled_mrows_sm120.cuh diff --git a/csrc/kernels/fp4_w4a4_mma_warpsplit_mrows_f32out_sm120.cu b/csrc/kernels/fp4_w4a4_mma_warpsplit_mrows_f32out_sm120.cu new file mode 100644 index 00000000..e0c28e46 --- /dev/null +++ b/csrc/kernels/fp4_w4a4_mma_warpsplit_mrows_f32out_sm120.cu @@ -0,0 +1,200 @@ +// SPDX-License-Identifier: Apache-2.0 +// +// See header. Kernel body proven in the llama.cpp SM120 adapter (bit-exact +// duplicated-token replay across M variants, perplexity-neutral). + +#include "fp4_w4a4_mma_warpsplit_mrows_f32out_sm120.cuh" + +#include + +#include "cute/arch/mma_sm120.hpp" +#include "cutlass/numeric_types.h" + +namespace flash_rt { +namespace gemm { + +namespace { + +#if defined(__CUDA_ARCH_FEAT_SM120_ALL) || !defined(__CUDA_ARCH__) +#define FR_WS_SM120A_OK 1 +#endif + +__device__ __forceinline__ void pdl_sync() { +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 900 + cudaGridDependencySynchronize(); +#endif +} +__device__ __forceinline__ void pdl_lc() { +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 900 + cudaTriggerProgrammaticLaunchCompletion(); +#endif +} + +using AtomType = cute::SM120::BLOCKSCALED::SM120_16x8x64_TN_VS< + cutlass::float_e2m1_t, cutlass::float_e2m1_t, float, + cutlass::float_ue4m3_t, 16>; + +__device__ __forceinline__ uint32_t fa(const uint8_t * s, int t0, int t1, int r) { + int ro = ((r & 1) ? (t1 + 8) : t1) * 32; + return *reinterpret_cast(s + ro + t0 * 4 + ((r >> 1) & 1) * 16); +} +__device__ __forceinline__ uint32_t fb(const uint8_t * s, int t0, int t1, int r) { + return *reinterpret_cast(s + t1 * 32 + t0 * 4 + r * 16); +} +__device__ __forceinline__ uint32_t fsa(const uint8_t * p, int u) { + return *reinterpret_cast(p + u * 4); +} +__device__ __forceinline__ void cpa(uint8_t * d, const uint8_t * s) { + uint32_t i = __cvta_generic_to_shared(d); + asm volatile("cp.async.ca.shared.global.L2::128B [%0], [%1], 4;\n" :: "r"(i), "l"(s)); +} +__device__ __forceinline__ void commit() { asm volatile("cp.async.commit_group;\n" ::); } +template __device__ __forceinline__ void waitg() { + asm volatile("cp.async.wait_group %0;\n" :: "n"(N)); +} + +template +__global__ void warpsplit_kernel_f32out( + const uint8_t * __restrict__ A, const uint8_t * __restrict__ B, + const uint8_t * __restrict__ SFA, const uint8_t * __restrict__ SFB, + float * __restrict__ D, float alpha, int N, int K) { + constexpr int M = MT; +#if defined(FR_WS_SM120A_OK) + pdl_lc(); pdl_sync(); + __shared__ uint8_t sA[WARPS][STAGES][16 * 32]; + __shared__ uint8_t sSFA[WARPS][STAGES][16 * 4]; + __shared__ uint8_t sB[WARPS][STAGES][8 * 32]; + __shared__ uint8_t sSFB[WARPS][STAGES][8 * 4]; + __shared__ float s_red[WARPS][4 * 8]; + + int tid = threadIdx.x, warp = tid >> 5, lane = tid & 31; + int my_n = blockIdx.x * 8; + const int KI = K / 64, KIw = KI / WARPS; + const int kt0 = warp * KIw; + const int KH = K / 2, ncs = (K / 16 + 3) / 4; + int t0 = lane & 3, t1 = lane >> 2, sau = (lane & 1) * 8 + (lane >> 2), sbu = lane >> 2; + float c0 = 0, c1 = 0, c2 = 0, c3 = 0; + + uint8_t (*mA)[16 * 32] = sA[warp]; + uint8_t (*mSFA)[16 * 4] = sSFA[warp]; + uint8_t (*mB)[8 * 32] = sB[warp]; + uint8_t (*mSFB)[8 * 4] = sSFB[warp]; + + if (lane >= 1 && lane < 16) { +#pragma unroll + for (int st = 0; st < STAGES; ++st) { + int4 * av = reinterpret_cast(mA[st]); int4 z{0, 0, 0, 0}; + av[lane * 2] = z; av[lane * 2 + 1] = z; + } + if (lane < 4) for (int st = 0; st < STAGES; ++st) + for (int i = 4 + lane; i < 64; i += 4) mSFA[st][i] = 0; + } + __syncwarp(); // M>1: row-1 cp.async below must not race the zero-init + auto ld = [&](int bf, int kt) { + int bo = kt * 32; + if (lane < 8) cpa(mA[bf] + lane * 4, A + bo + lane * 4); + if (lane == 0) cpa(mSFA[bf], SFA + kt * 512); +#pragma unroll + for (int rr = 1; rr < MT; ++rr) { // extra token rows: act tile + atom-layout scales (row r -> +r*16) + if (lane < 8) cpa(mA[bf] + rr * 32 + lane * 4, A + (size_t) rr * KH + bo + lane * 4); + if (lane == 0) cpa(mSFA[bf] + rr * 4, SFA + kt * 512 + rr * 16); + } + for (int c = 0; c < 2; ++c) { int ch = lane + c * 32, col = ch >> 3, off = ch & 7; + cpa(mB[bf] + ch * 4, B + (size_t)(my_n + col) * KH + bo + off * 4); } + if (lane < 8) { int col = my_n + lane, rb = col >> 7, ri = col & 127; + int si = rb * ncs + kt, ib = (ri & 31) * 16 + ((ri >> 5) & 3) * 4; + cpa(mSFB[bf] + lane * 4, SFB + (size_t)si * 512 + ib); } + }; +#pragma unroll + for (int st = 0; st < STAGES - 1; ++st) { if (st < KIw) ld(st, kt0 + st); commit(); } + for (int j = 0; j < KIw; ++j) { + int cb = j % STAGES, jp = j + STAGES - 1; + if (jp < KIw) ld(jp % STAGES, kt0 + jp); + commit(); waitg(); __syncwarp(); + uint32_t a0 = fa(mA[cb], t0, t1, 0), a1 = fa(mA[cb], t0, t1, 1); + uint32_t a2 = fa(mA[cb], t0, t1, 2), a3 = fa(mA[cb], t0, t1, 3); + uint32_t b0 = fb(mB[cb], t0, t1, 0), b1 = fb(mB[cb], t0, t1, 1); + uint32_t sfa_v = fsa(mSFA[cb], sau), sfb_v = fsa(mSFB[cb], sbu); + float d0, d1, d2, d3; + AtomType::fma(d0, d1, d2, d3, a0, a1, a2, a3, b0, b1, c0, c1, c2, c3, sfa_v, sfb_v); + c0 = d0; c1 = d1; c2 = d2; c3 = d3; + } + // m16n8 C fragment: {c0,c1} hold row (lane>>2) -> token t lives in lanes 4t..4t+3. + int q = lane >> 2, r = lane & 3; + if (q < M) { s_red[warp][q * 8 + r * 2] = c0; s_red[warp][q * 8 + r * 2 + 1] = c1; } + __syncthreads(); + if (warp == 0 && lane < 8) { + int col = my_n + lane; + if (col < N) { +#pragma unroll + for (int t = 0; t < MT; ++t) { + float acc = 0.f; +#pragma unroll + for (int w = 0; w < WARPS; ++w) acc += s_red[w][t * 8 + lane]; + D[(size_t) t * N + col] = acc * alpha; + } + } + } +#endif // FR_WS_SM120A_OK +} + +template +int launch(const uint8_t * A, const uint8_t * B, const uint8_t * SFA, + const uint8_t * SFB, float * D, float alpha, int N, int K, + bool pdl, cudaStream_t stream) { + const dim3 grid(N / 8), block(WARPS * 32); + if (pdl) { + cudaLaunchAttribute attr{}; + attr.id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr.val.programmaticStreamSerializationAllowed = 1; + cudaLaunchConfig_t cfg{}; + cfg.gridDim = grid; cfg.blockDim = block; cfg.dynamicSmemBytes = 0; + cfg.stream = stream; cfg.attrs = &attr; cfg.numAttrs = 1; + return (int) cudaLaunchKernelEx(&cfg, warpsplit_kernel_f32out, + A, B, SFA, SFB, D, alpha, N, K); + } + warpsplit_kernel_f32out<<>>( + A, B, SFA, SFB, D, alpha, N, K); + return (int) cudaGetLastError(); +} + +template +int launch_m(const uint8_t * A, const uint8_t * B, const uint8_t * SFA, + const uint8_t * SFB, float * D, float alpha, int M, int N, int K, + bool pdl, cudaStream_t stream) { + switch (M) { + case 1: return launch(A, B, SFA, SFB, D, alpha, N, K, pdl, stream); + case 2: return launch(A, B, SFA, SFB, D, alpha, N, K, pdl, stream); + case 3: return launch(A, B, SFA, SFB, D, alpha, N, K, pdl, stream); + default: return launch(A, B, SFA, SFB, D, alpha, N, K, pdl, stream); + } +} + +} // namespace + +int fp4_w4a4_mma_sm120_warpsplit_mrows_f32out( + const void * A_packed, const void * B_packed, float * D, int M, int N, + int K, const void * SFA, const void * SFB, float alpha, int warps, + int stages, bool pdl, cudaStream_t stream) { + if (M < 1 || M > 4 || N % 8 != 0 || K % 64 != 0) return -1; + if ((K / 64) % warps != 0) return -1; + const uint8_t * A = (const uint8_t *) A_packed; + const uint8_t * B = (const uint8_t *) B_packed; + const uint8_t * sfa = (const uint8_t *) SFA; + const uint8_t * sfb = (const uint8_t *) SFB; + const int cfg = stages * 10 + warps; + switch (cfg) { + case 34: return launch_m<3, 4>(A, B, sfa, sfb, D, alpha, M, N, K, pdl, stream); + case 44: return launch_m<4, 4>(A, B, sfa, sfb, D, alpha, M, N, K, pdl, stream); + case 64: return launch_m<6, 4>(A, B, sfa, sfb, D, alpha, M, N, K, pdl, stream); + case 38: return launch_m<3, 8>(A, B, sfa, sfb, D, alpha, M, N, K, pdl, stream); + case 48: return launch_m<4, 8>(A, B, sfa, sfb, D, alpha, M, N, K, pdl, stream); + case 32: return launch_m<3, 2>(A, B, sfa, sfb, D, alpha, M, N, K, pdl, stream); + case 62: return launch_m<6, 2>(A, B, sfa, sfb, D, alpha, M, N, K, pdl, stream); + case 42: return launch_m<4, 2>(A, B, sfa, sfb, D, alpha, M, N, K, pdl, stream); + default: return -1; + } +} + +} // namespace gemm +} // namespace flash_rt diff --git a/csrc/kernels/fp4_w4a4_mma_warpsplit_mrows_f32out_sm120.cuh b/csrc/kernels/fp4_w4a4_mma_warpsplit_mrows_f32out_sm120.cuh new file mode 100644 index 00000000..92c34698 --- /dev/null +++ b/csrc/kernels/fp4_w4a4_mma_warpsplit_mrows_f32out_sm120.cuh @@ -0,0 +1,30 @@ +// SPDX-License-Identifier: Apache-2.0 +// +// Small-M (M = 1..4) warp-split-K NVFP4 W4A4 GEMV/GEMM for sm_120, f32 +// output. Next generation of fp4_w4a4_mma_warpsplit_mrows_sm120 (additive: +// that entry is unchanged): the row count is a template parameter instead +// of a kernel argument (a runtime M in the MMA hot loop costs measurable +// time even at M=1), the epilogue writes f32 directly, and the launch can +// join the caller's programmatic-dependent-launch (PDL) chain — hosts that +// overlap every launch lose ground to any kernel that breaks the chain. +// The SM120_16x8x64 blockscaled MMA atom computes a 16-row tile, so rows +// 2..M ride the same weight HBM traffic as M=1: token t occupies A-tile +// row t (smem +t*32B), SFA atom-layout row t (+t*16B), and C-fragment +// lanes 4t..4t+3. +#pragma once +#include + +namespace flash_rt { +namespace gemm { + +// A_packed (M, K/2) row-major e2m1 pairs, B_packed (N, K/2), D f32 (M, N). +// SFA: atom-layout scales for rows 0..M-1 of problem (M, K); SFB: atom +// layout for (N, K). M in 1..4, warps in {2,4,8}, stages in {3,4,6}, +// N % 8 == 0, K % 64 == 0, (K/64) % warps == 0. Returns 0 on success. +int fp4_w4a4_mma_sm120_warpsplit_mrows_f32out( + const void * A_packed, const void * B_packed, float * D, int M, int N, + int K, const void * SFA, const void * SFB, float alpha, int warps, + int stages, bool pdl, cudaStream_t stream); + +} // namespace gemm +} // namespace flash_rt diff --git a/csrc/quantize/f32_act_to_nvfp4_swizzled_mrows_sm120.cu b/csrc/quantize/f32_act_to_nvfp4_swizzled_mrows_sm120.cu new file mode 100644 index 00000000..f3ff9ab4 --- /dev/null +++ b/csrc/quantize/f32_act_to_nvfp4_swizzled_mrows_sm120.cu @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: Apache-2.0 +// +// See header. The kernel is compile-time specialized on the row count and +// optionally joins the caller's programmatic-dependent-launch (PDL) chain: +// hosts that overlap every launch (llama.cpp CUDA backend) lose measurable +// time to any kernel that breaks the chain. + +#include "f32_act_to_nvfp4_swizzled_mrows_sm120.cuh" + +namespace flash_rt { +namespace quantize { + +namespace { + +__device__ __forceinline__ void pdl_sync() { +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 900 + cudaGridDependencySynchronize(); +#endif +} +__device__ __forceinline__ void pdl_lc() { +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 900 + cudaTriggerProgrammaticLaunchCompletion(); +#endif +} + +template +__global__ void f32_act_to_nvfp4_kernel( + const float * __restrict__ x, + uint2 * __restrict__ dst_packed, + uint8_t * __restrict__ dst_sfa, + int D, long long x_srow) { + pdl_lc(); pdl_sync(); +#pragma unroll + for (int r = blockIdx.x; r < MT; r += gridDim.x) // launch with grid = MT + f32_act_to_nvfp4_row(x + (size_t) r * x_srow, dst_packed, dst_sfa, D, r); +} + +template +int launch(const float * x, void * dst_packed, void * dst_sfa, + int D, long long x_srow, bool pdl, cudaStream_t stream) { + const dim3 grid(MT), block(256); + if (pdl) { + cudaLaunchAttribute attr{}; + attr.id = cudaLaunchAttributeProgrammaticStreamSerialization; + attr.val.programmaticStreamSerializationAllowed = 1; + cudaLaunchConfig_t cfg{}; + cfg.gridDim = grid; cfg.blockDim = block; cfg.dynamicSmemBytes = 0; + cfg.stream = stream; cfg.attrs = &attr; cfg.numAttrs = 1; + return (int) cudaLaunchKernelEx(&cfg, f32_act_to_nvfp4_kernel<256, MT>, + x, (uint2 *) dst_packed, (uint8_t *) dst_sfa, D, x_srow); + } + f32_act_to_nvfp4_kernel<256, MT><<>>( + x, (uint2 *) dst_packed, (uint8_t *) dst_sfa, D, x_srow); + return (int) cudaGetLastError(); +} + +} // namespace + +int f32_act_to_nvfp4_swizzled_mrows( + const float * x, void * dst_packed, void * dst_sfa, + int D, int M, long long x_srow, bool pdl, cudaStream_t stream) { + if (D % 16 != 0 || M < 1 || M > 4) return -1; + switch (M) { + case 1: return launch<1>(x, dst_packed, dst_sfa, D, x_srow, pdl, stream); + case 2: return launch<2>(x, dst_packed, dst_sfa, D, x_srow, pdl, stream); + case 3: return launch<3>(x, dst_packed, dst_sfa, D, x_srow, pdl, stream); + default: return launch<4>(x, dst_packed, dst_sfa, D, x_srow, pdl, stream); + } +} + +} // namespace quantize +} // namespace flash_rt diff --git a/csrc/quantize/f32_act_to_nvfp4_swizzled_mrows_sm120.cuh b/csrc/quantize/f32_act_to_nvfp4_swizzled_mrows_sm120.cuh new file mode 100644 index 00000000..21ad81f3 --- /dev/null +++ b/csrc/quantize/f32_act_to_nvfp4_swizzled_mrows_sm120.cuh @@ -0,0 +1,96 @@ +// SPDX-License-Identifier: Apache-2.0 +// +// F32 activation rows -> NVFP4 packed + swizzled SFA (Sm1xx atom layout), +// M rows in one launch (M = 1..4, speculative-decode verify batches). +// Framework-free: raw pointers + cudaStream_t; the device body is exposed +// so host-adapter kernels can inline the quantization into fused producers. +// Additive: new file + new entry point (the single-row bf16 weight/act +// quantizers are separate, older entries). +#pragma once +#include +#include +#include + +namespace flash_rt { +namespace quantize { + +// ---- device body (shared with fused producers) ---------------------------- + +__device__ __forceinline__ int nvfp4_sfa_offset_128x64(int row, int k, int dim) { + const int row_block = row >> 7; + const int row_in_block = row & 127; + const int k_block = k >> 6; + const int k_in_block = k & 63; + const int k_blocks = (dim + 63) >> 6; + return row_block * k_blocks * 512 + k_block * 512 + + (row_in_block & 31) * 16 + (row_in_block >> 5) * 4 + + (k_in_block >> 4); +} + +// activation-quant boundary convention (<=; the weight packers use strict <) +__device__ __forceinline__ uint8_t nvfp4_act_f32_to_e2m1(float x) { + uint8_t sign = (x < 0.f) ? 0x8u : 0x0u; + float ax = fabsf(x); + uint8_t mant; + if (ax <= 0.25f) mant = 0u; + else if (ax <= 0.75f) mant = 1u; + else if (ax <= 1.25f) mant = 2u; + else if (ax <= 1.75f) mant = 3u; + else if (ax <= 2.5f) mant = 4u; + else if (ax <= 3.5f) mant = 5u; + else if (ax <= 5.0f) mant = 6u; + else mant = 7u; + return sign | mant; +} + +// quantize one f32 row of length D into packed e2m1 + SFA; `row` selects the +// SFA atom-layout row and the packed output row (row-major, D/16 uint2). +// Callable from any single participating block of THREADS threads. +template +__device__ __forceinline__ void f32_act_to_nvfp4_row( + const float * __restrict__ x, + uint2 * __restrict__ dst_packed, + uint8_t * __restrict__ dst_sfa, + int D, int row = 0) { + const int n_blocks = D / 16; + uint2 * dst_row = dst_packed + (size_t) row * n_blocks; + for (int b = threadIdx.x; b < n_blocks; b += THREADS) { + float vals[16]; + float amax = 0.f; +#pragma unroll + for (int i = 0; i < 16; ++i) { + vals[i] = x[b * 16 + i]; + const float a = fabsf(vals[i]); + if (a > amax) amax = a; + } + float desired = amax / 6.f; + if (desired < 1e-12f) desired = 1e-12f; + __nv_fp8_e4m3 bs_q = __nv_fp8_e4m3(fmaxf(desired, 0.f)); + const float bs_dq = static_cast(bs_q); + dst_sfa[nvfp4_sfa_offset_128x64(row, b * 16, D)] = *reinterpret_cast(&bs_q); + const float inv_bs = 1.f / bs_dq; + uint2 out; + uint8_t * ob = reinterpret_cast(&out); +#pragma unroll + for (int p = 0; p < 8; ++p) { + const uint8_t lo = nvfp4_act_f32_to_e2m1(vals[2 * p] * inv_bs); + const uint8_t hi = nvfp4_act_f32_to_e2m1(vals[2 * p + 1] * inv_bs); + ob[p] = static_cast(lo | (hi << 4)); + } + dst_row[b] = out; + } +} + +// ---- host entry ------------------------------------------------------------ + +// Quantize M f32 rows (row t at x + t*x_srow) into packed [M, D/2] + SFA in +// the atom layout for problem rows 0..M-1. M in 1..4 (compile-time +// specialized; a runtime M in the hot loop costs measurable time). +// pdl: join the caller's programmatic-dependent-launch chain. +// Returns 0 on success. +int f32_act_to_nvfp4_swizzled_mrows( + const float * x, void * dst_packed, void * dst_sfa, + int D, int M, long long x_srow, bool pdl, cudaStream_t stream); + +} // namespace quantize +} // namespace flash_rt diff --git a/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu b/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu index 075e67f7..24a53738 100644 --- a/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu +++ b/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu @@ -23,6 +23,11 @@ #include "vecdotq.cuh" #include "convert.cuh" +// Heavy math comes from csrc (single source; the adapter only translates): +// the M-rows activation quantizer and the warp-split-K W4A4 GEMV. +#include "../../../../csrc/quantize/f32_act_to_nvfp4_swizzled_mrows_sm120.cuh" +#include "../../../../csrc/kernels/fp4_w4a4_mma_warpsplit_mrows_f32out_sm120.cuh" + // Model-specific constants come from the binding (single source: // flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml); regenerate // the header with tools/gen_binding_header.py after editing the binding. @@ -44,204 +49,41 @@ namespace frt { // ---------------- activation quantize (f32 row -> NVFP4 + swizzled SFA) --- +// device vocabulary of this window file, backed by the csrc single source __device__ __forceinline__ int sfa_offset_128x64(int row, int k, int dim) { - const int row_block = row >> 7; - const int row_in_block = row & 127; - const int k_block = k >> 6; - const int k_in_block = k & 63; - const int k_blocks = (dim + 63) >> 6; - return row_block * k_blocks * 512 + k_block * 512 + - (row_in_block & 31) * 16 + (row_in_block >> 5) * 4 + - (k_in_block >> 4); + return flash_rt::quantize::nvfp4_sfa_offset_128x64(row, k, dim); } - __device__ __forceinline__ uint8_t fp32_to_e2m1(float x) { - uint8_t sign = (x < 0.f) ? 0x8u : 0x0u; - float ax = fabsf(x); - uint8_t mant; - if (ax <= 0.25f) mant = 0u; - else if (ax <= 0.75f) mant = 1u; - else if (ax <= 1.25f) mant = 2u; - else if (ax <= 1.75f) mant = 3u; - else if (ax <= 2.5f) mant = 4u; - else if (ax <= 3.5f) mant = 5u; - else if (ax <= 5.0f) mant = 6u; - else mant = 7u; - return sign | mant; + return flash_rt::quantize::nvfp4_act_f32_to_e2m1(x); } - -// quantize a f32 row of length D into packed e2m1 + SFA (device body, callable -// from any single participating block). row selects the SFA atom-layout row and -// the packed output row (row-major, D/16 uint2 per row). template __device__ __forceinline__ void quant_act_fp4_f32_body( - const float * __restrict__ x, - uint2 * __restrict__ dst_packed, - uint8_t * __restrict__ dst_sfa, - int D, int row = 0) { - const int n_blocks = D / 16; - uint2 * dst_row = dst_packed + (size_t) row * n_blocks; - for (int b = threadIdx.x; b < n_blocks; b += THREADS) { - float vals[16]; - float amax = 0.f; -#pragma unroll - for (int i = 0; i < 16; ++i) { - vals[i] = x[b * 16 + i]; - const float a = fabsf(vals[i]); - if (a > amax) amax = a; - } - float desired = amax / 6.f; - if (desired < 1e-12f) desired = 1e-12f; - __nv_fp8_e4m3 bs_q = __nv_fp8_e4m3(fmaxf(desired, 0.f)); - const float bs_dq = static_cast(bs_q); - dst_sfa[sfa_offset_128x64(row, b * 16, D)] = *reinterpret_cast(&bs_q); - const float inv_bs = 1.f / bs_dq; - uint2 out; - uint8_t * ob = reinterpret_cast(&out); -#pragma unroll - for (int p = 0; p < 8; ++p) { - const uint8_t lo = fp32_to_e2m1(vals[2 * p] * inv_bs); - const uint8_t hi = fp32_to_e2m1(vals[2 * p + 1] * inv_bs); - ob[p] = static_cast(lo | (hi << 4)); - } - dst_row[b] = out; - } + const float * __restrict__ x, uint2 * __restrict__ dst_packed, + uint8_t * __restrict__ dst_sfa, int D, int row = 0) { + flash_rt::quantize::f32_act_to_nvfp4_row(x, dst_packed, dst_sfa, D, row); } -// MT is compile-time: a runtime token count in the hot path costs measurable -// time even at M=1 (handoff §5.2), so callers dispatch through FRT_M_DISPATCH. -template -__global__ void quant_act_fp4_f32( - const float * __restrict__ x, - uint2 * __restrict__ dst_packed, - uint8_t * __restrict__ dst_sfa, - int D, int64_t x_srow = 0) { - ggml_cuda_pdl_lc(); ggml_cuda_pdl_sync(); -#pragma unroll - for (int r = blockIdx.x; r < MT; r += gridDim.x) // launch with grid = MT - quant_act_fp4_f32_body(x + (size_t) r * x_srow, dst_packed, dst_sfa, D, r); +#ifdef GGML_CUDA_USE_PDL +constexpr bool frt_launch_pdl = true; +#else +constexpr bool frt_launch_pdl = false; +#endif + +// standalone M-rows activation quantize through the csrc entry +static void frt_quant_act_launch(const float * x, void * dst_packed, void * dst_sfa, + int D, int M, int64_t x_srow, cudaStream_t stream) { + const int rc = flash_rt::quantize::f32_act_to_nvfp4_swizzled_mrows( + x, dst_packed, dst_sfa, D, M, (long long) x_srow, frt_launch_pdl, stream); + if (rc != 0) fprintf(stderr, "frt: quant_act launch failed (%d)\n", rc); } -// switch a runtime M in [1,4] onto a compile-time MT inside __VA_ARGS__. #define FRT_M_DISPATCH(M, ...) do { switch (M) { \ case 1: { constexpr int MT = 1; __VA_ARGS__; } break; \ case 2: { constexpr int MT = 2; __VA_ARGS__; } break; \ case 3: { constexpr int MT = 3; __VA_ARGS__; } break; \ default:{ constexpr int MT = 4; __VA_ARGS__; } break; } } while (0) -// ---------------- warp-split-K NVFP4 W4A4 M=1 GEMV (f32 out) --------------- - -#if defined(__CUDA_ARCH_FEAT_SM120_ALL) || !defined(__CUDA_ARCH__) -#define FRT_SM120A_OK 1 -#endif - -using AtomType = cute::SM120::BLOCKSCALED::SM120_16x8x64_TN_VS< - cutlass::float_e2m1_t, cutlass::float_e2m1_t, float, - cutlass::float_ue4m3_t, 16>; - -__device__ __forceinline__ uint32_t fa(const uint8_t * s, int t0, int t1, int r) { - int ro = ((r & 1) ? (t1 + 8) : t1) * 32; - return *reinterpret_cast(s + ro + t0 * 4 + ((r >> 1) & 1) * 16); -} -__device__ __forceinline__ uint32_t fb(const uint8_t * s, int t0, int t1, int r) { - return *reinterpret_cast(s + t1 * 32 + t0 * 4 + r * 16); -} -__device__ __forceinline__ uint32_t fsa(const uint8_t * p, int u) { - return *reinterpret_cast(p + u * 4); -} -__device__ __forceinline__ void cpa(uint8_t * d, const uint8_t * s) { - uint32_t i = __cvta_generic_to_shared(d); - asm volatile("cp.async.ca.shared.global.L2::128B [%0], [%1], 4;\n" :: "r"(i), "l"(s)); -} -__device__ __forceinline__ void commit() { asm volatile("cp.async.commit_group;\n" ::); } -template __device__ __forceinline__ void waitg() { - asm volatile("cp.async.wait_group %0;\n" :: "n"(N)); -} - -template -__global__ void warpsplit_kernel_f32out( - const uint8_t * __restrict__ A, const uint8_t * __restrict__ B, - const uint8_t * __restrict__ SFA, const uint8_t * __restrict__ SFB, - float * __restrict__ D, float alpha, int N, int K) { - constexpr int M = MT; -#if defined(FRT_SM120A_OK) - ggml_cuda_pdl_lc(); ggml_cuda_pdl_sync(); - __shared__ uint8_t sA[WARPS][STAGES][16 * 32]; - __shared__ uint8_t sSFA[WARPS][STAGES][16 * 4]; - __shared__ uint8_t sB[WARPS][STAGES][8 * 32]; - __shared__ uint8_t sSFB[WARPS][STAGES][8 * 4]; - __shared__ float s_red[WARPS][4 * 8]; - - int tid = threadIdx.x, warp = tid >> 5, lane = tid & 31; - int my_n = blockIdx.x * 8; - const int KI = K / 64, KIw = KI / WARPS; - const int kt0 = warp * KIw; - const int KH = K / 2, ncs = (K / 16 + 3) / 4; - int t0 = lane & 3, t1 = lane >> 2, sau = (lane & 1) * 8 + (lane >> 2), sbu = lane >> 2; - float c0 = 0, c1 = 0, c2 = 0, c3 = 0; - - uint8_t (*mA)[16 * 32] = sA[warp]; - uint8_t (*mSFA)[16 * 4] = sSFA[warp]; - uint8_t (*mB)[8 * 32] = sB[warp]; - uint8_t (*mSFB)[8 * 4] = sSFB[warp]; - - if (lane >= 1 && lane < 16) { -#pragma unroll - for (int st = 0; st < STAGES; ++st) { - int4 * av = reinterpret_cast(mA[st]); int4 z{0, 0, 0, 0}; - av[lane * 2] = z; av[lane * 2 + 1] = z; - } - if (lane < 4) for (int st = 0; st < STAGES; ++st) - for (int i = 4 + lane; i < 64; i += 4) mSFA[st][i] = 0; - } - __syncwarp(); // M>1: row-1 cp.async below must not race the zero-init - auto ld = [&](int bf, int kt) { - int bo = kt * 32; - if (lane < 8) cpa(mA[bf] + lane * 4, A + bo + lane * 4); - if (lane == 0) cpa(mSFA[bf], SFA + kt * 512); -#pragma unroll - for (int rr = 1; rr < MT; ++rr) { // extra token rows: act tile + atom-layout scales (row r -> +r*16) - if (lane < 8) cpa(mA[bf] + rr * 32 + lane * 4, A + (size_t) rr * KH + bo + lane * 4); - if (lane == 0) cpa(mSFA[bf] + rr * 4, SFA + kt * 512 + rr * 16); - } - for (int c = 0; c < 2; ++c) { int ch = lane + c * 32, col = ch >> 3, off = ch & 7; - cpa(mB[bf] + ch * 4, B + (size_t)(my_n + col) * KH + bo + off * 4); } - if (lane < 8) { int col = my_n + lane, rb = col >> 7, ri = col & 127; - int si = rb * ncs + kt, ib = (ri & 31) * 16 + ((ri >> 5) & 3) * 4; - cpa(mSFB[bf] + lane * 4, SFB + (size_t)si * 512 + ib); } - }; -#pragma unroll - for (int st = 0; st < STAGES - 1; ++st) { if (st < KIw) ld(st, kt0 + st); commit(); } - for (int j = 0; j < KIw; ++j) { - int cb = j % STAGES, jp = j + STAGES - 1; - if (jp < KIw) ld(jp % STAGES, kt0 + jp); - commit(); waitg(); __syncwarp(); - uint32_t a0 = fa(mA[cb], t0, t1, 0), a1 = fa(mA[cb], t0, t1, 1); - uint32_t a2 = fa(mA[cb], t0, t1, 2), a3 = fa(mA[cb], t0, t1, 3); - uint32_t b0 = fb(mB[cb], t0, t1, 0), b1 = fb(mB[cb], t0, t1, 1); - uint32_t sfa_v = fsa(mSFA[cb], sau), sfb_v = fsa(mSFB[cb], sbu); - float d0, d1, d2, d3; - AtomType::fma(d0, d1, d2, d3, a0, a1, a2, a3, b0, b1, c0, c1, c2, c3, sfa_v, sfb_v); - c0 = d0; c1 = d1; c2 = d2; c3 = d3; - } - // m16n8 C fragment: {c0,c1} hold row (lane>>2) -> token t lives in lanes 4t..4t+3. - int q = lane >> 2, r = lane & 3; - if (q < M) { s_red[warp][q * 8 + r * 2] = c0; s_red[warp][q * 8 + r * 2 + 1] = c1; } - __syncthreads(); - if (warp == 0 && lane < 8) { - int col = my_n + lane; - if (col < N) { -#pragma unroll - for (int t = 0; t < MT; ++t) { - float acc = 0.f; -#pragma unroll - for (int w = 0; w < WARPS; ++w) acc += s_red[w][t * 8 + lane]; - D[(size_t) t * N + col] = acc * alpha; - } - } - } -#endif // FRT_SM120A_OK -} +// ---------------- warp-split-K NVFP4 W4A4 GEMV (csrc single source) -------- // runtime (STAGES, WARPS) selection for the region GEMVs: FRT_WS_CFG=sw // (default s4w2). K/64 must be divisible by W. @@ -253,28 +95,15 @@ static void frt_ws_launch(const uint8_t * A, const uint8_t * B, const char * e = getenv("FRT_WS_CFG"); env_cfg = 0; if (e) { - int s = 0, w = 0; - if (sscanf(e, "s%dw%d", &s, &w) == 2) env_cfg = s * 10 + w; + int sc = 0, w = 0; + if (sscanf(e, "s%dw%d", &sc, &w) == 2) env_cfg = sc * 10 + w; } } - const int cfg = env_cfg ? env_cfg : def_cfg; - dim3 grid(N / 8); -#define FRT_WS_CASE(S, W) do { \ - auto kf = M == 1 ? warpsplit_kernel_f32out : M == 2 ? warpsplit_kernel_f32out : \ - M == 3 ? warpsplit_kernel_f32out : warpsplit_kernel_f32out; \ - ggml_cuda_kernel_launch(kf, ggml_cuda_kernel_launch_params(grid, dim3(W * 32), 0, stream), A, B, SFA, SFB, D, alpha, N, K); \ - } while (0) - switch (cfg) { - case 34: FRT_WS_CASE(3, 4); break; - case 44: FRT_WS_CASE(4, 4); break; - case 64: FRT_WS_CASE(6, 4); break; - case 38: FRT_WS_CASE(3, 8); break; - case 48: FRT_WS_CASE(4, 8); break; - case 32: FRT_WS_CASE(3, 2); break; - case 62: FRT_WS_CASE(6, 2); break; - default: FRT_WS_CASE(4, 2); break; - } -#undef FRT_WS_CASE + const int cfg = env_cfg ? env_cfg : (def_cfg ? def_cfg : 42); + const int rc = flash_rt::gemm::fp4_w4a4_mma_sm120_warpsplit_mrows_f32out( + A, B, D, M, N, K, SFA, SFB, alpha, /*warps=*/cfg % 10, /*stages=*/cfg / 10, + frt_launch_pdl, stream); + if (rc != 0) fprintf(stderr, "frt: warpsplit launch failed (%d, cfg=%d)\n", rc, cfg); } // ---------------- W4A16 matvec (NVFP4 weight x f32 act, f32 out) ---------- @@ -567,8 +396,7 @@ static bool frt_region_serve(ggml_backend_cuda_context & ctx, int kind, int laye const int64_t key = ((int64_t) kind << 32) | layer; cudaStream_t stream = ctx.stream(); if (leader) { - FRT_M_DISPATCH(M, ggml_cuda_kernel_launch((frt::quant_act_fp4_f32<256, MT>), ggml_cuda_kernel_launch_params(dim3(M), dim3(256), 0, stream), - (const float *) src1->data, (uint2 *) g_reg.d_apack, g_reg.d_sfa, (int) r.K, (int64_t) r.K)); + frt_quant_act_launch((const float *) src1->data, g_reg.d_apack, g_reg.d_sfa, (int) r.K, M, (int64_t) r.K, stream); frt::frt_ws_launch(g_reg.d_apack, r.d_packed, g_reg.d_sfa, r.d_sf, g_reg.d_staging, r.alpha, (int) r.N, (int) r.K, M, stream); g_reg.leader_src = src1->data; @@ -1474,8 +1302,7 @@ bool ggml_cuda_frt_gdn_try_impl(ggml_backend_cuda_context & ctx, ggml_cgraph * c frt::g_reg.d_staging); } else { if (M != 1) return false; // W4A4 a/b staging fallback stays M=1 - ggml_cuda_kernel_launch((frt::quant_act_fp4_f32<256, 1>), ggml_cuda_kernel_launch_params(dim3(1), dim3(256), 0, stream), - (const float *) qkv_mm->src[1]->data, (uint2 *) frt::g_reg.d_apack, frt::g_reg.d_sfa, (int) reg.K, (int64_t) 0); + frt::frt_quant_act_launch((const float *) qkv_mm->src[1]->data, frt::g_reg.d_apack, frt::g_reg.d_sfa, (int) reg.K, 1, 0, stream); } const int gemv_n = (ab_ok || norm_anchor) ? 12288 : (int) reg.N; // ab rows owned by K0 when on frt::frt_ws_launch(frt::g_reg.d_apack, reg.d_packed, frt::g_reg.d_sfa, reg.d_sf, @@ -2713,8 +2540,7 @@ bool ggml_cuda_frt_head_mul_mat(ggml_backend_cuda_context & ctx, return true; } - FRT_M_DISPATCH(M, ggml_cuda_kernel_launch((frt::quant_act_fp4_f32<256, MT>), ggml_cuda_kernel_launch_params(dim3(M), dim3(256), 0, stream), - x, (uint2 *) frt::g_head.d_apack, frt::g_head.d_sfa, (int) frt::g_head.K, (int64_t) K)); + frt::frt_quant_act_launch(x, frt::g_head.d_apack, frt::g_head.d_sfa, (int) frt::g_head.K, M, (int64_t) K, stream); frt::frt_ws_launch(frt::g_head.d_apack, frt::g_head.d_packed, frt::g_head.d_sfa, frt::g_head.d_sf, out, frt::g_head.alpha, (int) N, (int) K, M, stream, 44); // head: s4w4 wins (+6 t/s) From 2954324c0b6aabb8fcc9962dfa85c68a1974a35d Mon Sep 17 00:00:00 2001 From: LiangSu8899 <7thuniversels@gmail.com> Date: Tue, 25 Aug 2026 14:27:53 -0400 Subject: [PATCH 27/32] ggml adapter sm120: windows default on (zero-configuration safe tier) Compiled-in windows now run by default: GGML_CUDA_FLASHRT_DISABLE=1 turns the whole layer off, GGML_FLASHRT_NO_=1 disables one window, and the historic FRT__SWAP switches remain as explicit A/B overrides. The in-process repack and the spec-draft head serving are part of the default set (the draft head builds lazily, only once a loaded draft model's head copy is actually seen, so plain runs spend nothing on it); the full-tier FP4 lm-head keeps its measured quality increment and stays opt-in, as do the archive windows. The draft model's own qkv projections match the kind-1 pack shapes and would be silently swept in; that is a separate acceptance-only judgment, gated off by default (FRT_DRAFT_REGIONS=1), measured flat here. Receipts: same-binary env vs zero-env tg128 identical; zero-env speculative server median matches the env-driven safe tier; zero-env 24-chunk perplexity bit-identical; whole-layer disable reproduces stock. --- flash_rt/structures/adapters/ggml/USAGE.md | 37 ++++-- .../adapters/ggml/fr_win_qwen36_sm120.cu | 114 ++++++++++++++---- 2 files changed, 115 insertions(+), 36 deletions(-) diff --git a/flash_rt/structures/adapters/ggml/USAGE.md b/flash_rt/structures/adapters/ggml/USAGE.md index ee6a872c..1b4eb30d 100644 --- a/flash_rt/structures/adapters/ggml/USAGE.md +++ b/flash_rt/structures/adapters/ggml/USAGE.md @@ -86,18 +86,31 @@ cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 \ -DCMAKE_CUDA_FLAGS="-I/include -gencode=arch=compute_120a,code=sm_120a" ``` -The target's windows are opt-in per mechanism (historic `FRT_*` switches; -unset = off). Safe-tier set: `FRT_INPROJ_SWAP=1 FRT_ATTNQKV_SWAP=1 -FRT_GDN_SWAP=1 FRT_MOEGLUE_SWAP=1 FRT_MOEFUSE_SWAP=1 FRT_MOEFUSE_SHEXP=1 -FRT_OUTNATIVE_SWAP=1 FRT_REGIONS_PACK=`; full tier adds -`FRT_HEAD_SWAP=1 FRT_HEAD_PACK=`. Speculative serving adds -`FRT_HEAD_DRAFT=1` (safe tier: FP4-serve only the draft's head copy) and the -host-side `LLAMA_GRAPH_SLOTS=6` + `--backend-sampling`. - -FP4 region weights repack **in-process** with `FRT_ONLINE_REPACK=1` (leave -`FRT_REGIONS_PACK` unset): the pre-capture hook dequantizes the GGUF members -on device and rebuilds the wire format byte-identically to the offline packer -(validated by `FRT_REPACK_CHECK=1` with `FRT_REGIONS_PACK_REF=`). The +The safe tier is the zero-configuration default: a compiled-in build runs +every quality-neutral window (fused packs, GDN span, MoE span, out-proj, +draft-head serving, in-process repack) with no environment at all — +`./llama-server -m [-md -bs]` is the whole story. +Switch semantics, most specific wins: + +| layer | variable | +|---|---| +| whole layer off (stock llama.cpp) | `GGML_CUDA_FLASHRT_DISABLE=1` | +| per-window disable | `GGML_FLASHRT_NO_{INPROJ,ATTNQKV,GDN,MOEGLUE,MOEFUSE,SHEXP_FOLD,OUTNATIVE,HEAD_DRAFT,ONLINE_REPACK}=1` | +| per-window A/B override | historic `FRT__SWAP=0/1` | + +Opt-in extras: `FRT_HEAD_SWAP=1 FRT_HEAD_PACK=` (full tier — the FP4 +lm-head trades a measured perplexity increment for speed, so it never +defaults on); `FRT_DRAFT_REGIONS=1` (also FP4-serve the draft model's own +qkv projections; judged flat, off by default); archive windows +(`FRT_SHEXP_SWAP`, `FRT_OUTPROJ_SWAP`, `FRT_MOE_SWAP`, `FRT_ATTNGATE_SWAP`, +`FRT_GDN_NORMFOLD`) stay opt-in. Recommended host-side flags for +speculative decode: `LLAMA_GRAPH_SLOTS=6` + `--backend-sampling`. + +FP4 region weights repack **in-process by default** (set `FRT_REGIONS_PACK` +to use an offline pack instead): the pre-capture hook dequantizes the GGUF +members on device and rebuilds the wire format byte-identically to the +offline packer (validated by `FRT_REPACK_CHECK=1` with +`FRT_REGIONS_PACK_REF=`). The lm-head is the exception: the shipped head pack is quantized from the BF16 checkpoint (the GGUF only holds Q6_K), and the BF16-sourced pack drafts and scores measurably better than an online Q6_K-sourced rebuild — keep diff --git a/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu b/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu index 24a53738..7f8448ab 100644 --- a/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu +++ b/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu @@ -33,6 +33,37 @@ // the header with tools/gen_binding_header.py after editing the binding. #include "fr_binding_qwen36_35b_sm120.h" +// ---- switch semantics ----------------------------------------------------- +// Compiled-in windows default ON. Three layers of control, most specific wins: +// FRT__SWAP=0/1 per-window A/B override (historic names) +// GGML_FLASHRT_NO_=1 per-window disable +// GGML_CUDA_FLASHRT_DISABLE=1 whole layer off (stock llama.cpp) +// Archive windows (judged off on this target) stay opt-in via their FRT_* +// switches only. The full-tier head swap changes output quality, so it too +// stays opt-in (FRT_HEAD_SWAP=1). +static bool frt_layer_enabled(void) { + static int on = -1; + if (on < 0) { const char * s = getenv("GGML_CUDA_FLASHRT_DISABLE"); on = (s && s[0] == '1') ? 0 : 1; } + return on == 1; +} +static bool frt_window_on(const char * frt_env, const char * no_env) { + if (!frt_layer_enabled()) return false; + const char * f = getenv(frt_env); + if (f) return f[0] == '1'; + const char * n = getenv(no_env); + return !(n && n[0] == '1'); +} +// The spec draft model's MTP layer reuses the target's tensor-name scheme at +// layer indices >= n_layer, and its attn q/k/v happen to match the kind-1 +// pack shapes. Serving the draft's projections from FP4 only moves +// acceptance, never output — but it is a separate judgment call, so it is +// gated off by default until judged (FRT_DRAFT_REGIONS=1 to enable). +static bool frt_layer_in_scope(int layer) { + static int draft_on = -1; + if (draft_on < 0) { const char * e = getenv("FRT_DRAFT_REGIONS"); draft_on = (e && e[0] == '1') ? 1 : 0; } + return layer < frt_binding::n_layer || draft_on == 1; +} + #include #include #include @@ -310,14 +341,12 @@ static bool frt_online_on(void); // defined with the in-process repack section static bool frt_regions_load(void) { if (g_reg.tried) return g_reg.ok; g_reg.tried = true; - const char * a = getenv("FRT_INPROJ_SWAP"); - const char * b = getenv("FRT_ATTNQKV_SWAP"); const char * c = getenv("FRT_SHEXP_SWAP"); const char * d = getenv("FRT_OUTPROJ_SWAP"); - g_reg.inproj_on = a && a[0] == '1'; - g_reg.attn_on = b && b[0] == '1'; - g_reg.shexp_on = c && c[0] == '1'; - g_reg.outproj_on = d && d[0] == '1'; + g_reg.inproj_on = frt_window_on("FRT_INPROJ_SWAP", "GGML_FLASHRT_NO_INPROJ"); + g_reg.attn_on = frt_window_on("FRT_ATTNQKV_SWAP", "GGML_FLASHRT_NO_ATTNQKV"); + g_reg.shexp_on = frt_layer_enabled() && c && c[0] == '1'; // archive: opt-in + g_reg.outproj_on = frt_layer_enabled() && d && d[0] == '1'; // archive: opt-in if (!g_reg.inproj_on && !g_reg.attn_on && !g_reg.shexp_on && !g_reg.outproj_on) return false; const char * path = getenv("FRT_REGIONS_PACK"); if (!path && frt_online_on()) { @@ -422,6 +451,7 @@ static bool frt_regions_mul_mat(ggml_backend_cuda_context & ctx, if (!frt_regions_load()) return false; int layer = -1; char rest[64] = {0}; if (sscanf(src0->name, "blk.%d.%63s", &layer, rest) != 2 || layer < 0 || layer >= frt_binding::layer_scan_max) return false; + if (!frt_layer_in_scope(layer)) return false; for (int kind = 0; kind < frt_binding::n_region_kinds; ++kind) { if (kind == 0 && !g_reg.inproj_on) continue; if (kind == 1 && !g_reg.attn_on) continue; @@ -467,9 +497,8 @@ static bool frt_head_load(void) { // storage; the target head stays Q6_K/stock) — draft logits only steer // acceptance, never the verified output, so this is quality-free. const char * sw = getenv("FRT_HEAD_SWAP"); - const char * dr = getenv("FRT_HEAD_DRAFT"); - const bool sw_on = sw && sw[0] == '1'; - const bool dr_on = dr && dr[0] == '1'; + const bool sw_on = frt_layer_enabled() && sw && sw[0] == '1'; // full tier: opt-in (quality) + const bool dr_on = frt_window_on("FRT_HEAD_DRAFT", "GGML_FLASHRT_NO_HEAD_DRAFT"); if (!sw_on && !dr_on) return false; g_head.draft_only = !sw_on; const char * path = getenv("FRT_HEAD_PACK"); @@ -634,7 +663,7 @@ __global__ void frt_w_pass2_bf16(const __nv_bfloat16 * __restrict__ w, const flo static bool frt_online_on(void) { static int on = -1; - if (on < 0) { const char * s = getenv("FRT_ONLINE_REPACK"); on = (s && s[0] == '1') ? 1 : 0; } + if (on < 0) on = frt_window_on("FRT_ONLINE_REPACK", "GGML_FLASHRT_NO_ONLINE_REPACK") ? 1 : 0; return on == 1; } @@ -745,8 +774,36 @@ static void frt_repack_check_region(int kind, int layer, const frt_region & r) { static void frt_online_prepare(ggml_backend_cuda_context & ctx, const ggml_cgraph * cgraph) { if (!frt_online_on()) return; static bool all_done = false; + static int evals_seen = 0; if (all_done) return; - if (!frt_regions_load() && !getenv("FRT_HEAD_SWAP") && !getenv("FRT_HEAD_DRAFT")) { all_done = true; return; } + ++evals_seen; + const char * hsw = getenv("FRT_HEAD_SWAP"); + const bool head_full = frt_layer_enabled() && hsw && hsw[0] == '1'; + const bool head_draft = frt_window_on("FRT_HEAD_DRAFT", "GGML_FLASHRT_NO_HEAD_DRAFT"); + if (!frt_regions_load() && !head_full && !head_draft) { all_done = true; return; } + + // once everything structural is built and only a draft-model head sighting + // is outstanding, drop to a light scan (name check only) — the full member + // scan on every eval costs measurable host time. + static bool structural_done = false; + static bool saw_draft_head = false; // a loaded spec draft's Q8_0 head copy + if (structural_done) { + // the draft model's graphs are small (single MTP layer); skip the walk + // on full-size target graphs entirely. + if (!saw_draft_head && cgraph->n_nodes < 512) { + for (int i = 0; i < cgraph->n_nodes; ++i) { + const ggml_tensor * n = cgraph->nodes[i]; + if (n->op == GGML_OP_MUL_MAT && n->src[0] && n->src[0]->type == GGML_TYPE_Q8_0 && + n->src[0]->ne[0] == frt_binding::d_model && strcmp(n->src[0]->name, frt_binding::head_name) == 0) { + saw_draft_head = true; + break; + } + } + } + if (!saw_draft_head && evals_seen < 4096) return; // cheap: small graphs only + if (!saw_draft_head) { all_done = true; return; } // no draft model; give up + // fall through: full scan once, to locate the pack source and build + } using namespace frt_binding; const ggml_tensor * mem[n_region_kinds][64][region_max_members] = {}; @@ -755,12 +812,14 @@ static void frt_online_prepare(ggml_backend_cuda_context & ctx, const ggml_cgrap const ggml_tensor * n = cgraph->nodes[i]; if (n->op != GGML_OP_MUL_MAT || !n->src[0]) continue; const ggml_tensor * w = n->src[0]; - if (strcmp(w->name, head_name) == 0 && w->type != GGML_TYPE_Q8_0 && w->ne[0] == d_model) { - head_w = w; + if (strcmp(w->name, head_name) == 0 && w->ne[0] == d_model) { + if (w->type != GGML_TYPE_Q8_0) head_w = w; + else saw_draft_head = true; continue; } int layer = -1; char rest[64] = {0}; if (sscanf(w->name, "blk.%d.%63s", &layer, rest) != 2 || layer < 0 || layer >= layer_scan_max) continue; + if (!frt_layer_in_scope(layer)) continue; for (int kind = 0; kind < n_region_kinds; ++kind) for (int m = 0; m < region_n_members[kind]; ++m) if (strcmp(rest, region_members[kind][m].name) == 0 && w->ne[1] == region_members[kind][m].rows) @@ -799,7 +858,10 @@ static void frt_online_prepare(ggml_backend_cuda_context & ctx, const ggml_cgrap } } - if (head_w && !g_head.ok && (getenv("FRT_HEAD_SWAP") || getenv("FRT_HEAD_DRAFT"))) { + // draft-only serving builds lazily: only once a draft model is actually + // loaded (its Q8_0 head copy shows up in a graph), so plain runs never + // spend VRAM on a head pack that would never serve. + if (head_w && !g_head.ok && (head_full || (head_draft && saw_draft_head))) { const int64_t N = head_w->ne[1], K = head_w->ne[0]; const size_t pkb = (size_t) N * K / 2; const size_t sfb = (size_t) ((N + 127) / 128) * ((K + 63) / 64) * 512; @@ -823,16 +885,20 @@ static void frt_online_prepare(ggml_backend_cuda_context & ctx, const ggml_cgrap } } - const bool head_pending = (getenv("FRT_HEAD_SWAP") || getenv("FRT_HEAD_DRAFT")) && !g_head.ok; + const bool head_pending = (head_full || (head_draft && saw_draft_head)) && !g_head.ok; bool region_pending = false; for (int kind = 0; kind < n_region_kinds; ++kind) { const bool on = (kind == 0 && g_reg.inproj_on) || (kind == 1 && g_reg.attn_on); region_pending = region_pending || (on && built[kind] < region_layers[kind]); } if (!head_pending && !region_pending) { - fprintf(stderr, "frt-repack: online repack complete (kind0=%d kind1=%d head=%d)\n", - built[0], built[1], (int) g_head.ok); - all_done = true; + structural_done = true; + const bool draft_wait = head_draft && !head_full && !saw_draft_head; + if (!draft_wait) { + fprintf(stderr, "frt-repack: online repack complete (kind0=%d kind1=%d head=%d)\n", + built[0], built[1], (int) g_head.ok); + all_done = true; + } } } @@ -1134,7 +1200,7 @@ __global__ void frt_gdn_epilogue( // Returns number of nodes consumed starting at i (0 = not ours). bool ggml_cuda_frt_gdn_try_impl(ggml_backend_cuda_context & ctx, ggml_cgraph * cgraph, int i, int * skip_count) { static int mode = -1; - if (mode < 0) { const char * m = getenv("FRT_GDN_SWAP"); mode = (m && m[0] == '1') ? 1 : 0; } + if (mode < 0) mode = frt_window_on("FRT_GDN_SWAP", "GGML_FLASHRT_NO_GDN") ? 1 : 0; if (!mode) return false; if (!frt::frt_regions_load() || !frt::g_reg.inproj_on) return false; @@ -1818,7 +1884,7 @@ static bool frt_router_span_try(ggml_backend_cuda_context & ctx, ggml_cgraph * c // (their fusion is M=1-only; the M=2 fallback is the unfused argsort chain). // "1" -> on for all M; "0" -> off entirely. static int mode = -1; - if (mode < 0) { const char * m = getenv("FRT_ROUTER_SWAP"); mode = m ? ((m[0] == '1') ? 2 : 0) : 1; } + if (mode < 0) { const char * m = getenv("FRT_ROUTER_SWAP"); mode = !frt_layer_enabled() ? 0 : (m ? ((m[0] == '1') ? 2 : 0) : 1); } if (!mode) return false; ggml_tensor * n0 = cgraph->nodes[i]; if (n0->op != GGML_OP_MUL_MAT || !n0->src[0] || @@ -1880,7 +1946,7 @@ static bool frt_router_span_try(ggml_backend_cuda_context & ctx, ggml_cgraph * c // fused MoE expert segment: anchor = MUL_MAT_ID on ffn_gate_exps.weight. static bool frt_moefuse_try(ggml_backend_cuda_context & ctx, ggml_cgraph * cgraph, int i, int * skip_count) { static int mode = -1; - if (mode < 0) { const char * m = getenv("FRT_MOEFUSE_SWAP"); mode = (m && m[0] == '1') ? 1 : 0; } + if (mode < 0) mode = frt_window_on("FRT_MOEFUSE_SWAP", "GGML_FLASHRT_NO_MOEFUSE") ? 1 : 0; if (!mode) return false; ggml_tensor * gate_id = cgraph->nodes[i]; if (gate_id->op != GGML_OP_MUL_MAT_ID || !gate_id->src[0] || @@ -1919,7 +1985,7 @@ static bool frt_moefuse_try(ggml_backend_cuda_context & ctx, ggml_cgraph * cgrap if (ids->ne[1] != M) return false; static int sh_mode = -1; - if (sh_mode < 0) { const char * m = getenv("FRT_MOEFUSE_SHEXP"); sh_mode = (m && m[0] == '1') ? 1 : 0; } + if (sh_mode < 0) sh_mode = frt_window_on("FRT_MOEFUSE_SHEXP", "GGML_FLASHRT_NO_SHEXP_FOLD") ? 1 : 0; const ggml_tensor * up_id = nullptr, * glu = nullptr, * down_id = nullptr; const ggml_tensor * wmul = nullptr; const ggml_tensor * shg = nullptr, * shu = nullptr, * shd = nullptr, * ginp = nullptr; @@ -2150,7 +2216,7 @@ static bool frt_moefuse_try(ggml_backend_cuda_context & ctx, ggml_cgraph * cgrap // (their path: quantize_q8_1 + mmvq + add = 3 launches) -> quant + gemv/add = 2. static bool frt_outproj_native_try(ggml_backend_cuda_context & ctx, ggml_cgraph * cgraph, int i, int * skip_count) { static int mode = -1; - if (mode < 0) { const char * m = getenv("FRT_OUTNATIVE_SWAP"); mode = (m && m[0] == '1') ? 1 : 0; } + if (mode < 0) mode = frt_window_on("FRT_OUTNATIVE_SWAP", "GGML_FLASHRT_NO_OUTNATIVE") ? 1 : 0; if (!mode) return false; ggml_tensor * n0 = cgraph->nodes[i]; if (n0->op != GGML_OP_MUL_MAT || !n0->src[0]) return false; @@ -2299,7 +2365,7 @@ bool ggml_cuda_frt_moeglue_try_impl(ggml_backend_cuda_context & ctx, ggml_cgraph if (frt_outproj_native_try(ctx, cgraph, i, skip_count)) return true; if (frt_shexp_span_try(ctx, cgraph, i, skip_count)) return true; static int mode = -1; - if (mode < 0) { const char * m = getenv("FRT_MOEGLUE_SWAP"); mode = (m && m[0] == '1') ? 1 : 0; } + if (mode < 0) mode = frt_window_on("FRT_MOEGLUE_SWAP", "GGML_FLASHRT_NO_MOEGLUE") ? 1 : 0; if (!mode) return false; ggml_tensor * n0 = cgraph->nodes[i]; From 65f9fb2e21adbabe25760ca1091d68223d0692f2 Mon Sep 17 00:00:00 2001 From: LiangSu8899 <7thuniversels@gmail.com> Date: Tue, 25 Aug 2026 14:33:43 -0400 Subject: [PATCH 28/32] qualification: one-command gate runner for the sm120/Qwen3.6 target Offline gates: binding manifest validation, structure-version pins, and binding-header freshness (the generated constants header must match the binding yaml). On-device gates re-establish their numbers by running: duplicated-token bit-exact selftest, online-repack byte-identity against an offline reference pack, 24-chunk perplexity (pinned bit value), and tg128 against the tier's pinned baseline with a tolerance band. First full run: six greens (selftest PASS, repack 40/40 byte-identical, ppl 6.1106 exact, tg128 within band on the full tier). --- .../ggml/qualification/pins_qwen36_sm120.yaml | 1 + .../run_qualification_qwen36_sm120.py | 174 ++++++++++++++++++ 2 files changed, 175 insertions(+) create mode 100644 flash_rt/structures/adapters/ggml/qualification/run_qualification_qwen36_sm120.py diff --git a/flash_rt/structures/adapters/ggml/qualification/pins_qwen36_sm120.yaml b/flash_rt/structures/adapters/ggml/qualification/pins_qwen36_sm120.yaml index 61e8e165..19a361c7 100644 --- a/flash_rt/structures/adapters/ggml/qualification/pins_qwen36_sm120.yaml +++ b/flash_rt/structures/adapters/ggml/qualification/pins_qwen36_sm120.yaml @@ -29,6 +29,7 @@ gates: spec_text_check: required # greedy spec output must be coherent text, never judged by tps alone perf: tg128_full_tier_bench: 377.8 # llama-bench r=5, full-tier env + tg128_default_bench: 368.0 # llama-bench r=5, zero-env default (safe tier) spec_full_tier_median: 444.6 # 4-prompt server battery, greedy 384 tok, LLAMA_GRAPH_SLOTS=6 -bs spec_safe_tier_median: 433.3 stock_reference: 283.07 # same-machine stock @95b8e33 diff --git a/flash_rt/structures/adapters/ggml/qualification/run_qualification_qwen36_sm120.py b/flash_rt/structures/adapters/ggml/qualification/run_qualification_qwen36_sm120.py new file mode 100644 index 00000000..ed7031e3 --- /dev/null +++ b/flash_rt/structures/adapters/ggml/qualification/run_qualification_qwen36_sm120.py @@ -0,0 +1,174 @@ +#!/usr/bin/env python3 +"""Qualification gates for the SM120/Qwen3.6-35B target. + +Offline gates (always run): + A. manifest — the pipeline binding validates against the live catalog. + B. pins — pinned structure versions match the live catalog. + C. header — the checked-in binding constants header is up to date with + the binding yaml (regenerate with tools/gen_binding_header.py). + +On-device gates (opt-in; each re-establishes its number by running, never by +quoting — see the gates block in pins_qwen36_sm120.yaml for the recorded +baseline): + D. selftest — duplicated-token bit-exact replay across batch variants + (FRT_MOEFUSE_SELFTEST) plus, when a reference pack is given, + the online-repack byte-identity check. + E. ppl — 24-chunk -ub 1 perplexity, safe tier; must match the pinned + value to the printed precision (bit-stable path). + F. bench — tg128 r=5; red below (1 - tol) x pinned. + +Usage: + python run_qualification_qwen36_sm120.py # A+B+C + python run_qualification_qwen36_sm120.py --device \\ + --bin --model --wikitext \\ + [--regions-ref ] [--tol 0.03] # + D+E+F +""" + +from __future__ import annotations + +import argparse +import os +import pathlib +import re +import subprocess +import sys + +import yaml + +_HERE = pathlib.Path(__file__).resolve().parent +_REPO = _HERE.parents[4] +if str(_REPO) not in sys.path: + sys.path.insert(0, str(_REPO)) + +from flash_rt.structures.binding import load_binding # noqa: E402 +from flash_rt.structures.registry import load as load_structure # noqa: E402 + +BINDING = "llamacpp_qwen36_35b_sm120" +PINS = _HERE / "pins_qwen36_sm120.yaml" + + +def gate_manifest() -> tuple[bool, str]: + try: + spec = load_binding(BINDING, require_pipeline_coverage=True) + except Exception as exc: # noqa: BLE001 + return False, f"binding failed validation: {exc}" + return True, (f"{spec.name} -> {spec.structure.name}@{spec.structure.version}, " + f"{len(spec.segments)} segments, contract {spec.coverage_contract}") + + +def gate_pins() -> tuple[bool, str]: + pinned = yaml.safe_load(PINS.read_text())["pins"] + drifted = [] + for name, version in pinned.items(): + try: + live = load_structure(name).version + except KeyError: + drifted.append(f"{name}: pinned @{version}, missing from catalog") + continue + if int(live) != int(version): + drifted.append(f"{name}: pinned @{version}, catalog is @{live}") + if drifted: + return False, "; ".join(drifted) + return True, f"{len(pinned)} structure versions match the catalog" + + +def gate_header() -> tuple[bool, str]: + gen = _HERE.parent / "tools" / "gen_binding_header.py" + header = _HERE.parent / f"fr_binding_{BINDING.split('llamacpp_')[-1]}.h" + if not header.is_file(): + return False, f"missing {header.name}" + current = header.read_text() + proc = subprocess.run([sys.executable, str(gen), BINDING], capture_output=True, text=True) + if proc.returncode != 0: + return False, f"generator failed: {proc.stderr.strip()}" + fresh = header.read_text() + if fresh != current: + header.write_text(current) # restore; the red asks for a deliberate regen+review + return False, "binding header is stale — regenerate with tools/gen_binding_header.py and review" + return True, f"{header.name} matches the binding" + + +# ---- on-device gates ------------------------------------------------------- + +def _run(cmd, env_extra=None, timeout=1800): + env = dict(os.environ) + if env_extra: + env.update(env_extra) + return subprocess.run(cmd, capture_output=True, text=True, env=env, timeout=timeout) + + +def gate_selftest(args) -> tuple[bool, str]: + env = {"FRT_MOEFUSE_SELFTEST": "1"} + if args.regions_ref: + env.update({"FRT_REPACK_CHECK": "1", "FRT_REGIONS_PACK_REF": args.regions_ref}) + proc = _run([f"{args.bin}/llama-cli", "-m", args.model, "-fa", "on", "-st", + "-n", "8", "-p", "Hello"], env) + out = proc.stdout + proc.stderr + ok_self = "PASS" in out and "FAIL" not in out + msgs = [f"selftest {'PASS' if ok_self else 'FAIL'}"] + ok = ok_self + if args.regions_ref: + n_ok = len(re.findall(r"packed=OK sf=OK", out)) + n_bad = out.count("MISMATCH") + msgs.append(f"repack byte-identity {n_ok} OK / {n_bad} mismatch") + ok = ok and n_bad == 0 and n_ok > 0 + return ok, "; ".join(msgs) + + +def gate_ppl(args, pinned: float) -> tuple[bool, str]: + proc = _run([f"{args.bin}/llama-perplexity", "-m", args.model, "-f", args.wikitext, + "-ub", "1", "-c", "512", "-b", "512", "--chunks", "24", "-fa", "1"]) + m = re.search(r"Final estimate: PPL = ([0-9.]+)", proc.stdout + proc.stderr) + if not m: + return False, "no PPL estimate in output" + got = float(m.group(1)) + ok = abs(got - pinned) < 5e-5 + return ok, f"ppl {got} vs pinned {pinned}" + + +def gate_bench(args, pinned: float, tol: float) -> tuple[bool, str]: + env = {} + if args.head_pack: + env = {"FRT_HEAD_SWAP": "1", "FRT_HEAD_PACK": args.head_pack} + proc = _run([f"{args.bin}/llama-bench", "-m", args.model, "-fa", "1", + "-p", "0", "-n", "128", "-r", "5"], env) + m = re.search(r"tg128\s*\|\s*([0-9.]+)", proc.stdout) + if not m: + return False, "no tg128 in output" + got = float(m.group(1)) + ok = got >= pinned * (1.0 - tol) + tier = "full" if args.head_pack else "safe/default" + return ok, f"tg128 {got} ({tier}) vs pinned {pinned} (tol {tol:.0%})" + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--device", action="store_true", help="run the on-device gates") + ap.add_argument("--bin", help="llama.cpp build/bin directory") + ap.add_argument("--model", help="target GGUF") + ap.add_argument("--wikitext", help="wikitext test file for the ppl gate") + ap.add_argument("--regions-ref", help="offline region pack for the byte-identity check") + ap.add_argument("--head-pack", help="full-tier head pack (bench gate then judges the full tier)") + ap.add_argument("--tol", type=float, default=0.03) + args = ap.parse_args() + + gates = [("manifest", gate_manifest()), ("pins", gate_pins()), ("header", gate_header())] + if args.device: + if not (args.bin and args.model): + ap.error("--device needs --bin and --model") + g = yaml.safe_load(PINS.read_text())["gates"] + gates.append(("selftest", gate_selftest(args))) + if args.wikitext: + gates.append(("ppl", gate_ppl(args, float(g["quality"]["ppl_24ch_ub1_safe_tier"])))) + key = "tg128_full_tier_bench" if args.head_pack else "tg128_default_bench" + gates.append(("bench", gate_bench(args, float(g["perf"][key]), args.tol))) + + red = False + for name, (ok, msg) in gates: + print(f"[{'GREEN' if ok else 'RED':5}] {name}: {msg}") + red = red or not ok + return 1 if red else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From 3b77186a039bce94ecbc75e33803b45ae1f27069 Mon Sep 17 00:00:00 2001 From: LiangSu8899 <7thuniversels@gmail.com> Date: Tue, 25 Aug 2026 14:34:34 -0400 Subject: [PATCH 29/32] docs: sm120 target invariants (PDL capability, template-M, speculative state discipline) --- .../structures/adapters/ggml/DEVELOPMENT.md | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/flash_rt/structures/adapters/ggml/DEVELOPMENT.md b/flash_rt/structures/adapters/ggml/DEVELOPMENT.md index 0c5f4742..8384e857 100644 --- a/flash_rt/structures/adapters/ggml/DEVELOPMENT.md +++ b/flash_rt/structures/adapters/ggml/DEVELOPMENT.md @@ -111,3 +111,38 @@ belong in `csrc/` or the structures catalog so every host adapter inherits them; this directory only translates. Nothing here may be copy-pasted into a host tree, and the host integration must stay behind its own opt-in build flag so stock builds are unaffected. + +## SM120 target: additional invariants (LLM decode, speculative) + +Learned on the Qwen3.6-35B window set; they generalize to any stateful or +speculative host integration. + +- **Host launch overlap is a capability, not a constant.** llama.cpp's CUDA + backend overlaps every launch through programmatic dependent launch + (sm90+); on such a host every adapter kernel must join the chain (device + trigger/sync + the launch attribute) or it stalls the pipeline — and once + the chain holds, pure launch-count reduction has near-zero marginal value, + so fusions must win on memory round-trips, byte reduction, or batch size. + On hosts without PDL the same fusions re-rank. Treat PDL as a per-target + capability flag (the csrc entries take a `pdl` bool). +- **Runtime dimensions out of hot loops.** A token-batch count as a kernel + argument instead of a template parameter costs measurable time even when + the value is 1; heavier instantiations degrade more. Dispatch runtime M + onto compile-time specializations at the launch boundary. +- **Speculative verify batches are a correctness regime, not a batch size.** + Stateful regions (recurrent state, conv windows) must write per-token + snapshots and leave the source slot pristine so the host can roll back to + any accepted position; in-place update produces degenerate output with + *inflated* acceptance and throughput, and perplexity-style gates do not + cover the speculative graphs at all. Judge on end-to-end text plus a + duplicated-token bit-exact replay across batch variants. +- **Zero-sized graph nodes can become real.** Checkpoint save nodes sit in + the host graph at zero size on most steps and materialize on checkpoint + steps; a region that silently skips them corrupts rollback invisibly. + Replay them inside the region or decline the whole span. +- **A second model shares the host's name scheme.** The speculative draft + model's tensors reuse the target's naming at shifted layer indices and can + collide with window/pack shapes; anything swept in from the draft model is + an acceptance-only substitution (never output-visible), but it is a + separate judgment — gate it explicitly instead of letting shape + coincidence decide. From ba8a5b73b915b75f3a35c9b5f34d4eab03dc340e Mon Sep 17 00:00:00 2001 From: LiangSu8899 <7thuniversels@gmail.com> Date: Tue, 25 Aug 2026 14:49:25 -0400 Subject: [PATCH 30/32] ggml adapter sm120: NVFP4-head artifact path (packless full tier) Add the FlashRT-edition GGUF flow: an NVFP4 lm-head quantized from the BF16 checkpoint is spliced into the shipping quantized body (tools/splice_nvfp4_head.py). The stock nvfp4 kernels then serve the head at pack-tier speed by themselves, the adapter's spec-draft head serving wires up from the same tensor by pure shuffle (scale passthrough, alpha = 1), and the last side-band pack file is gone: the artifact judges faster-tier speed with better perplexity than the pack-served full tier on the same body. Judged along the way, gates recorded in the pins file: serving the NVFP4 head through the fp4-activation GEMV adds no speed over stock's q8_1-activation mmvq and costs measurable perplexity (archive switch, opt-in); a whole-model standard requantization loses to the shipping body both at the floor and in FP4-region cost (region quality follows the source tensor's quantization); the draft model's shape-colliding projections stay gated. --- flash_rt/structures/adapters/ggml/USAGE.md | 9 +++ .../adapters/ggml/fr_win_qwen36_sm120.cu | 76 +++++++++++++++++-- .../ggml/qualification/pins_qwen36_sm120.yaml | 15 +++- .../adapters/ggml/tools/splice_nvfp4_head.py | 70 +++++++++++++++++ .../bindings/llamacpp_qwen36_35b_sm120.yaml | 2 +- 5 files changed, 164 insertions(+), 8 deletions(-) create mode 100644 flash_rt/structures/adapters/ggml/tools/splice_nvfp4_head.py diff --git a/flash_rt/structures/adapters/ggml/USAGE.md b/flash_rt/structures/adapters/ggml/USAGE.md index 1b4eb30d..f836386e 100644 --- a/flash_rt/structures/adapters/ggml/USAGE.md +++ b/flash_rt/structures/adapters/ggml/USAGE.md @@ -106,6 +106,15 @@ qkv projections; judged flat, off by default); archive windows `FRT_GDN_NORMFOLD`) stay opt-in. Recommended host-side flags for speculative decode: `LLAMA_GRAPH_SLOTS=6` + `--backend-sampling`. +**Model artifacts** (sm120 target): the safe tier runs any stock GGUF +as-is. The speed tier is itself just a GGUF — the FlashRT edition splices +an NVFP4 lm-head (quantized from the BF16 checkpoint via +`llama-quantize --output-tensor-type NVFP4` on a bf16 conversion) into the +shipping body with `tools/splice_nvfp4_head.py`; no side-band packs, no +environment. Do not requantize the whole body from scratch for this: the +FP4 regions inherit the source tensors' quantization quality, so the best +shipping body stays the best base. + FP4 region weights repack **in-process by default** (set `FRT_REGIONS_PACK` to use an offline pack instead): the pre-capture hook dequantizes the GGUF members on device and rebuilds the wire format byte-identically to the diff --git a/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu b/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu index 7f8448ab..83fa517a 100644 --- a/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu +++ b/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu @@ -499,7 +499,12 @@ static bool frt_head_load(void) { const char * sw = getenv("FRT_HEAD_SWAP"); const bool sw_on = frt_layer_enabled() && sw && sw[0] == '1'; // full tier: opt-in (quality) const bool dr_on = frt_window_on("FRT_HEAD_DRAFT", "GGML_FLASHRT_NO_HEAD_DRAFT"); - if (!sw_on && !dr_on) return false; + // opt-in: serving an NVFP4-typed head through the fp4-activation GEMV + // costs a measured perplexity increment over stock's q8_1-activation mmvq + // on the same weights, so it is a quality trade like the full-tier swap. + const char * nat = getenv("FRT_HEAD_NATIVE"); + const bool nat_on = frt_layer_enabled() && nat && nat[0] == '1'; + if (!sw_on && !dr_on && !nat_on) return false; g_head.draft_only = !sw_on; const char * path = getenv("FRT_HEAD_PACK"); if (!path && frt_online_on()) { @@ -661,6 +666,45 @@ __global__ void frt_w_pass2_bf16(const __nv_bfloat16 * __restrict__ w, const flo } } +// ggml GGML_TYPE_NVFP4 weight -> GEMV wire format, pure shuffle (no +// requantization): block_nvfp4 is 36 B / 64 elems = d[4] e4m3 sub-scales + +// qs[32] split-nibble codes (sub s at qs[s*8+j]: elem j low nibble, elem +// 8+j high). ggml's doubled e2m1 table and halved ue4m3 decode cancel, so +// the scale bytes pass through unmodified and the GEMV runs with alpha = 1. +// One thread per 16-element sub-block. +__global__ void frt_w_nvfp4_shuffle( + const uint8_t * __restrict__ src, uint8_t * __restrict__ packed, + uint8_t * __restrict__ sf_swz, int n_rows, int K, int row_base) { + const int64_t idx = (int64_t) blockIdx.x * blockDim.x + threadIdx.x; + const int64_t subs_per_row = K / 16; + if (idx >= (int64_t) n_rows * subs_per_row) return; + const int r = (int) (idx / subs_per_row); + const int sub = (int) (idx % subs_per_row); + const int blk = sub >> 2, s = sub & 3; + const uint8_t * b = src + ((size_t) r * (K / 64) + blk) * 36; + const uint8_t * qs = b + 4 + s * 8; + const int row = row_base + r; + uint8_t * out = packed + (size_t) row * (K / 2) + (size_t) sub * 8; +#pragma unroll + for (int p = 0; p < 4; ++p) + out[p] = (uint8_t) ((qs[2 * p] & 0x0F) | ((qs[2 * p + 1] & 0x0F) << 4)); +#pragma unroll + for (int p = 0; p < 4; ++p) + out[4 + p] = (uint8_t) ((qs[2 * p] >> 4) | ((qs[2 * p + 1] >> 4) << 4)); + sf_swz[sfa_offset_128x64(row, sub * 16, K)] = b[s]; +} + +// build wire buffers for an NVFP4-typed ggml tensor (shuffle only, alpha=1) +static bool frt_repack_shuffle_nvfp4(const ggml_tensor * t, int64_t N, int64_t K, + uint8_t * d_packed, uint8_t * d_sf, cudaStream_t stream) { + const int64_t total = N * (K / 16); + const int64_t blocks = (total + 255) / 256; + frt_w_nvfp4_shuffle<<>>( + (const uint8_t *) t->data, d_packed, d_sf, (int) N, (int) K, 0); + CUDA_CHECK(cudaStreamSynchronize(stream)); + return true; +} + static bool frt_online_on(void) { static int on = -1; if (on < 0) on = frt_window_on("FRT_ONLINE_REPACK", "GGML_FLASHRT_NO_ONLINE_REPACK") ? 1 : 0; @@ -861,15 +905,28 @@ static void frt_online_prepare(ggml_backend_cuda_context & ctx, const ggml_cgrap // draft-only serving builds lazily: only once a draft model is actually // loaded (its Q8_0 head copy shows up in a graph), so plain runs never // spend VRAM on a head pack that would never serve. - if (head_w && !g_head.ok && (head_full || (head_draft && saw_draft_head))) { + const char * hnat = getenv("FRT_HEAD_NATIVE"); + const bool head_native = head_w && head_w->type == GGML_TYPE_NVFP4 && + frt_layer_enabled() && hnat && hnat[0] == '1'; + if (head_w && !g_head.ok && (head_full || head_native || (head_draft && saw_draft_head))) { const int64_t N = head_w->ne[1], K = head_w->ne[0]; const size_t pkb = (size_t) N * K / 2; const size_t sfb = (size_t) ((N + 127) / 128) * ((K + 63) / 64) * 512; CUDA_CHECK(cudaMalloc(&g_head.d_packed, pkb)); CUDA_CHECK(cudaMalloc(&g_head.d_sf, sfb)); float alpha = 0.f; - frt_repack_src src = { head_w, N }; - if (frt_repack_build(&src, 1, N, K, g_head.d_packed, g_head.d_sf, &alpha, stream)) { + bool built; + if (head_w->type == GGML_TYPE_NVFP4) { + // FlashRT-edition GGUF: the head is already NVFP4 (quantized from + // the BF16 checkpoint by llama-quantize) — wire it up by shuffle, + // no requantization, alpha = 1. + built = frt_repack_shuffle_nvfp4(head_w, N, K, g_head.d_packed, g_head.d_sf, stream); + alpha = 1.0f; + } else { + frt_repack_src src = { head_w, N }; + built = frt_repack_build(&src, 1, N, K, g_head.d_packed, g_head.d_sf, &alpha, stream); + } + if (built) { g_head.N = N; g_head.K = K; g_head.alpha = alpha; CUDA_CHECK(cudaMalloc(&g_head.d_apack, 4 * (K / 2))); CUDA_CHECK(cudaMalloc(&g_head.d_sfa, 128 * (K / 16))); @@ -885,7 +942,7 @@ static void frt_online_prepare(ggml_backend_cuda_context & ctx, const ggml_cgrap } } - const bool head_pending = (head_full || (head_draft && saw_draft_head)) && !g_head.ok; + const bool head_pending = (head_full || head_native || (head_draft && saw_draft_head)) && !g_head.ok; bool region_pending = false; for (int kind = 0; kind < n_region_kinds; ++kind) { const bool on = (kind == 0 && g_reg.inproj_on) || (kind == 1 && g_reg.attn_on); @@ -2579,7 +2636,14 @@ bool ggml_cuda_frt_head_mul_mat(ggml_backend_cuda_context & ctx, } if (!frt::frt_head_load()) return false; if (strcmp(src0->name, frt_binding::head_name) != 0) return false; - if (frt::g_head.draft_only && src0->type != GGML_TYPE_Q8_0) return false; + // without the full-tier swap, serve only the spec draft's Q8_0 head copy + // and an NVFP4-typed main head (FlashRT-edition GGUF: same values as + // stock would dequantize, so the takeover is quality-neutral) + if (frt::g_head.draft_only && src0->type != GGML_TYPE_Q8_0) { + static int nat = -1; + if (nat < 0) { const char * e = getenv("FRT_HEAD_NATIVE"); nat = (e && e[0] == '1') ? 1 : 0; } + if (!(nat == 1 && src0->type == GGML_TYPE_NVFP4)) return false; + } if (src1->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) return false; const int M = (int) src1->ne[1]; // spec verify asks for logits at M = 1 + n_draft rows if (M < 1 || M > 4 || src1->ne[2] != 1 || src1->ne[3] != 1) return false; diff --git a/flash_rt/structures/adapters/ggml/qualification/pins_qwen36_sm120.yaml b/flash_rt/structures/adapters/ggml/qualification/pins_qwen36_sm120.yaml index 19a361c7..525e61a4 100644 --- a/flash_rt/structures/adapters/ggml/qualification/pins_qwen36_sm120.yaml +++ b/flash_rt/structures/adapters/ggml/qualification/pins_qwen36_sm120.yaml @@ -28,8 +28,21 @@ gates: repack_check: byte_identical # FRT_REPACK_CHECK=1 vs the offline packs (regions) spec_text_check: required # greedy spec output must be coherent text, never judged by tps alone perf: - tg128_full_tier_bench: 377.8 # llama-bench r=5, full-tier env + tg128_full_tier_bench: 377.8 # llama-bench r=5, full-tier env (BF16 head pack) tg128_default_bench: 368.0 # llama-bench r=5, zero-env default (safe tier) spec_full_tier_median: 444.6 # 4-prompt server battery, greedy 384 tok, LLAMA_GRAPH_SLOTS=6 -bs spec_safe_tier_median: 433.3 stock_reference: 283.07 # same-machine stock @95b8e33 + +# FlashRT-edition artifact (unsloth body + NVFP4-from-BF16 head, built with +# tools/splice_nvfp4_head.py; no side-band packs, zero-env): +artifact_spliced: + quality: + ppl_24ch_stock_floor: 6.1192 # head weight cost over the plain body's 6.0055 + ppl_24ch_default: 6.2290 # beats the pack-served full tier's 6.2889 + perf: + tg128_default: 377.7 # equals the pack-served full tier + spec_default_median: 439.4 # 2-prompt battery + text verified + judged_negative: + head_native_gemv: "FRT_HEAD_NATIVE: +0 speed (stock nvfp4 mmvq already + reads the 4.5bpw head), -1.1% ppl from the fp4 activation — archive" diff --git a/flash_rt/structures/adapters/ggml/tools/splice_nvfp4_head.py b/flash_rt/structures/adapters/ggml/tools/splice_nvfp4_head.py new file mode 100644 index 00000000..86db204b --- /dev/null +++ b/flash_rt/structures/adapters/ggml/tools/splice_nvfp4_head.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 +"""Build the FlashRT-edition GGUF: splice an NVFP4 lm-head into a quantized +body. + + python3 splice_nvfp4_head.py + +body: the shipping quantized model (its output.weight is replaced). +head-source: any GGUF whose output.weight is GGML_TYPE_NVFP4 quantized from +the BF16 checkpoint (e.g. `llama-quantize --output-tensor-type NVFP4` on a +bf16 conversion) — quantizing the head from BF16 measurably beats a rebuild +from an already-quantized head. + +The resulting artifact needs no side-band packs: the stock nvfp4 kernels +serve the head, and the adapter's online repack wires everything else. + +gguf-py pitfalls encoded here: ReaderTensor.shape is ne-order while +ReaderTensor.data.shape is the byte-shaped numpy-order the writer wants; +non-quantized tensors must go through dtype inference, not raw_dtype; and +field values may be numpy scalars the struct packer rejects. +""" + +import sys + +import numpy as np +from gguf import GGUFReader, GGUFWriter +from gguf.constants import GGUFValueType + + +def to_py(v): + if hasattr(v, "item"): + return v.item() + if isinstance(v, (list, tuple)): + return [to_py(x) for x in v] + return v + + +def main() -> None: + body, head_src, out = sys.argv[1], sys.argv[2], sys.argv[3] + hr = GGUFReader(head_src) + head_t = next(t for t in hr.tensors if t.name == "output.weight") + assert int(head_t.tensor_type) == 40, f"head-source output.weight is {head_t.tensor_type}, want NVFP4" + + br = GGUFReader(body) + w = GGUFWriter(out, br.fields["general.architecture"].contents()) + skip = {"GGUF.version", "GGUF.tensor_count", "GGUF.kv_count", "general.architecture"} + for key, field in br.fields.items(): + if key in skip: + continue + vt = field.types[0] + if vt == GGUFValueType.ARRAY: + w.add_key_value(key, to_py(field.contents()), vt, sub_type=field.types[1]) + else: + w.add_key_value(key, to_py(field.contents()), vt) + for t in br.tensors: + src = head_t if t.name == "output.weight" else t + if src.data.dtype == np.uint8: # quantized: raw bytes + explicit type + w.add_tensor(src.name, src.data, + raw_shape=[int(x) for x in src.data.shape], + raw_dtype=src.tensor_type) + else: # f32/f16/...: writer infers from dtype + w.add_tensor(src.name, np.ascontiguousarray(src.data)) + w.write_header_to_file() + w.write_kv_data_to_file() + w.write_tensors_to_file() + w.close() + print(f"wrote {out} (NVFP4 head spliced)") + + +if __name__ == "__main__": + main() diff --git a/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml b/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml index 8b751472..4fb7c6f0 100644 --- a/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml +++ b/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml @@ -165,5 +165,5 @@ coverage: seam: "host-tree change (llama-context): per-shape graph slots keyed (n_tokens, gtype, has_embd), each owning its scheduler" hosts: - llamacpp_dev_sm120: + llamacpp_sm120: versions: "upstream 95b8e33 + FlashRT hook guards (GGML_CUDA_FLASHRT_SM120)" From 79ce03bb26d448d9d74d0459cda377394e8dc153 Mon Sep 17 00:00:00 2001 From: LiangSu8899 <7thuniversels@gmail.com> Date: Fri, 11 Sep 2026 14:10:36 -0400 Subject: [PATCH 31/32] ggml adapter: move to hosts/ggml The native ggml host adapter is consumed at build time by a llama.cpp tree and takes its heavy math from csrc/ by include; it has no runtime dependency on the Python structures layer. It now lives beside the other native surfaces (exec/, runtime/, cpp/) instead of under flash_rt/structures/adapters/, ahead of the structures layer moving to its own repository. Path-only change: the csrc includes, the repo-root resolution in the qualification and tooling scripts, and the binding comments follow the move. Offline qualification gates (binding manifest, structure pins, binding-header freshness) pass unchanged. --- .../structures/bindings/jetson_pi_edge_pi05.yaml | 2 +- .../bindings/llamacpp_qwen36_35b_sm120.yaml | 2 +- .../adapters => hosts}/ggml/DEVELOPMENT.md | 0 .../structures/adapters => hosts}/ggml/README.md | 2 +- .../structures/adapters => hosts}/ggml/TESTING.md | 0 .../structures/adapters => hosts}/ggml/USAGE.md | 5 +++++ .../adapters => hosts}/ggml/fa4_aot/README.md | 0 .../ggml/fa4_aot/export_fa4_siglip.py | 2 +- .../ggml/fa4_aot/fa4_prefill_fwd.h | 0 .../ggml/fa4_aot/fa4_prefill_fwd.o | Bin .../ggml/fa4_aot/fa4_siglip_fwd.h | 0 .../ggml/fa4_aot/fa4_siglip_fwd.o | Bin .../structures/adapters => hosts}/ggml/fr_ada.cu | 0 .../ggml/fr_binding_qwen36_35b_sm120.h | 0 .../adapters => hosts}/ggml/fr_decode_attn.cu | 0 .../adapters => hosts}/ggml/fr_dispatch.cu | 0 .../adapters => hosts}/ggml/fr_fa4_shims.c | 0 .../adapters => hosts}/ggml/fr_fa4_vit.cu | 0 .../adapters => hosts}/ggml/fr_gemm_f32out.cu | 0 .../structures/adapters => hosts}/ggml/fr_ggml.cuh | 0 .../structures/adapters => hosts}/ggml/fr_kernels.h | 0 .../adapters => hosts}/ggml/fr_qkv_post.cu | 0 .../adapters => hosts}/ggml/fr_quant_act.cu | 0 .../structures/adapters => hosts}/ggml/fr_repack.cu | 0 .../adapters => hosts}/ggml/fr_win_qwen36_sm120.cu | 4 ++-- .../ggml/qualification/goldens/base.png | Bin .../qualification/goldens/pi05_thor_action.json | 0 .../ggml/qualification/goldens/wrist.png | Bin .../adapters => hosts}/ggml/qualification/pins.yaml | 0 .../ggml/qualification/pins_qwen36_sm120.yaml | 0 .../ggml/qualification/run_qualification.py | 2 +- .../qualification/run_qualification_qwen36_sm120.py | 2 +- .../ggml/tools/gen_binding_header.py | 2 +- .../ggml/tools/splice_nvfp4_head.py | 0 34 files changed, 14 insertions(+), 9 deletions(-) rename {flash_rt/structures/adapters => hosts}/ggml/DEVELOPMENT.md (100%) rename {flash_rt/structures/adapters => hosts}/ggml/README.md (98%) rename {flash_rt/structures/adapters => hosts}/ggml/TESTING.md (100%) rename {flash_rt/structures/adapters => hosts}/ggml/USAGE.md (96%) rename {flash_rt/structures/adapters => hosts}/ggml/fa4_aot/README.md (100%) rename {flash_rt/structures/adapters => hosts}/ggml/fa4_aot/export_fa4_siglip.py (99%) rename {flash_rt/structures/adapters => hosts}/ggml/fa4_aot/fa4_prefill_fwd.h (100%) rename {flash_rt/structures/adapters => hosts}/ggml/fa4_aot/fa4_prefill_fwd.o (100%) rename {flash_rt/structures/adapters => hosts}/ggml/fa4_aot/fa4_siglip_fwd.h (100%) rename {flash_rt/structures/adapters => hosts}/ggml/fa4_aot/fa4_siglip_fwd.o (100%) rename {flash_rt/structures/adapters => hosts}/ggml/fr_ada.cu (100%) rename {flash_rt/structures/adapters => hosts}/ggml/fr_binding_qwen36_35b_sm120.h (100%) rename {flash_rt/structures/adapters => hosts}/ggml/fr_decode_attn.cu (100%) rename {flash_rt/structures/adapters => hosts}/ggml/fr_dispatch.cu (100%) rename {flash_rt/structures/adapters => hosts}/ggml/fr_fa4_shims.c (100%) rename {flash_rt/structures/adapters => hosts}/ggml/fr_fa4_vit.cu (100%) rename {flash_rt/structures/adapters => hosts}/ggml/fr_gemm_f32out.cu (100%) rename {flash_rt/structures/adapters => hosts}/ggml/fr_ggml.cuh (100%) rename {flash_rt/structures/adapters => hosts}/ggml/fr_kernels.h (100%) rename {flash_rt/structures/adapters => hosts}/ggml/fr_qkv_post.cu (100%) rename {flash_rt/structures/adapters => hosts}/ggml/fr_quant_act.cu (100%) rename {flash_rt/structures/adapters => hosts}/ggml/fr_repack.cu (100%) rename {flash_rt/structures/adapters => hosts}/ggml/fr_win_qwen36_sm120.cu (99%) rename {flash_rt/structures/adapters => hosts}/ggml/qualification/goldens/base.png (100%) rename {flash_rt/structures/adapters => hosts}/ggml/qualification/goldens/pi05_thor_action.json (100%) rename {flash_rt/structures/adapters => hosts}/ggml/qualification/goldens/wrist.png (100%) rename {flash_rt/structures/adapters => hosts}/ggml/qualification/pins.yaml (100%) rename {flash_rt/structures/adapters => hosts}/ggml/qualification/pins_qwen36_sm120.yaml (100%) rename {flash_rt/structures/adapters => hosts}/ggml/qualification/run_qualification.py (99%) rename {flash_rt/structures/adapters => hosts}/ggml/qualification/run_qualification_qwen36_sm120.py (99%) rename {flash_rt/structures/adapters => hosts}/ggml/tools/gen_binding_header.py (99%) rename {flash_rt/structures/adapters => hosts}/ggml/tools/splice_nvfp4_head.py (100%) diff --git a/flash_rt/structures/bindings/jetson_pi_edge_pi05.yaml b/flash_rt/structures/bindings/jetson_pi_edge_pi05.yaml index ffa8b90d..f2e19fec 100644 --- a/flash_rt/structures/bindings/jetson_pi_edge_pi05.yaml +++ b/flash_rt/structures/bindings/jetson_pi_edge_pi05.yaml @@ -3,7 +3,7 @@ structure: vla_tick_pipeline # Host: ggml/llama.cpp server (PKU-SEC-Lab/Jetson-PI-Edge fork) running pi0.5 # on Jetson AGX Thor (SM110) with the native ggml adapter -# (flash_rt/structures/adapters/ggml/). Structure regions execute as fused +# (hosts/ggml/). Structure regions execute as fused # subgraph windows matched inside ggml-cuda's graph evaluation; the mapping # below is window <-> catalog structure. diff --git a/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml b/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml index 4fb7c6f0..51c64861 100644 --- a/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml +++ b/flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml @@ -3,7 +3,7 @@ structure: autoregressive_decode_pipeline # Host: stock llama.cpp (upstream master lineage) running Qwen3.6-35B-A3B # UD-Q4_K_M on RTX 5090 (SM120) with the native ggml adapter -# (flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu). Structure +# (hosts/ggml/fr_win_qwen36_sm120.cu). Structure # regions execute as fused subgraph windows matched inside ggml-cuda's # graph evaluation. Two quality tiers share one binary: the safe tier keeps # the target head and all matched math bit-comparable to stock (24-chunk diff --git a/flash_rt/structures/adapters/ggml/DEVELOPMENT.md b/hosts/ggml/DEVELOPMENT.md similarity index 100% rename from flash_rt/structures/adapters/ggml/DEVELOPMENT.md rename to hosts/ggml/DEVELOPMENT.md diff --git a/flash_rt/structures/adapters/ggml/README.md b/hosts/ggml/README.md similarity index 98% rename from flash_rt/structures/adapters/ggml/README.md rename to hosts/ggml/README.md index 1502c562..d95b9525 100644 --- a/flash_rt/structures/adapters/ggml/README.md +++ b/hosts/ggml/README.md @@ -49,7 +49,7 @@ through pattern-matched subgraph windows. Heavy math is single-source: - `fa4_aot/` — AOT FlashAttention-4 modules (vision and prefill shapes) plus their regeneration script and provenance notes. - `qualification/` — the release gates (see TESTING.md). -- `../../bindings/jetson_pi_edge_pi05.yaml` — the pipeline binding that +- `flash_rt/structures/bindings/jetson_pi_edge_pi05.yaml` — the pipeline binding that maps the host's hot path onto catalog structures under the complete-hot-path contract. diff --git a/flash_rt/structures/adapters/ggml/TESTING.md b/hosts/ggml/TESTING.md similarity index 100% rename from flash_rt/structures/adapters/ggml/TESTING.md rename to hosts/ggml/TESTING.md diff --git a/flash_rt/structures/adapters/ggml/USAGE.md b/hosts/ggml/USAGE.md similarity index 96% rename from flash_rt/structures/adapters/ggml/USAGE.md rename to hosts/ggml/USAGE.md index f836386e..8bbe3cf9 100644 --- a/flash_rt/structures/adapters/ggml/USAGE.md +++ b/hosts/ggml/USAGE.md @@ -2,6 +2,11 @@ ## Building a host against the adapter +The adapter sources live at `hosts/ggml/` in the FlashRT checkout (they +were under `flash_rt/structures/adapters/ggml/` before the structures +layer moved to its own repository). A host CMake that lists the +translation units by path must use `hosts/ggml/`. + The reference host is the Jetson-PI-Edge llama.cpp tree, which carries the integration side (CMake wiring, fuse-hook call sites, pi0 graph changes) on its FlashRT branch and consumes this repository as a submodule at diff --git a/flash_rt/structures/adapters/ggml/fa4_aot/README.md b/hosts/ggml/fa4_aot/README.md similarity index 100% rename from flash_rt/structures/adapters/ggml/fa4_aot/README.md rename to hosts/ggml/fa4_aot/README.md diff --git a/flash_rt/structures/adapters/ggml/fa4_aot/export_fa4_siglip.py b/hosts/ggml/fa4_aot/export_fa4_siglip.py similarity index 99% rename from flash_rt/structures/adapters/ggml/fa4_aot/export_fa4_siglip.py rename to hosts/ggml/fa4_aot/export_fa4_siglip.py index 0bb1e3e0..0818bd45 100644 --- a/flash_rt/structures/adapters/ggml/fa4_aot/export_fa4_siglip.py +++ b/hosts/ggml/fa4_aot/export_fa4_siglip.py @@ -13,7 +13,7 @@ from pathlib import Path _HERE = Path(__file__).resolve().parent -_REPO = _HERE.parents[4] +_REPO = _HERE.parents[2] sys.path.insert(0, str(_REPO)) import torch # noqa: E402 diff --git a/flash_rt/structures/adapters/ggml/fa4_aot/fa4_prefill_fwd.h b/hosts/ggml/fa4_aot/fa4_prefill_fwd.h similarity index 100% rename from flash_rt/structures/adapters/ggml/fa4_aot/fa4_prefill_fwd.h rename to hosts/ggml/fa4_aot/fa4_prefill_fwd.h diff --git a/flash_rt/structures/adapters/ggml/fa4_aot/fa4_prefill_fwd.o b/hosts/ggml/fa4_aot/fa4_prefill_fwd.o similarity index 100% rename from flash_rt/structures/adapters/ggml/fa4_aot/fa4_prefill_fwd.o rename to hosts/ggml/fa4_aot/fa4_prefill_fwd.o diff --git a/flash_rt/structures/adapters/ggml/fa4_aot/fa4_siglip_fwd.h b/hosts/ggml/fa4_aot/fa4_siglip_fwd.h similarity index 100% rename from flash_rt/structures/adapters/ggml/fa4_aot/fa4_siglip_fwd.h rename to hosts/ggml/fa4_aot/fa4_siglip_fwd.h diff --git a/flash_rt/structures/adapters/ggml/fa4_aot/fa4_siglip_fwd.o b/hosts/ggml/fa4_aot/fa4_siglip_fwd.o similarity index 100% rename from flash_rt/structures/adapters/ggml/fa4_aot/fa4_siglip_fwd.o rename to hosts/ggml/fa4_aot/fa4_siglip_fwd.o diff --git a/flash_rt/structures/adapters/ggml/fr_ada.cu b/hosts/ggml/fr_ada.cu similarity index 100% rename from flash_rt/structures/adapters/ggml/fr_ada.cu rename to hosts/ggml/fr_ada.cu diff --git a/flash_rt/structures/adapters/ggml/fr_binding_qwen36_35b_sm120.h b/hosts/ggml/fr_binding_qwen36_35b_sm120.h similarity index 100% rename from flash_rt/structures/adapters/ggml/fr_binding_qwen36_35b_sm120.h rename to hosts/ggml/fr_binding_qwen36_35b_sm120.h diff --git a/flash_rt/structures/adapters/ggml/fr_decode_attn.cu b/hosts/ggml/fr_decode_attn.cu similarity index 100% rename from flash_rt/structures/adapters/ggml/fr_decode_attn.cu rename to hosts/ggml/fr_decode_attn.cu diff --git a/flash_rt/structures/adapters/ggml/fr_dispatch.cu b/hosts/ggml/fr_dispatch.cu similarity index 100% rename from flash_rt/structures/adapters/ggml/fr_dispatch.cu rename to hosts/ggml/fr_dispatch.cu diff --git a/flash_rt/structures/adapters/ggml/fr_fa4_shims.c b/hosts/ggml/fr_fa4_shims.c similarity index 100% rename from flash_rt/structures/adapters/ggml/fr_fa4_shims.c rename to hosts/ggml/fr_fa4_shims.c diff --git a/flash_rt/structures/adapters/ggml/fr_fa4_vit.cu b/hosts/ggml/fr_fa4_vit.cu similarity index 100% rename from flash_rt/structures/adapters/ggml/fr_fa4_vit.cu rename to hosts/ggml/fr_fa4_vit.cu diff --git a/flash_rt/structures/adapters/ggml/fr_gemm_f32out.cu b/hosts/ggml/fr_gemm_f32out.cu similarity index 100% rename from flash_rt/structures/adapters/ggml/fr_gemm_f32out.cu rename to hosts/ggml/fr_gemm_f32out.cu diff --git a/flash_rt/structures/adapters/ggml/fr_ggml.cuh b/hosts/ggml/fr_ggml.cuh similarity index 100% rename from flash_rt/structures/adapters/ggml/fr_ggml.cuh rename to hosts/ggml/fr_ggml.cuh diff --git a/flash_rt/structures/adapters/ggml/fr_kernels.h b/hosts/ggml/fr_kernels.h similarity index 100% rename from flash_rt/structures/adapters/ggml/fr_kernels.h rename to hosts/ggml/fr_kernels.h diff --git a/flash_rt/structures/adapters/ggml/fr_qkv_post.cu b/hosts/ggml/fr_qkv_post.cu similarity index 100% rename from flash_rt/structures/adapters/ggml/fr_qkv_post.cu rename to hosts/ggml/fr_qkv_post.cu diff --git a/flash_rt/structures/adapters/ggml/fr_quant_act.cu b/hosts/ggml/fr_quant_act.cu similarity index 100% rename from flash_rt/structures/adapters/ggml/fr_quant_act.cu rename to hosts/ggml/fr_quant_act.cu diff --git a/flash_rt/structures/adapters/ggml/fr_repack.cu b/hosts/ggml/fr_repack.cu similarity index 100% rename from flash_rt/structures/adapters/ggml/fr_repack.cu rename to hosts/ggml/fr_repack.cu diff --git a/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu b/hosts/ggml/fr_win_qwen36_sm120.cu similarity index 99% rename from flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu rename to hosts/ggml/fr_win_qwen36_sm120.cu index 83fa517a..f6c2f4b6 100644 --- a/flash_rt/structures/adapters/ggml/fr_win_qwen36_sm120.cu +++ b/hosts/ggml/fr_win_qwen36_sm120.cu @@ -25,8 +25,8 @@ // Heavy math comes from csrc (single source; the adapter only translates): // the M-rows activation quantizer and the warp-split-K W4A4 GEMV. -#include "../../../../csrc/quantize/f32_act_to_nvfp4_swizzled_mrows_sm120.cuh" -#include "../../../../csrc/kernels/fp4_w4a4_mma_warpsplit_mrows_f32out_sm120.cuh" +#include "../../csrc/quantize/f32_act_to_nvfp4_swizzled_mrows_sm120.cuh" +#include "../../csrc/kernels/fp4_w4a4_mma_warpsplit_mrows_f32out_sm120.cuh" // Model-specific constants come from the binding (single source: // flash_rt/structures/bindings/llamacpp_qwen36_35b_sm120.yaml); regenerate diff --git a/flash_rt/structures/adapters/ggml/qualification/goldens/base.png b/hosts/ggml/qualification/goldens/base.png similarity index 100% rename from flash_rt/structures/adapters/ggml/qualification/goldens/base.png rename to hosts/ggml/qualification/goldens/base.png diff --git a/flash_rt/structures/adapters/ggml/qualification/goldens/pi05_thor_action.json b/hosts/ggml/qualification/goldens/pi05_thor_action.json similarity index 100% rename from flash_rt/structures/adapters/ggml/qualification/goldens/pi05_thor_action.json rename to hosts/ggml/qualification/goldens/pi05_thor_action.json diff --git a/flash_rt/structures/adapters/ggml/qualification/goldens/wrist.png b/hosts/ggml/qualification/goldens/wrist.png similarity index 100% rename from flash_rt/structures/adapters/ggml/qualification/goldens/wrist.png rename to hosts/ggml/qualification/goldens/wrist.png diff --git a/flash_rt/structures/adapters/ggml/qualification/pins.yaml b/hosts/ggml/qualification/pins.yaml similarity index 100% rename from flash_rt/structures/adapters/ggml/qualification/pins.yaml rename to hosts/ggml/qualification/pins.yaml diff --git a/flash_rt/structures/adapters/ggml/qualification/pins_qwen36_sm120.yaml b/hosts/ggml/qualification/pins_qwen36_sm120.yaml similarity index 100% rename from flash_rt/structures/adapters/ggml/qualification/pins_qwen36_sm120.yaml rename to hosts/ggml/qualification/pins_qwen36_sm120.yaml diff --git a/flash_rt/structures/adapters/ggml/qualification/run_qualification.py b/hosts/ggml/qualification/run_qualification.py similarity index 99% rename from flash_rt/structures/adapters/ggml/qualification/run_qualification.py rename to hosts/ggml/qualification/run_qualification.py index c1c61f8e..576a4382 100644 --- a/flash_rt/structures/adapters/ggml/qualification/run_qualification.py +++ b/hosts/ggml/qualification/run_qualification.py @@ -34,7 +34,7 @@ import yaml _HERE = pathlib.Path(__file__).resolve().parent -_REPO = _HERE.parents[4] +_REPO = _HERE.parents[2] if str(_REPO) not in sys.path: sys.path.insert(0, str(_REPO)) diff --git a/flash_rt/structures/adapters/ggml/qualification/run_qualification_qwen36_sm120.py b/hosts/ggml/qualification/run_qualification_qwen36_sm120.py similarity index 99% rename from flash_rt/structures/adapters/ggml/qualification/run_qualification_qwen36_sm120.py rename to hosts/ggml/qualification/run_qualification_qwen36_sm120.py index ed7031e3..22f6c5b8 100644 --- a/flash_rt/structures/adapters/ggml/qualification/run_qualification_qwen36_sm120.py +++ b/hosts/ggml/qualification/run_qualification_qwen36_sm120.py @@ -36,7 +36,7 @@ import yaml _HERE = pathlib.Path(__file__).resolve().parent -_REPO = _HERE.parents[4] +_REPO = _HERE.parents[2] if str(_REPO) not in sys.path: sys.path.insert(0, str(_REPO)) diff --git a/flash_rt/structures/adapters/ggml/tools/gen_binding_header.py b/hosts/ggml/tools/gen_binding_header.py similarity index 99% rename from flash_rt/structures/adapters/ggml/tools/gen_binding_header.py rename to hosts/ggml/tools/gen_binding_header.py index acf5ff97..e138bb1b 100644 --- a/flash_rt/structures/adapters/ggml/tools/gen_binding_header.py +++ b/hosts/ggml/tools/gen_binding_header.py @@ -17,7 +17,7 @@ import yaml _HERE = pathlib.Path(__file__).resolve().parent -_REPO = _HERE.parents[4] +_REPO = _HERE.parents[2] _BINDINGS = _REPO / "flash_rt" / "structures" / "bindings" diff --git a/flash_rt/structures/adapters/ggml/tools/splice_nvfp4_head.py b/hosts/ggml/tools/splice_nvfp4_head.py similarity index 100% rename from flash_rt/structures/adapters/ggml/tools/splice_nvfp4_head.py rename to hosts/ggml/tools/splice_nvfp4_head.py From 2e202d4287d1be3356f8d866f9d30b19a24335e7 Mon Sep 17 00:00:00 2001 From: LiangSu8899 <7thuniversels@gmail.com> Date: Fri, 11 Sep 2026 14:20:29 -0400 Subject: [PATCH 32/32] tests: the pipeline contract follows autoregressive_decode_pipeline v3 v3 added the optional spec_draft stage; the shared-schedule-family test still pinned version 2 and the v2 optional-stage set. --- tests/test_structures_pipeline_contracts.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/test_structures_pipeline_contracts.py b/tests/test_structures_pipeline_contracts.py index c1043155..13adf41d 100644 --- a/tests/test_structures_pipeline_contracts.py +++ b/tests/test_structures_pipeline_contracts.py @@ -89,10 +89,10 @@ def test_pipeline_catalogs_declare_shared_schedule_families(): assert autoregressive.kind == "stage_pipeline" assert autoregressive.family == "autoregressive_decode" - assert autoregressive.version == 2 + assert autoregressive.version == 3 assert _stage_sets(autoregressive) == ( {"prefill", "decode", "token_select"}, - {"input_prepare", "modality_encode"}, + {"input_prepare", "modality_encode", "spec_draft"}, ) assert vla.kind == "stage_pipeline"