Repository navigation
Conversation
sstamenk
marked this pull request as draft
November 16, 2026 15:46
sstamenk
force-pushed
the
feature/rocm-wmma-gemm-4bit-pr
branch
2 times, most recently
from
November 16, 2026 16:28
a8ca462 to
9d6b3f7
Compare
…nd RDNA4
Adds csrc/gemm_4bit_wmma.cu, a WMMA-based fused 4-bit (NF4/FP4) GEMM for
Radeon GPUs, and dispatches to it from the existing ROCm SIMT/dequant path.
It extends the fused path's advantage over dequant+F.linear from a few rows to
several hundred, mirroring the CUDA SIMT > MMA > dequant+BLAS dispatch.
Kernel
- gfx11 (RDNA3/RDNA3.5) v_wmma_f32_16x16x16_{f16,bf16}: a wave32 WMMA runs as
two independent 8x16x16 half-wave products, so each half-wave takes its own
32-K slice (summed with permlanex16); 8 real rows per WMMA.
- gfx12 (RDNA4) _w32_gfx12 WMMAs: one full 16x16x16 product; each half-wave
supplies its own K slice, summed inside the WMMA.
- Weights decoded through a 256-entry LDS byte -> centroid-pair table; absmax
applied in fp32 after the WMMAs (gfx11 fp16 large tiles pre-scale the B
operand with packed half math instead; nested statistics keep the fp32 path).
A is double-buffered in LDS, split-K across warps with a deterministic LDS
reduction, per-block K-chunk rotation against DRAM channel camping, and
row-tiles-first block order so row tiles of the same columns share B in L2.
- fp32 is emulated with 3-way split bf16 operands (6 WMMAs per tile) and stays
fp32-accurate (~3-5e-6 relative error vs a float64 reference).
Dispatch (K % 64 == 0, blocksize >= 32)
- Tile rules per target (gfx110x, gfx1103, gfx1151, gfx1201), fitted from tile
sweeps over 40 LLM linear-layer shapes at a pinned GPU performance level.
- SIMT vs WMMA in csrc/gemm_4bit.cu: bf16/fp16 WMMA from M = 1 (gfx11/gfx115x)
or M = 2 (gfx1201); fp32 picks SIMT or WMMA by M and N.
- Fused vs dequant+F.linear in _gemm_4bit_use_custom_rocm: gfx110x bf16/fp16
through M = 384 and fp32 through 128; gfx1103 through 384 for all dtypes;
gfx1151 through 512; gfx1201 bf16/fp16 through 384 and fp32 through 512.
gfx1101/gfx1102, gfx1150/gfx1152/gfx1153 and gfx1200 use the rules of their
family's measured chip (gfx1100, gfx1151, gfx1201): same WMMA instructions,
not measured.
Performance (speedup of the total time over 40 LLM linear-layer shapes vs the
previous dispatch, SIMT for M <= 8 and dequant+F.linear above; NF4 blocksize 64,
weights streamed from DRAM, pinned clocks; bf16/fp16 at M = 8 / 16 / 128 / 256):
- Radeon Pro W7900 (gfx1100): 2.8-2.9x / 5.2-5.8x / 1.6x / 1.2x
- Radeon AI PRO R9700 (gfx1201): 2.7x / 5.9x / 2.0x / 1.3x
- Radeon 8060S (gfx1151): 2.6-3.0x / 6.5-7.1x / 2.8-3.0x / 2.0-2.2x
- Radeon 760M (gfx1103): 2.3-2.5x / 8.6x / 2.7-2.8x / 1.6-1.7x
fp32 gains 1.5-2.1x at M = 8 and 4.4-9.2x at M = 16. At M = 1-2 the gain
is small and some shapes lose (up to 16% at M = 1 on gfx11; on gfx1201 at
M = 2, 6-7 of 40 shapes by up to 0.6x).
Within the caps the default dispatch's total time is within ~2% of a
per-shape oracle over SIMT, WMMA and dequant+F.linear at most M (largest gap
6.3%, gfx1151 fp32 M = 3).
Tests: extra odd shapes for the 4-bit GEMM and an fp32 precision test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sstamenk
force-pushed
the
feature/rocm-wmma-gemm-4bit-pr
branch
from
November 16, 2026 16:33
9d6b3f7 to
d24b85d
Compare
…es.cuh The tile selection was ~180 lines of nested if/else per target and dtype in launch_gemm_4bit_wmma_arch, with conditions written several ways, and the SIMT vs WMMA cutoffs were inline in gemm_4bit(). Both are now data in csrc/gemm_4bit_wmma_rules.cuh, next to the tuning-target constants: - wmma_preferred(target, fp32, M, N): SIMT or WMMA (bf16/fp16 rule plus the per-target fp32 cutoff tables). - Tile rule tables per target and dtype: a list of M buckets, each with up to three leaves (N < x, K < x or K/N < a/b) and a default tile, in the shape the tile sweeps are fitted in. wmma_pick() does the lookup; the launcher collects each table's distinct tiles at compile time and dispatches to their instantiations, so exactly the same kernels are built as before. No behaviour change. Host programs comparing the previous code with the new lookups agree on all 14.0e9 (target, dtype, M, N, K) tile choices (M = 1-1024, N and K in multiples of 64 up to 65536 plus every threshold +-1) and all 21.2e6 (target, dtype, M, N) SIMT/WMMA choices (N in multiples of 32 plus every cutoff +-1); the build contains the same 470 WMMA kernel symbols. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sstamenk
force-pushed
the
feature/rocm-wmma-gemm-4bit-pr
branch
from
November 16, 2026 17:35
68332d3 to
002dd31
Compare
…gets, review fixes - fp32: an Inf activation became NaN. split_bf16 kept Inf/NaN in the leading part, which is also multiplied by B's lower parts, and those are zero for every centroid exact in bf16 (Inf * 0). Non-finite values now go to the last part, which only meets B's leading part, so Inf/NaN propagate as in fp32 math. Finite results are unchanged. New test: test_matmul_4bit_fp32_special_values (fails on the WMMA path before this change). - Builds for gfx11-generic / gfx12-generic emitted no WMMA kernel bodies but the dispatch still selected WMMA by device name, so the first launch aborted on the missing symbol. The device pass now treats __gfx11_generic__ / __gfx12_generic__ like their family (one BNB_WMMA_DEVICE_ARCH instead of BNB_WMMA_RDNA3/4, which also drops most new -Wexpansion-to-defined warnings). - Document the input ranges of the WMMA path: absmax is applied to the fp32 sum of A x centroid over a quantization group, so |A| must stay below ~FLT_MAX / blocksize (fp32/bf16 inputs only); the three-part split-bf16 keeps fp32 accuracy for |A| >= ~1e-33. - Python: check K % 64 once; note that the M limits were calibrated with blocksize 64 and weights streamed from DRAM. - Named tuning targets instead of literal 11/12; fix stale test comments (the fp32 precision bound and its metric, the gfx12 wide-N tile). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On Radeon GPUs, the fused 4-bit GEMM (
gemm_4bit) is a SIMT kernel that only wins up to M = 8 rows. Above that, ROCm falls back to dequantizing the whole weight matrix and callingF.linear. That is slow for small and medium batches: the dequantized weights are written to and read back from DRAM, and the SIMT kernel's cost grows linearly with M.This PR adds a WMMA (matrix-core) kernel for the fused 4-bit GEMM on gfx11 (RDNA3/RDNA3.5) and gfx12 (RDNA4), and dispatches to it from the existing ROCm path. The fused path now stays faster than dequant +
F.linearup to M = 384-512 rows instead of 8. This mirrors the CUDA SIMT → MMA → dequant + BLAS split.Kernel (
csrc/gemm_4bit_wmma.cu)v_wmma_f32_16x16x16_{f16,bf16}): a wave32 WMMA runs as two independent 8x16x16 half-wave products. Each half-wave takes its own 32-element K slice and the halves are summed withpermlanex16, giving 8 real rows per WMMA without padding M to 16.*_w32_gfx12WMMAs): one full 16x16x16 product; each half-wave supplies its own K slice, summed inside the WMMA.Dispatch
Requires
K % 64 == 0andblocksize >= 32; anything else keeps the current path.SIMT vs WMMA (
wmma_preferred(), called fromcsrc/gemm_4bit.cu): bf16/fp16 use WMMA from M = 1 on gfx11/gfx115x and from M = 2 on gfx120x. fp32 picks SIMT or WMMA from a small per-target table by M and N.Tile selection: per-target rules for gfx110x/gfx1103, gfx115x and gfx120x as tables (per M bucket, up to three leaves on N, K or K/N, then a default tile), fitted from tile sweeps over 40 LLM linear-layer shapes at pinned GPU clocks. The launcher instantiates exactly the tiles the tables list.
Where the rules live: the SIMT-vs-WMMA rules, the tile tables and the tuning-target constants are in
csrc/gemm_4bit_wmma_rules.cuh. The second commit only moves them there from nested if/else in the launchers; the third applies review fixes (Inf handling in fp32, generic-target builds, documented input ranges). Host programs comparing old and new code give identical choices on all 14.0e9 (target, dtype, M, N, K) tile picks and 21.2e6 (target, dtype, M, N) SIMT/WMMA picks checked, and the build has the same 470 WMMA kernels.Fused vs dequant +
F.linear(_gemm_4bit_use_custom_rocm), as the largest M that stays on the fused path:Measured on gfx1100, gfx1103, gfx1151 and gfx1201. gfx1101/gfx1102 (RX 7800 XT/7700 XT/7600), gfx1150/gfx1152/gfx1153 (Strix Point, Krackan) and gfx1200 (RX 9060 XT) share the WMMA instructions of their family and use the rules of its measured chip (gfx1100, gfx1151, gfx1201) without having been measured. They have fewer CUs than those chips, so the tiles and M limits may be off for them, but the WMMA path should still be well ahead of SIMT and dequant +
F.linearfor M from about 9 to 128. Builds forgfx11-generic/gfx12-genericget the same kernels.The M limits were calibrated with blocksize 64 (the default) and weights streamed from DRAM. With blocksize 32 and weights already in cache, gfx110x at M ≈ 384 can be up to ~20% slower than dequant +
F.linear.CUDA, CDNA (gfx9) and CPU paths are unchanged.
Performance
Speedup of the default dispatch over
main(SIMT for M ≤ 8, dequant +F.linearabove). Each cell is the total time over 40 LLM linear-layer shapes (Llama 3 8B/70B, Qwen, Mistral, Gemma, ...). Setup: NF4, blocksize 64, weights streamed from DRAM (no L2 reuse between iterations), pinned clocks (amd-smi set -l HIGHorpower_dpm_force_performance_level=high).At M = 1-2 the gain is small and some shapes lose. On gfx11, up to ~10-15% of shapes are slower than SIMT at M = 1, by up to 16%. On gfx1201 at M = 2, 6-7 of the 40 shapes are slower, by up to 0.6x; WMMA starts at M = 2 there because it wins on total time. From M = 3-4 on, bf16/fp16 are faster on every shape. Within the M limits, the default dispatch's total time is within ~2% of a per-shape oracle over SIMT, WMMA and dequant +
F.linearat most M (largest gap 6.3%, gfx1151 fp32 at M = 3).Kernels vs the bandwidth limit
Median over the 40 shapes, with each kernel forced: latency (top) and achieved bandwidth (bottom) against the time to read the GEMM's bytes once at peak DRAM bandwidth (dashed).
Radeon Pro W7900 (gfx1100)

Radeon AI PRO R9700 (gfx1201)

Radeon 8060S (gfx1151)

Radeon 760M (gfx1103)

Tests
test_matmul_4bit_fp32_precision(fp32 against a float64 reference);test_matmul_4bit_fp32_special_values(Inf/NaN propagation on the SIMT and WMMA paths).pytest tests/test_functional.py tests/test_linear4bit.py -k "(4bit or Linear4bit or gemv) and not fsdp"on the head commit (test_fsdp_state_dict_save_4bitis skipped because it fails onmaintoo in this venv: its torchrun subprocess cannot import bitsandbytes):gfx11-generic;gfx12-genericbuild: 2422 passed, 23 failed (the WMMA kernels run)test_gemv_4bitcase (see below).test_gemv_4bitcases fail on gfx1100 and gfx1201, identically onmain(22 on gfx1151, wheremainwas not compared). The test comparesgemv_4bitto dequant +F.linear, and on these GPUs the BLAS fp32 result is the less accurate side. Against float64,F.linear(hipBLASLt on gfx11, rocBLAS/hipBLASLt on gfx1201) has ~4.8e-8 mean error andgemv_4bithas ~1.0e-8. WithTORCH_BLAS_PREFER_HIPBLASLT=0, all 64 fp32 cases pass on gfx1100. See also Fix fp32 4-bit GEMV test tolerance for ROCm/gfx1200 #2113.pre-commit run --all-filespasses.Full plots and data
The benchmark harness, raw measurements and all plots are on a separate branch that is not part of this PR:
benchmark/rocm-wmma-gemm-4bit(this PR plus two commits).bench.py, the tile sweeps, the rule and cutoff fitting scripts, and the CSVs the rules and plots come from.-DBNB_WMMA_DEV=ONbuild option with environment overrides for that (BNB_GEMM4BIT_KERNEL,BNB_WMMA_CFG,BNB_WMMA_ROW_TILES_FIRST,BNB_WMMA_PRESCALE). Apply it withgit applyon top of this PR to re-tune the rules or calibrate another GPU.🤖 Generated with Claude Code