int8bench

A microbenchmark for measuring int8 GEMM throughput (TOPS) and memory bandwidth on x86-64 CPUs with AVX2/AVX-512 support.

Features

• int8 GEMM benchmark: Measures peak TOPS using register-blocked micro-kernels
  • AVX2 (256-bit vpmaddubsw + vpaddd)
  • AVX-512F (512-bit vpmaddubsw + vpaddd)
  • AVX-512 VNNI (512-bit vpdpbusd dot product)
• Memory bandwidth benchmark: Measures read/write/write-nontemporal bandwidth at varying buffer sizes
• Automatic CPU topology detection for optimal thread pinning (spreads across physical cores first, then SMT siblings)
• Cross-platform: Linux only (uses /sys and /proc for topology/RAM info)

Building

make

Requires GCC or Clang with AVX2/AVX-512 support. Uses -march=native by default.

Usage

# Basic int8 GEMM benchmark (auto-detects best ISA, uses all physical cores)
./bench

# Run with specific ISA
./bench --isa avx2
./bench --isa avx512f
./bench --isa avx512_vnni

# Thread control
./bench --cores single      # 1 thread
./bench --cores half        # half physical cores
./bench --cores all         # all physical cores (default)
./bench --cores 8           # 8 threads
./bench -t 4                # explicit thread count

# Buffer size and iterations (per thread)
./bench 524288 500000       # 512 KiB buffer, 500k iterations

# Memory bandwidth benchmark (doubling sizes up to 90% RAM)
./bench --mem
./bench --mem --mem-from 1M

Output

isa:        avx512_vnni
threads:    8 (8 physical cores, 16 online CPUs)
size:       262144 bytes/core (total 2097152)
iters:      1000000
int8 ops:   16777216000000 (mac = 2 ops, 4x weight reuse)
peak TOPS:  42.317
acc sum:    3489217536

• isa: Instruction set used
• threads: Thread count (physical cores, logical CPUs)
• size: Working set per thread / total
• iters: Iterations per thread
• int8 ops: Total multiply-add operations (1 MAC = 2 ops)
• peak TOPS: Best of 5 runs (tera operations/sec)
• acc sum: Checksum to verify computation correctness

Methodology

The GEMM kernel uses a register-blocked approach: 4 weight rows (B0..B3) are kept in vector registers and reused across the entire input buffer. This makes the kernel compute-bound (limited by vpmaddubsw/vpdpbusd throughput) rather than memory-bound.

• AVX2: 32-byte A load × 4 weight rows = 128 MACs per 32 bytes loaded (4× reuse)
• AVX-512: 64-byte A load × 4 weight rows = 256 MACs per 64 bytes loaded (4× reuse)

License

MIT License - see LICENSE file