An energy-efficient FPGA accelerator for transformer inference using layer-wise precision optimization
No Thumbnail Available
Date
2026-09-06
Journal Title
Journal ISSN
Volume Title
Publisher
Elsevier B.V.
Abstract
Abstract
FPGA transformer accelerators routinely fix their multiply-accumulate (MAC) accumulators at 32 bits, echoing a theoretical 25–27-bit worst case for INT8 BERT-base matrix multiplication that has never been checked against actual per-layer behavior. We provide a transformer-specific layer-level study, profiling 215 linear layers across BERT-base, TinyBERT-4L, DistilBERT-base, ALBERT-base-v2, and ELECTRA-small-discriminator over WikiText-2 samples via PyTorch forward-hook INT8 simulation that tracks the true partial-sum accumulation trajectory (not only the final dot-product value) with a 2-bit margin. Across these 215 profiled layers, the maximum observed accumulator requirement is 20 bits under the specified profiling conditions, and 500 SST-2 samples show zero prediction changes, with δ=2 bits the best-performing margin among those tested. Synthesized on the Zynq™ UltraScale+™ MPSoC ZCU104, a 16-bit MAC cuts flip-flops 39% and LUTs 17% versus a 32-bit baseline while exceeding 400 MHz; a per-layer-instance resource lower bound for a 73-layer BERT-base array projects 34.5% fewer flip-flops and 15.4% fewer LUTs.