An energy-efficient FPGA accelerator for transformer inference using layer-wise precision optimization

Simple item page
dc.contributor.authorEiswar S, Darun
dc.contributor.authorN, Pugalbharani
dc.contributor.authorBharatula, Sitadevi
dc.contributor.authorReddy, B. Naresh Kumar.
dc.date.accessioned2026-10-31T07:19:26Z
dc.date.available2026-10-31T07:19:26Z
dc.date.issued2026-09-06
dc.description.abstractAbstract FPGA transformer accelerators routinely fix their multiply-accumulate (MAC) accumulators at 32 bits, echoing a theoretical 25–27-bit worst case for INT8 BERT-base matrix multiplication that has never been checked against actual per-layer behavior. We provide a transformer-specific layer-level study, profiling 215 linear layers across BERT-base, TinyBERT-4L, DistilBERT-base, ALBERT-base-v2, and ELECTRA-small-discriminator over WikiText-2 samples via PyTorch forward-hook INT8 simulation that tracks the true partial-sum accumulation trajectory (not only the final dot-product value) with a 2-bit margin. Across these 215 profiled layers, the maximum observed accumulator requirement is 20 bits under the specified profiling conditions, and 500 SST-2 samples show zero prediction changes, with δ=2 bits the best-performing margin among those tested. Synthesized on the Zynq™ UltraScale+™ MPSoC ZCU104, a 16-bit MAC cuts flip-flops 39% and LUTs 17% versus a 32-bit baseline while exceeding 400 MHz; a per-layer-instance resource lower bound for a 73-layer BERT-base array projects 34.5% fewer flip-flops and 15.4% fewer LUTs.
dc.identifier.issn25901230
dc.identifier.urihttps://doi.org/10.1016/j.rineng.2026.112807
dc.identifier.urihttp://nitt.ndl.gov.in/handle/123456789/379
dc.language.isoen
dc.publisherElsevier B.V.
dc.relation.ispartofseriesResults in Engineering; Vol. 32
dc.titleAn energy-efficient FPGA accelerator for transformer inference using layer-wise precision optimization
dc.typeArticle
Files
Original bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
An energy-efficient FPGA accelerator for transformer inference using layer-wise precision optimization.pdf
Size:
7.15 MB
Format:
Adobe Portable Document Format
Download
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
373 B
Format:
Item-specific license agreed upon to submission
Description:
Download
Collections
Journal Articles