Uploaded September 2026 | Updated November 2026, 3 days ago
Large Language Models (LLMs) are rapidly transforming hardware design, yet evaluating their true ability to generate correct Register-Transfer Level (RTL) code remains a moving target. Existing static benchmarks suffer heavily from data contamination, as newer models are continuously trained on open-source repositories containing the test questions themselves.
To break this cycle, we introduce NotSoTiny, an automated, living evaluation benchmark that turns the vibrant Tiny Tapeout open-source ecosystem into a continuous source of fresh evaluation data. Following its appearance at DAC 2026, this talk brings NotSoTiny home to the FOSSi community to explore how crowdsourced physical silicon designs and tools can drive the future of AI for chip desing.
NotSoTiny automatically ingests new designs from consecutive Tiny Tapeout shuttles. By leveraging an open-source evaluation pipeline built on formal verification and automated linting, it cleans duplicates and generates complex, structurally rich module-completion tasks. Because the dataset updates dynamically, it provides a contamination-resistant sandbox that reflects actual, human-designed hardware challenges.
In this presentation, we will detail the architecture of the NotSoTiny pipeline, share insights from evaluating state-of-the-art LLMs, and discuss how the open-source community can contribute to accelerate trustworthy AI-driven chip design.
Large Language Models (LLMs) are rapidly transforming hardware design, yet evaluating their true ability to generate correct Register-Transfer Level (RTL) code remains a moving target. Existing static benchmarks suffer heavily from data contamination, as newer models are continuously trained on open-source repositories containing the test questions themselves.
To break this cycle, we introduce NotSoTiny, an automated, living evaluation benchmark that turns the vibrant Tiny Tapeout open-source ecosystem into a continuous source of fresh evaluation data. Following its appearance at DAC 2026, this talk brings NotSoTiny home to the FOSSi community to explore how crowdsourced physical silicon designs and tools can drive the future of AI for chip desing.
NotSoTiny automatically ingests new designs from consecutive Tiny Tapeout shuttles. By leveraging an open-source evaluation pipeline built on formal verification and automated linting, it cleans duplicates and generates complex, structurally rich module-completion tasks. Because the dataset updates dynamically, it provides a contamination-resistant sandbox that reflects actual, human-designed hardware challenges.
In this presentation, we will detail the architecture of the NotSoTiny pipeline, share insights from evaluating state-of-the-art LLMs, and discuss how the open-source community can contribute to accelerate trustworthy AI-driven chip design.
