Uploaded August 2026 | Updated September 2026, 2 weeks ago
[EuroPython 2026 - S2 on 2026-07-15]
🎤 *The hardest test suite I ever built - a pytest case study by Grzegorz Kocjan*
🔗 https://ep2026.europython.eu/session/the-hardest-test-suite-i-ever-built-a-pytest-case-study
📝 Abstract:
For years, this real-time video system had no tests. Every change produced unpredictable side effects. Accuracy drifted. Production incidents and escalations followed. The only “verification” was manual inspection and hope.
When I joined the project, this was the reality - and building a proper integration test suite became my first priority.
In this talk, I’ll share how I designed and evolved the hardest integration test suite of my career using pytest - and kept it readable.
The system processed live streams in production. It was non-deterministic. Individual detections were only 80–90% accurate. For testing, we replayed recorded scenarios to make system behavior observable and comparable across runs. But binary assertions were not enough. A single failed event did not mean the whole system was broken - but we needed a way to measure when it actually was.
Instead of writing one massive test, I built a layered architecture:
- dual parametrization - recording scope and event scope
- orchestration in fixtures - assertions in tiny, single-purpose tests
- statistics collection during execution
- end-of-run aggregation that summarizes system accuracy
The result was a suite that could detect regressions in model changes and produce reproducible evidence - HTML reports, structured dumps, and a summary statistics file.
This talk explores how far pytest can be stretched beyond unit tests - into a framework for architecting complex integration systems.
---
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License: creativecommons.org/licenses/by-nc-sa/4.0
[EuroPython 2026 - S2 on 2026-07-15]
🎤 *The hardest test suite I ever built - a pytest case study by Grzegorz Kocjan*
🔗 https://ep2026.europython.eu/session/the-hardest-test-suite-i-ever-built-a-pytest-case-study
📝 Abstract:
For years, this real-time video system had no tests. Every change produced unpredictable side effects. Accuracy drifted. Production incidents and escalations followed. The only “verification” was manual inspection and hope.
When I joined the project, this was the reality - and building a proper integration test suite became my first priority.
In this talk, I’ll share how I designed and evolved the hardest integration test suite of my career using pytest - and kept it readable.
The system processed live streams in production. It was non-deterministic. Individual detections were only 80–90% accurate. For testing, we replayed recorded scenarios to make system behavior observable and comparable across runs. But binary assertions were not enough. A single failed event did not mean the whole system was broken - but we needed a way to measure when it actually was.
Instead of writing one massive test, I built a layered architecture:
- dual parametrization - recording scope and event scope
- orchestration in fixtures - assertions in tiny, single-purpose tests
- statistics collection during execution
- end-of-run aggregation that summarizes system accuracy
The result was a suite that could detect regressions in model changes and produce reproducible evidence - HTML reports, structured dumps, and a summary statistics file.
This talk explores how far pytest can be stretched beyond unit tests - into a framework for architecting complex integration systems.
---
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License: creativecommons.org/licenses/by-nc-sa/4.0
