Updated September 2026, 2 weeks ago
PyData
|
Silva-Enhancing RAG-based apps by constructing and leveraging knowledge graphs with open-source LLMs
š TAP TO UNMUTE
![How to Search Through 800 Billion Records in Real Time [PyCon DE & PyData 2026]](https://i.ytimg.com/vi/t0ZWNh-UXDs/mqdefault.jpg "How to Search Through 800 Billion Records in Real Time [PyCon DE & PyData 2026]
š Recorded at PyCon DE & PyData 2026, 14.04.2026
https://2026.pycon.de/talks/ZYUJH3/
š Watch Mirano Tuk and Filip Bacic reveal the architectural trade-offs and practical Kafka patterns used to transform 800 billion noisy records into a reliable, real-time searchable dataset.
Speakers:
Mirano Tuk, Filip Bacic
Description:
Searching through a threat repository containing one trillion records requires a high-throughput pipeline capable of aggregating data from hundreds of microservices. The primary challenge involves handling a flood of partial updates and duplicate file hashes, which can cause significant latency and system instability in search platforms like Apache Solr. An initial architecture using ClickHouse for daily aggregation resulted in a 36-hour data delay, which was unacceptable for users requiring real-time insights.
The current solution utilizes ScyllaDB as a high-performance key-value store to maintain the source of truth, where records are stored using a primary key composed of the file hash and feature type. When updates occur, file hashes are published to Kafka topics. A Python-based consumer service fetches the complete data set for each hash from ScyllaDB to build a comprehensive document for Solr. To prevent Solr from being overwhelmed by redundant updates, a custom deduplication buffer was implemented using a Python dictionary to track file hashes across batches. This buffer employs a Time-to-Live (TTL) mechanism and an eviction callback; data is only processed and sent to Solr when a hash is evicted from the buffer, ensuring that only the final state of a file is indexed.
To maintain system stability, the service limits the number of messages processed per iteration to prevent Kafka health-check timeouts during backlog clearing. Additionally, a secondary deduplication layer manages Solr commits and re-indexing across daily collections to avoid heap exhaustion and cluster failure. This architecture reduced update latency from 36 hours to under five minutes and decreased the ingestion volume from 100,000 messages per second to fewer than 2,000.
āļø About PyCon DE:
PyCon DE is the leading conference on open-source Python applications in AI and data science. It brings together industry professionals, researchers, AI and data science practitioners, and software engineering communities, providing a unique platform for collaboration, knowledge sharing, and innovation.
The PyCon DE & PyData 2026 conference delivered an exceptional experience, fostering stronger connections within the Python community while showcasing the latest advancements in artificial intelligence and data science. Attendees enjoyed a diverse and engaging program of talks, workshops, and networking opportunities, further establishing the conference as a premier event for Python, AI, and data science enthusiasts across Germany.
PyCon DE 2027 will take place in Heidelberg from 19 to 23 April 2027.
Follow us:
ā¢ Newsletter: https://2027.pycon.de/newsletter/
ā¢ LinkedIn: https://www.linkedin.com/company/pyconde
ā¢ X: https://www.x.com/pyconde
Links:
ā¢ Conference website: http://pycon.de
ā¢ Other sessions: https://2026.pycon.de/talks/
The conference was organized by
ā¢ Python Softwareverband e.V.: http://pysv.org
ā¢ Pioneers Hub gemeinnĆ¼tzige GmbH: http://pioneershub.org
in collaboration with NumFOCUS Inc.: http://numfocus.org
If you enjoyed this session, please like, and subscribe to our channel for more insightful talks and discussions.
Share this video with your network to spread the knowledge!
Hashtags:
#Python #PyConDE #PyData #OpenSource #AI #DataScience #MachineLearning #SoftwareEngineering #LLMs #Community #Sovereignty
Acknowledgements:
Special thanks to all the volunteers and sponsors who made this event possible.
About:
Python Softwareverband e.V.:
PySV is a non-profit that promotes the use and development of Python in Germany through events, education, and advocacy, fostering an open Python community.
Pioneers Hub gemeinnĆ¼tzige GmbH:
is a non-profit fostering innovation in AI and tech by connecting experts and promoting knowledge exchange through events and collaborative initiatives.
NumFOCUS Inc.
supports open-source scientific computing by providing financial and logistical support to key projects like NumPy and Jupyter, promoting sustainable development and collaboration.
www.pydata.org
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.")
![Dynamic Knowledge Graphs [PyCon DE & PyData 2026]](https://i.ytimg.com/vi/tDy8UzEO2cg/mqdefault.jpg "Dynamic Knowledge Graphs [PyCon DE & PyData 2026]
š Recorded at PyCon DE & PyData 2026, 16.04.2026
https://2026.pycon.de/talks/TST9LF/
š Watch Jakob Leander MĆ¼ller explain how to build a production-ready dynamic knowledge graph that overcomes the limitations of traditional RAG to handle real-time data evolution.
Speakers:
Jakob Leander MĆ¼ller
Description:
Dynamic knowledge graphs address the fragmentation of information across disconnected sources, such as Jira tickets, Git repositories, and Swagger documentation. While vector stores offer fast similarity searches, they lack global connectivity and struggle with conflicting entries. Static graph-based RAG systems provide holistic views but are slow to update and often lose traceability to source documents.
The proposed approach utilizes a property graph implemented in FalkyDB to maintain a machine-readable integration pipeline. The schema consists of four node types: document nodes (anchors with content hashes), entity nodes (deduplicated via similarity matching), fact nodes (atomic nuggets of information), and relationship nodes. Document nodes use outgoing edges to support all other nodes, ensuring full traceability and enabling precise deletions of outdated information without collapsing the entire graph.
Insertion involves a multi-stage pipeline using LLMs for structured output via Pydantic classes. To maximize connectivity, the system merges new nodes with existing ones using a hierarchy of matching techniques: exact matches, Jaro-Winkler string similarity, and semantic embedding comparisons, all verified by an LLM. Inference is handled by AI agents that explore the graph using tools for embedding queries, keyword searches, and neighborhood exploration, or by generating Cypher queries directly.
āļø About PyCon DE:
PyCon DE is the leading conference on open-source Python applications in AI and data science. It brings together industry professionals, researchers, AI and data science practitioners, and software engineering communities, providing a unique platform for collaboration, knowledge sharing, and innovation.
The PyCon DE & PyData 2026 conference delivered an exceptional experience, fostering stronger connections within the Python community while showcasing the latest advancements in artificial intelligence and data science. Attendees enjoyed a diverse and engaging program of talks, workshops, and networking opportunities, further establishing the conference as a premier event for Python, AI, and data science enthusiasts across Germany.
PyCon DE 2027 will take place in Heidelberg from 19 to 23 April 2027.
Follow us:
ā¢ Newsletter: https://2027.pycon.de/newsletter/
ā¢ LinkedIn: https://www.linkedin.com/company/pyconde
ā¢ X: https://www.x.com/pyconde
Links:
ā¢ Conference website: http://pycon.de
ā¢ Other sessions: https://2026.pycon.de/talks/
The conference was organized by
ā¢ Python Softwareverband e.V.: http://pysv.org
ā¢ Pioneers Hub gemeinnĆ¼tzige GmbH: http://pioneershub.org
in collaboration with NumFOCUS Inc.: http://numfocus.org
If you enjoyed this session, please like, and subscribe to our channel for more insightful talks and discussions.
Share this video with your network to spread the knowledge!
Hashtags:
#Python #PyConDE #PyData #OpenSource #AI #DataScience #MachineLearning #SoftwareEngineering #LLMs #Community #Sovereignty
Acknowledgements:
Special thanks to all the volunteers and sponsors who made this event possible.
About:
Python Softwareverband e.V.:
PySV is a non-profit that promotes the use and development of Python in Germany through events, education, and advocacy, fostering an open Python community.
Pioneers Hub gemeinnĆ¼tzige GmbH:
is a non-profit fostering innovation in AI and tech by connecting experts and promoting knowledge exchange through events and collaborative initiatives.
NumFOCUS Inc.
supports open-source scientific computing by providing financial and logistical support to key projects like NumPy and Jupyter, promoting sustainable development and collaboration.
www.pydata.org
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.")
![Making bad CLIs fun with Small Language Models [PyCon DE & PyData 2026]](https://i.ytimg.com/vi/tNAOyH1DS6o/mqdefault.jpg "Making bad CLIs fun with Small Language Models [PyCon DE & PyData 2026]
š Recorded at PyCon DE & PyData 2026, 16.04.2026
https://2026.pycon.de/talks/GMNE3E/
š Watch Senior Data Scientist Moritz Bauer demonstrate how to transform frustrating command-line interfaces into intuitive, natural-language experiences using fine-tuned Small Language Models.
Speakers:
Moritz Bauer
Description:
Complex command-line interfaces (CLIs) often suffer from steep learning curves due to intricate syntax and obscure flags. While large language models (LLMs) can translate natural language into CLI commands, they typically require cloud connectivity, API keys, and significant computational resources. To address this, a local implementation using small language models (SLMs) with fewer than one billion parameters enables fast, on-device inference without internet dependency.
The approach centers on supervised fine-tuning (SFT) using synthetic datasets. Initial data pairs of natural language instructions and corresponding CLI arguments are generated by prompting a coding LLM (such as Claude Opus) with the tools source code. To prevent the SLM from overfitting to rigid patterns, these pairs undergo prompt mutation, where a teacher model (such as Qwen 3.5 9B or GLM 4.5) generates multiple natural language variations for each command. To maintain data integrity, a filtering step removes variants that omit critical identifiers, such as customer names or dates, which would otherwise force the SLM to hallucinate.
The implementation utilizes the Gemma 2B or smaller variants (specifically a quarter-billion parameter model) fine-tuned via the Hugging Face stack on a MacBook M3 Pro. Training 100,000 pairs takes approximately 10 hours. Testing on an internal plotting tool and FFmpeg yielded an accuracy rate of roughly 85%, with errors typically manifesting as missing or extra flags. Using the Lama CPP inference engine, the system generates commands in under 2.5 seconds, demonstrating that SLMs can effectively map domain-specific natural language to complex technical syntax locally.
āļø About PyCon DE:
PyCon DE is the leading conference on open-source Python applications in AI and data science. It brings together industry professionals, researchers, AI and data science practitioners, and software engineering communities, providing a unique platform for collaboration, knowledge sharing, and innovation.
The PyCon DE & PyData 2026 conference delivered an exceptional experience, fostering stronger connections within the Python community while showcasing the latest advancements in artificial intelligence and data science. Attendees enjoyed a diverse and engaging program of talks, workshops, and networking opportunities, further establishing the conference as a premier event for Python, AI, and data science enthusiasts across Germany.
PyCon DE 2027 will take place in Heidelberg from 19 to 23 April 2027.
Follow us:
ā¢ Newsletter: https://2027.pycon.de/newsletter/
ā¢ LinkedIn: https://www.linkedin.com/company/pyconde
ā¢ X: https://www.x.com/pyconde
Links:
ā¢ Conference website: http://pycon.de
ā¢ Other sessions: https://2026.pycon.de/talks/
The conference was organized by
ā¢ Python Softwareverband e.V.: http://pysv.org
ā¢ Pioneers Hub gemeinnĆ¼tzige GmbH: http://pioneershub.org
in collaboration with NumFOCUS Inc.: http://numfocus.org
If you enjoyed this session, please like, and subscribe to our channel for more insightful talks and discussions.
Share this video with your network to spread the knowledge!
Hashtags:
#Python #PyConDE #PyData #OpenSource #AI #DataScience #MachineLearning #SoftwareEngineering #LLMs #Community #Sovereignty
Acknowledgements:
Special thanks to all the volunteers and sponsors who made this event possible.
About:
Python Softwareverband e.V.:
PySV is a non-profit that promotes the use and development of Python in Germany through events, education, and advocacy, fostering an open Python community.
Pioneers Hub gemeinnĆ¼tzige GmbH:
is a non-profit fostering innovation in AI and tech by connecting experts and promoting knowledge exchange through events and collaborative initiatives.
NumFOCUS Inc.
supports open-source scientific computing by providing financial and logistical support to key projects like NumPy and Jupyter, promoting sustainable development and collaboration.
www.pydata.org
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.")
![Using Sensor Fusion and ML to Navigate Underground When GPS Fails [PyCon DE & PyData 2026]](https://i.ytimg.com/vi/uyrlnwzsW1U/mqdefault.jpg "Using Sensor Fusion and ML to Navigate Underground When GPS Fails [PyCon DE & PyData 2026]
š Recorded at PyCon DE & PyData 2026, 16.04.2026
https://2026.pycon.de/talks/LVJXK3/
š Watch Ćtienne Tremblay explain how to leverage sensor fusion and ML to maintain precise navigation in GPS-denied underground environments.
Speakers:
Ćtienne Tremblay
Description:
Underground navigation is challenging because GPS signals cannot penetrate subway tunnels, leaving smartphones to rely on imprecise cell tower mapping or Wi-Fi scanning with accuracy radii often exceeding one kilometer. To solve this, a system was developed that estimates location by fusing motion sensor data, train schedules, and sparse device locations. The core logic treats the problem as a sequence of events: by detecting when a train moves and stops, the system can count stations traveled from a known starting point.
The technical approach utilizes a two-stage machine learning pipeline. First, a Convolutional Neural Network (CNN) is trained on millions of unlabeled user trips using a pretext task to classify general motion modes (stationary, walking, or automotive) based on accelerometer and gyrometer data. Second, transfer learning is applied to a smaller, high-quality dataset of 300 manually annotated trips to refine a binary classifier that specifically identifies moving metro states. This model is converted to TensorFlow Lite and deployed on-device via Core ML for iOS and Android to ensure functionality during network outages.
A mixer module integrates the binary motion predictions with offline train schedules and any available high-accuracy device locations to resolve edge cases, such as trains stopping between platforms. The system achieves approximately 90% accuracy, with predictions typically within one station of the true location. To manage uncertainty, the user interface employs warning banners and asymmetric confidence intervals, acknowledging a bias toward late rather than early predictions. The entire training pipeline is managed using Vertex AI to handle complex dependency graphs and parallel testing.
āļø About PyCon DE:
PyCon DE is the leading conference on open-source Python applications in AI and data science. It brings together industry professionals, researchers, AI and data science practitioners, and software engineering communities, providing a unique platform for collaboration, knowledge sharing, and innovation.
The PyCon DE & PyData 2026 conference delivered an exceptional experience, fostering stronger connections within the Python community while showcasing the latest advancements in artificial intelligence and data science. Attendees enjoyed a diverse and engaging program of talks, workshops, and networking opportunities, further establishing the conference as a premier event for Python, AI, and data science enthusiasts across Germany.
PyCon DE 2027 will take place in Heidelberg from 19 to 23 April 2027.
Follow us:
ā¢ Newsletter: https://2027.pycon.de/newsletter/
ā¢ LinkedIn: https://www.linkedin.com/company/pyconde
ā¢ X: https://www.x.com/pyconde
Links:
ā¢ Conference website: http://pycon.de
ā¢ Other sessions: https://2026.pycon.de/talks/
The conference was organized by
ā¢ Python Softwareverband e.V.: http://pysv.org
ā¢ Pioneers Hub gemeinnĆ¼tzige GmbH: http://pioneershub.org
in collaboration with NumFOCUS Inc.: http://numfocus.org
If you enjoyed this session, please like, and subscribe to our channel for more insightful talks and discussions.
Share this video with your network to spread the knowledge!
Hashtags:
#Python #PyConDE #PyData #OpenSource #AI #DataScience #MachineLearning #SoftwareEngineering #LLMs #Community #Sovereignty
Acknowledgements:
Special thanks to all the volunteers and sponsors who made this event possible.
About:
Python Softwareverband e.V.:
PySV is a non-profit that promotes the use and development of Python in Germany through events, education, and advocacy, fostering an open Python community.
Pioneers Hub gemeinnĆ¼tzige GmbH:
is a non-profit fostering innovation in AI and tech by connecting experts and promoting knowledge exchange through events and collaborative initiatives.
NumFOCUS Inc.
supports open-source scientific computing by providing financial and logistical support to key projects like NumPy and Jupyter, promoting sustainable development and collaboration.
www.pydata.org
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.")
