Open-source application platform inspired by Googles Boq Gerhard Lazu 2025-03-05 | Namespace.so was born from the need to run lots of tests efficiently, across a large amount of on-demand compute that spins up in seconds, and scales back down to 0.Hugo Santos shows us Foundation, an open-source Kubernetes application platform that powers Namespace's developer-optimized compute platform. We see a testing framework that has a:- Service-Oriented Architecture: The system manages how services are written, built, tested, deployed, and monitored in production- Container-Centric Approach: Everything runs in containers within Kubernetes clusters- Fast Test Execution: Tests that would normally be slow are dramatically accelerated through parallelization- Complete Isolation: Each test runs in its own dedicated Kubernetes cluster for complete isolationTECHNICAL INFRASTRUCTURE:1. Kubernetes Distribution: Uses customized k3s for speed2. Container Runtime: Custom-managed ContainerD (version 1.7)3. Operating System: Wolfi-based with custom kernel (6.7)4. Deployment Speed: New Kubernetes clusters become ready in under 2 seconds5. Scale: Can easily manage dozens of parallel test environmentsIf you want to watch the full conversation, as a 4k movie, go to 📺 makeitwork.tv/fast-infrastructure and join as a member.Alternatively, you can listen to the conversation at 🎧 https://makeitwork.fm/12There are 2 other videos that go well with this one:youtube.com/watch?v=zDJ-EE1f82wyoutube.com/watch?v=BSD-E8I7JNE
The nerdiest thing that Ive done this week Gerhard Lazu 2025-11-19 | The full video: youtube.com/watch?v=wi6Coi33_Lc . Live, small group discussions like this one: https://makeitwork.club
Converting an analog garage door to a Home Assistant-native one Gerhard Lazu 2025-11-17 | Have you ever wondered what it goes into converting a standard garage door into a smart one, integrated with Home Assistant?Nabeel Sulieman, Staff Software Engineer at GitHub, joins us for a makeitwork.club chat to talk about his 3‑month passion project that involves turning a very normal, very analog garage door into something that’s fully automated, observable, and Home Assistant–native.In this video we cover:* The initial plan and how the project evolved over 3 months* The background context: why even bother upgrading a working garage door?* Early ideas (including a few that didn’t make the cut) and what Nabeel learned from each* Choosing “the brains” of the setup* Microcontrollers, sensors, relays, and how everything talks to Home Assistant* Figuring out the physics* How do you safely and reliably sense door position, movement, and state?* Dealing with power, wiring runs, and a not‑so‑forgiving physical environment* What an optocoupler is, and why it matters for this build* Installing ESPHome on a chip and getting it onto your Wi‑Fi + into Home Assistant* Putting it all together into a reliable, testable system* How to control two doors with a single microcontroller* Mounting everything permanently so it’s safe, tidy, and spouse‑approved* ESPHome tips, YAML gotchas, and debugging strategies* The big question: was this actually worth it?* Reflections on “shipping the scrappy v1” vs. endlessly polishing* What surprised Nabeel the most during the build* Where he wants to take the project next (including a GitHub Action that ties into the setup)If you’re into Home Assistant, ESPHome, microcontrollers, or just love seeing a software engineer tackle a real‑world hardware problem, you’ll get a ton of practical ideas from this conversation—plus a realistic look at the tradeoffs, dead ends, and aha moments along the way.If you enjoyed this and want to explore more:- Small group discussions like this one in 🪩 https://makeitwork.club (the next one is on Nov. 21, 2025 where I cover my experience of KubeCon NA 2025)- Become a member at 📺 makeitwork.tv for the full length content (includes Jellyfin media server access)- Tune into 🎧 https://makeitwork.fm for the podcast00:00 - The plan for today00:51 - Intro02:09 - The Background04:19 - Idea 1: Replace the whole system05:12 - Idea 2: Use a pre-made kit05:47 - Idea 3: Build it yourself!07:30 - The brains of the setup09:17 - Figuring out the physics13:18 - What is an optocoupler?15:35 - Installing ESPHome on a chip18:28 - Putting it all together20:43 - How can you connect both doors with one microcontroller?22:05 - How to mount everything permanently?27:15 - ESPHome29:25 - Was this worth it?32:01 - I am wondering...33:05 - Ship the scrappy v133:37 - I forgot to mention...36:02 - What else surprised you during this build?38:56 - How are you thinking of continuing this?40:57 - A GitHub Action that...What did you think of Nabeel's passion project? Let us know in the comments below!
Learning from Matts homelab: Incus, Ansible and NO Kubernetes. Gerhard Lazu 2025-11-03 | Most people build a homelab slowly, over the course of many years, with the spare hardware that they have.After 3 years of tinkering, Matt Johnson, Senior Site Reliability Engineer, shares with us:- his motivations behind building a local, private environment for experimentation- the values that guided his homelab design- why he started with Incus - linuxcontainers.org/incus - and deferred Kubernetes for 3 years- how OpenTofu & Ansible are used to bootstrap hosts- the case for PowerDNS & WireGuardIn the second half, Matt shows us how he spins up a new Incus host, and what are some of the considerations for the entire homelab as this new host takes shape.If you enjoyed this, you are likely to also enjoy joining https://makeitwork.club where we discuss about topics like this one every 2 weeks.Homelab stats:- 64 vCPUs- 240GB RAM- 10.75TB storage- $5,000 over a 3 year period- $25/month running costs00:00 Intro00:44 Why I started a homelab02:47 What do I value in a homelab?05:26 What am I running in my homelab?09:09 How I settled on Incus?11:54 Could the kids help with the homelab?14:31 When do you have time to work on this?15:45 Does Incus provide scheduling & orchestration?17:27 What services do others depend on in your homelab?19:59 PowerDNS & WireGuard21:48 What do you use for observability?23:47 What am I working on now?27:26 A Christmas present for Gerhard28:47 Key learnings31:12 How much would 64 vCPUs cost in AWS?32:26 How do you install the host OS?34:41 What is the easiest way to consume these learnings?39:06 Let's spin up a new Incus host46:37 What would help understand this setup?48:44 Infrastructure diagram example51:41 Software for infrastructure diagrams55:20 What did you learn from your homelab?58:56 Wrap-up
Kubernetes: its still complicated. Gerhard Lazu 2025-11-02 | Whether you love it or hate it, it's worth using Kubernetes in anger and figuring out the setup that works best for you. Full conversation ships on 2025-11-03.
Is this the fastest HTTP benchmark that youve seen? Gerhard Lazu 2025-10-29 | 👉 This is a follow-up from youtube.com/watch?v=9cEcX_a8on8Have you wondered how fast is Varnish HTTP Cache (soon to be renamed to Vynil) when CPU & network are not a bottleneck? FWIW, this HTTP benchmark is for the open-source CDN that powers changelog.com: github.com/thechangelog/pipelyMy 10Gbit connection was the bottleneck last time, so I went to work:- Ryzen 7 5800X for the client, github.com/hatoo/oha- Threadripper 9970X for the server, Varnish 7.7- 2 x 100gbit NVIDIA Mellanox ConnectX-5 connecting both hosts via QSFP28 DAC- Adam Stacoviak & Jerod Santo as my "partners in crime"If you enjoyed this, there's more to it:- Full conversation: youtube.com/watch?v=hYahjUGoy5Y- Join the most loyal members for small group discussions every 2 weeks 🪩 https://makeitwork.club- Subscribe at 📺 makeitwork.tv for the full length content- Tune into 🎧 https://makeitwork.fm for the podcast00:00 - Intro00:33 - 64vCPU @ 5.4GHz max boost01:36 - 16vCPU @ 4.7GHz max boost02:26 - MCX556A-ECAT04:05 - 20Gbps HTTP benchmark06:16 - 40Gbps HTTP benchmark08:24 - 80Gbps HTTP benchmark10:01 - changelog.com on Vynil on Fly.io11:11 - What's next?14:45 - One more thing...
HTTP bench peaked at 40Gbps. I need faster CPUs. #homelab #http #benchmark #cdn Gerhard Lazu 2025-10-27 | Subscribe & catch the full video in a few days time. We made it to 80Gbps 🤫
Tiny improvements for my new 100Gbit/s homelab Gerhard Lazu 2025-10-23 | This is a short follow-up to youtube.com/watch?v=9cEcX_a8on8For my first pair of 100gbit network cards, I didn't know what to expect. I installed drivers the WRONG way and hit an issue on the first kernel upgrade. That bend was also driving me crazy! What I didn't think of is to check just how hot this network card actually runs.Until the next https://makeitwork.club when I will dig into all the improvements - SPOILER(3D printed tall bracket, LACP bond, 40mm Noctua fans, CDN benchmarks) - this is a quick peek into what is coming.Thank you for all the great comments, and especially those that addressed the lack of active cooling on the Mellanox ConnectX-5 card, and suggested a temporary fix until I get a 3D printed tall bracket.00:00:00 Intro00:00:29 I was doing MLNX_EN wrong00:01:34 _jodi33 has a solution for the bend00:02:41 WillFul noticed that there is no active cooling00:04:37 99Gbits/s for Marcos00:07:31 Does the storage support 100Gbit/s?00:09:46 Hopefully NO MORE Kubernetes
Containers integrated with your host Gerhard Lazu 2025-10-20 | Distrobox makes it easy to integrate containers with your host, especially those that require access to X11 / Wayland, external storage / USB devices, HOME directory etc. https://distrobox.it/
starship explain Gerhard Lazu 2025-10-17 | I've been using starship.rs for many years now, but I didn't know about the explain command.
Supercharge your development workflow with these CLI tools Gerhard Lazu 2025-10-12 | Join us on this treasure trove of CLI tools that will improve your development environment.Rio Kierkels, Head of Platform Engineering for Tribe.ai, shares his expert setup, from a lightning-fast shell prompt (you know it as starship.rs) to reproducible dev containers and seamless dotfile management.🛠️ TOOLS MENTIONED IN THIS VIDEO1. starship.rs - the minimal, blazing-fast, and infinitely customizable prompt for any shell!Learn more about Starship2. Bluefin OS - a custom image of Fedora Silverblue, designed for developers who need a reliable and powerful workstation.3. Distrobox - use any Linux distribution inside your terminal. Enable both backward and forward compatibility with software and freedom to use whatever you want.4. Devpod - create reproducible developer environments with a single command.5. chezmoi - a powerful dotfile manager that helps you manage your personal configuration files across multiple machines.6. mise - a next-generation version manager for your development tools, replacing asdf, nvm, and more.7. Bat - a cat clone with syntax highlighting and Git integration.If you enjoyed this and want to explore more:- Become a member at 📺 makeitwork.tv for the full length content- Tune into 🎧 https://makeitwork.fm for the podcast- Join the most loyal members for small group discussions every 2 weeks 🪩 https://makeitwork.club00:00:00 - Intro00:00:37 - starship.rs00:02:10 - Bluefin OS00:03:55 - Distrobox00:08:41 - xeyes & cmatrix00:10:38 - devpod.sh00:12:22 - chezmoi00:15:42 - mise00:18:27 - setup.sh00:23:28 - How do you run databases in dev?00:24:48 - What do you use for live-reloading K8s in dev?00:26:28 - Let's build a make-it-work devpod00:27:25 - What do teams often miss?00:28:50 - What is good team communication?00:32:10 - Thank you Rio for sharing all these tips with us
Reminder to alias cat=bat Gerhard Lazu 2025-10-08 | Rio Kierkels will be showing off a few tricks with chezmoi, mise, devpod, devcontainers and a bunch of other tools on 2025-10-10. Subscribe & catch the long version.#cli #unix #linux #terminal #dev #devlife #devops
Keep it simple & janky 🦾 Gerhard Lazu 2025-10-06 | Until I get a 3D printed tall bracket, this simple & janky fix does it! The bigger problem is the lack of active cooling. These NVIDIA Mellanox ConnectX-5 cards run VERY hot. Fix coming up...
A homelab network FASTER than an NVMe Gerhard Lazu 2025-10-03 | 👉 The follow-up: youtube.com/watch?v=uNsNKvR1miAHave you ever looked at your shiny new, maxed-out MacBook Pro and thought: "This is great, but where's my 100Gbit dongle?" Yeah, me neither.My first goal at the new startup was to build a 100Gbit homelab. I needed to benchmark high-performance software for live, cross-host process migrations (memory & all), remote GPUs and a bunch of other things which are as ridiculous and awesome as they sound.In this https://makeitwork.club conversation, I take you on my journey to build a home network that's 2x as fast as the fastest MacBook Pro NVMe available today. We'll cover:- 100Gbit dongles- Why most consumer grade motherboards are not enough- The great Linux distro debate (the answer might surprise you!)- Jumbo frames (MTU 9000) and a few more tuning tips- A live unpeeling that was definitely not what it looked likeIf you are a networking nerd, a homelab enthusiast, or just someone who enjoys watching nerds getting way too excited about PCIe lanes, then this video is for you.This is about half of the full conversation that we had in https://makeitwork.club, a place where we started meeting every 2 weeks. Everyone is welcome to join!There is more to this:- Become a member at 📺 makeitwork.tv for the full length content- Tune into 🎧 https://makeitwork.fm for the podcast00:00:00 Intro00:00:45 New company = new Apple hardware00:02:20 Where is my 100gbit dongle?00:02:59 Would this hardware work for 100gbit?00:05:38 The second half of the homelab00:07:44 Which Linux distro did I pick?00:09:47 Check network card PCIe speed00:12:05 netplan00:13:10 jumbo frames00:15:13 Why only 95Gbps?00:17:10 mlnx_tune00:18:00 How much is 98Gbps?00:18:52 What will this new host be used for?00:20:57 100Gbit networking00:24:58 Thunderbolt 500:27:15 OWC 10Gbps dock00:28:09 One more thing...00:29:47 Intro to next meeting's topic
Best homelab moments are shared Gerhard Lazu 2025-10-02 | Full video ships Friday, Oct. 3, 2025. We talk about topics like this one every 2 weeks in https://makeitwork.club
Whats wrong with this network card? Gerhard Lazu 2025-09-30 | Full video ships Friday, Oct. 3, 2025. We talk about topics like this one every 2 weeks in https://makeitwork.club. That landing page is still WIP, but getting the 100Gbps up & running was more important!
Lessons learned from building Fly.io Machines Gerhard Lazu 2025-09-27 | Join us for a deep dive into the architecture of Fly.io's global infrastructure with JP Phillips, one of the key engineers behind Fly.io Machines.What started as a fascinating "Exit Interview" blog post - fly.io/blog/the-exit-interview-jp - uncovering the human element behind a massive piece of technology, turned into this in-depth conversation about building and scaling a global, multi-tenant system.In this conversation, JP shares his reflections on the evolution of Fly.io Machines, the critical design decisions that shaped the platform, and the trade-offs made along the way. We explore the journey from a Nomad-based architecture to the custom-built Fly.io Machines, the reasoning behind choosing BoltDB over SQLite, and the crucial role of fly-proxy in handling traffic at a massive scale.We also take a look at some of the open-source components that power Fly.io, including a code walkthrough of the Finite State Machine (FSM) that manages the lifecycle of machines and volumes. JP explains how this simple yet powerful component ensures reliability and simplifies operations.Whether you're a software engineer, a DevOps professional, or just curious about how large-scale distributed systems are built, this conversation is packed with valuable insights and hard-earned lessons.If you enjoy this, there is more:- Join 🪩 https://makeitwork.club for regular members-only discussions- Tune into 🎧 https://makeitwork.fm for the podcast- Sign-up at 📺 makeitwork.tv for more content00:00:00 — It all started with The Exit Interview00:03:24 — Have any of your thoughts refined since the blog post?00:09:12 — How did Fly.io Machines start?00:17:20 — Initial design considerations00:28:21 — SQLite vs BoltDB00:36:07 — Why no database for flaps?00:38:50 — fly-proxy00:46:21 — Open-source components00:54:09 — Let’s look at the FSM code01:06:57 — Something simple that works as designed01:11:07 — Our take aways
Tweaks required for 100Gbps Gerhard Lazu 2025-09-23 | This is the highlight from last week's makeitwork.club. Subscribe to not miss the longer video when it publishes. We talk about topics like this one every 2 weeks in https://makeitwork.club. That landing page is still WIP, but getting the 100Gbps up & running was more important!
How to Run the Simplest Talos Linux Cluster on Oracle Cloud Gerhard Lazu 2025-09-17 | Join us as we explore the "why" behind Kubernetes and Talos, and then get hands-on with a step-by-step guide to getting your own single-node cluster up and running. We'll cover everything from generating an Oracle Cloud Image to configuring your instance and deploying your first applications.Whether you're new to Talos or looking for a streamlined way to run K8s on OCI, this video is for you!This is about half of the full conversation that we had in https://makeitwork.club, a place where we started meeting every 2 weeks. Everyone is welcome to join!There is more to this:- Become a member at 📺 makeitwork.tv for the full length content- Tune into 🎧 https://makeitwork.fm for the podcast00:00 - Introduction00:30 - Context for Talos Linux on Oracle Cloud02:25 - Why Kubernetes?04:42 - Why Talos?05:34 - What's the first thing that you like about Talos?07:05 - A look at the Talos Linux docs08:11 - Improving the single-node Talos install09:45 - Creating an Oracle Cloud Image with factory.talos.dev12:48 - The Oracle Cloud Image (OCI)13:49 - Let's create an instance15:15 - Configuring the new Talos instance18:11 - Using talosctl gen config18:44 - Setting up a single-node Talos cluster20:48 - Exploring the talosctl dashboard21:12 - Where do you keep configs with credentials?22:54 - Generating the talosctl kubeconfig23:11 - Did it work?25:15 - Let's start deploying stuff!27:05 - Who would like to see part 2?29:00 - Troubleshooting connection issues31:09 - Key takeaways32:51 - What's next?
Everything is date-based, except... Gerhard Lazu 2025-09-17 | Date-based & immutable runs deep - with one exception - and it works really well with no in-place upgrades.
What is eCapture? Gerhard Lazu 2025-09-13 | eCapture.cc allows you to capture SSL/TLS in plaintext without needing a CA certificate. It uses eBPF and it requires Linux/Android kernel on either amd64 or arm64.Marcos Nils shows us how to use it to debug all TLS requests from Docker to any container image repository.eCapture also supports:- openssi, libressl, boringssl, gnutls, gotls & nspr(nss)- capturing bash & zsh commands for Host Security Audit.- sql queries from mysald 5.6 & postgres 10- pcap modeThere is more to this:- Sign-up at 📺 makeitwork.tv for the full length content- Tune into 🎧 https://makeitwork.fm for the podcast- Join 🪩 https://makeitwork.club for regular members-only discussions
AWS CloudFront for a container image registry? Gerhard Lazu 2025-09-12 | We deploy a new AWS CloudFront distribution in less than 10 minutes while we talk serverless with Matias Pan & Marcos Nils. We then test everything with live traffic using Route53 DNS geolocation. We troubleshoot issues, celebrate this milestone & discuss the future of this infrastructure setup.If you found this video helpful, please give it a thumbs up, and don't forget to subscribe for more content on cloud infrastructure, DevOps, and all things tech! Let us know in the comments if you have any questions or what you'd like to see next.There is more to this:- Sign-up at 📺 makeitwork.tv for the full length content- Tune into 🎧 https://makeitwork.fm for the podcast- Join 🪩 https://makeitwork.club for regular members-only discussions00:00:00 Intro00:00:33 Serverless talk while we deploy a new AWS CloudFront distribution00:02:48 Let's test this AWS CloudFront00:05:06 AWS Route53 Geolocation records00:06:05 Let's test everything by selectively resolving registry.dagger.io to the new setup00:08:06 So what did we do today?00:09:10 How do you want to continue with this?00:12:03 This is a huge milestone
eCapture.cc SSL/TLS in plaintext #ebpf #ssl #tls Gerhard Lazu 2025-09-09 | ecapture.cc came in really handy when trying to understand all the HTTPS calls that the docker CLI is making when pulling down container images. More context: youtube.com/watch?v=n44Jiy3OjBc&t=465s
Vector.dev for all your logs & metrics Gerhard Lazu 2025-09-08 | Have been using it for years, highly recommended!
Live migrating PostgreSQL every 3s #postgresql #aws #migration Gerhard Lazu 2025-09-01 | Have you seen a live PostgreSQL process jump between physical hosts every three seconds without losing a single connection or in-progress transaction?
We made our ghcr.io 2x faster with AWS CloudFront Gerhard Lazu 2025-08-31 | Together with Matias Pan & Marcos Nils, our primary goal was to establish a container registry operating through CloudFront, effectively positioning CloudFront as a frontend for ghcr.io.While ghcr.io provides a generous free tier that served us well for years, the existing analytics on ghcr.io offer limited insights, latency is sometimes poor, and unavailability hits hard. A custom domain allows us to gather more granular data, makes requests consistently faster & more reliable anywhere in the world, and we have the option of switching container registries without disrupting users.A significant challenge in this project was understanding the HTTPS traffic between the user's machine and the CloudFront endpoint. To do this without the complexity of a man-in-the-middle proxy, we used eCapture, a powerful tool that leverages eBPF (Extended Berkeley Packet Filter) to capture SSL/TLS traffic in plaintext. This allowed us to inspect the requests and responses, understand the authentication flow, and identify which requests to cache.By using eCapture, we demonstrate that the new setup works as expected. We show how the Docker CLI interacts with the CloudFront endpoint, how the token is fetched, and how the manifest and blobs are pulled. We also demonstrate that the caching is working correctly, with subsequent requests for the same image being served from the CloudFront cache, resulting in a significant performance improvement - it's 100% faster.We plan to continue optimizing this setup including:- Caching the token: To limit the impact of ghcr.io service disruption- Fine-tuning the caching policies: To further improve performance and reduce costs- Origin shield: To reduce the load on ghcr.ioThere is more to this:- Sign-up at 📺 makeitwork.tv for the full length content- Tune into 🎧 https://makeitwork.fm for the podcast- Join 🪩 https://makeitwork.club for regular members-only discussions00:00 Intro00:36 What are we trying to achieve?01:44 Why custom domain for ghcr.io?03:34 registry-redirect05:48 vector.dev07:45 ecapture14:44 ecapture new image from registry.dagger.io21:56 Does it work?29:47 How can we improve on this?31:56 Is it faster than ghcr.io?36:59 How to set this up42:58 Follow-up optimisations
It was Usher #azure #cdn #varnish Gerhard Lazu 2025-08-25 | Subscribe & get notified when the conversation ships.
Migrating PostgreSQL between hosts with no downtime Gerhard Lazu 2025-08-20 | Have you seen a live PostgreSQL process jump between physical hosts every three seconds without losing a single connection or in-progress transaction?In this demo, Shivansh Vij, CEO of Loophole Labs, introduces us to architect.run, a groundbreaking live migration technology in action.We start with a standard, off-the-shelf PostgreSQL OCl image inside a virtual machine. We then live-migrate it between two different AWS instances - back and forth, every three seconds.Watch our real-time dashboards as we track:- The VM's location bouncing between hosts.- Continuous, successful read/write operations from a single psql client.- Stable, persistent network connections that survive every migration.- The most incredible part? The client application is completely unaware of the chaos. It connects to the exact same IP address the entire time, thanks to our intelligent network plane that handles all the magic behind the scenes. No custom database builds, no application code changes required.Features worth emphasizing:1. Zero-Downtime Migration: We move a stateful PostgreSQL database between hosts in seconds, demonstrating true, seamless live migration.2. Complete State Preservation: In-memory data, disk state, and active network connections are all perfectly preserved and transferred during every migration.3. Cross-Cloud Ready: While this demo uses two AWS instances, our custom kernel is built to normalize CPU features, enabling migrations between different cloud providers (like AWS and GCP) and even different CPU architectures-a problem that makes traditional snapshots non-portable.4. Instant Fault Tolerance: We discuss how this technology provides incredible high availability. If a host machine were to disappear without warning, the workload could be restarted from its last snapshot (in this case, from just 3 seconds prior) on another machine.00:00 - Introduction: The GCP Kernel Update Story00:58 - Explaining the Live Demo Dashboard Setup02:06 - The "Wow" Moment: Migrating a Live Database Every 3 Seconds02:21 - How Network Connections Survive the Migration03:04 - Proof: Real-Time Read/Write Operations to PostgreSQL04:36 - The Magic: How the Client Connects to a Single, Unchanging IP05:14 - No Custom Builds: Using an Official, Off-the-Shelf PostgreSQL Image06:17 - The Secret to Multi-Cloud: How We Handle Different CPU Architectures07:20 - What Happens if a Host Fails? (High Availability & Fault Tolerance)07:48 - How Snapshots Enable Near-Instant Recovery10:08 - Visualizing the Migration Time#LiveMigration #CloudComputing #HighAvailability #PostgreSQL #AWS #DevOps #ZeroDowntime #CloudMigration #Virtualization
We shipped an open-source CDN Gerhard Lazu 2025-08-13 | This is a shorter version of Kaizen 20, the Changelog Live recording from Denver: youtu.be/CZK57gTxbKo18 months ago with a simple question: "Can we do this?" Watch as we culminate months of work by migrating our entire production traffic from Fastly to our new, open-source CDN, "Pipely," live on stage.We started with a cache hit ratio of just 17.93%, meaning over 80% of our requests were slow and prone to failure. Follow along as we debug a last-minute memory issue, push the fix to production in under three minutes, and perform a live DNS switchover. The result? A staggering 99.1% cache hit rate!This is a real-world demonstration of continuous improvement, embracing public learning, and the power of building with friends.00:00 - The story of Pipely: How it all began 18 months ago.01:12 - Welcome to Kaizen 20: The first in-person meetup after a decade of remote collaboration.02:20 - All our work is open source! Find the code and discussions on GitHub.04:10 - The Problem: Our abysmal 17.93% cache hit ratio.06:21 - Let's fix it live! Debugging an out-of-memory crash that happened the night before.07:04 - Pushing the fix to production with a simple Git tag.08:41 - Watching the live, zero-downtime, blue-green deployment across our global infrastructure on Fly.io.11:47 - The Moment of Truth: Switching DNS from Fastly to our new CDN, Pipely.14:20 - The Result: From 17% to a 99.1% cache hit rate!15:05 - Verifying the DNS change has propagated worldwide.Learn more about the project: https://pipely.tech/Dive into the code and technical details (Discussion #546): github.com/changelog/changelog.com/discussions/546A huge thank you to our friends and contributors who made this possible, and to Fly.io and DNSimple for their amazing platforms.#Kaizen #LiveCoding #DevOps #CDN #Varnish #CloudNative #OpenSource #WebPerf
Eat, pray & launch a thing Gerhard Lazu 2025-08-08 | A few Changelog Friends met in Denver to eat, pray & launch something that has been in the works for a decade! Watch the full thing on https://changelog.com.
Just command runner on Windows Gerhard Lazu 2025-08-06 | It all started with a single, frustrating bug. Our development environment was supposed to be cross-platform, but for our Windows users, it was anything but simple. Running commands in PowerShell led to bizarre errors, creating broken file paths deep within the Windows Subsystem for Linux (WSL).Follow the journey as we:- Start with a completely fresh Windows virtual machine to hunt down the root cause.- Discover how containerizing our entire local environment with Dagger was a game-changer.- Uncover the subtle but critical differences between how PowerShell and WSL handle home directories, leading to our "aha!" moment.- Make the tough call to simplify, rather than over-engineer, by standardizing our workflow on WSL for everyone.This isn't just about fixing a bug; it's a story about simplifying complexity, the power of good documentation, and the satisfaction of deleting code that's no longer needed. We improved the developer experience, cleaned up our repository, and made Pipely better ever.What is your experience running just.systems in Windows? Let us know in the comments below!makeitwork.tv 👈 full version shipping any week now0:00 Intro0:43 Windows dev env1:35 Update docs for local dev tests2:27 Just home_dir() does not work correctly in Windows5:22 Read your own PRs8:12 tmux is not the easiest9:38 Just in Powershell vs WSL11:15 Another issue...11:59 Working from home FTW
GitHub push protection saves the day #github Gerhard Lazu 2025-08-01 | If you've been coding long enough, you will have committed secrets AT LEAST once. This happened the last time I did it...See the context: youtube.com/watch?v=rFcC8cAbW-MWatch the full pairing session: makeitwork.tv
Trust me, the docs are good and up-to-date Gerhard Lazu 2025-07-30 | We review the github.com/thechangelog/pipely dev experience with Nabeel Sulieman & Matt Johnson.From outdated docs, to outdated forks, to just & dagger. It's a small excerpt from the final version that will ship any week now on makeitwork.tvWDYT, are the dev docs better now? github.com/thechangelog/pipely/blob/v1.0.0-rc.5/docs/local_dev.md
Do you git fork? #codinglife #codingtips Gerhard Lazu 2025-07-21 | Remember to always git pull. And keep those docs up-to-date.
CDN PURGE with Elixir Gerhard Lazu 2025-07-19 | makeitwork.tv 👈 full version shipping any week nowWe implement the following one-liner bash script the Elixir Way™:dig cdn-2025-02-25.internal AAAA +short | while read ipv6; do curl -so /dev/null -w "%{http_code} %{method} %{url_effective}\n" -X PURGE "http://[$ipv6]:9000/"; doneFind the code and the conversation in this pull request 🐙 github.com/thechangelog/changelog.com/pull/549🤔 THE PROBLEM: CDN is serving stale content, with some entries expired for over 4 minutes. We need a way to explicitly purge cache across all CDN instances rather than waiting for user requests to trigger refreshes.🙇♂️ THE SOLUTION: DNS lookups to discover all instances and coordinate purges across them.🧑🔧 THE DNS THING (02:26): Using Fly.io's internal DNS system, we implement IPv6 lookups to discover all CDN instances. Claude Code helps. It works the first time 🎉🐞 BUGS & PREDATORS (3:00): Undefined function errors, mock testing issues, and the classic double-slash URL bug. No Claude Code, just experience.🧐 DID IT WORK? (6:56): Let's force push straight into production and see what happens 😱This session isn't just a tutorial - it's an authentic look at collaborative coding, where DNS magic meets Elixir elegance to conquer distributed caching.If you're into DevOps, CDNs, or just love seeing bugs squashed in real time, this gives you the tools to understand (and maybe implement) your own purge system.00:00 The Problem00:46 The Solution02:26 The DNS thing03:00 Bugs & Predators04:43 Let's fire this up06:56 Let's force push it07:25 Is it working?08:53 No no no
Who is PHK? #varnish #freebsd #bikeshed Gerhard Lazu 2025-07-18 | FreeBSD... FreeBSD Jails... NTP timecounters... Varnish HTTP Cache... The Bikeshed?
I LOVE TLS Gerhard Lazu 2025-07-11 | makeitwork.tv 👈 full version shipping any week nowOpen-source Varnish HTTP Cache does not support TLS backends. What does the simplest solution look like? Why are we even doing this? And "Why no SSL?"We go all the way back to 2011 to understand the reasons behind this, learn about PHK, The Bikeshed and conclude what's next for Pipely, "the 20-line Varnish config that we deploy around the world and: Look at our CDN!"This is a follow-up to youtube.com/watch?v=8bDgWvyglno00:00 What is Pipely?00:34 Three years ago00:57 Resolving the Varnish TLS issue03:41 Hitch05:44 Why no SSL?07:17 Who is PHK?09:10 The Bikeshed
Hunting double slashes #elixir #varnish #cdn Gerhard Lazu 2025-07-04 | Hunting double slashes. Purging global caches. No misses. makeitwork.tv
The Bear that Dreams of DevOps Gerhard Lazu 2025-04-12 | If you want to become exceptional at DevOps instead of just competent, you need to discover the mindset that transforms technicians into masters. Most people think it's about learning Kubernetes, mastering CI/CD pipelines, and deploying the right tools, so they focus on technical skills while missing what truly matters.But here's the thing: technical expertise alone won't make you remarkable. It's your dedication to perfecting your craft that creates true mastery. The greatest experts, like Jiro who's been making sushi daily for 80 years straight, understand something most technologists never grasp.This is an excerpt from the 🍱 DevOps Sushi - Part 1 🎬 movie that is now available on https://makeitwork.tv. Join as a member & put this conversation into context by watching the full-length movie.
What does Mischa run in his Kubernetes Homelab? Gerhard Lazu 2025-03-30 | Mischa van den Burg takes us on a quick tour of his sustainable Kubernetes homelab setup built from recycled enterprise hardware, all while maintaining a minimalist approach.☸️ 4-node Kubernetes cluster built with affordable refurbished hardware💻 Control plane running on a ThinkPad T430🗃️ Three worker nodes using HP EliteDesks (~€170 each)🐧 Minimalist Arch Linux setup with Wayland and Hyperland🏡 Homepage as his central dashboard for all services🦘 Wallabag for saving and storing articles👔 Linkding for browser-agnostic bookmark management🐘 PostgreSQL databases for data persistenceThis is an excerpt from the 🍱 DevOps Sushi - Part 1 🎬 movie that is now available on https://makeitwork.tv. Join as a member & put this conversation into context by watching the full-length movie.
Is Kubernetes Overkill for Your Homelab? Gerhard Lazu 2025-03-16 | Most homelab enthusiasts think Kubernetes is excessive, so they stick with simpler tools like Docker and Ansible. When containers fail, they intervene manually and miss out on true automation.When Mischa van den Burg created his first Kubernetes deployment with three (3) replicas, killed a container, and watched it automatically resurrect itself, he knew that this was the setup he really wanted. That feeling of seeing a rolling update happen flawlessly is something that you can't understand until you experience it first-hand. The automatic rollback when something goes wrong is the true epiphany.If you want a homelab that runs as reliably as professional cloud infrastructure, you need to consider what begins as overkill, but starts making more sense after some hands-on experience.This is an excerpt from the 🍱 DevOps Sushi - Part 1 🎬 movie that is now available on https://makeitwork.tv. Join as a member & put this conversation into context by watching the full-length movie.
Build containers & preview deployments on a fast app platform Gerhard Lazu 2025-03-05 | Gerhard demonstrates a simple web application deployment workflow:- First, he runs a local build using yarn (takes ~10 seconds)- Then packages the build into a container using his local Docker setup- He then compares this with a remote Docker build on Namespace's infrastructure.Gerhard then showcases Namespace's preview environment capabilities:- Using nsc run creates temporary deployment environments in seconds (~7 seconds)- These environments get unique URLs for testing- By default, previews require authentication, but can be made public with --ingress *=noauth- Preview instances automatically get cleaned up after a default duration (4 hours)Some highlights discussed:- Extremely fast container builds in the cloud- Quick preview environment creation- Namespace's infrastructure optimizes for throughput- Upcoming features like deferred starting (scale-to-zero) that only spins up when someone accesses the URLThe conversation includes interesting implementation details:- Namespace uses advanced caching for fast instance creation (~50-100ms for cloning cache volumes)- Deployments use global capacity planning rather than region selection- Preview environments support various protocols including TCP, TLS passthrough, and TLS terminationIf you want to watch the full conversation, as a 4k movie, go to 📺 makeitwork.tv/fast-infrastructure and join as a member.Alternatively, you can listen to the conversation at 🎧 https://makeitwork.fm/12There are 2 other videos that go well with this one:youtube.com/watch?v=Um77B1RiLaIyoutube.com/watch?v=BSD-E8I7JNE
Why is this 25 gigabit home internet SO SLOW? Gerhard Lazu 2025-03-05 | Hugo Santos is a happy init7.net customer. His home internet connection is capable of reaching 25 gigabit / s. So why is it slow?If you want to watch the full conversation, as a 4k movie, go to 📺 makeitwork.tv/fast-infrastructure and join as a member.Alternatively, you can listen to the conversation at 🎧 https://makeitwork.fm/12There are 2 other videos that go well with this one:youtube.com/watch?v=Um77B1RiLaIyoutube.com/watch?v=zDJ-EE1f82w
How to Build Alert Workflows that Actually Work | KeepHQ Demo Gerhard Lazu 2025-01-26 | This is a quickstart to Keep, an open-source platform for alert management and workflow automation. Matvey Kukuy shows us how to:1. Set up Keep with a MySQL database using Docker2. Create a workflow that: - Receives alerts - Queries a MySQL database - Sends webhooks based on query resultsTal Borenstein sprinkles a high level perspective, and helps fix a misconfiguration.Key features highlighted:- Workflows can be defined in YAML (similar to GitHub Actions syntax)- Supports multiple trigger types: alerts, incidents, manual triggers, and intervals- Includes deduplication and correlation of alerts- Integrates with external systems via "providers" (MySQL, HTTP webhooks, etc.)Orbstack & webhook.site are worth exploring on your own.Listen to the conversation before & after this demo 🎧 https://makeitwork.fm/11If you enjoyed this content and want to watch the full movie in 4k, go to 📺 https://makeitwork.tv. Offline download is available.00:16 Setup Keep locally01:21 Orbstack02:20 Our first alert03:07 What makes Keep different04:30 Let me try to understand this06:21 Our first workflow13:29 GitHub Actions for monitoring tools19:50 I can't believe it worked22:43 Something bad happened28:01 Reporting capabilities
Lets build a CDN - Part 2 Gerhard Lazu 2024-12-16 | This is a follow-up to youtube.com/watch?v=8bDgWvyglnoMost of this is captured in 🐙 github.com/thechangelog/changelog.com/pull/518We made IAD the primary region (it's the closest one to the origin), discussed VARNISH_SIZE and exposed environment variables as custom HTTP headers. We then struggle with IPv6 DNS resolution. At the end, we are surprised when this starts working all of a sudden.Listen to the conversation before the screen share 🎧 https://makeitwork.fm/10If you enjoyed this content and want to watch the full movie in 4k, go to 📺 https://makeitwork.tv. Offline download is available.00:00 Intro00:57 Who is James?02:12 Who is Matt?03:23 What are we trying to achieve today?04:26 GitHub Pull Request 51805:53 Make IAD the primary region06:31 How do we think about VARNISH_SIZE?09:43 Expose env var as custom HTTP header12:15 503 backend fetch failed22:59 We could create a GitHub issue...24:27 Why does this work?
Ninjastructure: Move fast & break nothing Gerhard Lazu 2024-10-26 | Matias Pan, a professional maté drinker & Senior Software Engineer at Dagger, is showing us an approach to Infrastructure as Code built with Pulumi.We look at Go code, discuss procedural (imperative) vs. declarative, spend some time on state management & introduce the concept of Ninjas in the context of infrastructure: move fast & break nothing.In the second half, Matias uses diagrams to talk through different ideas of rolling this out into production. Which of the two approaches would you choose?Testing Pulumi programs: pulumi.com/docs/iac/concepts/testingListen to the audio version 🎧 https://makeitwork.fm/9The full content experience 📺 makeitwork.tv00:00 Why Pulumi instead of Terraform?02:40 Procedural or declarative?07:19 What is Ninjastructure?08:47 First thing that gets provisioned in an AWS account11:18 How does the network module work?14:29 Biggest advantage to using Pulumi over Terraform17:02 Stacks = different environments18:20 Where is state stored?20:18 Where did you choose to store the state?21:46 How to use this in production?24:32 The GitOps approach29:17 Outro
From Homelab to Production - TalosCon 2024 Gerhard Lazu 2024-09-28 | As you move from homelab to production with Talos, you will be faced with several challenges: non-disruptive upgrades, backup & restore, securing secrets, observability at multiple levels & host security. Let me show you how I reached a simple & efficient productions K8s setup with Talos.Listen to 3 other conversations from TalosCon 2024 🎙 https://makeitwork.fm/800:00 Intro01:06 How this started01:55 What are the two things wrong?02:48 LattePanga Sigma03:29 Portable homelab04:15 What is Omni?04:42 How do I bootstrap my homelab?06:20 What is running on this homelab?08:15 Daggerverse on the Tailnet09:36 How many requests per second?13:40 Can we squeeze more requests per second?14:45 300B requests per month on a K8s phone15:09 Why Single Node?16:33 Why Kubernetes?17:29 Let's scale this homelab into production19:02 What is Dagger?20:55 The digitalocean function24:22 dagger call digitalocean28:38 We have a problem29:49 How to bootstrap db from backup30:59 Production network benchmark31:34 Takeaways34:22 Outro35:53 Questions from the audience40:39 One more thing
