Distributed Embeddings At Scale: Processing 10+ million rows per day with Ray and GPUs @socallinuxexpo
Distributed Embeddings At Scale: Processing 10+ million rows per day with Ray and GPUs  @socallinuxexpo
Uploaded April 2026 | Updated November 2026, 2 hours ago
Talk by Justin Miller

socallinuxexpo.org/scale/23x/presentations/distributed-embeddings-scale-processing-10-million-rows-day-ray-and-gpus

In this talk, we’ll describe a production-grade NLP pipeline that processes millions of pieces of social media content across TikTok, YouTube, and Instagram using Ray and GPU acceleration. Learn how we use Ray's distributed computing model to orchestrate scalable embedding generation, sharded batch writes to Qdrant for vector search, and end-to-end pipeline tracking with Snowflake. We'll also talk about selecting a vector store and how to best evaluate the many options available.
Distributed Embeddings At Scale: Processing 10+ million rows per day with Ray and GPUsAsk and You Shall Debug: Conversational Troubleshooting for KubernetesThe State of Immutable LinuxKnowing is Half the Battle: End to End Observability in OpenStack on KubernetesConquering Events with Streaming AnalyticsFrom SQL to NoSQL? Building a MongoDB-Compatible Engine on PostgreSQLKeeping FreePBX Secure: A Walkthrough of Finding and Mitigating VulnerabilitiesAT: The Billion-Edge Open Social GraphReproducibility as a Social ContractReal-time billing using the new bidirectional ARI for AsteriskWhen AI Agents Meet Production InfrastructureRoom 106 Saturday Mar. 07 - SCaLE 23x
Southern California Linux Expo |

Distributed Embeddings At Scale: Processing 10+ million rows per day with Ray and GPUs

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER