Live Panel: Building at the Roofline — Planning and Delivering Gigawatt-Scale AI Infrastructure@Scale2026-11-19 | Live Panel: Building at the Roofline — Planning and Delivering Gigawatt-Scale AI InfrastructureClosing Remarks | Santosh Janardhan@Scale2023-05-18 | Closing Remarks | Santosh JanardhanFIRESIDE CHAT: The Future of AI Infra: The Opportunities and Challenges That Await Us On Our Journey@Scale2023-05-18 | FIRESIDE CHAT: The Future of AI Infra: The Opportunities and Challenges That Await Us On Our Journey | Irina Kofman, Alexis Björlin, Aparna Ramani, Kim Hazelwood & Rachel Peterson
This panel discussion will focus on The Future of AI Infra: The Opportunities and Challenges That Await Us On Our Journey. Moderated by Irina Kofman, head of XAI and responsible for cross-company AI efforts at Meta, this panel features leaders across Meta's infrastructure organization and will discuss the challenges and opportunities they see with building world-class, custom infrastructure specially built for AI.Gen AI-Assisted Code Authoring at Meta | Michael Bolin@Scale2023-05-18 | At Meta, we have built upon existing research published by FAIR to develop our own AI-Assisted code authoring tools. The freedom to experiment with the model combined with the ability to train on first-party code has enabled us to deliver tooling that has had a measurable impact on developer productivity.MSVP: Metas Scalable Video Processor | Harikrishna Reddy & Ioannis Katsavounidis@Scale2023-05-18 | This presentation will introduce MSVP (Meta's Scalable Video Processor), the first generation server grade video processing hardware accelerator of its kind developed at Meta. We will describe the motivation behind it, the architecture, and some of the novel algorithms that are in the video encoder and other video processing blocks to achieve high video quality. We will also describe how the hardware accelerators are used in Meta’s data center to support processing and transcoding billions of videos every day and provide premium video quality to end users, while saving power.MTIA: Metas First Generation of AI Accelerators@Scale2023-05-18 | MTIA: Meta's First Generation of AI Accelerators | Roman Levenstein, Amin Firoozshahian, Joel Coburn & Olivia Wu
Meta has traditionally relied on using CPU-based servers for running AI workloads, but the increasing compute and memory requirements of these models have pushed the company towards using specialized solutions such as GPUs or other hardware accelerators. This talk describes the company's effort in constructing its first silicon designed for its internal AI workloads and systems; It describes the accelerator architecture and platform design, and the software stack for enabling and optimizing workloads. It also touches upon the upcoming challenges and evolving requirements that need to be accommodated moving forward.PyTorch 2.0 | Peng Wu@Scale2023-05-18 | What makes PyTorch beloved makes it harder to compile. After almost five years, we finally cracked the technologies that made it possible to compile any PyTorch model, resulting in a step-function change in PyTorch’s approach to execution efficiency. We called it PyTorch 2.0.
PyTorch 2.0 delivers significant performance improvements over a wide variety of models, often with just a simple one-liner change. This talk focuses on the two critical technologies underlying PyTorch 2.0, TorchDynamo and TorchInductor.
PyTorch 2.0 was released in March. But do not mistake it as the end of the story. The first release of PyTorch 2.0 marks the beginning of a roadmap for improving PyTorch execution efficiency via compiled mode.Metas Research SuperCluster(RSC): Accelerating AI Research at Scale | Scott & Kalyan@Scale2023-05-18 | Meta is facing a challenging and exciting future as it expands beyond its current capabilities in the social space. Ensuring our platform is open to as many diverse cultures, languages and perspectives is a significant challenge that requires intensive large-scale AI models. The complexities of adding a virtual reality Metaverse further increases the challenge space, requiring much larger models with greater numbers of modalities and parameters.
Meta anticipated these challenges and has built a dedicated high-performance state-of-the art cluster to accelerate AI research. We present the architectural choices that went into building the cluster composed of 16K GPUs, high-performance storage and a non-blocking Infiniband network. We will discuss some of the lessons learned and how they have been applied to Meta in general.
Finally, we reflect on the impact the RSC has had on our research projects, and provide some insight into future directions.Opening Remarks | Santosh Janardhan@Scale2023-05-18 | Opening Remarks | Santosh JanardhanNext-Generation Data Center Design | Alan Duong@Scale2023-05-18 | Building AI capacity is essential to the future of our company, and supporting AI workloads at scale requires a different approach than scaling to support our regular online services. Our new data center design will support the next generation of AI systems. We are building an increased level of flexibility into our design, which will allow us to pivot in response to shifts and changes in the AI space.
The new design will have fewer but denser racks to support large scale AI clusters, allowing us to have a smaller footprint while serving the same capacity as our previous data center designs. This design was created with efficiency at the forefront. Each data center going forward will be optimized for water and energy usage depending on the site/region, and will continue to incorporate sustainable features to ensure efficient facilities. We anticipate this design will also be faster and cheaper to build.Lessons Learned from Scaling Infrastructure as Code | Rosemary Wang@Scale2023-05-15 | You adopted an infrastructure as code tool like Terraform. What started as one person writing some configuration and deploying new infrastructure scales to everyone in the company writing their own infrastructure configuration and deploying their own systems. In this talk, we’ll share some of the lessons learned across the Terraform community when scaling infrastructure as code practices from one team to an entire company and its users. We’ll cover the patterns and practices that help address challenges of updating infrastructure, managing infrastructure modules, maintaining security, streamlining cost, and even upgrading and migrating tools.AI Infra @Scale 2023 - Speaker Highlights@Scale2023-05-11 | AI Infra @Scale 2023 will be hosted virtually and feature a range of speakers from Meta’s engineering and infrastructure teams. Meta's Head of Infrastructure, Santosh Janardhan, will deliver opening and closing remarks, while also guiding us through six exciting technical presentations on some of Meta's latest AI infrastructure investments. Additionally, we’re thrilled to host a fireside chat with a panel of Meta AI infrastructure leaders as they discuss "The Future of AI Infra: The Opportunities and Challenges That Await Us On Our Journey.” The event will take place via webcast on May 18, 2023.
To register visit atscaleconference.com/events/meta-ai-infra-scaleAI Infra @Scale 2023 - Fireside Chat Preview@Scale2023-05-11 | AI Infra @Scale 2023 will be hosted virtually and feature a range of speakers from Meta’s engineering and infrastructure teams. Meta's Head of Infrastructure, Santosh Janardhan, will deliver opening and closing remarks, while also guiding us through six exciting technical presentations on some of Meta's latest AI infrastructure investments. Additionally, we’re thrilled to host a fireside chat with a panel of Meta AI infrastructure leaders as they discuss "The Future of AI Infra: The Opportunities and Challenges That Await Us On Our Journey.” The event will take place via webcast on May 18, 2023.
To register visit atscaleconference.com/events/meta-ai-infra-scaleSystems @Scale Summer Live Q&A Session #4 | Moderated by Yun Jin@Scale2023-05-11 | Featuring David Meisner, Rosemary Wang, Kenny Yu, Ranjith Kumar S Moderated by Yun JinSystems @Scale Summer Live Q&A Session #3 | Moderated by Sherry Chen@Scale2023-05-11 | Featuring Joseph Eley, Nathan Bronson, Louis Brandy, Patricia McKenzie, Rohit Ahuja Moderated by Sherry ChenHosting Open Source Relational Databases at Scale on Microsoft Azure | Rakesh Gujjula@Scale2023-05-11 | Hosting managed relational database services in the cloud with the level of availability, reliability guarantees demanded by mission critical workloads and doing it at scale presents a set of interesting challenges. This talk will walk you through the evolution journey of Open Source relational database services hosting platform on Azure and how these challenges were addressed while meeting the agility, performance, stability, and cost goals from the business and customers.The Ent Framework: Meta’s Object-Relational Mapping | Patricia McKenzie and Rohit Ahuja@Scale2023-05-11 | When you think about Meta’s family of apps, what comes to mind? Maybe the over 6 thousand photos and videos created per second on Instagram, the 5 trillion photos on Facebook, or the 60 million group posts loaded each second. It’s challenging to manage all of the associated databases, schemas, queries and constraints at scale like ours. How do we keep this data consistent? How do we handle who can access or read this data in different contexts with different roles and permissions? How do we make it so engineers across teams can easily understand and onboard to another team’s database model? These challenges are why Meta created the Ent Framework, our Object-Relational Mapping layer. The Ent Framework simplifies development for tens of thousands of engineers by automating and simplifying how they integrate with multitudes of different storage systems. Learn about how Meta keeps databases more secure, code less repetitive and the family of apps more robust via the Ent Framework.Cache Made Consistent – Cache Invalidation Might No Longer Be a Hard Thing in Computer Science@Scale2023-05-11 | Cache Made Consistent – Cache Invalidation Might No Longer Be a Hard Thing in Computer Science | Lu Pan
Cache invalidation is considered one of the hardest things in Computer Science. We, at Meta, operate some of the world's largest cache deployments (e.g. Memcache and TAO), serving more than one quadrillion queries a day. We have developed a systemic approach at diagnosing inconsistencies from cache invalidations at scale.Getting from Schemaless Ingest to Fast SQL at Rockset | Nathan Bronson and Louis Brandy@Scale2023-05-11 | Rockset provides low-latency SQL access to schemaless data that is ingested in real-time. Immediate access to dynamically structured data is very powerful, enabling rapid development and iteration for products built on top, but it needs to be baked into the design and implementation from the start. Rockset’s architecture incorporates ideas from search engines, OLTP databases, and OLAP engines. In this talk we will present some of the database and low-level C++ programming techniques we use to get excellent performance and efficiency over dynamic data.Scaling End to End Reliability Tracking Across Large Scale, Multiplexed Products and Services@Scale2023-05-11 | Scaling End to End Reliability Tracking Across Large Scale, Multiplexed Products and Services | Kewei Qu
This talk introduces a new user experience-focused reliability measurement that exposes end-to-end reliability guarantees across the vertical service stack used by Meta’s family of Apps. The talk discusses the difference between the new reliability approach versus the industry standard and our successes and what comes next.Global Capacity Management at Meta | Kenny Yu and Ranjith Kumar S@Scale2023-05-11 | Meta currently operates more than 15 data center regions around the world. This rapidly expanding global datacenter footprint poses new challenges for service owners and for our infrastructure management systems. In this talk, we will present the challenges with managing a global-scale infrastructure and our approach for global service and capacity management. In particular, we’ll focus on the abstractions and guarantees we present to service owners with global capacity, and we’ll walk through our current design and implementation for how we manage our workloads across 10s of regions. We’ll also present our future plans with Infra Cloud as we build towards our longer term vision of transparent automated global capacity management.Leveraging Data in Motion in a Cloud-first World | Jun Rao@Scale2023-05-11 | Apache Kafka has emerged to the de-facto standard for event streaming platform in enterprise architectures. Many business applications are moving away from data-at-rest to an event-driven architecture so that they could leverage the data in real time as new events occur. More than 80% of the Fortune 100 are building their businesses on this new platform. In this talk, I will first share the story behind Kafka: how it was invented, what problem it was trying to solve and how it has been evolving. Then, I will talk about how making Kafka Cloud native creates new opportunities for building a system of record and some real world use cases.Configuration Safety at Scale with Ads | Joseph Eley@Scale2023-05-11 | The Configerator repository provides Meta developers with a way to make changes easily and quickly to production services. By default, it pushes changes to all services at Meta in a matter of seconds, and doesn’t have the traditional safeguards that most services have of being able to do extensive testing before releasing a new version of the service. It is one of the main things that enables Meta’s move-fast culture and lets developers iterate quickly and have fast development cycles. However, if not done properly, this can result in many reliability issues. This talk goes over the reliability issues that the Ads organization has had due to these config changes and how we have improved the reliability over time by investing into various safety measures like build time validations, diff-time testing, and helped develop and use a new service that can gradually push config changes.DADI @ Scale: Deploying Containers at Scale in Alibaba | Huiba Li@Scale2023-05-11 | Alibaba Cloud offers a comprehensive suite of elastic computing services that are based on container technology. Alibaba Group is one of the key customers of Alibaba Cloud and all of the major applications across its large and diverse set of businesses are run in containers. In this talk, we present DADI, the image system that underpins Alibaba’s containers, and share our experience with deploying it at scale worldwide to serve all of Alibaba Group and a large and rapidly growing number of external customers on the Alibaba Cloud. DADI is a block-level image system that replaces the waterfall model of starting containers (downloading image, unpacking image, starting container) with fine-grained on-demand transfer of remote images, realizing instant start of containers. DADI relies on a peer-to-peer architecture in large clusters to balance network traffic among all the participating hosts. One of the unique features of DADI is that it is based on the standard block device so that the image system is file system and platform agnostic, enabling one image system to handle the many application and container platforms that inevitably span very large organizations including Alibaba. The system is high-extensible, allowing us to quickly add features including trace-based prefetching and custom acceleration of container provisioning for different computing services such as serverless computing or Function-as-a-Service (FaaS). As part of this talk, we highlight the ease with which DADI can support new container technologies including those based on Kata Containers, firecracker and gVisor. We conclude with a discussion of ongoing efforts towards more secure containers by leveraging the small attack surface of the DADI block device, and decrypting the container image only within the container.How Meta Keeps its Large-scale Infrastructure Hardware Up and Running | Fred Lin@Scale2023-05-11 | Internet services like Facebook, Instagram, and Whatsapp rely on large-scale infrastructure to support the various compute, storage, and AI workloads. With the support of data and ML techniques, we can scale our infrastructure successfully by improving the efficiency of our tooling and workflows. In this presentation we’ll share our recent work on hardware remediation, automated anomaly detection and root cause analysis, error reporting interrupt tuning for minimizing performance overhead, near-real time at-scale server reboot detection, and an ML framework for predicting repairs for hardware failures. The data and ML solutions help us engage people less but with more context, so we can focus people on the real challenging work while the repetitive tasks are automated.Systems @Scale Summer Live Q&A Session #1 | Moderated by Ernesto Gonzalez@Scale2023-05-11 | Featuring Lu Pan, Jun Rao, Ali Zaveri, Suyog Mapara, Rakesh Gujjula Moderated by Ernesto GonzalezInfra Cloud Service Platform (ICSP) | David Meisner@Scale2023-05-11 | Building and operating a service is challenging and complex. At scale, service owners need to consider a number of responsibilities including how they develop, deploy, scale and monitor their service in production. Each of these concerns may require a service owner to understand, configure, and operate multiple underlying supporting systems to accomplish their goal. Service owners desire a solution that's simpler to develop and manage. Infra Cloud Service Platform (ICSP) is a holistic infrastructure product aimed at reducing the complexity of developing and operating services at Meta. As a platform, ICSP provides an integrated ecosystem that streamlines and orchestrates the pieces for the user. The platform is built on three important pillars: 1) a common configuration and control system (the Control Plane), 2) a logical model of operation (Service Data Model), and 3) code framework (Service Code Experience).Dont Ship the Org Chart: Rebuilding Istio for User Maintainability | Craig Box and Sam Naser@Scale2023-05-11 | While the cry of "breaking apart the monolith" can be heard throughout the industry, the Istio service mesh took a different tack, and consolidated its control plane microservices into one binary. How did we get here? In this talk, Google DA and Istio Steering Committee member Craig Box will talk about how the team building the service mesh designed it based on the internal Google services it was emulating, why that turned out to be the wrong choice, and how the ship was righted. Google engineer and Istio Environments working group lead Sam Naser will talk about how the new model allows for safer upgrades of the mesh, letting users test new versions with a canary model before rolling out to the whole fleet.Systems @Scale Summer Live Q&A Session #2 | Moderated by Amaya Booker@Scale2023-05-11 | Featuring Fred Lin, Huiba Li, Kewei Qu, Craig Box, Sam Nasar Moderated by Amaya BookerIntroducing Zelos - Zookeeper API leveraging Delos | Ali Zaveri and Suyog Mapara@Scale2023-05-11 | In this presentation we will introduce Zelos. Zelos provides the exact same semantics as ZooKeeper but is built using Delos. ZooKeeper forms the foundation of Meta's infrastructure stack and we have been using it over a decade. Over the decade we have improved the performance of ZooKeeper to meet our scale requirement but every such improvement has been non trivial amount of work especially given the monolithic design of ZooKeeper. In this talk we will deep dive into Zelos's architecture and show how it enables to solve some of the scaling limits we were hitting with ZooKeeper. Migrating with 0 downtime has been an important goal and we will give an overview on how we went about doing it.Video @Scale Fall 2022 LIVE Q&A #3 | Moderated by Venus Montes@Scale2023-05-09 | Featuring Yuriy Reznik, Naizhi Li, Anastasis Germanidis Moderated by Venus MontesReliably Changing Configuration @ Scale | Avery Berchek@Scale2023-05-09 | Thousands of services at Meta use Configuration Management, so it is important we change that configuration reliably. Tune in for a story spanning several years, covering how we exponentially grew coverage of a protection mechanism for one our most critical developer workflows. Along the way, we'll dive into some specifics of challenges we faced and overcame to reliably change configuration at scale.Shrinking the Impact of Production Incidents | Yuri Grinshteyn@Scale2023-05-09 | Shrinking Production Incidents details an organized approach for reducing the overall impact of production outages. Attendees can expect to learn how to prioritize reliability-related engineering tasks based on incident postmortem data, focusing on tasks that: * Reduce time to detection of the incident * Shorten the time to repair * Expand the time between failuresService Degradation at Scale: Creating Instant Capacity | Thote Gowda & Yi Yu@Scale2023-05-09 | We will talk about what factors made us realize that service degradation is necessary for our infrastructure and the challenges we faced while implementing service degradation at scale. We will also speak about how we are changing our Fault Tolerance Strategy to use service degradation instead of provisioning extra buffer.How We Drained Every Backbone Router Simultaneously | Santosh Janardhan@Scale2023-05-09 | Presented by: Santosh JanardhanAWS Infrastructure: Engineering for Resiliency at Scale | Prasad Kalyanaraman@Scale2023-05-09 | On December 16, 2021, an unlikely series of distributed system events slowed down customer requests through the Cloudflare system for a period of approximately 30 minutes. This talk gives an overview of how unexpected outcomes are available at any point in a large, complicated and heavily interdependent system.Lessons Learned from the Halloween Outage | Max Ross@Scale2023-05-09 | In this talk, VP of Engineering Max Ross will discuss the 73 hour outage that impacted Roblox late last year. He will also share some of the ways that a multi-day outage can turn conventional reliability wisdom on its head.Improving Reliability @ Meta: By Analyzing Historical Events That Led to SLO Violations@Scale2023-05-09 | Improving Reliability @ Meta: By Analyzing Historical Events That Led to SLO Violations | Kostiantyn Tsaregradskyi & Keshav Varma
Learn about culture of tracking Service Level Indicators/Service Level Objectives at Instagram specifically and Meta in general, the tools that we use and how teams' SLI/SLO workflows can be improved by annotating SLO violations and analysing them later. In the talk we will briefly cover history of SLI/SLO tracking at Meta, then talk about how Instagram team used data annotations to tackle some of the reliability issues they had and how we're expanding this approach to the whole company.Metas SEV Culture: How Todays SEVs Create Tomorrows Reliability | Joe Gasperetti & Nick Egebo@Scale2023-05-09 | Would you believe us if we said the more SEVs we have, the more reliable we are? In this talk we'll talk about the reasons why we love SEVs at Meta, and how our culture around SEVs has allowed us to build reliable services at scale. We'll start by exploring research from other industries about how incident culture shapes how reliable they are. We'll then share how we've applied these lessons to our own culture. Along the way we'll give a peek at our SEV tool, some insight into our SEV review process, and describe how we encourage a "culture of SEVs" from the very first day an engineer arrives at Meta.PIPEFAIL: How a Missing Shell Option Slowed Cloudflare Down | Jeremy Hartman@Scale2023-05-09 | On December 16, 2021, an unlikely series of distributed system events slowed down customer requests through the Cloudflare system for a period of approximately 30 minutes. This talk gives an overview of how unexpected outcomes are available at any point in a large, complicated and heavily interdependent system.Lessons From Long-Running Investigations | Jana Iyengar & Hossein Lotfi@Scale2023-05-09 | In this talk, we share some lessons from several of our long-running investigations. Some of them are well-known, but are worth repeating, and some of them are things we learned and want to share.Service Incident Deep Dive: Technical Overview & Learnings | James Kretchmar@Scale2023-05-09 | This talk will provide a technical overview of a service incident on the Akamai platform in July 2021 which, despite layers of safety technologies, nevertheless impacted some of Akamai’s customers. In addition to exploring the technical underpinnings of the incident, there will be discussion of lessons learned and actions taken to broadly reduce the risk of recurrence.QUIC Exit: Exposing a New Class of Outage | Ian Swett@Scale2023-05-09 | A crash bug in QUIC handshake code exposed a new class of bugs we termed ‘contagion bugs’. For these bugs, a tiny number of tasks can cause a huge outage and rollbacks don’t work as expected. This talk details what contagion bugs are, discusses the details of the outage, and what we did to prevent and mitigate them going forward.Live Q&A Session #2 | Moderated by Christian Monzon@Scale2023-05-09 | Featuring Kostiantyn Tsaregradskyi, Keshav Varma, Thote Gowda, Yi Yu, Avery Bercheck, Nick Egebo, Joe Gasperetti Moderated by Christian MonzonLive Q&A Session #1 | Moderated by Anca Agape@Scale2023-05-09 | Featuring Santosh Janardhan, Max Ross, Ian Swett, James Kretchmar, Jana Iyengar, Hossein Lotfi, Prasad Kalyanaramn, Jeremy Hartman Moderated by Anca AgapeML Infrastructure for Autonomous Vehicles @ Cruise | Alexander Sidorov@Scale2023-05-04 | In this talk I will present the machine learning infrastructure that supports the large scale data preparation and training at Cruise. And how it enables the model improvements in an effective, reliable, continuous and automatic fashion.Fireside Chat | Yann LeCun & Ludovic Hauduc@Scale2023-05-04 | Fireside Chat | Yann LeCun & Ludovic HauducModel Performance Monitoring - A Practitioners Perspective | Krishna Gade@Scale2023-05-04 | Artificial Intelligence is increasingly playing an integral role in determining our day-to-day experiences. Moreover, with the proliferation of AI-based solutions in areas such as hiring, lending, criminal justice, healthcare, and education, the resulting personal and professional implications of AI are far-reaching. The dominant role played by AI models in these domains has led to a growing concern regarding potential performance drift and bias in these models, and demand for model transparency and interpretability. Model Monitoring has become a prerequisite for building trust and adoption of AI systems in high-stakes domains requiring reliability and safety such as healthcare and automated transportation, and critical industrial applications with significant economic implications such as predictive maintenance, exploration of natural resources, and climate change modeling. In this talk, we will be talking about how Fiddler.AI is building Model Performance Management that solves this problem by continuously monitoring AI algorithms for performance and bias issues and reports actionable insights with explanations to the entire organization. For more information, you can visit www.fiddler.ai or follow us on Twitter @fiddlerlabs.AI @Scale at Microsoft | Yuan Yu@Scale2023-05-04 | Microsoft’s AI @Scale encompasses multiple dimensions. It gives customers access to state-of-the-art large-scale AI models, training and inferencing optimization tools, and supercomputing resources; It provides cross-platform AI runtime that enables customers to infuse AI into every aspect of their platforms and products, across cloud and edge. In this talk, I will give an overview of Microsoft AI @Scale. I will focus on the roles that AI software plays, and discuss the challenges we are facing and the areas for potential collaborations between AI software and hardware.Content Steering with MPEG DASH | Will Law@Scale2023-05-04 | Content distributors routinely use multiple concurrent CDNs to distribute their live and VOD content. For performance, contractual and failover reasons, there are requirements to switch dynamically between these distribution channels at run time. A new specification being developed by the DASH Industry Forum provides a standardized means for a third-party steering service to switch a player between alternate content sources, both at start-up and dynamically while the stream is underway. This talk investigates the mechanics of this steering workflow, including manifest enhancements, player behavior, local steering for 3GPP EMBMS compatibility, steering the manifest itself, and steering ads separately from primary content. We’ll demo a working steering server and discuss compatibility and interop with HLS Content Steering.Scaling AV1 End-to-End Delivery at Meta | Ryan Lei@Scale2023-05-04 | AV1 was the first generation royalty-free coding standard developed by Alliance for Open Media, of which Meta is one of the founding members. Since its release in 2018, we have worked closely with the open source community to implement and optimize AV1 software decoder and encoder. Early in 2022, we believed AV1 was ready for delivery at scale for key VOD applications such as Facebook(FB) Reels and Instagram (IG) Reels. Since then, we have started delivering AV1 encoded FB/IG Reels videos to selected iPhone and Android devices. After roll out, we have observed great engagement win, playback quality improvement, and bitrate reduction with AV1. In this talk, we will share our journey on how we enabled AV1 end-to-end from Meta servers to users' mobile screens around the world. First, we will talk about AV1 production, including encoding configuration and ABR algorithms. Further, since the main delivery challenge is on the decoder and client side, we will also talk about the learnings on integrating AV1 software decoder on both iOS and Android devices and the current state. Finally, some ongoing and future work will also be presented.