Designing a location database: QuadTrees and Hilbert Curves Gaurav Sen 2024-10-21 | Designing a location database: QuadTrees and Hilbert Curves
Scaling Memcache at Facebook: Research Paper Walkthrough Gaurav Sen 2024-10-19 | In this video, we walk through the 2010 research paper "Scaling Memcache at Facebook." This paper has been very influential in companies building distributed caching systems.For example, Uber and Twitter have caching systems inspired by Memcache. The requirements of Memcache were:1. Easy adoption, minimal APIs2. Low Latency3. Very High Availability (without sacrificing too much on consistency)4. Large globally distributed systemThe design choices of Memcache are influenced by the above, as we will see in this video.Chapters:00:00 Background02:30 Design Choices05:35 Encouraging Adoption09:58 Cache Architecture12:59 Optimizations17:33 Scaling The Cache21:39 Hail Availability!26:30 Scaling The Database31:29 Thank you!Scaling Memcached at Facebook: usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdfMy Website: interviewready.ioDesigning Data-Intensive Applications Book: amzn.to/3SyNAOyYou can follow me on:Github: github.com/InterviewReady/system-design-resourcesInstagram: instagram.com/interviewready_LinkedIn: linkedin.com/company/interview-readyTwitter: twitter.com/gkcs_#SystemDesign #InterviewReady #Coding
5 questions in 4 minutes: Gaurav Sen interviews Hemant Pandey Gaurav Sen 2024-10-14 | I asked a series of (hopefully relevant) questions to Senior Software Engineer Hemant Pandey, who currently works at Meta. Watch this short video to understand what a day in their life is like.Hemant's Profiles: https://linktr.ee/hemant.pandey00:00 1. What does a senior engineer do?00:48 2. How do you motivate the team?01:27 3. Individual Contributor or Engineering Manager?02:06 4. How do you stand out in FAANG?02:35 5. Is ChatGPT affecting the job market?03:53 Thank you!Links:Github: github.com/InterviewReady/system-design-resourcesTwitter: twitter.com/gkcs_InterviewReady: interviewready.io#RapidFireQuestions #SeniorEngineer
Algorithm Deep Dive: Realtime Audio Matching In Shazam Gaurav Sen 2024-10-11 | Have you ever been at a restaurant, and noticed a song playing in the background? You may want to know the original song to listen to later!This is the use case of Shazam, an audio search app that takes recorded clips from users in their natural environment. It then finds the original song of the recorded clip.I have personally used it often, and the algorithm beneath fascinates me :DAlgorithm:- Graph the frequency and amplitude of each point in time in the song.- Find the points with high amplitude and frequency variation in this graph.- Take all k consecutive points to get a set of chunks.- Take the combinatorial hash of each chunk.- Store these hashes in the DB.Resources:medium.com/@treycoopermusic/how-shazam-works-d97135fb4582http://coding-geek.com/how-shazam-workshttps://www.ee.columbia.edu/~dpwe/papers/Wang03-shazam.pdf00:00 Usecase 00:20 Storing Songs01:32 Storage Considerations02:02 Representing Songs02:44 Points of Interest04:02 Example05:59 Time Delta Variation06:53 Algorithm Optimization07:32 Searches Between Chunks09:11 Hashes - Song Signatures10:19 Thank you!You can follow me on:My Website: interviewready.ioGithub: github.com/InterviewReady/system-design-resourcesTwitter: twitter.com/gkcs_#Algorithm #Audio #Hashing
System Design of a Ticket Booking System: BookMyShow Gaurav Sen 2024-09-24 | Let's try to design the ticket booking system of BookMyShow.You can read the blog here: interviewready.io/blog/ticket-booking-system-design
System Design Components List Gaurav Sen 2024-07-18 | Link: interviewready.io/blog/system-design-component-examples#systemdesign #interviews #examples
Software Systems: What is a load balancer? Gaurav Sen 2024-07-15 | Load balancing is distributing work evenly onto a set of resources.In this video, we explain the main reason for load balancing, along with some common algorithms:1. Round robin2. Geo-distributed3. Least Connection00:00 What is load balancing? 00:18 Example02:14 Resource Optimization03:40 Round robin04:39 Geo-distributed load balancing05:31 Least connections06:23 Practical example07:08 ConclusionLooking to ace your next interview? Try this System Design video course! π₯interviewready.io#LoadBalancer #Algorithms #SystemDesign
Interview Question: Tell me about yourself Gaurav Sen 2024-07-14 | She is the CEO of Edelweiss, and she isn't alone.Many people believe this is a good question to ask. Because it's open-ended and can be answered by anyone (no barrier to entry).Here is how you can answer the question: Highlight a story from your recent past, that explains why you are good for the applied job.If you talk about irrelevant things, the interviewer will have a great conversation for 45 minutes, but they will not move forward. Because they won't know if you are right for the role or not. -- From the interviewer's perspective.Unstructured questions have very biased evaluations. Someone who plays chess impresses me, someone who plays the guitar impresses you. We will not be able to come to a consensus on who is right for the role.You have limited time. You have a small set of questions to ask. The direction you set in the interview will matter for the next 2 years.Ask yourself: Will the answer to this question help me understand whether this candidate is to be accepted or rejected?A heuristic is okay. A substitute is okay. But random, unstructured interview questions are like shooting darts blindfolded.mcgill.ca/psychology/files/psychology/structuredinterviews.pdfhttps://www.hbs.edu/faculty/Pages/item.aspx?num=56600hbr.org/2016/04/how-to-take-the-bias-out-of-interviewsSo if you are the interviewee, answer the question with a smile while continuing to look for other roles. If you are the interviewer, replace this question with a more specific job-relevant one.00:00 The question00:16 How to answer01:14 Is this a good question?Behavioral Interview Course: interviewready.io/learn/behavioural-interview-course/Behavioural-Intro/course-intro-behavioralYou can find me at:Twitter: twitter.com/gkcs_#Interview #behavioralinterviewquestions
Caching in distributed systems: A friendly introduction Gaurav Sen 2024-07-13 | Caching is an amazingly effective technique to reduce latency. It helps build scalable, distributed systems. We first discuss what is a cache and why we use it. We then talk about the key features of a cache in a distributed system.Cache management is important because of its relation to cache hit ratios and performance. We talk about various scenarios in a distributed environment.Benefits of a cache:1. Saves network calls2. Avoids repeated computations3. Reduces DB loadDrawbacks of a cache:1. Can be expensive to host2. Potential thrashing3. Eventual consistencyCache Write Policies:1. Write-through2. Write-back3. Write-aroundCache Replacement Policies:1. LRU2. LFU3. Segmented LRU00:00 What is a cache?00:20 Caching use cases03:42 Caching limitations06:33 Drawbacks09:42 Cache PlacementCaching resources listed together (click on the resources tab):interviewready.io/learn/system-design-course/caches-deep-dive/caching-basics You can follow me on:Github: github.com/InterviewReady/system-design-resourcesInstagram: instagram.com/interviewready_LinkedIn: linkedin.com/company/interview-readyTwitter: twitter.com/gkcs_#Caching #DistributedSystems #SystemDesign
System Design of Zamzar File Convertor #SystemDesign #OnlineJudge Gaurav Sen 2024-07-11 | Me playing a system design game at InterviewReady for an online file convertor like Zamzar.System Design Game: interviewready.io/practice/system-design-judgeDesigning Data Intensive Applications Book: amzn.to/3SyNAOy#SystemDesign #InterviewReady #Coding
What is a CDN (Content Delivery Network)? Gaurav Sen 2024-07-09 | Content Delivery Networks are a bunch of servers spread across the globe to serve information. These networks are available on rent to deliver static content quickly to nearby users.Some examples of CDNs are Amazon CloudFront and the Akamai CDN. They are (relatively) cheap to rent and have high availability. They also provide pluggable algorithms to invalidate and fetch data.We discuss why content delivery networks are useful, and how they serve data with examples.References:System design course: interviewready.ioAmazon Study: conductor.com/academy/page-speed-resources/faq/amazon-page-speed-studyGoogle Study: glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html00:00 What is covered here?00:23 Use case00:57 Basic Idea01:20 Challenges with central caching02:32 Localized Geodistributed cache02:51 Benefits04:01 Content Delivery Networks04:56 CDN example - AWS CloudFront05:28 TLDRSocial Media:GitHub: github.com/InterviewReady/system-design-resourcesLinkedIn: linkedin.com/in/gkcs#SystemDesign #CDN #SoftwareEngineering
How to get promoted to Senior Software Engineer roles Gaurav Sen 2024-06-30 | What gets you promoted to senior engineer?-- As a senior, find simple solutions to problems.Simple is fast. Simple is extendible. Simple is easy to explain.-- As a senior, do things (including boring things) that have an impact.Business considerations come before our preferences, even before our learning opportunities.-- Senior engineers have a reputation for reliability. Maybe not 100% tested code, but they check for basic edge cases and monitor prod for problems.If you are trying to be promoted to senior, go for the most impactful projects, and do them without complaint. Yes, the code sucks. Yes, the vendor APIs suck. Damn, it's just a CRUD app with zero innovation.That's where the money is though. And once you have your manager's trust, you will see the most interesting projects come to you. Even better, you will be promoted quicker, having a bigger and bigger impact on your organization.Ironically, with the above attitude, you are likely to get the most exciting projects when they come up.Cheers!System Design Course at InterviewReady: interviewready.ioDesigning Data-Intensive Applications Book: amzn.to/3SyNAOyYou can follow me on:Github: github.com/InterviewReady/system-design-resourcesLinkedIn: linkedin.com/in/gkcsTwitter: twitter.com/gkcs_#SoftwareEngineering #Promotions
This is why Senior Software Engineers arent clearing interviews Gaurav Sen 2024-06-08 | Senior engineers are finding it harder and harder to clear interviews.Here is why.(I think it's a problem in our methodology).#SoftwareInterviews #SeniorEngineers
30 [Software Engineering] research papers you should read Gaurav Sen 2024-05-15 | You can find all the papers listed here:interviewready.io/blog/white-papers-worth-reading-for-software-engineersHere are 30 whitepapers worth reading for software engineers. These papers are dense sources of information.Most are from Google and Meta, but you see the occasional Amazon or LinkedIn paper. Keep an eye out for the topics you like.Gotta read them all!00:00 Whitepapers00:12 30. Scalability at what COST00:50 29. Detection of Silent Data Corruption01:20 28. Prophet01:57 27. Napa02:26 26. Cubrick03:00 25. Monitoring and Root Cause Analysis04:00 24. Presto04:40 23. Foundation DB05:16 22. F1 Lightning06:16 21. LinkedIn Distributed Graph07:15 20. TensorFlow07:30 19. The monolith strikes back - Istio08:25 18. HALP - YouTube CDN09:18 17. Monarch10:16 16. Dapper10:51 15. Gorilla DB11:17 14. Zanzibar11:55 13. Scaling Memcache at Facebook12:45 12. SIEVE vs. LRU13:40 11. Twitter cache clusters 14:28 10. Amazon Firecracker15:10 9. Spanner15:30 8. Dremel16:08 7. Google File System16:50 6. BigTable17:24 5. Hive Warehousing Solution17:52 4. Pregel18:18 3. TAO19:00 2. End-to-End Path CDN Performance19:52 1. Google Backend SubsettingUseful links:Github: github.com/InterviewReady/system-design-resourcesTwitter: twitter.com/gkcs_#SystemDesign #WhitePapers #Research
Launch Party: System Design Online Judge Gaurav Sen 2024-04-27 | You can try the judge here: interviewready.io/question-list/system-design-judgeIt's my dream to build an online system design judge for this community. A place to practice, learn and prepare for system design interviews. This judge has big aims:1. To democratize system design preparation -- provide an alternative to expensive mock interviews -- go beyond languages and time zones, with an automated judge -- reduce bias by bringing objectivity into system design tests2. Cater to all engineers, especially those who are new to system designLet's try it live!You can us here:Github: github.com/InterviewReady/system-design-resourcesInstagram: instagram.com/interviewready_LinkedIn: linkedin.com/company/interview-readyTwitter: twitter.com/gkcs_#SystemDesign #InterviewReady #OnlineJudge
Find the distance between friends/connections - LinkedIn System Design with @KeertiPurswani Gaurav Sen 2024-04-08 | Have you noticed a small icon on the right of LinkedIn profiles?It tells you how closely you are related to a user (1st, 2nd or 3rd-degree connection).These distances are calculated using graph algorithms. Specifically, LinkedIn uses Bi-directional BFS.This is a two-way algorithm where a search is started from both source and destination, and terminated midway. The total distance traveled from source+destination is the degree of the connection.But at scale, this solution brings challenges. Let's limit ourselves to finding 3rd-degree connections:1. Every query needs a graph traversal on our social network.2. A naive BFS would take O(n^3) for search. A bi-directional BFS needs O(n^2) for searching users + O(n^2) for merging results.3. To avoid performing these queries repeatedly, we cache the second-degree connections of users. This saves up on the first O(n^2) factor for subsequent queries.Great, but how do we store all this data in a single cache?Well, we can't. LinkedIn is forced to scale out, with multiple cache nodes storing second-degree connections. User-Connection data is sharded into caches according to user_id.But what if a machine crashes? All queries to the shards hosted by that machine will fail.To avoid this, we replicate shards into different machines. Even if one of the machines fails, the shard can be found on another machine.And here we meet our main villain: High Latency.We don't want to hit all the machines with our shards. We want to hit the SMALLEST possible number of machines, that contain all our shards.For the mathematically inclined, this is a set-cover problem. LinkedIn uses a modified greedy version of the problem to reduce its compute costs in half!The basic idea is to find machines that host most of the users who are your 2nd-degree connections, and avoid those that have few 2nd-degree connections.Less effort, more work done!00:00 Introduction00:35 Capacity Estimation01:20 Bruteforce SQL query04:00 Bidirectional BFS06:37 Caching connection data10:00 Potential databases11:11 Redundancy for fault tolerance12:12 Cold starts13:48 Removing connections14:37 Replicated Caches16:03 Ideal replication factor?17:25 SummaryHere is the paper: usenix.org/system/files/conference/hotcloud13/hotcloud13-wang.pdfReferences:Sharding: youtube.com/watch?v=5faMjKuB9bcSystem Design at InterviewReady: interviewready.ioDesigning Data-Intensive Applications Book: amzn.to/3SyNAOyYou can follow me on:Github: github.com/InterviewReady/system-design-resourcesTwitter: twitter.com/gkcs_#SystemDesign #InterviewReady #SocialNetworks
System Design of a Delivery System like Zomato with @KeertiPurswani Gaurav Sen 2024-03-30 | In this video, we look at what powers the delivery and tracking systems of e-commerce apps like Zomato, Swiggy, and Amazon. Along the way, we go through relevant spatial algorithms, network protocols, and design components used in location-based systems.Cheers!00:00 Problem Statement00:45 Requirements02:00 High-Level Design04:01 Capacity Estimation05:36 Location-Based Queries13:30 Tracking riders20:00 APIs24:05 Conclusion, thank you!Quad Trees: youtube.com/watch?v=OcUKFIjhKu0System Design Course at InterviewReady: get.interviewready.ioDesigning Data Intensive Applications Book: amzn.to/3SyNAOyYou can follow me on:Github: github.com/InterviewReady/system-design-resourcesLinkedIn: linkedin.com/company/interview-readyTwitter: twitter.com/gkcs_#SystemDesign #InterviewReady #Coding
System Design of GitHub Code Search - SDC Episode 1 with @KeertiPurswani Gaurav Sen 2024-03-16 | Github code search allows developers to view and edit code online. This is particularly useful when debugging code in remote locations (like on vacation)!GitHub manages permissions, storage, and retrieval through a set of services. In this video, we look at systems that power its search APIs.If you have any doubts or suggestions, please share them in the comments below.This is the first episode of the System Design Charcha series. Subscribe for notifications and updates!00:00 Problem Statement01:55 Capacity Estimations02:52 Brute Force Approach04:00 High Level Architecture06:30 API calls11:04 Form of Response Object?16:10 API flow17:40 Search Engine31:10 Summary33:10 Peek under the hood36:14 Final thoughts37:00 Thank you!References:Numbers every programmer should know: gist.github.com/jboner/2841832Github statistics: https://github.blog/2023-01-25-100-million-developers-and-counting/Useful Resources:InterviewReady: interviewready.ioDesigning Data-Intensive Applications Book: amzn.to/3SyNAOySocial Links:Github: github.com/InterviewReady/system-design-resourcesLinkedIn: linkedin.com/company/interview-readyTwitter: twitter.com/gkcs_#SystemDesign #InterviewReady #Coding
Date Night with System Design Gaurav Sen 2024-02-15 | Try the new SD question at InterviewReady: interviewready.io/learn/system-design-course/netflix_movie_onboarding/complete-the-design-netflix-content-distribution-networkWe are going to solve this new question at InterviewReady related to Valentine's Day!#SystemDesign #InterviewReady #valentinesday
Software Engineering Whitepapers that are worth reading. Live Readings at InterviewReady. Gaurav Sen 2024-01-18 | ...
System Design Recommendation #ytshorts @sudocode Gaurav Sen 2023-12-19 | A recommendation for system design study.
Latency Numbers Every Programmer Should Know - 1000x slow-down Gaurav Sen 2023-11-29 | These are some latency numbers every programmer should know.00:00 Memory and SSD00:13 Sequential Disk Read00:33 Roundtrip CA to Netherlands02:48 Roundtrip CA to India06:08 EndsGit Gist: gist.github.com/jboner/2841832System Design: interviewready.ioDesigning Data-Intensive Applications Book: amzn.to/3SyNAOyTwitter: twitter.com/gkcs_#SystemDesign #Software #Coding
Resource Lists for Senior Engineers: WhitePapers, Blogs and Videos Gaurav Sen 2023-11-27 | We share some fantastic technical papers to level up your skills!From must-read whitepapers to game-changing blogs, we've got it all covered. πGrab your favorite drink, kick back, and explore the latest tech trends together!00:00 WhitePapers01:16 Company cultures reflected in papers04:28 Example - Scalability at what COST06:06 Sample Walkthrough08:26 Engineering Blogs10:00 Difference between papers and blogs10:56 Example - Uber Engineering Blog11:30 Resource links!#softwareengineering #systemdesign #resources Resources:WhitePapers: interviewready.io/blog/white-papers-worth-reading-for-software-engineersSystem Design Course: interviewready.ioSystem Design Book: amzn.to/3SyNAOyGithub: github.com/InterviewReady/system-design-resourcesSystem Design Judge: interviewready.io/learn/system-design-course/hotstar-live-video-streaming-design/complete-the-design---live-streaming-appSocial Media:Twitter: twitter.com/gkcs_#SystemDesign #InterviewReady #Coding
The Most Advanced Multiplication Algorithms: Why Karatsuba and FFT beat high-school mathematics Gaurav Sen 2023-11-08 | For folks interested in multiplication algorithms (turns out this is a big deal in computer science and math).1. High school method - O(n^2)2. Karatsuba - O(n^1.58)3. Toom-Cook - O(n^1.46)4. FFT - O(n*log(n)*log(log(n)))The last algorithm is pretty complex. We explain the intuition behind it in this video.00:00 Who should watch this?00:18 Example Problem01:23 Karatsuba Algorithm02:38 Time complexity analysis03:57 Toom-Cook Algorithm04:48 Problem - Polynomial Multiplication05:27 Plotting points on a graph05:57 Plotting nth degree curve07:00 Multiplication Strategy07:28 Optimisation Strategy08:50 Choosing the right points09:25 Complex Numbers!10:26 The nth roots of unity12:56 The impact of complex numbers14:57 FFT Conclusion15:52 Thank you!InterviewReady: interviewready.ioKaratsuba: en.wikipedia.org/wiki/Karatsuba_algorithmToom-Cook: en.wikipedia.org/wiki/Toom%E2%80%93Cook_multiplicationFFT: en.wikipedia.org/wiki/Sch%C3%B6nhage%E2%80%93Strassen_algorithmYou can follow me on:Github: github.com/InterviewReady/system-design-resourcesTwitter: twitter.com/gkcs_#SystemDesign #InterviewReady #Coding
The painfully outdated practice of software interviews Gaurav Sen 2023-11-01 | Software Interviews need a revamp. Coding rounds are IRRELEVANT to our jobs.On average, software developers spend three months preparing for an interview.A good engineer has to choose between:1. Focusing on the job.2. Practice useless DSA questions.It's time to ask job-relevant questions and test candidates' WORKPLACE capabilities.At InterviewReady, we are building tools to test engineering capabilities like:1. Fault Tolerance2. Async Programming3. API DesignThese are fundamental concepts that all engineers should know.00:00 Agenda00:16 Coding Interviews are broken01:00 Company Perspective01:19 Are Coding Questions Relevant?02:24 Hiring in the 2000s02:56 Hiring in the 2010s03:44 Hiring in 2020s04:10 Proposed Solution Features05:00 Idea - Software Judge05:31 Idea - System Design Judge06:03 Launch Date06:16 Why are we doing this?06:51 Where we are now07:17 Thank you!We want to enable senior engineers to focus on their jobs rather than spending three months revising binary trees.We want to help young engineers improve through self-practice using our automated tools.We want to make hiring tests relevant.Save the date: November 21st, 2023.Register at InterviewReady for updates.System Design Judge V1: interviewready.io/learn/system-design-course/8-map-reduce-and-stream-processing/complete-the-design---recommendation-engineSend your suggestions to:contact[at]interviewready[dot]io#softwareengineering #softwareinterviews #codinginterview
How ANOMALY DETECTION works in time series using the Holt-Winters Algorithm Gaurav Sen 2023-10-26 | This video explains how anomalies are detected in a time-series graph.The algorithm's name is Holt-Winters. The idea is simple, and the results are often useful.Cheers!00:00 What is a time series?00:30 How can we find anomalies?01:00 Anomaly Detection Algorithm02:25 Repeated Differentiation03:36 Example04:14 Thank you!#anomalydetection #timeseries #monitoringInterviewReady: interviewready.ioDesigning Data Intensive Applications Book: amzn.to/3SyNAOyGithub: github.com/InterviewReady/system-design-resourcesLinkedIn: linkedin.com/company/interview-readyTwitter: twitter.com/gkcs_#SystemDesign #InterviewReady #Coding
20 AWS services you should know [as a Software Engineer] Gaurav Sen 2023-10-18 | Here are some AWS services you should know as a software developer or freelancer. These services can help you quickly build and manage the technical muscle of your organization!InterviewReady: interviewready.ioEach of these services solves a different business and tech problem. Let's dive in!00:00 Who should watch this?00:32 20. AWS Neptune Graph Database01:17 19. AWS Cognito Authenticator01:45 18. AWS TimeStream Database02:18 17. AWS OpenSearch Engine03:02 16. AWS Serverless Lambda04:16 15. AWS Kinesis Streams05:05 14. AWS Glacier Storage06:23 13. AWS Simple Queue Service07:06 12. AWS Data Migration Service08:10 11. AWS DynamoDB09:13 10. AWS ElastiCache09:55 9. AWS Simple Email Service10:57 8. AWS Elastic Load Balancer11:37 7. AWS CloudWatch Monitoring12:39 6. AWS API Gateway13:00 5. AWS Route53 DNS13:48 4. AWS CloudFront CDN14:41 3. AWS RDS Database15:45 2. AWS S316:27 1. AWS EC218:03 Engineering Perspective18:57 Conclusion19:27 Thank you!Designing Data-Intensive Applications: amzn.to/3SyNAOyGithub: github.com/InterviewReady/system-design-resourcesLinkedIn: linkedin.com/company/interview-readyTwitter: twitter.com/gkcs_#SystemDesign #InterviewReady #Coding
System Design of a Startup: How to host a website with AWS Gaurav Sen 2023-10-11 | Here is practical advice for software engineers working in small to medium organizations. We take a deep dive into the architecture of InterviewReady to understand how tech works in startups.InterviewReady: interviewready.ioThis video has been made after studying the tech of more than 60 startups, most of whom face similar problems!00:00 Who is this for?00:30 War Story00:52 Getting a Website02:50 Outsourcing04:52 Migration to AWS05:57 CICD pipeline06:35 Cost Optimisations08:39 Frontend Deployments10:37 Backend Deployments12:19 Overall System Design14:11 Payments Webhooks15:23 Amazon SES15:43 AWS CloudWatch16:05 Benefits of moving to AWS16:45 Practical Considerations18:18 TLDR - Conclusion19:46 Thank you!Github: github.com/InterviewReady/system-design-resourcesTwitter: twitter.com/gkcs_#Startups #architecture #InterviewReady
20 Whitepapers that changed the world [For Senior Software Engineers] Gaurav Sen 2023-10-04 | Here are some whitepapers that you must read as a software engineer. These technologies have changed the world and inspired hundreds of systems through their ideas to tackle scale, reliability, and performance.WhitePapers: interviewready.io/blog/white-papers-worth-reading-for-software-engineersWhitepapers have the benefit of discussing practical implementation details and the related tradeoffs to engineering products. The best engineers worldwide have written them after years of work, so don't miss any of them!00:00 Who should watch this?00:29 Memcached Design Tradeoff01:40 20. TikTok Monolith02:55 19. Meta FlexiRaft03:54 18. Google Spanner04:30 17. Meta Minesweeper05:33 16. Apache Cassandra06:07 15. Apple FoundationDB06:36 14. Amazon AuroraDB07:11 13. Google Pregel07:45 12. Google Dapper08:37 11. Google Chubby09:12 10. Google Megastore10:04 9. Google Bigtable10:44 8. Google Map-Reduce11:38 7. Google File System12:25 6. Meta TAO13:05 5. Meta Memcached13:40 4. Google Monarch14:23 3. Meta GorillaDB15:22 2. Amazon DynamoDB15:52 1. Google Zanzibar17:00 Links in DescriptionDesigning Data-Intensive Applications Book: amzn.to/3SyNAOyGithub: github.com/InterviewReady/system-design-resources#SystemDesign #WhitePapers #SeniorEngineer
Design Patterns for High Availability: What gets you 99.999% uptime? Gaurav Sen 2023-09-27 | In this video, we discuss the topic of availability in distributed systems. We categorize organizations based on their acceptable levels of availability, ranging from startups to mature companies aiming for five to six nines of availability.InterviewReady: interviewready.ioWe share a real-world example of a startup facing availability challenges with its database hosted in the wrong region. The solution involves migrating the database to a more suitable location and implementing a step-by-step process to minimize downtime.Here are five principles for building highly available systems:1. Simplicity over Perfection2. Downtime Over Loss3. Lesser Moving Parts4. Chaos Engineering5. Incident Reports and Root Cause AnalysisWe also touch upon fault tolerance strategies such as redundancy, load balancing, and database replication to ensure high availability in distributed system components. Engineers should either leverage existing highly available systems or adopt a principled approach to building and maintaining availability in their systems.00:00 Who is this video for?00:20 The 9s of availability02:43 War Story at InterviewReady06:14 Principles for High Availability09:05 Design Patterns for Availability12:31 Conclusion12:49 Thank you!Designing Data-Intensive Applications Book: amzn.to/3SyNAOyYou can follow me on:Github: github.com/InterviewReady/system-design-resourcesInstagram: instagram.com/interviewready_LinkedIn: linkedin.com/company/interview-readyTwitter: twitter.com/gkcs_#HighAvailability #SystemDesign #SoftwareEngineering
System Design PRACTICE Platform - Online Judge for High Level Design Gaurav Sen 2023-09-23 | Learn system design by practicing it.00:00 Ignore - waiting05:57 Announcement - Online Design Judge07:46 GAME - Live Streaming16:08 Feedback18:07 GAME - Recommendation Engine26:30 Who should play this game?27:35 Break28:20 GAME - Chess Website38:00 Ignore - Screen Share Fail41:20 Anomaly Detection Algorithm43:50 How to play - Instructions47:00 GAME - Email Design55:06 Live FeedbackLink: interviewready.io/learn/system-design-course/hotstar-live-video-streaming-design/complete-the-design---live-streaming-appYou can follow me at:Twitter: twitter.com/gkcs_#SystemDesign #OnlineJudge #Practice
Tech that DOESNT WORK for Start-ups and Mid-sized companies Gaurav Sen 2023-09-13 | Tech advice is like refactoring. It's easy to talk about, but hard to implement.In small companies, picking up low-impact projects can derail the entire organization. In this video, we look at what are the worst pieces of tech advice for startups, and how to avoid them.At the end of this video, you should be able to detect superficial hype-fueled advice from a distance. The overall idea of the video can be summarised in two acronyms: KISS and YAGNI.The video explains the above with examples :D00:00 Who is this for?00:51 5. Containers02:20 4. NoSQL04:35 3. Microservices05:59 2. Quality Obsession08:00 1. Automation11:13 TLDR11:38 Thank you! System Design at InterviewReady: interviewready.ioDesigning Data-Intensive Applications Book: amzn.to/3SyNAOyYou can follow me on:Github: github.com/InterviewReady/system-design-resourcesInstagram: instagram.com/interviewready_LinkedIn: linkedin.com/company/interview-readyTwitter: twitter.com/gkcs_#Startups #NoSQL #Containers
Authorization across Distributed Systems: The OAuth Protocol Gaurav Sen 2023-09-06 | Websites are like castles, with large moats around them. You need a password at the gate to get in.And the average person has to remember 27 passwords!Remembering passwords is a pain. About 30% of all customer queries are "How do I reset my password?"So websites found a unique solution: outsource the authentication problem to the πππ castles.---------------------------------------------Instead of asking users to enter an email and password, websites now ask users to "connect" them with Google for registration.πΏππππ ππππ£πππ: π»ππ¦, πππ’ππ π¦ππ’ π‘πππ ππ π‘βπ ππππ ππ πππ@πππππ.πππ? πβππ¦ πππππ π‘π ππ π¦ππ’π π’π ππ.πΊπππππ: πππππ π π βππ€ ππ π‘βπππ π πππππ ππππ’ππ π‘.πΏππππ ππππ£πππ: ππ’ππ, βπππ π¦ππ’ ππ.πΊπππππ: πππ , π‘βππ‘'π ππ’π π’π ππ. πβπππ ππππ ππ π½πβπ π·ππ.πΏππππ ππππ£πππ: πΊππππ‘. πΌ'ππ πππ‘ π‘βππ ππ. πΆππ’ππ π¦ππ’ π βπππ π‘βπππ πππππππ ππππ‘π’ππ π‘ππ?πΊππππ: πππππ¦, ππ’π‘ πΌ πππ'π‘ π ππ π‘βππ‘ ππ π‘βπ π πππππ ππππ’ππ π‘.πΏππππ ππππ£πππ: πππ£ππ ππππ, π‘βππππ .The user is now authenticated, and a session token can be sent for further auth requests. This process of outsourcing user authentication (technically authorization, since the user authorized you to view their name) is called OAuth.---------------------------------------------Third-party sign-in reduces login hesitance, ease of mobile registration, and password reset issues.It also consolidates data power into a few companies, which know exactly which websites you visited to tailor your ads (Did you register on FirstCry? Let me show you a diaper ad).You can learn more about OAuth, SSO, and Access Control Lists at InterviewReady.Cheers!00:00 What will we learn?00:20 The Problem with Passwords01:25 OAuth Flow04:22 War story: OAuth Doubles Signups06:43 Advantages of OAuth08:55 Drawbacks of OAuth11:31 Conclusion12:13 Distributed Security Terms15:30 Thank you!System Design at InterviewReady: interviewready.ioUse the special DISCOUNT coupon of "HELLOWORLD" to avail an exclusive YouTuber channel offer!#OAuth #Security #DistributedSystems
How Stored Procedures make databases FAST Gaurav Sen 2023-08-31 | Stored Procedures are functions which can run on a database. Here, we discuss how stored procedures work, when and why they should be used, and the potential drawbacks of this approach.We illustrate these points with a real-world example: a war story!00:00 What are stored procedures?00:40 Where and how do they run?01:25 Real-world example02:30 Request-Response Flow03:20 First Suggestion - BiDirectional BFS04:14 Cache results05:30 Debugging06:02 The Problem07:08 Solution - Stored Procedures08:03 Benefits of Stored Procedures09:59 Drawbacks of Stored Procedures12:00 Thank you!System Design Course at InterviewReady: interviewready.ioYou can follow me on:Github: github.com/InterviewReady/system-design-resourcesInstagram: instagram.com/interviewready_LinkedIn: linkedin.com/company/interview-readyTwitter: twitter.com/gkcs_#Databases #StoredProcedures #QueryOptimization
Garbage Collection Algorithms in Java: Concurrent Updates with Optimistic Locking - Part III Gaurav Sen 2023-08-24 | We discuss how concurrent compaction using Compare and Swap (CAS) operations helps speed up garbage collection algorithms and reduce response latencies.This is the last piece of the puzzle in the garbage collection series. Cheers!Garbage collection is a way of automatic memory management provided by modern programming languages like Java, Go, Python and JavaScript.00:00 Who should watch this00:43 Recap - Garbage Identification02:00 The hunt for low latency02:33 Generational Hypothesis03:27 Concurrent Mark and Sweep04:05 Partitioning into regions04:50 Part III - Concurrent Compaction05:17 Problem: Lost Writes06:28 Solution: Forwarding Pointers08:21 Garbage Collectors in Production08:50 Conclusion09:50 Thank you!Garbage Collection in Java Series: youtube.com/playlist?list=PLMCXHnjXnTnsrjHPvnvXdQV3_6lFfEYi3System Design Course at InterviewReady: interviewready.ioYou can follow me on:Github: github.com/InterviewReady/system-design-resourcesLinkedIn: linkedin.com/company/interview-readyTwitter: twitter.com/gkcs_#GarbageCollection #Java #ConcurrentCompaction
How 100 milliseconds cost Amazon 3 BILLION DOLLARS: Latency, Concurrency and Parallelism Gaurav Sen 2023-08-19 | Can a 500-millisecond increase in page load times cost a website 20% of its incoming search traffic? Yes. The impact of latency on user experience is disproportionately large, and both front-end and back-end software engineers are expected to know how to write async programs.This helps businesses respond to user demands as soon as possible. There are two types of async programming techniques:1. Concurrency2. ParallelismIn this video, we explain the benefits and drawbacks of each technique using manufactured and real-world examples. 00:00 Who should watch this00:20 What are Async Processes?00:50 The Impact of Reduced Latency01:54 Concurrency Example02:28 Parallelism Example03:02 Putting them together03:40 Drawbacks of Async Processes04:26 The Impact of Increased Complexity05:36 Real-world examplesBlog Link: developer.chrome.com/blog/inside-browser-part3System Design Course at InterviewReady: interviewready.ioYou can follow me on:Github: github.com/InterviewReady/system-design-resourcesLinkedIn: linkedin.com/company/interview-readyTwitter: twitter.com/gkcs_#SystemDesign #InterviewReady #Coding
Impeccable API Design: What you MUST CONSIDER before deploying APIs to production Gaurav Sen 2023-08-16 | In this video, we talk about what APIs are, how they help software engineers expose data for query and updates, and what are some best practices when exposing APIs on the server side.We touch upon various topics including:00:00 What is an API?00:27 Indian Government Aadhaar APIs00:40 How do APIs work?01:16 REST vs. GraphQL01:31 Webhooks02:04 Writing Good APIs02:10 What is Atomicity?02:53 What is Idempotency?03:35 Error Codes for HTTP APIs05:28 OpenAPI Specs with Swagger05:54 War Story - Aadhaar API IntegrationWe end with a war story of API design gone wrong. Let's make our software clients live in a better place :pAre you looking to master system design?InterviewReady: interviewready.ioYou can follow me on:Github: github.com/InterviewReady/system-design-resourcesLinkedIn: linkedin.com/company/interview-readyTwitter: twitter.com/gkcs_#SystemDesign #InterviewReady #Coding
What is an API Gateway? Gaurav Sen 2023-08-10 | What's an API gateway? It is a server that exposes internal service APIs to external client devices, using a protocol like HTTP.The gateway also uses design patterns like REST and GraphQL to expose APIs, which allow clients to get and update data through servers with databases.In this video, we will talk about API gateways, their use cases, and how they work with CDNs and DNSes to ensure fast and reliable user request handling.Some of their most common use cases are:1. Authentication and Authorization2. Request Transformation3. Request Validation4. Rate Limiting5. Routing6. Load BalancingLooking to ace your system design interviews?This course has over 300+ detailed videos on systems design.InterviewReady: interviewready.ioYou can follow me on:Github: github.com/InterviewReady/system-design-resourcesLinkedIn: linkedin.com/in/gkcsTwitter: twitter.com/gkcs_00:00 What is an API Gateway?01:02 Authentication on the Gateway01:30 Request Transformation02:07 Request Validation02:43 Rate Limiting03:09 Request Routing04:20 Load Balancing05:32 Architecture Overview06:00 DNS07:50 CDNs09:30 Architecture Overview with AWS10:31 War Story - Restart EVERY Week!#SystemDesign #InterviewReady #Coding
Career Choices: Four differences between Managers and ICs Gaurav Sen 2023-06-11 | This video compares individual contributors and managerial roles. The critical points covered include the following:Evaluating individual contributors and managers based on specific metrics and performance.Individual contributors' roles include technical knowledge, implementing solutions, and gathering product requirements.The role of managers focuses on influence, integrity, transparency, fairness, and helping teammates succeed.Handling uncertainty as a manager, dealing with volatility, uncertainty, complexity, and ambiguity (VUCA).People management skills for managers, including hiring, onboarding, and creating a conducive work environment.The impact of individual contributors across teams (horizontal impact) and managers within their vertical impact.Managers' Competencies include task management, presence, ownership, and monitoring team health.Mitigating risks as a manager, including avoiding single points of failure and minimizing risks to team success.00:00 Introduction00:38 Evaluation Metrics02:05 Day in the life - IC02:27 Influence as an Engineering Manager03:38 What is Transparency?04:13 What is Fairness?04:49 Willingness to help05:45 Managing Uncertainty - VUCA07:18 People - Hiring, Appraisals and leaves07:50 Impact on teams08:27 Competencies08:40 Task Management09:09 Presence09:35 What is Ownership?10:08 Team Health - Motivation10:32 Team Strength11:05 Managing Risks12:00 Share your feedback!InterviewReady: interviewready.ioDesigning Data-Intensive Applications Book: amzn.to/3SyNAOyVUCA: hbr.org/2014/01/what-vuca-really-means-for-youLinkedIn: linkedin.com/company/interview-ready#IndividualContributor #EngineeringManagement #EM
How custom banners are shown INSTANTANEOUSLY: Contextual Advertising at Scale Gaurav Sen 2023-06-05 | This is a lesson from the free chapter of "live-streaming app system design" at InterviewReady.Original InterviewReady Video: interviewready.io/learn/system-design-course/hotstar-live-video-streaming-design/virtual-advertising-for-live-sports-eventsContext-based advertising during sports events is a big challenge, especially when serving millions of users with diverse interests.In this video, we dive into what works behind the scenes, like recording calibrations, object recognition, ad ranking, user segmentation and video rendering.The question answered here is: How do you serve dynamic ads to different user cohorts in real-time?Designing Data-Intensive Applications Book: amzn.to/3SyNAOyVirtual Advertising articles:forbes.com/sites/robertkidd/2018/08/24/how-virtual-advertising-is-helping-brands-reach-international-soccer-fansen.wikipedia.org/wiki/Virtual_advertisingVirtual Advertising Video: twitter.com/UltraLinx/status/1411807406728925196Supponor: supponor.comYou can follow me on:Github: github.com/InterviewReady/system-design-resourcesInstagram: instagram.com/interviewready_LinkedIn: linkedin.com/company/interview-readyTwitter: twitter.com/gkcs_Chapters.00:00 Intro00:07 What are contextual ads?00:44 Recommendation System01:44 Rendering Options02:22 Keying out with green screens03:17 Fixed Camera Stands04:29 Keeping the players in video06:00 Object Recognition using AI07:08 Thank you!#SystemDesign #Contextual #Advertisements
Running a bootstrapped startup for 3 Years: The story of InterviewReady! Gaurav Sen 2023-06-02 | It's been three years of running the awesome bootstrapped Edtech startup that we all know as InterviewReady!Thank you for your support and love! We are here for the long run :D00:00 Introduction00:35 What is this video about?01:18 What does bootstrapping mean?04:05 Were there any tough times?07:15 What makes you different from other EdTechs?10:13 What was the vision when you started?13:11 What's your daily routine?16:11 Advice for young entrepreneurs18:02 Advice for successful entrepreneurs18:55 How did you inform your family?21:10 How did you get such a high product rating?24:50 Any setbacks?26:48 Rapid Fire!27:00 Texting vs. Talking27:07 Favourite day of the week27:20 Recent whitepaper you read27:35 Recent song you are listening to28:04 Who inspires you?28:27 What's for dinner tonight?!InterviewReady: interviewready.ioYou can follow us on:Github: github.com/InterviewReady/system-design-resourcesInstagram: instagram.com/interviewready_LinkedIn: linkedin.com/company/interview-readyTwitter: twitter.com/gkcs_#SystemDesign #InterviewReady #Coding
Dark Mode: Pilot Launch at InterviewReady Gaurav Sen 2023-05-25 | Hey guys, I am proud that we have this exclusive pilot feature to share with you:Dark mode for videos.At InterviewReady, we care about your learning and comprehension. Some of us study at night, and bright contrasting lights can be challenging to adjust to.So, this is what we have (please play the attached video).Cheers!#learning #experience #darkmode Link: interviewready.io/learn/system-design-course/consistency-in-distributed-systems/what-is-data-consistency
Zerodha Stock Broker System Design with @KeertiPurswani Gaurav Sen 2023-05-17 | In this video, we discuss the high-level design of Zeroda, a stock broker app. We start by talking about the app's capabilities, including features like watchlists, market depth, stock performance analysis, buying and selling stocks, and portfolio management. We also mention the order book and trade book, which track orders and transactions.We explain that Zeroda acts as a broker between clients and stock exchanges like the National Stock Exchange (NSE) and Bombay Stock Exchange (BSE). Zeroda takes orders from clients and places them with the exchanges. The market depth feature allows users to see the number of orders and their prices in the market. This information helps users make informed decisions about buying or selling stocks.We discuss different types of orders, such as market orders and limit orders. Market orders involve buying or selling at the current market price, while limit orders allow users to set a specific price at which we want to buy or sell stocks. We also mention the importance of real-time stock price updates and the need for a fast and reliable system architecture.Towards the end of the video, we briefly touch on the challenges of connecting to stock exchanges, including the cost of acquiring permissions and the need for reliable communication lines. We mention that exchanges strive for fairness by ensuring equal wire lengths for all brokers to minimize the latency of updates.The video sets the stage for a discussion on designing the high-level system architecture of Zeroda, focusing on real-time data updates and communication with stock exchanges.00:00 Intro00:48 Stockbroker Requirements12:11 Joke - Message queues aren't free?12:44 Some trivia about stock machines15:00 WebSockets for stock updates?20:00 Capacity estimation of updates23:30 Order Management26:09 Some more trivia - War Story28:30 Deciding on service boundaries36:36 Message Queue - needed?40:55 Cache - Global caching?44:39 Executing orders50:53 Real-life connection purchasing52:52 Deep dive into watch lists59:14 Eventual consistency for updates?1:02:40 TCP vs. UDP for watch lists?1:03:48 Validating the architecture1:05:44 InterviewReady Discount!System Design Course at InterviewReady: https://interviewready.io?_aff=KEERTIDesigning Data-Intensive Applications Book: amzn.to/3SyNAOyKeerti is at:YouTube: @KeertiPurswani LinkedIn: linkedin.com/in/keertipurswaniInstagram: instagram.com/keerti.purswaniYou can follow me on:Github: github.com/InterviewReady/system-design-resourcesInstagram: instagram.com/interviewready_LinkedIn: linkedin.com/company/interview-readyTwitter: twitter.com/gkcs_#SystemDesign #InterviewReady #Coding
Will Developers be replaced by ChatGPT and other AI tools (like Github Copilot)? Gaurav Sen 2023-04-05 | Many are wondering if their jobs will be taken over by machines. In the software development industry, this concern with AI is growing as ChatGPT gets popular.ChatGPT is capable of generating code to a variety of input problems (it can even solve leetcode). Github Copilot is an AI-powered coding assistant that suggests code based on comments!While this is impressive, the question remains: will they replace human developers?The answer is not straightforward.Slido Link: https://app.sli.do/event/n6fbRxNiUxkb8WUoZDjQh8InterviewReady: interviewready.ioDesigning Data-Intensive Applications Book: amzn.to/3SyNAOyYou can follow me on:Github: github.com/InterviewReady/system-design-resourcesInstagram: instagram.com/interviewready_LinkedIn: linkedin.com/company/interview-readyTwitter: twitter.com/gkcs_#SystemDesign #InterviewReady #Coding
Task Management - The one thing you must know as a software engineer Gaurav Sen 2023-03-29 | In this video, Gaurav Sen explains that software engineers are usually assigned to a project or task from a business, whether in a large company or a startup. He emphasizes the importance of documenting and tracking tasks, even in large organizations, and introduces the use of task management systems like JIRA to do so. These tools allow teams to discuss and track tasks, and provide a stage management lifecycle system for tasks to go through different stages, such as testing and code review, before being marked as complete.00:00 Software Engineering Series00:19 First task scenario01:30 Tracking Tasks02:00 Task Management Systems02:21 Use cases03:06 Ticketing managementInterviewReady: interviewready.ioDesigning Data-Intensive Applications Book: amzn.to/3SyNAOyYou can follow me on:Github: github.com/InterviewReady/system-design-resourcesInstagram: instagram.com/interviewready_LinkedIn: linkedin.com/company/interview-readyTwitter: twitter.com/gkcs_#SoftwareEngineering #TaskManagement #JIRA
Microservices vs Monoliths Tradeoffs Gaurav Sen Gaurav Sen 2023-03-27 | System Design Course at InterviewReady: interviewready.ioDesigning Data-Intensive Applications Book: amzn.to/3SyNAOyYou can follow me on:Github: github.com/InterviewReady/system-design-resourcesInstagram: instagram.com/interviewready_LinkedIn: linkedin.com/company/interview-readyTwitter: twitter.com/gkcs_#Monoliths #Microservices #SystemDesign
