Die Türme des Februar • Musical (Teil 2) Martin Kleppmann 2019-10-04 | Ein musikalisches Drama nach dem Buch “Die Türme des Februar” von Tonke Dragt.Weitere Informationen unter: martin.kleppmann.com/die-tuerme-des-februarDieses Video ist Teil 2 (Akt IV, V). Teil 1 (Akt I, II, III) befindet sich hier: youtube.com/watch?v=Xjd31R_3TXMText und Musik: Martin KleppmannArrangement und Leitung: Norbert LocherRegie: Ines KreutterLive-Aufzeichnung der Aufführung am 13. Februar 2007 in der Stadthalle Aalen. Eine Kooperation des Pfarrwiesen-Gymnasiums Sindelfingen, des Theodor-Heuss-Gymnasiums Aalen, und der Oratorienvereinigung Aalen. Siehe martin.kleppmann.com/die-tuerme-des-februar/mitwirkende.html für die Liste aller Mitwirkenden.
Rêverie by Claude Debussy Martin Kleppmann 2023-04-05 | Plenty of wrong notes, which just make it all the more authentic!
Discussion session on Peritext, a CRDT for rich text Martin Kleppmann 2022-09-16 | A discussion session for my Patreon supporters, about the paper "Peritext: A CRDT for Collaborative Rich Text Editing" with Geoffrey Litt, Sarah Lim, and Peter van Hardenberg.Paper: inkandswitch.com/peritext/static/cscw-publication.pdfYou can join future discussion sessions by supporting me on Patreon: patreon.com/martinkl
Automerge: a new foundation for collaboration software Martin Kleppmann 2021-11-28 | Local-first software is an effort to make collaboration software less dependent on cloud services, and Automerge is an open-source library for realising local-first software. In this talk I explain our motivation for creating Automerge, and map out 7 years worth of research projects that are feeding into this project.Recording of a talk given at the University of Cambridge SRG Seminars on 25 Nov 2021.cl.cam.ac.uk/research/srg/seminars
Distributed Systems 6.2: Raft Martin Kleppmann 2021-10-13 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2021/ConcDisSys/materials.htmlNOTE: There is an earlier version of this video at youtube.com/watch?v=IPnesACYRck but it contains several mistakes that are corrected in this video.
Thinking in Events: From Databases to Distributed Collaboration Software (ACM DEBS 2021) Martin Kleppmann 2021-07-01 | Keynote by Martin Kleppmann at the 15th ACM International Conference on Distributed and Event-based Systems (ACM DEBS 2021)Paper: dl.acm.org/doi/10.1145/3465480.3467835Alternative paper link: martin.kleppmann.com/papers/debs21-keynote.pdfSlides: speakerdeck.com/ept/thinking-in-events-from-databases-to-distributed-collaboration-softwareIf you found this useful, please consider becoming a supporter on Patreon: patreon.com/martinkl
Maple Leaf Rag (played by beginner) Martin Kleppmann 2021-01-24 | My lockdown project for the last year has been to learn the piano. Thanks to my partner for teaching me. Here is my attempt at Scott Joplin's Maple Leaf Rag, with plenty of wrong notes — that's how you know it's played by human, not by computer.
Distributed Systems 8.1: Collaboration software Martin Kleppmann 2020-11-16 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 8.2: Googles Spanner Martin Kleppmann 2020-11-16 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 7.3: Eventual consistency Martin Kleppmann 2020-10-28 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 7.2: Linearizability Martin Kleppmann 2020-10-28 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 7.1: Two-phase commit Martin Kleppmann 2020-10-28 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 6.2: Raft Martin Kleppmann 2020-10-28 | NOTE: There are some mistakes in this video. Please watch this one instead, in which the bugs are fixed: youtube.com/watch?v=uXEYuDwm7e4This video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge.Accompanying lecture notes: cl.cam.ac.uk/teaching/2021/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiB
Distributed Systems 6.1: Consensus Martin Kleppmann 2020-10-28 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 5.3: State machine replication Martin Kleppmann 2020-10-28 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 5.2: Quorums Martin Kleppmann 2020-10-28 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 5.1: Replication Martin Kleppmann 2020-10-28 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 4.3: Broadcast algorithms Martin Kleppmann 2020-10-28 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 4.2: Broadcast ordering Martin Kleppmann 2020-10-28 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 4.1: Logical time Martin Kleppmann 2020-10-28 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 3.3: Causality and happens-before Martin Kleppmann 2020-10-28 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 3.2: Clock synchronisation Martin Kleppmann 2020-10-28 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 3.1: Physical time Martin Kleppmann 2020-10-28 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 2.4: Fault tolerance Martin Kleppmann 2020-10-28 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 2.3: System models Martin Kleppmann 2020-10-28 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 2.2: The Byzantine generals problem Martin Kleppmann 2020-10-28 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 2.1: The two generals problem Martin Kleppmann 2020-10-28 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 1.3: RPC (Remote Procedure Call) Martin Kleppmann 2020-10-28 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 1.2: Computer networking Martin Kleppmann 2020-10-28 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
Distributed Systems 1.1: Introduction Martin Kleppmann 2020-10-28 | Accompanying lecture notes: cl.cam.ac.uk/teaching/2122/ConcDisSys/dist-sys-notes.pdfFull lecture series: youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiBThis video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge. It is preceded by an 8-lecture course on concurrent systems for which videos are not publicly available, but slides can be found on the course web page: cl.cam.ac.uk/teaching/2122/ConcDisSys/materials.html
CRDTs: The Hard Parts Martin Kleppmann 2020-07-06 | A talk on the latest research on CRDTs, originally given at the Hydra distributed computing conference on 6 July 2020.References: martin.kleppmann.com/2020/07/06/crdt-hard-parts-hydra.htmlSlides: speakerdeck.com/ept/crdts-the-hard-partsAbstract:Conflict-free Replicated Data Types (CRDTs) are an increasingly popular family of algorithms for optimistic replication. They allow data to be concurrently updated on several replicas, even while those replicas are offline, and provide a robust way of merging those updates back into a consistent state. CRDTs are used in geo-replicated databases, multi-user collaboration software, distributed processing frameworks, and various other systems.However, while the basic principles of CRDTs are now quite well known, many challenging problems are lurking below the surface. It turns out that CRDTs are easy to implement badly. Many published algorithms have anomalies that cause them to behave strangely in some situations. Simple implementations often have terrible performance, and making the performance good is challenging.In this talk Martin goes beyond the introductory material on CRDTs, and discusses some of the hard-won lessons from years of research on making CRDTs work in practice.Bio:Dr Martin Kleppmann is a researcher in distributed systems at the University of Cambridge, and author of the acclaimed "Designing Data-Intensive Applications" (O'Reilly Media, 2017). He mainly works on collaboration software, CRDTs, and formal verification of distributed algorithms. Previously he was a software engineer and entrepreneur at Internet companies including LinkedIn and Rapportive, where he worked on large-scale data infrastructure.martin.kleppmann.comtwitter.com/martinkl
Tied the Rain into a Bow Martin Kleppmann 2020-05-03 | Lyrics:We used to think we were running the showThe apex predator, the ones in the knowNow one tenth of a micron has taken controlThirty thousand bases bring lockdown and furlough and doleWe tied the rain into a bowEnjoyed the sunshine through the windowForces of nature, barely containedTook all away, the essential remainedLocked indoors from the invisible floodThis spring was not nipped in the budTumultuous days… are fears overblown?Our civic duty is drinking at homeWe're in this together, this isn't a warAttempting homeschooling and video meetings galoreWe tied the rain into a bowEnjoyed the sunshine through the windowForces of nature, barely containedTook all away, the essential remainedLocked indoors from the invisible floodThis spring was not nipped in the budThe under-appreciated is now suddenlyAn essential worker and a crucial employeeNow pay them a fair wage and give them PPENo visitors allowedNo grandchild at this sideNo miracles, no cureA shortage nationwideTwelve-hour shiftsA lot to askNo brave smile hiddenBehind her maskWe tied the rain into a bowEnjoyed the sunshine through the windowForces of nature, barely containedTook all away, the essential remainedLocked indoors from the invisible floodThis spring was not nipped in the budWords, music, vocals: Martin KleppmannPiano: Rita ChowSheet music: martin.kleppmann.com/music/rain.pdf© 2020 Martin Kleppmann. This work is made freely available under a Creative Commons Attribution-ShareAlike 4.0 International License.
Correctness proofs of distributed systems with Isabelle/HOL Martin Kleppmann 2019-12-11 | This is an extended version of a talk given at Strange Loop 2019 and Code Mesh London 2019.Isabelle code: gist.github.com/ept/b6872fc541a68a321a26198b53b3896bSlides & more info: martin.kleppmann.com/2019/09/14/strange-loop-isabelle.htmlAbstract:Testing systems is great, but tests can only explore a finite set of inputs and behaviours. Many real systems, especially distributed systems, have a potentially infinite state space. If you want to be sure that a program does the right thing in all possible situations, testing is not sufficient: you need proof. Only mathematical proof, e.g. by induction, can cover an infinite state space.Pen-and-paper proofs are well established in mathematics, but they need to be laboriously checked by hand, and humans sometimes make mistakes. Automated theorem provers and computerised proof assistants can help here. This talk introduces Isabelle/HOL, an interactive proof assistant that can be used to formally prove the correctness of algorithms. It is somewhat like a programming language and interactive console for proofs.In this talk we will explore how Isabelle can be used to analyse algorithms for distributed systems, and prove them correct. We will work through some example problems in live demos, and prove real theorems about some simple algorithms. Proof assistants still have a pretty steep learning curve, and this talk won’t be able to teach you everything, but you will get a sense of the style of reasoning, and maybe you will be tempted to try it for yourself.
Die Türme des Februar • Musical (Teil 1) Martin Kleppmann 2019-10-04 | Ein musikalisches Drama nach dem Buch “Die Türme des Februar” von Tonke Dragt.Weitere Informationen unter: martin.kleppmann.com/die-tuerme-des-februarDieses Video ist Teil 1 (Akt I, II, III). Teil 2 (Akt IV, V) befindet sich hier: youtube.com/watch?v=U1rWzPewB-YText und Musik: Martin KleppmannArrangement und Leitung: Norbert LocherRegie: Ines KreutterLive-Aufzeichnung der Aufführung am 13. Februar 2007 in der Stadthalle Aalen. Eine Kooperation des Pfarrwiesen-Gymnasiums Sindelfingen, des Theodor-Heuss-Gymnasiums Aalen, und der Oratorienvereinigung Aalen. Siehe martin.kleppmann.com/die-tuerme-des-februar/mitwirkende.html für die Liste aller Mitwirkenden.
Data liberation and data integration with Kafka — Strata New York 2015 Martin Kleppmann 2015-11-01 | Even the best data scientist can't do anything if they cannot easily get access to the necessary data. Simply making the data available is step 1 towards becoming a data-driven organization. In this talk, we'll explore how Apache Kafka can replace slow, fragile ETL processes with real-time data pipelines, and discuss best practices for data formats and integration with existing systems.Talk given at Strata+Hadoop World New York, 30 September 2015.http://martin.kleppmann.com/2015/09/30/data-liberation-with-kafka-at-strata.htmlhttp://conferences.oreilly.com/strata/big-data-conference-ny-2015/public/schedule/detail/42723Slides: speakerdeck.com/ept/data-liberation-and-data-integration-with-kafkaAbstract:Apache Kafka is a popular open source message broker for high-throughput real-time event data, such as user activity logs or IoT sensor data. It originated at LinkedIn, where it reliably handles around a trillion messages per day.What is less widely known: Kafka is also well suited for extracting data from existing databases, and making it available for analysis or for building data products. Unlike slow batch-oriented ETL, Kafka can make database data available to consumers in real-time, while also allowing efficient archiving to HDFS, for use in Spark, Hadoop or data warehouses.When data science and product teams can process operational data in real-time, and combine it with user activity logs or sensor data, that turns out to be a potent mixture. Having all the data centrally available in a stream data platform is an exciting enabler for data-driven innovation.In this talk, we will discuss what a Kafka-based stream data platform looks like, and how it is useful:* Examples of the kinds of problems you can solve with Kafka * Extracting real-time data feeds from databases, and sending them to Kafka * Using Avro for schema management and future-proofing your data * Designing your data pipelines to be resilient, but also flexible and amenable to change
Systems that enable data agility Martin Kleppmann 2015-06-23 | Talk given at Strata + Hadoop World, London, 6 May 2015.Slides: speakerdeck.com/ept/systems-that-enable-data-agilityAbstract:Congratulations, you’ve got a lot of data! Now what? How do you enable your organisation to create value from that data? What tools do your data scientists need in order to create data-driven products? How do you empower your teams to experiment, to innovate, and to be agile in response to customer needs?In this session we will discuss LinkedIn’s approach to solving these problems, and the open source tools that were created at LinkedIn to support data agility in a large organisation. The approach boils down to a few simple ideas:1. Make all data available centrally, in real time. If it’s too difficult to access data across silos, you can’t derive value from it. For this purpose, LinkedIn created Apache Kafka, which can be the data exchange backbone of your organisation.2. Make it easy to analyse and process that data. You’ve hired smart people, now empower them to easily try out new ideas for data-driven products, and rapidly get them into production if they are good. To support this, LinkedIn created Apache Samza, a stream processing framework that provides powerful, reliable tools for working with data in Kafka.Since Kafka and Samza are open source, you can apply these lessons and start implementing your own agile data pipeline today. In this talk you’ll learn about:- How Kafka and Samza reliably scale to millions of messages per second- What kinds of real-time data problems you can solve with Samza- How Samza compares to other stream processing frameworks- How data streams support collaboration between different data science, product and engineering teams within an organisation- Lessons learnt on how to move fast without breaking things
Building real-time data products at LinkedIn with Apache Samza Martin Kleppmann 2014-11-18 | Presented at Strata+Hadoop World, New York, 16 October 2014 http://strataconf.com/stratany2014/public/schedule/detail/36045Slides: speakerdeck.com/ept/building-real-time-data-products-at-linkedin-with-apache-samzaAbstract:The world is going real-time. MapReduce, SQL-on-Hadoop and similar batch processing tools are fine for analyzing and processing data after the fact — but sometimes you need to process data continuously as it comes in, and react to it within a few seconds or less. How do you do that at Hadoop scale?Apache Samza is an open source stream processing framework designed to solve these kinds of problems. It is built upon YARN/Hadoop 2.0 and Apache Kafka. You can think of Samza as a real-time, continuously running version of MapReduce.Samza has some unique features that make it powerful. It provides high performance for stateful processing jobs, including aggregation and joins between many input streams. It is designed to support an ecosystem of many different jobs written by different teams, and it isolates them from each other, so that one badly behaved job can’t affect the others.At LinkedIn, we have been using Samza in production for some time, both for internal analytics purposes and for data products that are served on the live site. In this talk, we’ll discuss our experience of working with Samza. You’ll learn about:- What kinds of real-time data problems you can solve with Samza - How Samza reliably scales to millions of messages per second - How Samza compares to other stream processing frameworks - How Samza can help collaboration between different data science, product, and engineering teams within an organization - How to avoid implementing the same data pipeline twice (once for offline/batch processing and once for real-time/stream processing) - Lessons we learnt on how to structure real-time data pipelines for scale and flexibility
