JupyterCon
Tanya Rai - Introducing Bento: Jupyter Notebooks @ Facebook | JupyterCon 2020
updated
Over the past 2.5 years, I’ve been building a spreadsheet extension for JupyterLab called Mito. This extension needs to share state between the frontend and the backend, and also have the backend/frontend communicate constantly. As such, we naturally built Mito as a Jupyter Widget.
Being a widget comes with many benefits, including easy-to-extend templates, automatic shared state, pre-setup comm channels, and a handling of kernel restarts and page refreshes. For the first 2 years of development, widgets were perfect.
Limitations of Widgets
But recently, we've hit into some limitations of the widgets framework. Namely, opinionated state management practices, complexity with integrating with other UI libraries, additional Python dependencies, and install rates that leave a bit to be desired.
As a result of these complexities, we decided to move from being a widget to being a rich output: a javascript frontend that manually establishes comms with the Python kernel, and shares state with it.
I'll share details on how to make this transition away from widgets, and also talk about what it gives you as widget author: reduced dependencies, potentially-easier installation, more manual (but customizable) state management, and more!
Towards a Unified Rich Output
Jupyter contributors are currently exploring (github.com/jupyterlab/richoutput-js) a more minimial rich output interface. I’ll close by reflecting on the benefits of this more minimal interface, and why it might be the right way forward for the entire notebook ecosystem :)
We will cover the following topics:
Strategies for gathering and prioritizing user feedback, including surveys, polls, and short interviews.
Techniques for analyzing and synthesizing user feedback, including data visualization and text analysis tools.
Methods for communicating and following up on user feedback, including the use of transparent and consistent processes for decision-making.
Best practices for engaging with users and fostering a sense of community ownership.
By the end of this session, attendees will have a better understanding of how to effectively manage user feedback and use it to drive the growth and success of their projects.
Audience:
This conference proposal is suitable for community managers, moderators, and other professionals responsible for online communities. It is also relevant for anyone interested in understanding how to effectively process and use user feedback to shape the direction of their communities.
Objectives:
To provide attendees with strategies for gathering and prioritizing user feedback.
To discuss techniques for analyzing and synthesizing user feedback.
To share community practices for communicating and acting on user feedback.
To explore ways to engage with users and foster a sense of community ownership and responsibility.
In this talk, we will talk about the last evolutions of the xeus stack, and how the flexible architecture of xeus made it easy to develop kernels that run entirely in the browser.
We will first give an overview of the xeus ecosystem and the different kernels based on it. We will then dive into the detail of xeus and its architecture, and how users can author new kernels with the library.
In the next section, we will discuss the specifity of WASM kernels, and demonstrate how the change in xeus made it easy to generate kernels that run entirely in the browser.
We will conclude with a roadmap for future developments.
Outline:
Overview of the xeus-based kernels
Details of the xeus architecture
Specificity of WASM kernels and how we adapted xeus
WASM kernel generation
Pluto.jl is a new, open source notebook programming environment for Julia, written in Julia and JavaScript. Our mission is to make Julia more accessible and fun! 🎈
In this talk, we would like to introduce Pluto.jl to the JupyterCon audience, and we will talk specifically about our approach to reproducibility and reactivity. While Pluto.jl is not directly connected to the Jupyter ecosystem, we think that our position (Julia-only, beginners-first) has led to new discoveries and solutions that are exciting to discuss!
Reproducibility 1 – Package Management
We see package management as one of the major hurdles for beginner programmers. It can be intimidating to set up an environment to start programming, but it is especially difficult to set it up in a reproducible way. We want to flip this paradigm: a simple, reproducible environment should be the default, and more advanced users can set up an environment themselves. As a whole, 'scientific computing' has an awful onboarding process, and we scare away so many creative and wonderful people before they are able to contribute. Let's fix that!
One of our goals is to make notebooks reproducible by default. Each notebook file (or HTML export) contains the Manifest.toml file that can be used to exactly recreate the package environment. When you open a Pluto notebook file, the embedded package information is used to automatically recreate the package environment that was used to write it.
A second big feature is automatic package management: instead of a terminal interface, packages are automatically installed and removed as they are used in code. We show package GUI inline in code, and we relay installation progress to the user visually. As a user, it feels like you can simply import any package you want (we even autocomplete all registered package names!), and Pluto takes care of installation and reproducibility.
Reproducibility 2 – Reactivity
Pluto notebooks are reactive, which means that – just like a spreadsheet – your notebook forms a computational graph, and cells re-run automatically when one of their dependencies changes. We also have a "managed scope": we delete variables from scope when the definition disappears.
Reactivity makes Pluto fun and interactive, but it also avoids effects from old code lingering around until a restart. Reactivity and managed scope mean that the notebook is always in its correct state, the same state you would get if you would restart the notebook. At any instant, the notebook state is completely defined by the code you see.
Reproducibility 3 – Binder
Pluto can be installed as a JupyterLab extension, which means that we also run on Binder, the free cloud compute service. We went one step further, and integrated the Binder startup directly into our notebook UI. Inspired by the Thebe project, this allows users to launch a Binder session directly from the website where they are reading a notebook!
Every HTML export file has the original code, and an embedded project environment. But as part of our Binder integration, each HTML file also contains a reference to a version-pinned Binder image, meaning that the exports from Pluto can be re-run in exactly the same environment years into the future.
Lessons and discussion
Pluto is deeply integrated with Julia's package manager, metaprogramming and runtime, something that we were able to freely explore by limiting ourselves to a single language. Originally inspired by Jupyter, which supports more languages and use cases each year, our experiment is to see what happens when we really narrow down our scope: we focus on one language and one audience (Julia newcomers and educators).
We hope to offer an interesting new take on existing topics in the Jupyter ecosystem, and we really look forward to hear what you think!
Minimap
One approach is to create a map of cells in the notebook linked according to their dependencies; a “minimap” reduces each cell to a point labeled by a truncated line of code and mirrors the top-down cell ordering in the notebook. Edges connect these points to show dependencies between cells, but most edges are hidden until a cell is selected. When a cell is selected, cells that are upstream or downstream of it shift left or right, respectively, to show dependencies, and immediate dependencies are connected with lines. These interactions allow users to easily understand the impact of changing cells and resolve misunderstandings about dependencies. For example, when given a notebook with a cell that loads in data, we can quickly know all the cells that this data depends on in our notebook and which cells must be re-run as a result of our changes. Observable, a Javascript notebook environment, introduced this style of minimap and inspired our approach. In an Observable notebook, however, there is at most one output per cell. In a Jupyter IPython environment, a single cell may generate many outputs, making the visualization more challenging. Having a minimap allows for faster understanding and navigation based on this information without the screen clutter of a traditional graph and can easily be viewed while casually creating a notebook. This additional knowledge and understanding can prevent tricky situations leading to notebook irreproducibility as users now know which cells directly impact others and can make better decisions about when to re-execute.
Cell Dependency Graph
However, users may be more familiar with standard node-link graph diagrams, where cells are nodes with variables nested within them, and dependencies are links. This structure allows the topology of the notebook to be more easily understood from the graph as links are not hidden like in the minimap. This topological information helps prevent users from the misunderstanding that cells they have placed out of order in the notebook may have to be re-run before reaching their previous results. This practice allows for exporting partial notebooks that may be shared with colleagues. Microsoft’s Gather provides one way to export portions of notebooks to share with others. However, we provide a way to view and understand what is being exported by selecting and highlighting inside the graph. Using graphs in the notebook encourages interaction in new ways from structural points of view and allows users more control over the notebook via graphs. The linking of notebook and graph interactivity promotes better practices in notebooks by preventing unwanted behavior and provides a step forward in better understanding.
Spyder, a community developed, open source IDE written in and for Python, aims to bridge the gap in that area. It blends the debugger with the interpreter to allow data exploration at any point during code execution, not just at the end. For that, Spyder's debugger attempts to offer the same functionality as a full IPython interpreter, so that its users can debug their code in the same way they are used to doing the rest of their scientific programming.
In this talk, I will cover the debugger features available in Spyder 5, as well as those planned for future versions. Specifically, I will present a live demo to showcase the following features (described in depth in this blog post and the Spyder documentation):
How to start the debugger and set breakpoints.
How to better understand the code at a certain frame by writing code snippets in the debugger itself, facilitated by syntax highlighting, code completion, multi-line editing and command history.
How to move up and down in the call stack to explore other frames in the same way.
How to use Spyder's Variable Explorer to browse the contents of objects.
How to generate Matplotlib plots while in the debugger.
How to use IPython magics while debugging to profile code using %timeit, explore the filesystem with %cd and %ls, and open files with %edit.
Thanks to these improvements, Spyder transforms debugging from a task that feels foreign to scientific programming, to be almost second nature. By adding breakpoint calls or setting breakpoints in the debugger, users can easily prototype different solutions for their problems at any point during code execution, and not just at end. This also increases programming speed by allowing to constantly check the correctness of code during development.
At the end of this talk, I hope attendees will learn that they don't need resort to print() or other workarounds to debug their code. Instead, they can rely on the robust debugging methods used by professional developers, supported by the interactivity and workflow they're familiar with in IPython/Jupyter.
Technology teams usually have an on-call rotation with static wiki-style documentation to guide the on-call engineer. Jupyter Notebooks can replace static documentation with executable notebooks. E.g. “fetch service logs” and “rollback last deployment” can simply mean executing a code cell that’s available alongside the markdown instructions.
What are the benefits of executable vs. static documentation for DevOps -
Quick e.g. “check DB latency” is 1-click notebook code cell execution to plot latency graph vs. going to a third-party UI in the middle of an incidence
Precise e.g. “promote read replica to master” can mean a series of steps & possibility of human error; codifying the steps in advance removes ambiguity & results in precise action.
“Executable documentation” is a simple yet powerful concept that can extend to other use cases such as - API documentation, developer onboarding, data visualization & reporting, scheduling routine tasks & so on. Think of it as executable GoogleDocs powered by Jupyter!
In this talk, we’d like to,
- Introduce the concept of Jupyter powered “executable documentation” platform, particularly for DevOps and Incident Response,
- Show a demo of how it’d work - (youtube.com/watch?v=vvLXSAHCGF8)
- Talk about important challenges, and propose a way forward to make this a mainstream application of Jupyter notebooks.
Since their creation, Jupyter and IPython have always had an architecture and user interface for code generation through autocompletion. Indeed Jupyter users have been well trained to hit “tab” as they are coding to get contextual autocompletions for functions, variable, methods, properties, classes, etc. This architecture has already been used to integrate AI provided autocompletions into JupyterLab and the Jupyter Notebook (Kite, Tabnine). As generative AI models expand to perform other tasks, a generalization of the Jupyter autocompletion architecture is needed to enable users to perform potentially any task in Jupyter’s applications through a generative AI. In this talk we will describe an architecture and user experience for generative AI tasks in Jupyter.
The Jupyter generative AI architecture is founded on an extensible Jupyter Server API for registering generative AI models and the tasks they perform (represented as task prompts) with Jupyter. This enables third parties to quickly integrate their generative AI models into Jupyter and for end users to enable the models and tasks using the JupyterLab extension manager UI or a simple pip/conda install. Once models have been enabled, users working in JupyterLab or the Jupyter Notebook can 1) select anything (text, notebook cell, image, file, etc), 2) type a prompt to perform a task with the selection, and 3) insert the AI generated response in a chosen location (replace, insert below, new file, etc.). The task system is integrated with Jupyter’s MIME type output system, which enables GAI tasks to work with inputs and outputs of any MIME type. From the end users perspective, this enables them to enable and use models to perform tasks on text, code, images, data, audio, video, etc. This architecture also has APIs in Jupyter Server and JupyterLab that enable third parties to extend and integrate with the models and UI.
In the talk we will also give concrete examples of the AI driven tasks that can now be performed in JupyterLab. Examples include code refactoring, debugging, code transformation (e.g., Python to C/C++), code explanation (“generate and insert a markdown cell that describes this code”), synthetic data generation, data transformation, API documentation generation, technical content generation, and more. The focus of these examples will be less on the capabilities of the underlying AI models, and more on the seamless integration of the models into Jupyter to perform various tasks. Our assumption is that generation AI models will continue to improve and that Jupyter users will want a simple user experience that works with any such model.
Another potential application of this architecture is as a general purpose “playground” for exploring the capabilities of generative AI models. While not the primary goal of this work, the architecture effectively turns JupyterLab into a general-purpose generative AI playground. This enables users to quickly integrate models, and then use the familiar interface of Jupyter, with its support for editing files/notebooks, version control, real-time collaboration, etc., to play with generative AI. We discuss the pros and cons of this user experience and pose questions for future research and development.
This talk will be useful and of interest to 1) researchers who are building generative AI models to perform different tasks, 2) JupyterLab extension authors who would like to leverage generative AI, and 3) end users of Jupyter, who would like explore generative AI models and use them to perform tasks in their work.
The same can be said about Jupyter deployments, the de-facto standard for data analysis in the scientific and research communities: each institution has its own configuration, deployment strategy and, more importantly, their customized way of giving users access to their (siloed) data and code.
The EU-fundend project CS3MESH4EOSC was started to address these major technical, but also societal challenges. Science Mesh, its main asset, was idealized to provide an interoperable platform that easily integrates and extends sync & share services, applications (like Jupyter) and software components within the full CS3 community. Such federated service mesh provides a frictionless collaboration platform for hundreds of thousands of users, offering easy access to data across institutional and geographical boundaries.
This presentation will focus on the development of the cs3api4lab, a plugin created by the project to connect Jupyter to the Science Mesh. It brings features like easy to configure access to CS3 services’ backends, sharing and parallel access to notebooks right from within the Jupyterlab interface. We will also discuss its applicability outside of the Mesh and, finally, on the project vision for collaborative scientific analysis.
The MyST (Markedly Structured Text) project has grown out of the ExecutableBooks team, which has been working on MyST Markdown and JupyterBook as new ways to publish notebooks. Originally based on Sphinx and RST, over the past year the ExecutableBooks team has been working on a MyST Specification to coordinate development of the markup language and extensions across multiple languages & parsers (e.g. implementations in Python & Javascript). The new mystjs libraries run directly in the browser, opening up new workflows for components to be used in web-based editors, directly in Jupyter and in JupyterLite. The libraries work with current MyST Markdown documents/projects and can export to LaTeX/PDF, Microsoft Word and JATS as well as multiple website templates using a modern React-based renderer. There are currently over 400 scientific journals that are supported through templates, with new LaTeX templates that can be added easily using a Jinja-based templating package, called jtex.
In our presentation we will give an overview of the MyST ecosystem, how to use MyST tools in conjunction with existing Jupyter Notebooks, markdown documents, and JupyterBooks to create professional PDFs and interactive websites, books, blogs and scientific articles. We give special attention to the additions around structured data, standards in publishing (e.g. efforts in representing Notebooks as JATS XML), rich frontmatter and bringing cross-references and persistent IDs to life with interactive hover-tooltips (ORCID, RoR, RRIDs, DOIs, intersphinx, wikipedia, JATS, GitHub code, and more!). This rich metadata and structured content can be used directly to improve science communication both through self-publishing books, blogs, and lab websites — as well as journals that incorporate Jupyter Notebooks.
Our talk is aimed at attendees who are looking to communicate their Jupyter Notebooks, markdown documents or write scientific articles/papers. Our talk will assume basic familiarity with command-line tools and markdown. Throughout the presentation we will practically demonstrate the myst command-line interface to work with Jupyter Notebooks & markdown articles to create PDFs & Word documents, add citations & cross-references, and deploy modern websites with multiple themes that preserves the structured content.
We also offer some observations about how to help Jupyter be more flexible in enterprise contexts.
This talk is for intermediate to advanced developers/administrators wanting to customize or build on Jupyter standards or software to deploy in an enterprise context.
JupyterLite is a Jupyter distribution that runs entirely in the web browser without any server components. A significant benefit of this approach is the ease of deployment. With JupyterLite, the only requirement to provide a live computing environment is a collection of static assets. In this talk, we will show how you can create such static website and deploy it to your users.
We will cover the basics of JupyterLite, including how to use its command-line interface to generate and customize the appearance and behavior of your Jupyter website. This will be a guided walkthrough with step-by-steps instructions for adding content, extensions and configuration.
By the end of this talk, you will have a good understanding of how JupyterLite works and how you can use it to create interactive websites.
Outline:
Introduction to Jupyter and JupyterLite
Examples of JupyterLite used for interactive documentation and educational content (NumPy, Try Jupyter, SymPy)
Step-by-step demo for creating a Jupyter website
Quickstart with the demo repository
Adding content: notebooks, files and static assets
Adding extensions to the user interface
Adding packages to the Python runtime
Customization and custom settings
Deploy JupyterLite as a static website on GitHub Pages, Vercel or your own server
Conclusion and next steps for learning more about the Jupyter ecosystem
The talk will be based on resources already publicly available:
try JupyterLite in your browser: jupyterlite.github.io/demo
the JupyterLite documentation: jupyterlite.readthedocs.io/en/latest/quickstart/deploy.html
the JupyterLite repositories: github.com/jupyterlite
We will start by introducing the core Jupyter projects, including the Jupyter Notebook and JupyterLab, and explaining how they provide a platform for interactive computing and data visualization. We will then discuss some of the key sub-projects within the Jupyter ecosystem, such as JupyterHub for enabling multi-user access to notebooks and nbconvert for converting notebooks to other formats, and we will explain how they fit into the overall landscape of Jupyter.
Next, we will delve into the underlying projects and libraries that make Jupyter and its related projects possible, such as the Jupyter server, the core APIs projects and the traitlets library. We will discuss the different protocols used for communication between the applications and the kernels, and show how it makes Jupyter agnostic to the language. We will also cover some of the key technologies used by Jupyter and its related projects, such as the Tornado web framework or the ZeroMQ messaging library, and we will explain how these technologies fit into the Jupyter landscape.
Throughout the talk, we will provide examples of how these tools and technologies can be used in practice and discuss the latest developments and future directions of the Jupyter ecosystem. By the end of the talk, attendees will have a better understanding of the Jupyter ecosystem and how its various projects and libraries fit together to enable interactive computing and data science.
Outline:
Introduction to the Jupyter ecosystem
Overview of the core Jupyter projects (Jupyter Notebook, JupyterLab)
Overview of key sub-projects within the Jupyter ecosystem (JupyterHub, nbconvert)
Explanation of the underlying projects and libraries used by Jupyter (Jupyter server, jupyter_client, traitlets)
Introduction to the Jupyter protocol and the widget protocol
Overview of key technologies used by Jupyter and its related projects (ZeroMQ, Tornado framework)
Discussion of the latest developments and future directions of the Jupyter ecosystem
Conclusion and next steps for learning more about the Jupyter ecosystem
Capytale provides several web-based interfaces well suited for the various use cases of the high-school curriculum, one of which is the Jupyter notebook.
Now adopted by most school districts in the country, the notebook service provided by Capytale serves over 80,000 users per week.
The deployment model of Capytale, built upon Pyodide and the Jupyter notebook user interface, proved remarkably reliable and scalable, allowing tens of thousands of concurrent user sessions with small hosting requirements.
In this presentation, we detail the technical constraints and pedagogical motivations that lead to this specific design and discuss ongoing and future improvements to the platform.
One strategy we employed from the start of the grant-writing process was to secure funding for four rounds of Outreachy, with two interns per round, over the grant duration of two years. Outreachy is a mission-aligned organisation dedicated to placing interns from backgrounds that are underrepresented in tech, into open source projects. The mentorship these interns receive is the bedrock on which sustainable entry-level pathways into the community can be built. Since Outreachy supports more than only coding projects, we can also provide other pathways into the community that do not rely on being a “coder” or “software developer”.
This kind of “Mountain of Engagement” work is important to any community-led project, whether within the Jupyter ecosystem or beyond, and as such we have been capturing lessons learned in a guide as we go. This will ensure that the process of participating in Outreachy as a community is a little more repeatable with each round, and provide clear pathways for other community members to become involved in the processes after the term of the grant. We also hope that by sharing our experiences, this resource becomes usable by other Jupyter subprojects, or elsewhere, to begin their own internship initiatives.
Repository: github.com/jupyterhub/outreachy
Website: jupyterhub-outreachy.readthedocs.io
By the time JupyterCon 2023 arrives, JupyterHub will have completed the first Outreachy round funded by the CZI grant. We have already learned, and will continue to learn, a great deal around the processes required for running these internships, which we have captured in the above guide. During this talk, we will discuss some strategies the JupyterHub team implemented during this initial round, such as:
Establishing partnerships with other mentoring organisations, such as Open Life Science, to deliver support through mentor training and cohort calls for interns
Developing processes during the Outreachy contribution period to manage and evaluate applications
In this talk, we will discuss a recent overhaul of 2i2c’s tooling that facilitates the centralisation of information and optimal operation of the engineering team, whilst protecting a community’s Right to Replicate their infrastructure. Critical to protecting the Right to Replicate is a configuration schema for both clusters and JupyterHubs, where these files should live in the repository, and how the contents should be structured. Each individual JupyterHub we deploy is defined by its own individual set of configuration files which enables simple extrication from the repository, and they can be deployed independently with a basic command. There is no added magic in the rest of 2i2c’s specific tooling that would prevent this.
Further tooling to optimise the deployment and management of these JupyterHubs for 2i2c’s engineering team includes:
A Python “deployer” module that knows how to read the configuration for a given JupyterHub on a given cluster and can perform an upgrade action
A function within the deployer module that can extrapolate which JupyterHubs on which cluster require an upgrade from a list of changed files in the repository (e.g. from a Pull Request)
A GitHub Actions workflow that can deploy to multiple clusters in parallel, deploy production JupyterHubs in parallel, implement Canary deployments using staging JupyterHubs, and intelligently prevent a Canary deployment failure affecting the deployments on an unrelated cluster
Details of these efforts were first published in the “Tech update: Multiple JupyterHubs, multiple clusters, one repository” blog post.
The adoption of real-time collaboration in JupyterLab has been a challenge for many developers over the years. From the very beginning, RTC was in JupyterLab's roadmap. Still, it was only in v3.x that it became a reality, and in v4.0, that it shows its real power. We want to describe the feature in detail with everyone to give extension developers the knowledge to leverage it into their plugins.
This talk will go through the RTC implementation and describe the role of the packages used.. The various entry points to use and extend real-time collaboration on documents will be highlighted. And it will show the corner cases and the restrictions it enforces.
Finally, we will offer a glimpse into how real-time collaboration works in JupyterCAD and JupyterLite. JupyterCAD is a JupyterLab extension using a non-default document type for 3D geometry modeling that supports the FreeCAD format. JupyterLite is a lightweight serverless version of JupyterLab, which changes the paradigm of collaboration. It adds a new challenge by removing the central authority that's the server, requiring the use of peer-to-peer communication to synchronize clients. The document is leveraging the real-time collaboration API to allow collaborative editing.
As part of this transition, I stepped down as "project BDFL". This was a decision driven by many reasons; among others, I think the context of open source communities today is radically different from 20 years ago, and our view with Jupyter is centered around a multi-stakeholder community, where no single actor should concentrate too much authority. From day one, IPython, then Jupyter, were collaborative efforts. Many people put in their ideas, code, feedback and energy to create something we can all collectively own, adapt for our needs, and extend in new directions. While it's necessary to build a sense of vision and direction for any project to be successful in the long run, a "dictator" is entirely the wrong metaphor to base that work on. Python itself has moved on from that model, and we hope more projects will continue to find better ways to harness the energy of their entire community.
The space where Jupyter exists today is challenging: we have had phenomenal success on many fronts. From users of Juptyer itself to those who build on our foundations, standards and protocols in new directions, the impact of the project goes far beyond our original hopes and expectations. But, as well stated in the classic Innovator's dilemma, this type of large-scale success is itself a potential trap. While not a commercial entity, Jupyter still needs to provide value to all of its stakeholders, with the added challenge that its own strategic directions are not decided by any single actor.
This conference is an opportunity to learn from each other on both what has worked so far, and how the next decade will be very different. If we are to continue providing value, these are some of the challenges I hope we'll tackle together:
Jupyter beyond the "Global North": Jupyter should be a tool for democratizing access to science and education, but in a way that empowers all to participate, not only to access and consume. This means we need to continue growing our community of developers, contributors and leaders in countries beyond those most traditionally represented (which for Jupyter, is highly concentrated in the US and Europe). Important efforts have made progress on this front already, but we should make it a priority to build culturally-relevant, contextually appropriate spaces for local communities to develop new tools and use cases, as well as contributing back into the main project.
AI: as of this writing, 2022 is shaping up as a watershed year in AI, with advances like ChatGPT causing ripples in the industry and even general media. These tools open a new front on the idea of "interactive conversations with humans and the computer" that Jupyter has already explored. They also tend to be developed and controlled by a few large and powerful corporations. Jupyter should explore its own version of how to ensure these tools can empower everyone, while innovating with use cases that capitalize on our architecture.
What are the layers of the "core of Jupyter" that its stakeholders should agree upon, and continue building as 100% open-source, vendor-independent technology, while so many of these stakeholders have their own business goals? These goals often depend on Jupyter-based products, so there is an inherent tension between what goes into the "Jupyter commons" and what is the value added by a specific vendor. Rather than wishing this tension away, we should recognize it and work productively within these constraints.
To explore these and related questions, we should reflect on identifying the scope of Jupyter - Jupyter tools cover many different aspects of computing, yet there's a common theme. I'll present my own take on this, informed by my interactions with many of you for 20+ years, in the hope that it can lead to useful conversations for our next period.
I hope this talk will be a useful point of reference for some of the ideas that the project has built so far, but focused on thinking of the future. While I will discuss them from my limited, personal perspective, my goal is to motivate thinking at the strategic, community and technical levels.
In this talk, we’ll discuss authoring these dynamic, computational documents with Quarto and Python that bring code, output, and prose together, leveraging integrations with both Jupyter and the Quarto VS Code extension. Whether you’re new to Jupyter or have thousands of notebooks already, we’ll walk you through using a single source document to target multiple formats - transforming a simple document into a presentation, a scientific manuscript, a website, a blog, and a book in a variety of formats including HTML, PDF and MS Word. We’ll also show how you can change themes and styling, and publish these artifacts directly from the command line to the web, so they’re immediately available online.
Let us step in the shoes of a data scientist, and with a Jupyter Notebook try to perform a classification or regression task on data coming from a collection of raw tables.
In this tutorial, we will demonstrate how dirty_cat, an open source Python package developed in our team, can help with table preparation for machine learning tasks and improve results of prediction tasks in the presence of dirty data.
Some of the common problems we will be tackling are:
- joining groups of tables on inexact matches;
- de-duplicating values;
- encoding dirty categories with interpretable results.
And all of this on dirty categorical columns that will be transformed into numerical arrays ready for machine learning.
Examples of individual features can be seen here:
- dirty-cat.github.io/stable
- github.com/dirty-cat/dirty_cat/tree/main/examples (link to Jupyter notebooks)
Dask is a popular library for parallel and distributed computing in Python that was co-developed alongside Jupyter with human interaction and interactivity in mind. In this talk we'll discuss Dask in the context of interactive data science, highlighting the ways in which Dask and Jupyter leverage each other to achieve a powerful and scalable user experience that fits easily into your hand. In particular we'll highlight rich notebook outputs, JupyterLab dashboard extensions, and JupyterHub deployment integrations, and how leveraging the extensibility of Jupyter can result in a first-class open source experience
In this talk we will present the latest updates on the Voilà open-source project, including the recent move to using components from the JupyterLab project for the rendering, enabling support for rich mime type renderers. We will also demo the ongoing work on using in-browser Python kernels from JupyterLite.
community. At NASA’s Jet Propulsion Laboratory, we have built upon those foundations to
answer engineering challenges and prove that with innovative approaches, the use cases that
can benefit from the Jupyter ecosystem are endless. With modular extensions and novel
methods, we are able to perform system-level and physics-based modeling, trade studies,
automated pipelines of engineering reports, hardware testing procedures, and more.
In this talk, we share our experience of using Jupyter as an integration platform and multi-domain
engineering tool, and why we believe that with the correct approach, Jupyter can be extended
to do just about anything.
The UX of computational notebooks
Computational notebooks have changed the way we think about interactive computing. By blending together narration and code, they are widely regarded as a concrete implementation of the concept of literate programming [1]. Very popular in educational contexts, notebooks are a unique tool in promoting computational thinking, the ability to present problems and solutions in the form of algorithmic steps.
We will present some of the challenges with the UX of computational notebooks, that is distinct from both document processing tools and development environments.
JupyterLab, an open application framework
While a lot of websites are focused on guiding users towards a predefined goal (such as a purchase for e-commerce websites), JupyterLab provides a space for open-ended exploration and creation, creating unique user experience challenges that have been sparsely explored.
Photo editing software, development environments, CAD tools, all fall in this category and have similar requirements, such as allowing for a complex tiled layout or displaying a lot of personalised information on the screen.
We will show how the foundations of JupyterLab can be reused to build other tools of that category, and present two examples with the JupyterCAD and Glue-Jupyter projects.
Improving the UX of Jupyter
We are launching a new initiative focused on the UX of JupyterLab. How can we uncover issues and improve user experience through UX research without breaking what Jupyter got right and what enabled its global adoption.
We will present the first steps of our initiative, ranging from the triaging of issues to the set up of user tests.
[1] Knuth, Donald E. (1992). Literate Programming. California: Stanford University Center for the Study of Language and Information. ISBN 978-0-937073-80-3.
Link Git
Link is a JupyterLab extension that allows users to create pipelines on Jupyter by connecting different cells. The user can connect cells in their desired order into a DAG structure to run the code. Link also provides a Link-git extension, which provides git features on Jupyter for ipynb files with pipelines.
Git diff check: Users can visualize the commit history of an ipynb file. The feature shows all changes made at the code level for each commit, and also how the structure of the pipelines changed. When collaborating, users are able to review the history of previous works before moving on to the next stage, and also decide from which commit they wish to begin.
Merge conflict management: Link-git contains a merge driver, which resolves all merge conflicts at cell levels - for both the code and pipeline structure - when a conflict occurs between different users working from their respective local environments. Using this feature, a team of developers can create an overall code framework in the form of a pipeline to facilitate the merge process of pipelines after writing codes at cell levels on their respective local environments.
Sharing pipelines and cache
Link provides features to facilitate code sharing in file formats. Conventionally, Jupyter users often share codes as ipynb files to make modifications on existing codes or to re-use them. However, on Jupyter, code cells are linearly listed and contains code cells that are not well organized, leading to difficulties in reproducing results or to make changes or additions. Link provides the following file export, import features to resolve these complications.
Pipeline export, import: Link users can export entire pipelines or component cells as json files. When another Link user imports these json files, they can re-open the pipeline or component cell on Jupyter. As the code includes the DAG structure, users can re-open the ordered pipelines together with the code cells, allowing them to more efficiently understand the flow of the code without any additional textual explanations. Also, users can share only the relevant code cells that are required for execution.
Cache export, import: Users can store the cache of each component after executing a whole pipeline, and export the cache as an archived name file(.tar.gz). When another user imports this file, they can use the pipeline results without having to reiterate the pipeline execution. This feature is a time-saving feature for users when they need to repeat certain jobs on their own pipelines, or for different users to reproduce the same code.
Finishing up
In short, Link provides a useful framework for easier collaboration with its Link-git feature and the pipeline/cache export and import features.
Science is not composed of isolated groups of practitioners, but is rather an interconnected network of communities of practice, with members who fluidly move between them. Infrastructure for scientific research and collaboration should leverage this structure to make science more productive and inclusive. You will learn about the JupyterHub tools, community, and best practices being developed to achieve these goals around a specific NASA science mission and research objectives in a project known as the CryoCloud.
Outline:
For NASA’s upcoming Year of Open Science, NASA and other federal funding agencies have begun to transition their data stores and computing into the cloud. However, substantial barriers exist for individual users to make the transition from their local systems to the cloud to accomplish research goals and fully utilize new cloud capabilities: cloud cost opacity, infrastructure deployment complexity, and a general lack of community awareness and knowledge, among others.
To address these challenges, we have built a persistent JupyterHub in partnership with the International Interactive Computing Collaboration (2i2c), called CryoCloud (cryointhecloud.com), for NASA Cryosphere communities. We provide a persistent hub space across a series of hackathon-style workshop events to help the NASA ICESat-2 Science Team and related Cryosphere researchers build community and transition to a collaborative cloud workspace. CryoCloud models how to build a research community around a specific science mission and research objectives, while enabling learning and creation of the technical knowledge to facilitate NASA’s open-source, interconnected, and science-accelerated vision of the future. We gather user data to understand research needs, imagine tools required to streamline collaboration, and build community best practices. We share examples of how these JupyterHub cloud tools make scientific computing more intuitive, cost- and time-efficient, and open for all.
This presentation is intended for cloud engineers, scientific programmers, data scientists, and all other interested people. Attendees are expected to have no specific knowledge background. Familiarity with geospatial datasets and Earth science is useful but not required.
The Copernicus Climate Change Service (C3S) Climate Data Store (CDS) is an entry point to a broad spectrum of data related to climate and meteorology, from output produced by weather and climate models to observation data from satellites and weather stations. Even when strict data standards are followed, each dataset has its own peculiarities and pitfalls which require some degree of documentation support. Jupyter notebooks, provide the perfect complement to user-guides that typically accompany the datasets available in the CDS. The notebooks provide examples of how to access, download and explore the data, and provide a platform where the producers of the data can demonstrate the qualities of the data and applied use cases.
The C3S is embracing this concept and building a JupyterBook-based training material to support users of the datasets available in the CDS. Such a library will instigate greater use of climate and meteorology in sectoral fields (e.g. health, tourism and transport) and could be used as an educational resource in universities and schools. This library will feature directly in the JupyterHub-based online development environment that will be integrated into the modernised version of the CDS.
This session will present the training material we have produced, and the framework we have in place such that data producers are able to easily contribute to our growing library of content. We will demonstrate the importance of this material in providing data users the information required to make use of the data, and how this fits in the plans for modernising our online development environment. This session is aimed at people who may be interested in using the training material we have produced to gain experience handling climate, weather and observation data; and aimed at people who may want to construct a similar, community developed, training material resource.
The CDS is implemented by the European Centre for Medium-Range Weather Forecasts (ECMWF) on behalf of the European Commission.
TheJupyter ecosystem is becoming central to support users in discovering, using and analysing this data.
The European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT) is an implementer of the Copernicus programme and this talk will show how Jupyter is used to provide know-how and access toCopernicus Earth Observation data on the following aspects:
Training and awareness - jupyter-based platforms used in events to self-paced approach over a wide range of domains, including atmosphere, climate change, and marine. For example in 2022 have been held more than 50 international events making use of Jupyter environment with more than 5000 users
Data ready - using jupyter notebooks to showcase applications and grant a seamless access to data - providing notebooks and access to hubs through advanced and open platforms .
Among these, the European Weather Cloud and the WEkEO Copernicus Data and Information Access Services (DIAS) as collaboration among key international organisations and agencies (ECMWF, EUMETSAT, EEA and Mercator Ocean International)
Communications and outreach- make use of Jupyter to engage with a large public scientifically literate. This includes competition of jupyter notebooks applied to innovative data analysis and access and practical sessions using Jupyter Notebooks in largely attended massive on-line courses (MOOCs)
1) A wiki to serve as user guide with full description of the tools involved;
2) Jupyter notebook templates and examples of the workflows of case studies;
3) Access points to the models, data needed and tools for data manipulation and visualisation for the Climate Risk Assessment;
4) Access to computational and storage resources.
Jupyter ecosystem will be the heart of the CLIMAAX toolbox with Jupyter lab enabled for users and Jupyter books for wiki, documentation and templates.
The project will start in January 2023 and in this poster we will share first design and implementation of the toolbox.
This poster will examine:
Accessibility issues with Jupyter Notebooks and how these problems frequently keep disabled people out of careers in data and bench sciences.
Our user-centered method of tackling this difficult problem by cycling between paid usability feedback sessions with impacted users and implementation of accessibility enhancements. We repeat this until we create a benchmark notebook built with a more accessible structure to contribute back to the community.
Stories of how Jupyter accessibility impacts community members, including case studies of people with disabilities deterred from pursuing careers in STEM by inaccessible tools.
Specific problems we have uncovered so far and solutions we have tried for those problems. Solutions will focus on both individual authors and the wider codebase.
Our team from a variety of organizations (Space Telescope Science Institute, Iota School, Quansight) with a diverse array of skills used in the project.
How members of the Jupyter community can contribute to further work towards accessibility and inclusion in this area.
The platform is composed of a Docker stack with the following containers :
a Django application to manage users and permissions,
a Jupyter Lab instance to provide an integrated IDE,
a Voilà container to allow live kernels when accessing dashboards,
worker containers for housekeeping tasks, data management, etc.
as many database containers, of as many flavors as needed.
The results of notebooks can be rendered in a polished environement for the stakeholders with the following modalities :
precomputed notebooks with nbconvert,
live notebooks with Voilà that can use Dash or Panel technologies,
and we plan to add the support for pure Dash application deployment for use cases that need a better scalability.
The platform is used in production for several clients and in different contexts such as :
monthly updated dashboards for marketing evaluation based on open data and sales forecast,
hourly updated Web application using statistical models to triggers warnings about landslide and rockfall risks around mountain roads.
Although it was first developped for our internal needs, we believe that the platform can be of interest for organisations that need an easy to deploy environment, that can deliver production ready solutions, for data science and business intelligence projects.
The code and resources are publicly available under the GNU AGPLv3 licence : gitlab.com/tetras-lab
A public demo illustrating the capabilities of the Tétras Lab platorm will be deployed for demo and testing during the conference. Meanwhile, a demo of the application for landslide alertness is available : sigale.pinea.sage-ingenierie.com/dashboard/public/yrnhmikkticoepshxnmaibnrsslifkxljgbpzltdudlsneeazeanmwnbrtlcjizq.
In this talk I will discuss Two Sigma's partnership with QuantStack's team to deliver direct-to-upstream contributions to improve JupyterLab's performance and furtheir its Real-Time Collaboration (RTC) initiative, all to be soon seen in JupyterLab 4.
Beyond just a teaser of new features to get excited for, this talk is a testimonial of a fruitful and successful relationship between two private entities, Two Sigma and QuantStack, which not only furthers each partner's own internal goals but also improves quality of life of all Jupyter users.
Key takeaways
Get excited for JupyterLab 4, which will bring large performance improvements throughout the whole application.
JupyterLab 4 will pave the way for a Real-Time Colalboration mode in JupyterLab, similar to Google Docs. Come join the effort!
For small and medium-size organizations, an external partnership with a team like QuantStack can help them exponentiate their impact in the open-source community.
Making sure your organization runs on the latest version of Jupyter is key to shipping your contributions directly to your users.
JupyterLab's extensibility is a great way to deliver value to an organization's internal users. But it can also undermine your organization's ability to stay current with open-source.
Call to action
Two Sigma, QuantStack, and many other members of the Jupyter community are actively contributing to Jupyter's general performance and making Real-Time Collaboration a reality. Use the lessons you learn in this talk to get your organization involved, or get involved yourself!
Audience
This talk is aimed at 1. Jupyter users looking to hear about new and exciting features and 2. users of Jupyter at private entities who are interested in ways to get their organization involved in open-source development. You do not need any specific knowledge of how Jupyter works to get value from this talk. If you're interested in contributing to Jupyter, the talk will also give you an idea of features that are being actively developed by the project's contributors and hopefully get you excited about contributing!
To promote the diffusion of interactive computational practices, we developed five proof of concept notebooks that aim not only to demonstrate the possible uses of Jupyter Notebooks and Python machine learning tools, but also to foster their adoption by the HSS community. This project was initiated by the large digital research infrastructure in HSS, Huma-Num, in the context of testing a jupyter hub deployment. However, despite our best efforts, these notebooks were woobly and not very useful. We can say that we failed to match our expectations.
In this communication, we propose to dissect this failure as an attempt to gain a better understanding of the current uses of Notebooks by French researchers. We would like to emphasize the need to better explore research practices. Drawing on Science and Technological Studies, we suggest the hypothesis that many notebooks are primarily "intermediate objects" that allow for the coordination of the research process. If this is the case, reproducibility is not the primary goal, nor is diffusion. For this reason, the adoption of Jupyter Notebooks by HSS researchers would require primarily effort on the general computational literacy to foster their integration into their research process.
However, the base autoreload algorithm suffers from a number of reliability issues and can easily crash, thereby necessitating a costly restart. In this presentation, we’ll describe references reload, which is our new and improved autoreload algorithm with a number of benefits over the basic autoreload functionality. We’ll show how we use references reload in production to save developers hours of time, and we’ll close with a concrete use case for development on top of Bento server, which is Meta’s internal version of the Jupyter notebook server.
Developing & Iterating
In the first section we will go over the development of a rich and powerful data application in a notebook. We will demonstrate how we can quickly build and preview individual components by displaying them inline and previewing the entire application using a JupyterLab extension.
Once we have built the application we will go over recommendations for structuring the application to make them easier to maintain but also to achieve a good look and feel and get the best performance out of the application.
Jupyter Widgets integration
Next we will go over the integrations of the Panel and Lumen stack with the Jupyter Widgets ecosystem. The ability to leverage this existing ecosystem allows users to leverage the tools they love and allows taking Jupyter Widgets out of Jupyter and ship them in a standalone application.
Deployment
Lastly we will demonstrate how we can easily deploy Panel applications in a JupyterHub environment using existing extensions like CDSDashboards and how we can begin to extend this with Jupyter extensions that provide UIs to deploy to a cloud provider.
Our platform provides a virtual JupyterHub server for each year from undergraduate to graduate. Each virtual server hosts all the classes of that year. This lets us optimise and customise the servers depending on the use case:
numerous students (Greater than 1000) for the first undergraduate year, with low memory and CPU needs (discovery of python and notebook in science)
a few students (~30) for the last graduate year, with large memory and CPU needs (modelling, simulation, machine learning in mechanical engineering)
all the courses remain active all the semester, and are accessible 24 hours a day, 7 days a week by students using only a web-browser.
Our platform augments the classical Jupyter experience by offering features helping teachers run and manage courses, such as:
1. simple sharing of documents (notebooks, data, python library..) back and forth between teachers and students
2. multiple courses on the same JupyterHub server with the ability to set which students see which courses. Also useful to divide a course in groups for practical work
3. automatic grading of notebooks, python code, ...
4. simple evaluation of students work (notebooks, reports in markdown or LaTeX...) directly in the web browser without download or file transfers
5. plagiarism test on the students work
6. link with TOMUSS, the educational monitoring platform of the Lyon 1 University, to input student lists and output student grades
Our platform is made of three main components: JupyterHub, nbgrader, and a custom web application written in Flask running as JupyterHub services. To implement it, we have followed the Unix philosophy KISS, by using simple components such as bash scripts and python programs. The documentation written in French using Jupyter-book is available here: https://perso.univ-lyon1.fr/marc.buffat/2022/BOOK_VALIDATION.
This platform has grown as a prototype in the Mechanical Engineering department of the Lyon 1 University. Thanks to the AMI INCLUDE project, it is being refined, redeployed on new servers and will eventually be available to other educational institution around Lyon.
During the talk, a live demonstration will illustrate how students and teachers use the platform. Finally, we will share feedback from teachers that have run courses on the platform.
This includes assignments during the semester as well as exams. Nbgrader provides basic tools for creating, grading, and exchanging assignments. We developed e2xgrader, an nbgrader add-on, to address the needs of three types of users: (i) students, (ii) graders, and (iii) teachers.
Students, especially in undergraduate STEM courses, come with varying levels of knowledge in programming and how to use Jupyter Notebooks. The freedom provided by Jupyter Notebooks has led to several problems in grading and teaching. Using JupyterHub eliminates issues with setting up local environments and ensures fairness by providing a standardized computing environment. However, students may still accidentally delete or alter nbgrader cells or notebooks. In interpreted languages, students may import libraries in a later cell and then use it in an earlier cell, causing further problems.
To address these issues, we have implemented measures to restrict the notebook and provide clear instructions for students. We provide a visual indicator in the form of a cell toolbar to help students know where to write solutions. During exams, we replace the standard view with a simple toolbar that includes a save and submit button and links to provided resources. Students can view their submission in HTML, and are given a restricted Python kernel with preloaded libraries and disabled terminal commands and magics. This kernel is fully configurable.
To support the transfer of paper grading processes to electronic systems, we added the ability for graders to annotate student submissions by drawing on them and display them in feedback.
We also implemented a new grading view that allows graders to view submissions horizontally (e.g. looking at one exercise for all students).
After partially grading a notebook, graders may realize that the automatic test cases need improvement. To address this, we added the option to autograde only specific cells. After updating a test case, the grader can run the autograding process again for the changed cells, and only the relevant grades will be updated. Graders can also export grades in CSV format for use in other university systems. This provides greater control and ease of use in the grading process.
Teachers need tools that make it easier to create assignments. To address this, we created a tool that organizes simple exercises into pools by topic. Teachers can choose from a variety of preset question types and create their own, and can use templates to quickly assemble assignments from the pools. We also added version control to the pools and exercises to make the process more transparent. This provides greater efficiency and flexibility in assignment creation.
Teachers need specific cell types for different courses. For example, in a software engineering class, they may want students to create diagrams, while in a mathematics class they may want to give students the ability to solve exercises on paper and upload an image of the solution. To support these needs, we added multiple and single choice cells, a fully integrated diagram editor cell based on diagrams.net, and a cell for uploading files and taking pictures from a webcam.
Finally, teachers wanted greater standardization and ease of use in authoring test cases. For this, we provide the assignmenttest package which can assign partial grades and provides an easy way to author test cases for variables, functions and classes in Python.
Summing up, the Jupyter ecosystem provides the needed to flexibility to enhance the effectivity of learning, teaching and grading for students, staff and professors, with e2xgrader.


