DSPy Explained! Connor Shorten 2024-01-29 | Hey everyone! Thank you so much for watching this explanation of DSPy! DSPy is a super exciting new framework for developing LLM programs! Pioneered by frameworks such as LangChain and LlamaIndex, we can build much more powerful systems by chaining together LLM calls! This means that the output of one call to an LLM is the input to the next, and so on. We can think of chains as programs, with each LLM call analogous to a function that takes text as input and produces text as output.DSPy offers a new programming model, inspired by PyTorch, that gives you a massive amount of control over these LLM programs. Further, the Signature abstraction wraps prompts and structured input / outputs to clean up LLM program codebases. DSPy then pairs the syntax with a super novel *compiler* that jointly optimizes the instructions for each component of an LLM program, as well as sourcing examples of the task.Here is my review of the ideas in DSPy, covering the core concepts and walking through the introduction notebooks showing how to compile a simple retrieve-then-read RAG program, as well as a more advanced Multi-Hop RAG program where you have 2 LLM components to be optimized with the DSPy compiler! I hope you find it useful!Please send me a message @CShorten30 on X, I would love to discuss what you are working on and offer any help with DSPy and/or Weaviate!Useful links:Join the DSPy Discord! discord.gg/ckYTFvAPe3DSPy GitHub: github.com/stanfordnlp/dspyDSPy Paper: arxiv.org/abs/2310.03714DSPy Assertions: arxiv.org/abs/2312.13382A Guide to LLM Abstractions: twosigma.com/articles/a-guide-to-large-language-model-abstractionsJason Liu: Pydantic is all you need: youtube.com/watch?v=yj-wSRJwrrcLlamaIndex Query Engines: docs.llamaindex.ai/en/stable/module_guides/deploying/query_engine/root.html#modules LangChain: js.langchain.com/docs/use_casesChapters0:00 DSPy!1:37 The DSPy Programming Model5:07 LLM Programs8:56 Programming, Not Prompting13:48 PyTorch Analogy19:16 The DSPy Compiler27:50 Metrics30:13 Teleprompters30:50 Code Example50:37 Start using DSPy!
Chunking with Generative Feedback Loops Connor Shorten 2024-08-12 | Hey everyone! I am super excited to share a quick notebook on using Generative Feedback Loops to chunk code files and better structure how they are indexed in the Weaviate Vector Database! Chunking is one of the key topics in Vector Search. We need to break up long documents into smaller parts that we can encode with a pre-trained embedding model and index in a vector index, such as HNSW-PQ. Most solutions use some form of a rolling token window such as taking every 300 tokens as a chunk, with say 50 tokens overlapping between each window. Unfortunately, this solution doesn't work that well for code particularly. We don't want the chunk to cut off in the middle of a function or class definition. Thus, this tutorial employs Generative Feedback Loops to analyze the best places to chunk the code file and write a natural language description of what the code in the chunk does.I hope this inspires your interest in Generative Feedback Loops, soon more! Also please let us know if you have any issues with running the code in the notebook, more than happy to help!Code in Weaviate Recipes - github.com/weaviate/recipesThe dataset used is the DSPy repository, which can be found here - github.com/stanfordnlp/dspy/tree/mainHere are some experiments we've done on OPRO prompt optimization to achieve JSON outputs without Structured Decoding - github.com/weaviate/structured-ragChapters:0:00 Introduction3:07 Where to find the code5:15 What is Semantic Chunking?6:22 Code deep dive
Google Gemini 1.5 Pro and Flash - Demo of Long Context LLMs! Connor Shorten 2024-05-17 | Hey everyone! Thanks so much for watching this video exploring Gemini Pro 1.5 and Gemini Flash! Long Context LLMs!! This video covers 3 key tests, the classic "Lost in the Middle" exploration, using Long Context LLMs as Re-rankers in Search, and finally, testing Many-Shot In-Context Learning! I am really excited about the potential of Many-Shot In-Context Learning with DSPy's `BootstrapFewShot` and Gemini, curious to know what you think!Notebook: github.com/weaviate/recipes/blob/main/integrations/llm-frameworks/dspy/llms/Gemini-1.5-Pro-and-Flash.ipynbGemini 1.5 Technical Report: storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdfChapters0:00 Gemini 1.5!!1:25 Setup and Overview4:30 Lost in the Middle test7:18 Gemini for Re-ranking9:22 Many-Shot In-Context Learning
Llama 3 RAG Demo with DSPy Optimization, Ollama, and Weaviate! Connor Shorten 2024-04-18 | Hey everyone! Thank you so much for watching this overview of Llama 3 looking at the release notes and seeing a demo of how to integrate it with DSPy through Ollama and how to use DSPy's MIPRO to find the optimal prompt when using this new large language model for RAG!We are hosting an event in San Francisco on May 1st with Arize AI and Cohere, featuring a talk from Omar Khattab, the lead author of DSPy! Hope to see you there! https://lu.ma/dspyIntroducing Meta Llama 3: ai.meta.com/blog/meta-llama-3Ollama Llama 3: ollama.com/library/llama3Weaviate Recipes: github.com/weaviate/recipes/blob/main/integrations/llm-frameworks/dspy/llms/Llama3.ipynbChapters0:00 Llama3!!1:28 Release Notes5:35 Performance Reporting9:50 Training Details17:32 DSPy Demo!
Building RAG with Command R+ from Cohere, DSPy, and Weaviate! Connor Shorten 2024-04-04 | Hey everyone! Thank you so much for watching this overview of Command R+ showing you how you can use the new model in DSPy and a quick RAG demo, as well as walking through the details of the release post! Congratulations to the Cohere team! Super exciting times to be working with LLM systems!Introducing Command R+: A Scalable LLM Built for Business - txt.cohere.com/command-r-plus-microsoft-azureLink to demo notebook - github.com/weaviate/recipes/blob/main/integrations/llm-frameworks/dspy/llms/Command-R-Plus.ipynbChapters0:00 Welcome! Command R+!1:12 Demo with Cohere, DSPy, and Weaviate6:06 Command R+ Announcement Post9:24 LLM Evals
Structured Outputs with DSPy Connor Shorten 2024-04-01 | The code for this notebook can be found here! - github.com/weaviate/recipes/blob/main/integrations/llm-frameworks/dspy/4.Structured-Outputs-with-DSPy.ipynbUnfortunately, Large Language Models will not consistently follow the instructions that you give them. This is a massive problem when you are building AI systems that require a particular type of output from the previous step to feed into the next one! For example, imagine you are building a blog post writing system that first takes a question and retrieved context to output a list of topics. These topics have to be formatted in a particular way, such as a comma-separated list or a JSON of Topic objects, such that the system can continue writing the blog post!I am SUPER excited to share the 4th video in my DSPy series, diving into 3 solutions to structuring outputs in DSPy programs: (1) TypedPredictors, (2) DSPy Assertions, and (3) Custom Guardrails with the DSPy programming model!TypedPredictors follow the line of thinking around JSON mode and using Pydantic BaseModels to interface types and custom objects into a JSON template for LLMs. The output can then be validated to provide a more structured retry prompt to correct the output structure!DSPy Assertions are one of the core building blocks of DSPy, offering an interface to input a boolean-valued function and a retry prompt which is templated alongside the past output to retry the call to the LLM!Custom Guardrails with the DSPy Programming Model are one of the things I love the most about DSPy — we have unlimited flexibility to control these systems however we want. The video will also show you how to write custom guardrails and retry Signatures and discussion around using TypedPredictors for your Custom Guardrails and potentially feeding your Custom Guardrails into a DSPy Assertion.I had so much fun exploring this topic! Further seeing how well OpenAI’s GPT-4 and GPT-3.5-Turbo, Cohere’s Command R, and Mistral 7B hosted with Ollama perform with each Structured Output strategy! I also found monitoring structured output retries to be another fantastic application of Arize Phoenix! I hope you find the video useful!If interested, all the code examples for this DSPy series can be found here! - github.com/weaviate/recipes/tree/main/integrations/llm-frameworks/dspyOpenAI Function Calling: openai.com/blog/function-calling-and-other-api-updatesGorilla LLM Function Calling Leaderboard: https://gorilla.cs.berkeley.edu/leaderboard.htmlInstructor Examples: github.com/jxnl/instructor/tree/main/examples**ERRATA** (Massive thank you to Thomas Ahle for sending some notes and clarifications of content covered in the video)1. `dspy.TypedPredictor` can be used directly instead of `dspy.functional.TypedPredictor`2. When creating a pydantic type, `list[Topic]` can be used directly in the Signature without needing the `Topics` wrapper.3. The default `max_retry` for TypedPredictor is 3, and can be set when creating the TypedPredictor.4. Setting `TypedPredictor(explain_errors=True)` can help with retry errors by providing clearer descriptions of what needs to change.Chapters0:00 Welcome! Let’s Structure LLM Outputs!3:12 Background (Instructor, Function Calling, JSON mode)9:08 TypedPredictors Demo16:00 Logging with Arize Phoenix18:15 DSPy Assertions22:15 Custom Guardrails
Adding Depth to DSPy Programs Connor Shorten 2024-03-04 | Hey everyone! Thank you so much for watching the 3rd edition of the DSPy series, Adding Depth to DSPy Programs!! This video begins with some DSPy news such as STORM, DSPy Assertions, and Typed Signatures! We then dive into the concept of adding depth to DSPy programs, taking a further look at what it means to have unique input-output examples for each component and how we can compose DSPy programs with different LLMs per component! We then dive into two notebooks illustrating adding depth to RAG programs and a 4-layer question to blog post writer!Demo #1 Notebook: github.com/weaviate/recipes/blob/main/integrations/llm-frameworks/dspy/3.Adding-Depth-to-RAG-Programs.ipynbDemo #2 Notebook: github.com/weaviate/recipes/blob/main/integrations/llm-frameworks/dspy/2.Writing-Blog-Posts-with-DSPy.ipynbYou can find the examples and links to community resources / news on github.com/weaviate/recipes!Chapters0:00 Intro0:50 Chapters Overview5:06 Weaviate Recipes5:24 DSPy News and Community Notes13:51 Adding Depth to RAG Programs18:40 Multi-Model DSPy Programs20:18 DSPy Optimizers25:30 Deep Dive Optimizers27:55 Into the Optimizer Code!37:48 Demo #1: Adding Depth to RAG1:05:25 Demo #2: Questions to Blogs1:07:48 Thank you so much for watching!
Getting Started with RAG in DSPy! Connor Shorten 2024-02-12 | Hey everyone! Thank you so much for watching this tutorial on getting started with RAG programming in DSPy! This video will take you through 4 major aspects of building DSPy programs (1) Installation, settings, and Datasets with dspy.Example, (2) LLM Metrics, (3) The DSPy programming model, and (4) Optimization!!The notebook used in the video can be found here: github.com/weaviate/recipes/blob/main/integrations/llm-frameworks/dspy/1.Getting-Started-with-RAG-in-DSPy.ipynbAll future videos, as well as additional utils like data import scripts, will be in this folder: github.com/weaviate/recipes/tree/main/integrations/llm-frameworks/dspyPlease leave a star, it helps a lot!DSPy on GitHub: github.com/stanfordnlp/dspyJoin the DSPy Discord! discord.gg/HZwtqNzKCuDSPy GPT by Sean Chatman: chat.openai.com/g/g-cH94JC5NP-dspy-guide-v2024-2-7New Signature class based on Pydantic by Thomas Ahle: github.com/stanfordnlp/dspy/pull/368Knox, Ollama in DSPy: twitter.com/JR_Knox1977/status/1756018720818794916Stephen Byrne, Why I'm excited about DSPy: https://substack.stephen.so/p/why-im-excited-about-dspyChapters0:00 Intro!1:35 Where to find the code2:08 Community Notes4:00 Getting Started with RAG7:56 0. DSPy Settings and Installation10:00 1. DSPy Datasets11:56 2. LLM Metrics19:22 3. The DSPy Programming Model23:32 4. DSPy Optimization30:45 Recap
Approximate Nearest Neighbor Benchmarks - Weaviate Podcast Recap Connor Shorten 2022-05-30 | Please check out the full podcast here: youtube.com/watch?v=kG3ji89AFyQThis video is a commentary on the latest Weaviate Podcast with Etienne Dilocker on ANN Benchmarks. ANN search -- short for Approximate Nearest Neighbors -- describes algorithms that enable efficient distance comparison between an encoded query vector and a vector database. For example, we may have 1 billion vectors to search through -- we don't want to do a dot product distance between our query and 1 billion candidate vectors! This podcast describes Weaviate's efforts to benchmark HNSW within the Weaviate system and give users a sense of how performance varies with respect to each dataset (and their respective properties), as well as the hyperparameters of the HNSW algorithm.I hope you find this useful, happy to answer any questions / hold any discussion! Thank you for watching!
Search through Y Combinator startups with Weaviate! Connor Shorten 2022-04-03 | Please check out Eric Jang's article "Ranking YC Companies with a Neural Net": evjang.com/2022/04/02/yc-rank.htmlPlease subscribe to SeMI Technologies on YouTube! youtube.com/c/SeMI-and-WeaviateTimecodes0:00 Introduction0:58 Weaviate Demo3:40 Article Overview10:45 NLP for Venture Capital and Data-Centric AI
MosaicML Composer for faster and cheaper Deep Learning! Connor Shorten 2022-03-28 | Please leave a star! github.com/mosaicml/composerThank you so much for watching! This video presents some details of MosaicML's Composer launch and how to use it in Python. I am really excited about this company and their mission to deliver faster and cheaper Deep Learning training! I hope you find this video useful, happy to answer any questions you might have about this or these ideas in Efficient Deep Learning generally!The full Weaviate podcast with Jonathan Frankle will be uploaded very soon on SeMI Technologies YouTube, please subscribe!youtube.com/c/SeMI-and-WeaviateChapters0:00 Introduction1:45 Documentation Intro4:20 Composer Notebooks5:35 Functional API10:25 Composer Trainer15:35 Methods Overview16:58 Jonathan Frankle18:25 Podcast Clip
Jina AI DocArray - Documentation Overview Connor Shorten 2022-03-19 | I hope you found this useful, please let me know if you have any questions or ideas!Docarray Documentation: docarray.jina.aiFull-Length Podcast: youtube.com/watch?v=HIGAQAE_xaICode Tutorial (Weaviate + Jina AI for Image Search): youtube.com/watch?v=rBKvoIGihnYPlease check out Jina AI on YouTube: youtube.com/c/JinaAIPlease check out SeMI Technologies on YouTube: youtube.com/c/SeMI-and-Weaviate/videos
What lead Jina AI CEO Han Xiao to Neural Search? Connor Shorten 2022-03-17 | This video explains one of the biggest lessons for me in interviewing Han Xiao from Jina AI. I hope this was a good explanation of the preprocessing / granularity of embeddings and how that can enable different kinds of search applications.Full-Length Podcast: youtube.com/watch?v=HIGAQAE_xaICode Tutorial (Weaviate + Jina AI for Image Search): youtube.com/watch?v=rBKvoIGihnYPlease check out Jina AI on YouTube: youtube.com/c/JinaAIPlease check out SeMI Technologies on YouTube: youtube.com/c/SeMI-and-Weaviate/videosChapters0:00 Introduction
Full Stack Neural Search Connor Shorten 2022-03-17 | This video explains one of the biggest lessons for me in interviewing Han Xiao from Jina AI. I hope this was a good explanation of the preprocessing / granularity of embeddings and how that can enable different kinds of search applications.Full-Length Podcast: youtube.com/watch?v=HIGAQAE_xaICode Tutorial (Weaviate + Jina AI for Image Search): youtube.com/watch?v=rBKvoIGihnYPlease check out Jina AI on YouTube: youtube.com/c/JinaAIPlease check out SeMI Technologies on YouTube: youtube.com/c/SeMI-and-Weaviate/videosChapters0:00 Please check out SeMI YouTube!0:15 My takeaways on Full Stack Neural Search11:04 Podcast Clip - Han Xiao
Python Tutorial: How to use Weaviate and Jina AI for Image Search! Connor Shorten 2022-03-16 | I hope this video helps you get started with Image Search using Weaviate and Jina AI - happy to answer any questions / help solve problems!Check out the full tutorial explanation from Laura Ham: youtube.com/watch?v=rBKvoIGihnYNew podcast with Jina AI CEO Han Xiao! youtube.com/watch?v=HIGAQAE_xaIFull notebook code: github.com/laura-ham/HM-Fashion-image-neural-search/blob/main/hm-fashion-image-neural-search.ipynbGet started with the Weaviate Cloud Service: console.semi.technology
Causal Inference in Deep Learning (Podcast Overview with Brady Neal) Connor Shorten 2022-03-01 | Hey everyone! Hopefully this video helps supplement the new Weaviate podcast with Brady Neal, I hope you find this interesting / useful!Check out Brady Neal on YouTube! youtube.com/c/BradyNealCausalInference/featuredWeaviate Podcast: youtube.com/watch?v=t7g9s1GWcB80:00 New Weaviate Podcast!0:42 Brady Neal Causal Inference1:34 Oogway.ai2:45 Whiteboard Ideas5:35 Discussion Topics
OpenAI Embeddings API - (Interview Recap and Background) Connor Shorten 2022-02-12 | Hey everyone! I recently interviewed Arvind Neelakantan from OpenAI about the new OpenAI Embeddings API on the Weaviate Podcast! This video provides some additional detail for the different topics that were discussed. If you find this video to be informative, please check out SeMI technologies on youtube where we are working hard on developing content explaining concepts in Deep Learning for Search.Full Podcast: youtube.com/watch?v=uFxfZ0vLsoUSeMI Technologies on YouTube: youtube.com/channel/UCJKT6kJ3IFYybWnL7jbXxhQ
AI Weekly Update - February 7th, 2022 Connor Shorten 2022-02-07 | Thanks for watching! Please subscribe for more Deep Learning and AI videos, the list of papers is below under "Content Links"Please subscribe to SeMI Technologies (creators of the Weaviate Vector Search Engine) on YouTube: youtube.com/c/SeMI-and-WeaviateContent Links:Fully Online Meta-Learning without Task Boundaries: arxiv.org/abs/2202.00263Datamodels: Predicting Predictions from Training Data: arxiv.org/abs/2202.00622Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantization: arxiv.org/abs/2202.01334Competition-Level Code Generation with AlphaCode: storage.googleapis.com/deepmind-media/AlphaCode/competition_level_code_generation_with_alphacode.pdfGPT-NeoX-20B: blog.eleuther.ai/announcing-20bPromptSource: arxiv.org/abs/2202.01279Chain of Thought Prompting Elicits Reasoning in Large Language Models: arxiv.org/abs/2201.11903Scaling Laws for Routed Language Models: arxiv.org/abs/2202.01169Active Learning over Multiple Domains in Natural Language Tasks: arxiv.org/abs/2202.00254Can Robots Follow Instructions for New Tasks? ai.googleblog.com/2022/02/can-robots-follow-instructions-for-new.htmlBLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation: arxiv.org/pdf/2201.12086.pdfHow to Leverage Unlabeled Data in Offline Reinforcement Learning: arxiv.org/abs/2202.01741The Challenges of Exploration for Offline Reinforcement Learning: arxiv.org/pdf/2201.11861.pdfETSformer: Exponential Smoothing Transformers for Time-series Forecasting: arxiv.org/pdf/2202.01381.pdfCoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting: arxiv.org/abs/2202.01575Can Transformers be Strong Treatment Effect Estimators? arxiv.org/abs/2202.01336Chapters:0:00 Introduction0:18 Weaviate Vector Search0:34 Fully Online Meta-Learning6:03 Datamodels7:38 Dynamic Vector Quantization9:57 AlphaCode14:35 GPT-NeoX-20B15:32 PromptSource17:52 Chain of Thought Prompting19:52 Scaling Laws for Routed Language Models21:12 Active Learning over Multiple Domains22:35 BC-Z (Vision-Based Robots and Language)24:10 Bootstrapping Language-Image Pre-training (BLIP)25:38 How to Leverage Unlabeled Offline RL Data26:24 Challenges of Exploration for Offline RL27:04 ETSformer27:34 CoST28:32 Can Transformers be Strong Treatment Effect Estimators?
Deep Learning for Podcast Content Search (Summary of Interview with Alex Canan at Zencastr) Connor Shorten 2022-02-04 | This video gives an overview of the latest Weaviate podcast, please subscribe to see future episodes!youtube.com/c/SeMI-and-Weaviate/videosThanks for watching!Chapters0:00 Overview7:53 Ideas for Podcast Search10:44 Weaviate Podcast so far
AI Weekly Update - January 31st, 2022 Connor Shorten 2022-01-31 | Thank you so much for watching, please subscribe for more Deep Learning and Ai videos! Please check out SeMI Technologies on YouTube as well, where I am hosting a podcast on Deep Learning for Search!Paper Links:Text and Code Embeddings by Contrastive Pre-Training: cdn.openai.com/papers/Text_and_Code_Embeddings_by_Contrastive_Pre_Training.pdfIntroducing Text and Code Embeddings in the OpenAI API (Blog Post): openai.com/blog/introducing-text-and-code-embeddingsNils Reimers - OpenAI GPT-3 Text Embeddings - Really a new state-of-the-art in dense text embeddings? medium.com/@nils_reimers/openai-gpt-3-text-embeddings-really-a-new-state-of-the-art-in-dense-text-embeddings-6571fe3ec9d9Training language models to follow instructions with human feedback: cdn.openai.com/papers/Training_language_models_to_follow_instructions_with_human_feedback.pdfReasoning Like Program Executors: arxiv.org/pdf/2201.11473.pdfSynchromesh: Reliable Code Generation from Pre-Trained Language Models: arxiv.org/pdf/2201.11227.pdfArtefact Retrieval: Overview of NLP Models with Knowledge Base Access: arxiv.org/pdf/2201.09651.pdfGreaseLM: Graph Reasoning Enhanced Language Models for Question Answering: arxiv.org/pdf/2201.08860.pdfRelational Memory Augmented Language Models: arxiv.org/pdf/2201.09680.pdfCodeRetriever: Unimodal and Bimodal Contrastive Learning for Code Search: arxiv.org/pdf/2201.10866.pdfOut-of-Domain Semantics to the Rescue! Zero-Shot Hybrid Retrieval Models: arxiv.org/pdf/2201.10582.pdfNatural Language Descriptions of Deep Visual Features: arxiv.org/pdf/2201.11114.pdfDecoupling the Role of Data, Attention, and Losses in Multimodal Transformers: arxiv.org/pdf/2102.00529.pdfEnvironment Generation for Zero-Shot Compositional Reinforcement Learning: arxiv.org/pdf/2201.08896.pdfAI in health and medicine: nature.com/articles/s41591-021-01614-0.pdfChapters0:00 Introduction1:28 Text and Code Embeddings by Contrastive Pre-Training10:20 Training LMs to Follow Instructions12:34 Reasoning Like Program Executors14:36 Synchromesh17:38 Artefact Retrieval19:38 GreaseLM21:58 Relational Memory Augmented Language Models23:45 CodeRetriever24:48 Out-of-Domain Semantics to the Rescue!27:36 Natural Language Descriptions of Deep Visual Features (MILAN)30:05 Decoupling the Role of Data, Attentino, and Losses in Multimodal Transformers32:10 Environment Generation for Zero-Shot Compositional Reinforcement Learning34:52 AI in Health and Medicine
AI Weekly Update - January 24th, 2022 Connor Shorten 2022-01-24 | Thank you so much for watching, please subscribe for more Deep Learning and AI videos! Please check out SeMI Technologies on YouTube as well!Paper Links:CM3: arxiv.org/abs/2201.07520data2vec: scontent.fmia1-1.fna.fbcdn.net/v/t39.8562-6/271974914_483120576492438_4239522333319653600_n.pdf?_nc_cat=107&ccb=1-5&_nc_sid=ae5e01&_nc_ohc=4-cMR5tUq4QAX-Of7fj&_nc_ht=scontent.fmia1-1.fna&oh=00_AT9ymN9dNPt1p8zWQClW6MSZikaCTT8gobc2LqxW4OhzZQ&oe=61F3F7D1LaMDA: arxiv.org/abs/2201.08239PromptBERT: arxiv.org/abs/2201.04337UnifiedSKG: arxiv.org/abs/2201.05966Collapse by Conditioning: arxiv.org/abs/2201.06578GradTail: arxiv.org/abs/2201.05938CLIP-TD: arxiv.org/abs/2201.05729CoAuthor: arxiv.org/pdf/2201.06796.pdfMemory-assisted prompt editing to improve GPT-3 after deployment: arxiv.org/abs/2201.06009ZeroPrompt: arxiv.org/abs/2201.06910HyperTransformer: arxiv.org/abs/2201.04182Chapters0:00 Introduction0:45 ak92501 on Twitter1:05 Weaviate Sponsorship1:38 CM35:08 UnifiedSKG7:22 data2vec10:05 PromptBERT12:45 ZeroPrompt14:40 CLIP-TD16:10 Memory-Assisted prompt editing for GPT-318:35 LaMDA22:02 CoAuthor24:20 GradTail25:43 Collapse by Conditioning27:32 HyperTransformer
Deep Learning for Search - January 15th, 2022 Connor Shorten 2022-01-16 | I have spent the last 3 months studying the Weaviate Vector Search Engine. I am a PhD student who has been studying Deep Learning and making videos about emerging ideas. Search really caught by attention when studying Deep Learning applications for COVID-19. I was heavily inspired by the CO-Search system from Salesforce research, as well as parallel work in NLP such as the Text-to-Text Transfer Transformer (T5) and Retrieval-Augmented Generation (RAG). I was delighted when Bob van Luijt contacted me to collaborate and have since learned so much about Deep Learning for Search. This video presents an overview of these ideas. I intend to update this as I continue making podcasts for SeMI Technologies / Weaviate. Please subscribe to SeMI on YouTube to stay up to date with these podcast! Thank you so much for watching!SeMI YouTube: youtube.com/c/SeMI-and-WeaviateCO-Search: arxiv.org/abs/2006.09595Haystack Pipelines: haystack.deepset.ai/components/pipelinesRetrieval-Augmented Generation: ai.facebook.com/blog/retrieval-augmented-generation-streamlining-the-creation-of-intelligent-natural-language-processing-modelsChapters0:00 Introduction0:38 Please Subscribe!1:08 Architecture Overview11:22 CO-Search14:20 Haystack Pipelines16:45 Retrieval-Augmented Generation18:00 CLIP
Weaviate and Haystack Connor Shorten 2022-01-09 | Here are some ideas on how to combine Weaviate and Haystack! Weaviate can be used as the DocumentStore, serving as the database for Haystack retrievers to search through. I am really excited about the Neurosymbolic Search Weaviate can add to Haystack retrieval pipelines. Haystack has built a Query Classifier to route symbolic and neural queries, but I think it would be even more exciting to have a Query Annotator that adds symbolic filtering to neural search which is something offered in Weaviate and that I think could have a really interesting plug-in functionality for Haystack pipelines! Please share any thoughts or questions you have on these ideas!Please subscribe to SeMI Technologies on YouTube for the full podcast with Malte Pietsch discussing several ideas like this: youtube.com/channel/UCJKT6kJ3IFYybWnL7jbXxhQHaystack Documentation on DocumentStores combining Weaviate with Haystack: haystack.deepset.ai/components/document-storeLaura Ham on Data Science with Weaviate and GraphQL: youtube.com/watch?v=9sFBrcpKDGMChapters0:00 Introduction0:34 Haystack Docs3:17 Data Science with Weaviate and GraphQL4:55 Haystack Query Classifier ideas7:38 clip from Interview with Malte Pietsch
General Purpose Readers Connor Shorten 2022-01-09 | This video explains some ideas around General Purpose Reader models. The idea is to split up the tasks Deep Learning has gone after into "retrieve" and then "read" components. The retriever component offers a lot of flexibility and I am more convinced the read component will have the general adaptability promised by things like the GPT-3 API. Please share any thoughts you have on these ideas!Please subscribe to SeMI Technologies on YouTube for the full podcast with Malte Pietsch discussing several ideas like this: youtube.com/channel/UCJKT6kJ3IFYybWnL7jbXxhQHaystack Pipelines: haystack.deepset.ai/components/pipelinesSQuAD: rajpurkar.github.io/SQuAD-explorer/explore/v2.0/dev/Oxygen.html?model=nlnet%20(single%20model)%20(Microsoft%20Research%20Asia)&version=v2.0OpenAI API: openai.com/blog/openai-apiDecision Transformer: arxiv.org/pdf/2106.01345.pdfCO-Search: arxiv.org/abs/2006.09595Chapters0:00 Introduction2:36 SQuAD motivating example of retrieval4:02 CO-Search, Vector Search ImageNet moment7:05 Haystack Pipelines7:42 Benefits of Retrieve-then-Read9:29 Research Ideas13:15 Precision Medicine Applications14:55 OpenAI API16:10 Interview with Malte Pietsch (Haystack)
Binary Passage Retrieval in Weaviate (32x Memory Savings) Connor Shorten 2021-12-24 | This video presents a quick overview of Binary Passage Retrieval in Weaviate! Huge congratulations to Etienne Dilocker (CTO and Co-Founder of Weaviate) for putting this together, it was really awesome getting to watch his thought process and brainstorming around putting this together! Really excited to discuss this further on the Weaviate Podcast, please subscribe to SeMI Technologies to be informed when this is published!LinksBPR Weaviate Docs: https://www.semi.technology/developers/weaviate/current/architecture/binary_passage_retrieval.htmlEfficient Passage Retrieval with Hashing for Open-domain Question Answering: arxiv.org/abs/2106.00882TechCrunch Apple XNOR.ai: techcrunch.com/2020/01/15/apple-buys-edge-based-ai-startup-xnor-ai-for-a-reported-200mSimCLR: arxiv.org/abs/2002.05709Barlow Twins: arxiv.org/abs/2103.03230HuggingFace Spaces: huggingface.co/spacesWeaviate Wikipedia Demo: github.com/semi-technologies/semantic-search-through-Wikipedia-with-WeaviateWeaviate Wikidata Demo: github.com/semi-technologies/biggraph-wikidata-search-with-weaviateHappy Holidays!
Keras Code Search with Weaviate Connor Shorten 2021-12-22 | This video illustrates how to upload the Keras Code Examples into the Weaviate Vector Search Engine!Keras Code Examples: keras.io/examplesWeaviate Quick Start: https://www.semi.technology/developers/weaviate/current/getting-started/I would really appreciate a like on this HuggingFace spaces if you have a chance, thanks in advance! huggingface.co/spaces/merve/KerasBERTv1
Healthsea from Spacy on HuggingFace Spaces Connor Shorten 2021-12-19 | Check it out for yourself! huggingface.co/spaces/spacy/healthsea-pipelineHere is another interesting paper about this kind of system -SUPP.AI: Finding Evidence for Supplement-Drug Interactions, authored by Lucy Lu Wang, Oyvind Tafjord, Arman Cohan, Sarthak Jain, Sam Skjonsberg,Carissa Schoenick, Nick Botner, Waleed Ammar from the Allen Institute of Artificial Intelligence in Seattle, Washington.This video does not contain medical advice.
Open-Source Deep Learning Connor Shorten 2021-12-18 | This video presents some ideas Open-Source Deep Learning, and particularly a recent trend in creating more User Interfaces of the capabilities of Deep Learning! We are very excited about presenting demonstrations of Deep Learning Search with Weaviate! This video previews Bob and I's full length discussion about all things Vector Search, please check out the SeMI Technologies YouTube channel for the full video:youtube.com/c/SeMI-and-WeaviateHuggingFace Spaces: huggingface.co/spaces
Deep Learning in Context (Thoughts on OpenAI WebGPT and DeepMind Retro) Connor Shorten 2021-12-18 | It feels like a really exciting to be following Deep Learning research! This video explains some ideas around Information Retrieval with Supervised Learning and how this can improve the abilities of Deep Learning. This video previews Bob and I's full length discussion about all things Vector Search, please check out the SeMI Technologies YouTube channel for the full video:youtube.com/c/SeMI-and-WeaviateLinks:WebGPT: openai.com/blog/improving-factual-accuracyRETRO: deepmind.com/research/publications/2021/improving-language-models-by-retrieving-from-trillions-of-tokens
Wikipedia Vector Search Demo with Weaviate Connor Shorten 2021-12-08 | Check out the demo for yourself! github.com/semi-technologies/semantic-search-through-Wikipedia-with-WeaviateBob's Article - Semantic search through Wikipedia with Weaviate (GraphQL, Sentence-BERT, and BERT Q&A): towardsdatascience.com/semantic-search-through-wikipedia-with-weaviate-graphql-sentence-bert-and-bert-q-a-3c8a5edeacf6This video will present the Wikipedia Vector Search Demo with Weaviate. Wikipedia is a great information source to test the next generation of search systems built on Deep Learning and implemented in Weaviate This is a really great visualization of neural search because it’s very intuitive to compare this to search systems we are all familiar with such as google search.This presentation will describe how information retrieval can enhance supervised learning tasks. Particularly, this is done with the Weaviate Question Answering Module, from the demo you will know exactly how to do this yourself. We’ll further present the generalization of these ideas to AlphaFold2, Frozen, GPT-3, more examples of how information retrieval can supplement supervised learning tasks across data domains from proteins to images and text.As a reminder, this web demo is freely accessible across the world, so please check out the GraphQL console for yourself before or after watching this video. This demo is linked in the video description as well as a pinned comment on the video.Additional Links:SQuAD: rajpurkar.github.io/SQuAD-explorerKILT: ai.facebook.com/blog/introducing-kilt-a-new-unified-benchmark-for-knowledge-intensive-nlp-tasksCO-Search: arxiv.org/pdf/2006.09595.pdfChallenges in Open-Domain Question Answering Generalization: arxiv.org/pdf/2109.01156.pdfAlphaFold2: deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biologyGPT-3: arxiv.org/pdf/2005.14165.pdfFrozen: arxiv.org/pdf/2106.13884.pdfMetaICL: arxiv.org/pdf/2110.15943.pdfChapters0:00 Introduction1:17 What is Wikipedia?2:27 Wikipedia NLP Tasks5:18 Wikipedia Dataset Statistics7:07 How this was setup7:55 Demo Query #19:24 Demo Query #211:20 Demo Query #312:08 Demo Query #413:26 CO-Search15:11 Retrieve-then-Read for NLP17:50 Retrieve-then-Read for AlphaFold2 and more20:28 General Ideas
Vector Search through Wikidata with Weaviate Connor Shorten 2021-12-03 | I’ve recently been working with the Weaviate team on Vector Search Engines. One of our projects that I’m the most excited about is creating Web Demos where anyone across the world can see the power of Neural Search for themselves -- for free.I’m really to excited to share that there are 2 new web demos in WeaviateWikidata Weaviate Demo (subject of this video): github.com/semi-technologies/biggraph-wikidata-search-with-weaviateDemo: http://biggraph-wikidata-in-weaviate.vectors.network/Wikipedia Weaviate Demo: github.com/semi-technologies/semantic-search-through-Wikipedia-with-WeaviateThis video will explains all sorts of things around this such as: (1) the difference between Wikidata and Wikipedia, (2) how graph embeddings are computed with Facebook AI’s PyTorch-BigGraph, (3) a visualization of the demo itself, and (4) a conclusion where I present my ideas on how I’m using graph-structured data and how I’m currently about thinking about the field of Graph Neural Networks and Knowledge Graphs.Thank you so much for watching, I really hope you enjoy this presentation and would be more than happy to answer any questions you might have about these ideas.Content Links:Wikidata Weaviate Demo:Wikipedia Weaviate Demo: Wikidata Properties: wikidata.org/wiki/Help:PropertiesPyTorch-BigGraph: ai.facebook.com/blog/open-sourcing-pytorch-biggraph-for-faster-embeddings-of-extremely-large-graphsDeep Learning for Graph-Structured Representations, Thomas Kipf PhD Thesis: https://pure.uva.nl/ws/files/46900201/Thesis.pdfChapters0:00 Intro0:48 Wikidata versus Wikipedia4:10 Word2Vec versus BERT5:05 PyTorch-BigGraph7:07 Wikidata Knowledge Graph8:00 Weaviate Demo10:07 Nearest Neighbors to “Deep Learning”10:52 Graph Vector Search11:08 Drug Repurposing Graphs12:22 Citation Network Keras Example13:53 KerasBERT and Graph Data15:48 Thoughts on Graph Embeddings19:55 Weaviate Web Demos!
Demonstrations of Deep Learning Connor Shorten 2021-11-30 | Demonstrations of Deep Learning inspire our creativity and help us understand where we’re really at with Deep Learning - such as the strengths and weaknesses of these systems.Traditionally, sharing Deep Learning systems at a large-scale required serious software engineering skills... But! There’s a very exciting trend in overcoming bottlenecks to deployment, and enabling easy sharing of Deep Learning models and applications at scale!Bob van Luijt - "Semantic search through Wikipedia with Weaviate (GraphQL, Sentence-BERT, and BERT Q&A)": towardsdatascience.com/semantic-search-through-wikipedia-with-weaviate-graphql-sentence-bert-and-bert-q-a-3c8a5edeacf6Wikipedia Weaviate Demo: github.com/semi-technologies/semantic-search-through-wikipedia-with-weaviateWikidata Weaviate Demo: github.com/semi-technologies/biggraph-wikidata-search-with-weaviateHuggingFace Spaces: huggingface.co/spacesApocalyptify: huggingface.co/spaces/Norod78/ApocalyptifyDALL-E Mini: huggingface.co/spaces/flax-community/dalle-miniKaggle Datasets: kaggle.com/datasetsUCI ML Repository: https://archive.ics.uci.edu/ml/datasets.phpHuggingFace Datasets: huggingface.co/datasetsPaperswithcode: paperswithcode.com/datasetsMosaicML Methods: mosaicml.com/methodsStreamlit + Transformers article in AI Brewery: ai-brewery.medium.com/conversational-chatbot-using-transformers-and-streamlit-73d621afde9Gradio: https://www.gradio.app/ml_examplesChapters0:00 Intro0:21 HuggingFace Spaces1:02 Dataset Sharing1:52 PaperswithCode2:18 MosaicML2:40 Streamlit and Gradio3:30 HuggingFace Spaces (again)3:50 Contribution from Weaviate5:10 TDS Wikipedia with Weaviate (Bob van Luijt)
KerasBERT on the Weaviate Podcast Connor Shorten 2021-11-26 | Look out for the podcast on SeMI YouTube: youtube.com/channel/UCJKT6kJ3IFYybWnL7jbXxhQKerasBERT quick intro video: youtube.com/watch?v=J3P8WLAELqk (Full Explanation coming soon)
Graph Embeddings and PyTorch-BigGraph Connor Shorten 2021-11-15 | This video provides an overview of Graph Embeddings and how PyTorch-BigGraph enables learning graph embeddings for very large graphs. The key challenge with this is you would need a lot of memory to store the vectors for each node in a large graph. To solve this, BigGraph uses novel partitioning, distributed execution, and negative sampling algorithms. I hope this is a decent introduction to graph embeddings and PyTorch-BigGraph, really excited about the upcoming release of the Wikidata Weaviate web demo!PyTorch-BigGraph (blog post) - ai.facebook.com/blog/open-sourcing-pytorch-biggraph-for-faster-embeddings-of-extremely-large-graphsPyTorch-BigGraph (paper) - arxiv.org/abs/1903.12287Paperswithdata (graph filter) - paperswithcode.com/datasets?mod=graphs&page=1
Dmitry Kan on Vector Search Engines Connor Shorten 2021-11-10 | Subscribe to the Vector Podcast: youtube.com/channel/UCCIMPfR7TXyDvlDRXjVhP1gNot All Vector Databases are Equal: towardsdatascience.com/milvus-pinecone-vespa-weaviate-vald-gsi-what-unites-these-buzz-words-and-what-makes-each-9c65a3bd0696Get started with Weaviate!https://www.semi.technology/developers/weaviate/current/getting-started/quick-start.htmlChapters0:00 Introduction2:55 What inspired your interest?6:24 Weaviate Modules - Search + QA and more7:40 BERT + Elasticsearch13:42 FAISS and recent developments17:37 Mixed Precision18:21 Symbolic Filters and Vector Search24:34 Approximate Nearest Neighbor Benchmarks27:38 ANN vs. Information Retrieval32:58 Robustness in Vector Representations34:20 Semantic Gap between Encoded Objects34:50 Data Domains for Vector Encodings35:20 Not All Vector Databases are Equal42:24 Dmitry’s overview of Weaviate46:20 Hacker News Viral Blog Post Experience49:00 Vector Search and MLOps emerging spaces49:50 NGT better than HNSW?50:53 Self-Hosted versus Managed Hosting55:45 GraphQL in Weaviate1:00:55 Custom Hardware for Vector Search1:04:26 Thank you for watching!
Presenting... The Weaviate Podcast! Connor Shorten 2021-11-03 | This is a prelude to the Weaviate Podcast (moving to SeMI YouTube).Etienne Dilocker and I discuss the Weaviate Slack, the Weaviate console for Dataset Interfaces, HNSW, and many more slack topics such as encoding longer than 512 tokens for vector similarity search. Thank you so much for watching, please head over to the SeMI YouTube for future episodes and check out the Weaviate slack if you would like your questions answered here!***ERRATA*** -- Apologies, forgot to add this on the screen. At 15:02, Etienne and I are talking about these graphs -- http://ann-benchmarks.comWeaviate Slack: join.slack.com/t/weaviate/shared_invite/zt-goaoifjr-o8FuVz9b1HLzhlUfyfddhwWeaviate Web Demo: https://www.semi.technology/developers/weaviate/current/getting-started/quick-start.htmlANN Benchmarks: http://ann-benchmarks.comEtienne Dilocker's article in Towardsdatascience - Effects of filtered HNSW searches on Recall and Latency: towardsdatascience.com/effects-of-filtered-hnsw-searches-on-recall-and-latency-434becf8041cChapters0:00 Preview3:20 Opening and Weaviate Slack5:30 Web Demo10:25 ANN Benchmarks19:35 HNSW29:40 Symbolic Filters in HNSW40:05 Effects of filtered HNSW searches on Recall and Latency47:08 Searching with 2 Vectors48:14 Beyond 512 Tokens53:25 Knowledge Graphs
AugMax Explained! Connor Shorten 2021-10-28 | There has been a new breakthrough in using Data Augmentation in Deep Learning. Researchers from NVIDIA, Caltech, UT Austin, and Arizona State have published AugMax.The acronym AugMax communicates the use of Data Augmentation and Maximizing the Diversity and Difficulty of Augmented Examples.AugMax is basically free lunch in Deep Learning meaning that it is a very strong technique to add to any existing Deep Learning workflow. But more particularly, Data Augmentation has been heavily studied with Image Data in Computer Vision.AugMax is the sequel to AugMix.This predecessor tree of research in Data Augmentation is roughly AutoAugment to RandAugment to AugMix, and now AugMax.AugMax further utilizes Friendly Adversarial Training -- rather than using an adversarial search that augments images to be as challenging as possible, the authors utilize an Entropy proxy to better ensure that these adversarial images aren’t just static noise maps.Further, AugMax utilizes a new normalization scheme. From StyleGAN to GauGAN, researchers at NVIDIA have made great use of these normalization layers and they didn’t disappoint in this one.Solving Robustness and Distribution Shift wlll be huge for the trust-worthiness of Deep Learning, as well as all the applications highlighted in the WILDS benchmark -- which I highly recommend checking out if interested in Deep Learning research.Links: Weaviate Web Demo: https://console.semi.technology/Henry AI Labs Weaviate Query Tutorial: youtube.com/watch?v=K_2X48Tln9UYou might think this is cool as well - Mathcha.io (LaTex Equation Writing Tool) - mathcha.io/editorAugMax Paper Link: arxiv.org/pdf/2110.13771.pdfAugMax Code Repo: github.com/VITA-Group/AugMaxAugMix Paper Link: arxiv.org/pdf/1912.02781.pdfAttacks Which Do Not Kill Training Make Adversarial Learning Stronger: arxiv.org/pdf/2002.11242.pdfExplaining and Harnessing Adversarial Examples: arxiv.org/pdf/1412.6572.pdfMixUp: Beyond Empirical Risk Minimization: arxiv.org/pdf/1710.09412.pdfMixUp Keras Code Examples from Sayak Paul: keras.io/examples/vision/mixupRandAugment: arxiv.org/pdf/1909.13719.pdfBenchmarking Neural Network Robustness to Common Corruptions and Perturbations: arxiv.org/pdf/1903.12261.pdfAutoAugment: arxiv.org/pdf/1805.09501.pdfEvolving Normalization-Activation Layers: arxiv.org/abs/2004.02967Chapters0:00 Introduction0:08 AugMax in 2 Minutes3:22 Presentation Overview5:20 Sponsored by Weaviate!6:22 Data Augmentation11:03 Overfitting and Robustness13:08 RandAugment14:18 AugMix15:53 MixUp17:08 Friendly Adversarial Training18:58 AugMax Algorithm21:23 Normalization - DuBIN23:28 Vectorized Image Visualization24:06 Experiments and Metrics30:13 Research Context30:55 Ideas34:18 AugMax and Weaviate
Weaviates GraphQL API for Neurosymbolic Search! Connor Shorten 2021-10-22 | This video walks through some example queries to help you get started with Weaviate's GraphQL API. If you have any questions, ideas, or want advice on your personal use case, please leave it in the comments! Stay tuned for a tutorial on using a custom dataset with Weaviate!Quick Start Tutorial: https://www.semi.technology/developers/weaviate/current/getting-started/quick-start.htmlHow Weaviate's GraphQL API was designed: hackernoon.com/how-weaviates-graphql-api-was-designed-t93932tlChapters0:00 Neurosymbolic Systems0:55 Query 12:00 Query 22:33 Query 33:27 Query 4 - Symbolic Filters!4:16 Query 55:09 Query 6 - Neural Search!6:24 Query 7 - Neurosymbolic Query!7:43 Query 88:41 Grocery Store Example9:55 Custom Dataset - Coming Soon!
Introducing the Weaviate Vector Search Engine! Connor Shorten 2021-10-20 | This video presents Weaviate Vector Search Engine! Please leave a comment if you have any ideas or questions!Weaviate Documentation: https://www.semi.technology/developers/weaviate/current/SeMI Technologies on YouTube: youtube.com/channel/UCJKT6kJ3IFYybWnL7jbXxhQ/videosChapters0:00 Overview of Chapters5:16 Semantic Similarity7:55 Symbolic Queries9:45 Vector + Symbolic Search10:23 Weaviate Modules13:44 Demo Dataset and GraphQL15:12 Integration Project16:16 Research Ideas19:18 Connect with us!
Etienne Dilocker on Vector Search Engines and Weaviate Connor Shorten 2021-10-11 | Vector Search Engines are powering the the next generation of search. Instead of relying on things like BM25 or TF-IDF representations, we use neural representations to compare semantic similarity. Dot products to calculate the similarity between all vectors would be very time consuming. Hierarchical Navigable Small World Graphs (HNSW) are a cutting-edge new algorithm used by Weaviate to speed up this search. This podcast explores HNSW, Neurosymbolic search and filtering, and many more!I hope you find this interesting, happy to answer any questions left on the YouTube video!Check out this Introduction to Weaviate: Very well organized, no headaches getting started!https://www.semi.technology/developers/weaviate/current/Chapters0:00 Introduction1:30 Why Vector Search Engines?3:02 Hierarchical Navigable Small World Graphs and Scaling9:00 Setup Questions12:57 What is the compute heavy part?17:15 Neurosymbolic Databases23:53 Weaviate for Language Models26:17 Weaviate for HuggingFace or PapersWithCode Datasets30:43 Weaviate Modules33:58 Text-to-Image Vector Search36:54 What inspired Etienne to work on this?
Robust Fine-Tuning of Zero-Shot Models Connor Shorten 2021-09-16 | Researchers from the University of Washington, Columbia University, Open AI, the Allen Institute of Artificial Intelligence, and Toyota Research have teamed up to present a new method for fine-tuning these pre-trained models such as GPT-3, BERT, DALL-E, EfficientNet, or CLIP for application specific datasets. The key insight is that as you fine-tune these models, you gain in-distribution accuracy, but sacrifice the zero-shot flexibility, or out-of-distribution generalization, of these pre-trained “foundation” models. The authors present Weight-Space Ensembling, where you take a linear interpolation between the weights of the zero-shot and fine-tuned model to make new inference. This achieves a balance between in and out of distribution accuracy. The authors connect this to Linear Mode Connectivity to explain why it works compared to random weight-space ensembles, which do not work. This is another very interesting study on the Generalization capability of Deep Neural Networks. This includes solving problems of adversarial attacks that destroy public trust in these systems, as well as the general problem of models that fail with new distributions of data. The authors collect several examples of these distribution shifts such as the WILDS dataset, adversarial injections, or style transfers, to give a few examples. The findings of this paper should be useful for anyone working with Deep Learning.Weaviate GitHub: github.com/semi-technologies/weaviateWeaviate YouTube: youtube.com/channel/UCJKT6kJ3IFYybWnL7jbXxhQAI Weekly Update: https://ebony-scissor-725.notion.site/AI-Weekly-Update-September-15th-2021-eb65324ed2f640a5974c369ccf33af8eAI Weekly Updates: github.com/CShorten/AIWeeklyUpdatesContent Links:Paper Link: arxiv.org/pdf/2109.01903.pdfOpenAI Clip: openai.com/blog/clipWILDS Dataset Page: https://wilds.stanford.edu/datasets/DomainNet: http://ai.bu.edu/M3SDA/Supermasks in Superposition: arxiv.org/pdf/2006.14769.pdfChapters0:00 Overview1:28 Weaviate / SeMI Technologies2:26 Motivation - Foundation Models5:03 Out-of-Distribution Test Sets9:33 Weight-Space Ensembling13:54 Linear Mode Connectivity18:00 Results26:30 Takeaways and Ideas
Generalization in Open-Domain Question Answering Connor Shorten 2021-09-08 | AI Weekly Update Notion Page: https://ebony-scissor-725.notion.site/AI-Weekly-Update-September-8th-2021-a2119851b5b74470b4971d064665e77eNew AI Weekly Update GitHub Repo: github.com/CShorten/AIWeeklyUpdatesI am looking for collaborators for two survey papers on Text-to-Image Generation and Data Augmentation Controllers. If you are interested in helping out and being a co-author of either paper, please send me a quick overview of what you think about the topic to cshorten2015@fau.edu!Content Links in the Video:Challenges in Generalization in Open Domain Question Answering: arxiv.org/pdf/2109.01156.pdfHurdles to Progress in Long-Form Question Answering: arxiv.org/pdf/2103.06332.pdfGoogle AI Blog Post for ^: ai.googleblog.com/2021/03/progress-and-challenges-in-long-form.htmlHuman-level concept learning through probabilistic program induction: science.org/doi/abs/10.1126/science.aab3050Transformers as Soft Reasoners over Language: arxiv.org/pdf/2002.05867.pdfCLEVR: arxiv.org/pdf/1612.06890.pdfPointer Value Retrieval: arxiv.org/pdf/2107.12580.pdfMultimodal Few-Shot Learning with Frozen Language Models: arxiv.org/pdf/2106.13884.pdfHow Much Knowledge Can You Pack into the Parameters of a Language Model? arxiv.org/abs/2002.08910Time-Aware Language Models as Temporal Knowledge Bases: arxiv.org/pdf/2106.15110.pdfGPT-3: arxiv.org/abs/2005.14165Finetuned Language Models are Zero Shot Learners: arxiv.org/pdf/2109.01652.pdfZero-Shot Text-to-Image Generation: arxiv.org/abs/2102.12092On the 'steerability' of generative adversarial networks: arxiv.org/abs/1907.07171Classification Accuracy Score: arxiv.org/abs/1905.10887QA Dataset Explosion: arxiv.org/pdf/2107.12708.pdfTurning Tables: arxiv.org/abs/2107.07261Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data: aclanthology.org/2020.acl-main.463Chapters:0:00 Introduction2:07 Generalization4:53 Generalization for Open-Domain Question Answering6:56 Question Decomposition10:18 Datasets13:05 Models Tested15:25 Results and Takeaways18:23 Question Pattern Frequency23:16 Challenges in Long-Form ODQA24:48 Compositional Generalization26:14 Novel-Entity Generalization27:22 WILDS28:36 Data Augmentation Solution31:12 Looking for Survey Writers!
AI Weekly Update - August 7th, 2021 (#40) Connor Shorten 2021-08-07 | Notion Link: https://ebony-scissor-725.notion.site/AI-Weekly-Update-August-7th-2021-3b21331c5c6a45e1a5638955dda7923cChapters:0:00 Introduction0:06 Open-Ended Learning2:38 Persistent Reinforcement Learning3:34 Domain-Matched Pre-training for Retrieval4:22 Growing Knowledge Culturally5:06 Language Grounding with 3D Objects6:03 Pre-train, Prompt, and Predict6:32 QA Dataset Explosion7:32 ProtoTransformer9:32 Don’t sweep your Learning Rate under the Rug10:34 AAVAE11:40 Domain-Agnostic Contrastive Learning12:43 Dataset Distillation14:04 Pointer Value Retrieval16:01 Go Wider Instead of Deeper16:52 Geometric Deep Learning on Molecules18:37 Simulation Framework for Label Noise19:59 A Tale of Two Long Tails21:07 DALL-E Mini on HF Spaces
Reasoning with Language Models - Turning Tables Connor Shorten 2021-07-23 | Notion Link: https://ebony-scissor-725.notion.site/Henry-AI-Labs-Weekly-Update-July-22nd-2021-0c43042b93a3459c901f7f5973b949bfThanks for watching! Please Subscribe!
Deduplicating Training Data makes Language Models Better Connor Shorten 2021-07-22 | Notion Link: https://ebony-scissor-725.notion.site/Henry-AI-Labs-Weekly-Update-July-22nd-2021-0c43042b93a3459c901f7f5973b949bfFollow Katherine Lee on Twitter @katherine1eeTwitter Thread Link: twitter.com/katherine1ee/status/1415496898241339400Thanks for watching! Please Subscribe!
Using HTML for Language Modeling Connor Shorten 2021-07-22 | Notion Link: https://ebony-scissor-725.notion.site/Henry-AI-Labs-Weekly-Update-July-22nd-2021-0c43042b93a3459c901f7f5973b949bfThanks for watching! Please Subscribe!
Writing with AI - Wordcraft Text Editor Connor Shorten 2021-07-22 | Notion Link: https://ebony-scissor-725.notion.site/Henry-AI-Labs-Weekly-Update-July-22nd-2021-0c43042b93a3459c901f7f5973b949bfDemo Video: youtube.com/watch?v=9p4mfA0Fyd8Thanks for watching! Please Subscribe!
Internet-Augmented Dialogue Generation Connor Shorten 2021-07-22 | Notion Link: https://ebony-scissor-725.notion.site/Henry-AI-Labs-Weekly-Update-July-22nd-2021-0c43042b93a3459c901f7f5973b949bfThank you for watching! Please Subscribe!
Beyond Goldfish Memory! Connor Shorten 2021-07-22 | Notion Link: https://ebony-scissor-725.notion.site/Henry-AI-Labs-Weekly-Update-July-22nd-2021-0c43042b93a3459c901f7f5973b949bfThumbnail Goldfish Image Credit - Photo by zhengtao tang on UnsplashThanks for watching! Please Subscribe!Chapters:0:00 Introduction1:54 Multi-Session Chat4:12 Models6:40 Drawbacks of Info Retrieval9:40 Memory Augmented Models10:09 Comparison with Language Models
Blender Bot 2.0 Connor Shorten 2021-07-22 | Notion Link: https://ebony-scissor-725.notion.site/Henry-AI-Labs-Weekly-Update-July-22nd-2021-0c43042b93a3459c901f7f5973b949bfBeyond Goldfish Memory explained: youtube.com/watch?v=GdNWpbPVRA8Full explanations of these papers coming shortly!Thanks for watching! Please Subscribe!
