Deep dive: model merging, part 2 @juliensimonfr

Uploaded July 2024 | Updated September 2026, 2 weeks ago

Model merging is an increasingly popular technique for adding or removing capabilities from transformer models without additional training.

⭐️⭐️⭐️ Don't forget to subscribe to be notified of future videos. Follow me on Medium at julsimon.medium.com or Substack at https://julsimon.substack.com. ⭐️⭐️⭐️

In a previous video (youtu.be/cvOpX75Kz4M), we introduced model merging and studied several merging algorithms implemented in the mergekit library (github.com/arcee-ai): model soups, SLERP, Task Arithmetic, TIES, DARE, and Franken-merging.

This new video builds upon the previous one and explores new merging methods: model breadcrumbs, model stock, and DELLA. We also quickly look at model merging in Arcee Cloud, which you can run for free as part of the free tier! For a deeper look, please watch youtu.be/xObTcelxb24

Slides: fr.slideshare.net/slideshow/julien-simon-deep-dive-model-merging/270921708

00:00 Introduction
01:20 Model breadcrumbs
10:15 Model stocks
19:55 DELLA
30:20 Merge models for free in Arcee Cloud!

Sign up for Arcee Cloud at https://www.arcee.ai, and please follow Arcee.ai on LinkedIn to stay on top of the latest Small Language Model action! linkedin.com/company/99895334

#ai #aws #slm #llm #openai #chatgpt #opensource #huggingface

Arcee AI Drops Trinity Nano & Mini MoE Models!

SLM in Action: Arcee-Scribe, a 7.7B model for creative writing

Unmask the Truth Behind AI Marketing Claims! 🚨

SLM in Action: SuperNova-Medius, a high-performance 14B model

Unlocking the Future: The Best AI Career Path Revealed!

$Trinity Large: An Open 400B MoE Model I built a code review app powered by @ArceeAIs new Trinity Large model and deployed it to a @HuggingFace Space. ⭐️⭐️⭐️ More content on Substack at https://julsimon.substack.com ⭐️⭐️⭐️ In this video, I walk through a live demo — paste a GitHub PR URL, hit review, and watch it catch real issues: security flaws, logic bugs, missing edge cases. Its fast, free to try, and the 512K context window lets it process entire files without chunking. Code Review App (HF Space): https://huggingface.co/spaces/juliensimon/trinity-code-reviewer Trinity Large is a 400B parameter Mixture-of-Experts model trained in 33 days for $20M — a fraction of what frontier labs spend. It has 256 experts but activates only 4 per token, so only 13B active parameters do the work. That translates to 2-3x faster inference than anything in its weight class, which you can feel in the demo when reviews come back in seconds. In this video: - 🔴 Live demo: code review app catching real issues on real PRs - 🏗️ How I built the app — stack, prompting strategy, handling large diffs - 🧠 Architecture deep dive: 256 experts, 4 active, extreme sparsity - 📦 Why three checkpoints exist: Preview, Base, and TrueBase - ⚡ Why MoE makes this practical to actually deploy and run at scale Try the app yourself and let me know what it catches in your code. Hugging Face model page: https://huggingface.co/arcee-ai/Trinity-Large-Preview OpenRouter model page: https://openrouter.ai/arcee-ai/trinity-large-preview Arcee AI blog post: https://www.arcee.ai/blog/trinity-large Arcee AI technical report: https://github.com/arcee-ai/trinity-large-tech-report VentureBeat article: https://venturebeat.com/technology/arcees-u-s-made-open-source-trinity-large-and-10t-checkpoint-offer-rare-look$

Summarizing legal documents with Hugging Face and Amazon SageMaker

My AI found 15 critical issues in my code!

Hugging Face / AWS roadshow - Zurich 🇨🇭🇨🇭🇨🇭