Uploaded August 2024 | Updated September 2026, 2 weeks ago
In this video, you will learn about Arcee-Lite, a small yet powerful 1.5B model created with Distilkit, an open-source project for model distillation.
⭐️⭐️⭐️ Don't forget to subscribe to be notified of future videos. Follow me on Medium at julsimon.medium.com or Substack at https://julsimon.substack.com. ⭐️⭐️⭐️
Arcee-Lite outperforms Qwen2 1.5B, and is currently the best 1.5B model.
First, I run an 8-bit version on my M3 MacBook with ollama and OpenWeb UI. Then, I deploy the model on AWS with Amazon SageMaker. I run both synchronous and streaming inference. I also show you how to use the OpenAI Messages API, allowing you to invoke the model with the OpenAI prompting format.
* Model page (full precision model): huggingface.co/arcee-ai/arcee-lite
* Model page (quantized models): huggingface.co/arcee-ai/arcee-lite-GGUF
* Notebook: gitlab.com/juliensimon/arcee-demos/-/blob/main/sagemaker/deploy_lite_gpu.ipynb
00:00 Introduction
00:55 Introducing Arcee-Lite
04:40 Running Arcee-Lite locally with ollama and OpenWeb UI
06:20 Deploying Arcee-Lite on AWS with Amazon SageMaker
08:10 100+ tokens per second on g5.xlarge !
08:45 Streaming inference
09:40 Use cases for this model
Configuration file for ollama:
FROM ./arcee-lite-Q8_0.gguf
Sign up for Arcee Cloud at https://hubs.li/Q02Kh_YQ0 and please follow Arcee.ai on LinkedIn to stay on top of the latest Small Language Model action!
#ai #aws #slm #llm #openai #chatgpt #opensource #huggingface
In this video, you will learn about Arcee-Lite, a small yet powerful 1.5B model created with Distilkit, an open-source project for model distillation.
⭐️⭐️⭐️ Don't forget to subscribe to be notified of future videos. Follow me on Medium at julsimon.medium.com or Substack at https://julsimon.substack.com. ⭐️⭐️⭐️
Arcee-Lite outperforms Qwen2 1.5B, and is currently the best 1.5B model.
First, I run an 8-bit version on my M3 MacBook with ollama and OpenWeb UI. Then, I deploy the model on AWS with Amazon SageMaker. I run both synchronous and streaming inference. I also show you how to use the OpenAI Messages API, allowing you to invoke the model with the OpenAI prompting format.
* Model page (full precision model): huggingface.co/arcee-ai/arcee-lite
* Model page (quantized models): huggingface.co/arcee-ai/arcee-lite-GGUF
* Notebook: gitlab.com/juliensimon/arcee-demos/-/blob/main/sagemaker/deploy_lite_gpu.ipynb
00:00 Introduction
00:55 Introducing Arcee-Lite
04:40 Running Arcee-Lite locally with ollama and OpenWeb UI
06:20 Deploying Arcee-Lite on AWS with Amazon SageMaker
08:10 100+ tokens per second on g5.xlarge !
08:45 Streaming inference
09:40 Use cases for this model
Configuration file for ollama:
FROM ./arcee-lite-Q8_0.gguf
Sign up for Arcee Cloud at https://hubs.li/Q02Kh_YQ0 and please follow Arcee.ai on LinkedIn to stay on top of the latest Small Language Model action!
#ai #aws #slm #llm #openai #chatgpt #opensource #huggingface
