Techno Tim Tinkers | How to Self-Host Your Own Private AI Stack @TechnoTimTinkers | Uploaded July 2024 | Updated October 2024, 4 hours ago.
In this tutorial we'll walk through my local, private, self-hosted AI stack so that you can run it too.
If you're looking for the overview of this stack, you can out the video here: youtube.com/watch?v=GrLpdfhTwLg
Video Notes: https://technotim.live/posts/ai-stack-tutorial/
Products:
GPUs
12GB is probably "good enough" for small models.
3000 Series
👇 My value pick. 3060, 12GB RAM, affordable, can play games too.
- MSI Gaming GeForce RTX 3060 12GB: amzn.to/3WerOUI
- GIGABYTE GeForce RTX 3060 Gaming OC 12G: amzn.to/3WdqHV9
- NVIDIA GeForce RTX 3090 Founders Edition Graphics Card Renewed 24GB: amzn.to/4cTIS7Q
- Zotac Gaming GeForce RTX 3090 Trinity OC, 24GB: amzn.to/3WdhJqV
4000 Series
- ASUS TUF Gaming GeForce RTX™ 4090 OG OC Edition Gaming Graphics Card 24GB: amzn.to/463LHRF
- MSI Gaming GeForce RTX 4060 8GB: amzn.to/4eVxrON
- ASUS Dual GeForce RTX™ 4070 White OC Edition 12GB: amzn.to/460rqN0
CPUs
You'll want a modern CPU, if you are going desktop class here are a few I would choose
- Intel Core i7-12700K Gaming Desktop Processor: amzn.to/3XTj1Zu
- Intel Core i7-13700K Gaming Desktop Processor: amzn.to/3Lg7M5V
- Intel Core i7-14700K Gaming Desktop Processor: amzn.to/3WftehU
Storage
For flash storage, I always go with these SSDs
- Samsung 870 EVO SATA: amzn.to/3XTW6gA
- SAMSUNG 990 PRO SSD NVMe: amzn.to/4cW1xA9
(Affiliate links may be included in this description. I may receive a small commission at no cost to you.)
Support me on Patreon: patreon.com/technotim
Sponsor me on GitHub: github.com/sponsors/timothystewart6
Subscribe on Twitch: twitch.tv/technotim
Merch Shop 🛍️: https://l.technotim.live/shop
Gear Recommendations: https://l.technotim.live/gear
Get Help in Our Discord Community: https://l.technotim.live/discord
Main channel: youtube.com/@TechnoTim
Talks Channel: youtube.com/@TechnoTimTalks
00:00 - Intro
01:14 - Hardware Specs
02:23 - GPU Considerations
03:10 - GPU Perspective
04:54 - Proxmox
07:05 - Server OS
07:25 - Drivers
08:31 - NVIDIA Container Toolkit
09:44 - Docker
10:25 - Folder Layout
11:20 - Stack Overview
12:15 - Traefik & Docker Networking Considerations
12:55 - Ollama
17:29 - Open WebUI
20:55 - Starting the Stack
23:08 - Ollama Models
25:50 - Chatting with Ollama & Performance
32:05 - searXNG
33:33 - Stable Diffusion & ComfyUI
39:54 - Stable Diffusion Models
44:42 - ComfyUI Workflows
47:00 - Verifying Model Checksum
48:48 - Integrating ComfyUI into Open WebUI
51:59 - Whisper
56:22 - Home Assistant Assist - Chat, Voice, and Text to Voice
01:04:15 - Code Suggestions and Chat Assistant in VSCode (Free Co-pilot)
Thank you for watching!
In this tutorial we'll walk through my local, private, self-hosted AI stack so that you can run it too.
If you're looking for the overview of this stack, you can out the video here: youtube.com/watch?v=GrLpdfhTwLg
Video Notes: https://technotim.live/posts/ai-stack-tutorial/
Products:
GPUs
12GB is probably "good enough" for small models.
3000 Series
👇 My value pick. 3060, 12GB RAM, affordable, can play games too.
- MSI Gaming GeForce RTX 3060 12GB: amzn.to/3WerOUI
- GIGABYTE GeForce RTX 3060 Gaming OC 12G: amzn.to/3WdqHV9
- NVIDIA GeForce RTX 3090 Founders Edition Graphics Card Renewed 24GB: amzn.to/4cTIS7Q
- Zotac Gaming GeForce RTX 3090 Trinity OC, 24GB: amzn.to/3WdhJqV
4000 Series
- ASUS TUF Gaming GeForce RTX™ 4090 OG OC Edition Gaming Graphics Card 24GB: amzn.to/463LHRF
- MSI Gaming GeForce RTX 4060 8GB: amzn.to/4eVxrON
- ASUS Dual GeForce RTX™ 4070 White OC Edition 12GB: amzn.to/460rqN0
CPUs
You'll want a modern CPU, if you are going desktop class here are a few I would choose
- Intel Core i7-12700K Gaming Desktop Processor: amzn.to/3XTj1Zu
- Intel Core i7-13700K Gaming Desktop Processor: amzn.to/3Lg7M5V
- Intel Core i7-14700K Gaming Desktop Processor: amzn.to/3WftehU
Storage
For flash storage, I always go with these SSDs
- Samsung 870 EVO SATA: amzn.to/3XTW6gA
- SAMSUNG 990 PRO SSD NVMe: amzn.to/4cW1xA9
(Affiliate links may be included in this description. I may receive a small commission at no cost to you.)
Support me on Patreon: patreon.com/technotim
Sponsor me on GitHub: github.com/sponsors/timothystewart6
Subscribe on Twitch: twitch.tv/technotim
Merch Shop 🛍️: https://l.technotim.live/shop
Gear Recommendations: https://l.technotim.live/gear
Get Help in Our Discord Community: https://l.technotim.live/discord
Main channel: youtube.com/@TechnoTim
Talks Channel: youtube.com/@TechnoTimTalks
00:00 - Intro
01:14 - Hardware Specs
02:23 - GPU Considerations
03:10 - GPU Perspective
04:54 - Proxmox
07:05 - Server OS
07:25 - Drivers
08:31 - NVIDIA Container Toolkit
09:44 - Docker
10:25 - Folder Layout
11:20 - Stack Overview
12:15 - Traefik & Docker Networking Considerations
12:55 - Ollama
17:29 - Open WebUI
20:55 - Starting the Stack
23:08 - Ollama Models
25:50 - Chatting with Ollama & Performance
32:05 - searXNG
33:33 - Stable Diffusion & ComfyUI
39:54 - Stable Diffusion Models
44:42 - ComfyUI Workflows
47:00 - Verifying Model Checksum
48:48 - Integrating ComfyUI into Open WebUI
51:59 - Whisper
56:22 - Home Assistant Assist - Chat, Voice, and Text to Voice
01:04:15 - Code Suggestions and Chat Assistant in VSCode (Free Co-pilot)
Thank you for watching!
