TechTechPotato | ChatGPT Comes From a Foundation Model | Talking With Talia, Episode 2 @TechTechPotato | Uploaded December 2022 | Updated October 2024, 1 hour ago.
Note, this was recorded before chatGPT became huge. I've been playing with it - it could probably write some basic future scripts already. Large Language Models (LLM), and Natural Language Processing (NLP) is crazy. Scary stuff.
But, for certain applications, we have too much data. There are ways to train AI models with lots of unlabelled data, and the result is what we call a Foundation Model. With this super large model, you can fine tune it a little to get a more focused version - chat bot for your onsite support? autocomplete engine for your homework? an easy way to replace low level journalists? In this video, with Dr. Talia Gershon, we cover what a Foundation Model is, and why they're becoming a big thing in modern AI.
Timestamps
[0:08] What is a Foundation Model
[2:30] Is this AI by brute force with unlabeled data?
[3:40] Self-supervised training
[4:22] Trends in Foundation Models
[5:24] Limits
[6:28] Evolution of Models
[8:35] Fine-tuning and prompt-tuning
[9:58] Who can create these models, and how?
[11:00] Interfacing with Foundation Models
[12:44] Cat Tax
IBM Foundation Models: research.ibm.com/topics/foundation-models
IBM Blog: What Are Foundation Models?: research.ibm.com/blog/what-are-foundation-models
IBM AI Hardware Center: research.ibm.com/collaborate/ai-hardware-center
Stanford paper: arxiv.org/abs/2108.07258
-----------------------
more-moore.com : Sign up to the More Than Moore Newsletter!
patreon.com/TechTechPotato : Patreon gets you access to the TTP Discord server!
Follow Ian on Twitter at twitter.com/IanCutress
Follow TechTechPotato on Twitter at twitter.com/TechTechPotato
If you're in the market for something from Amazon, please use the following links. TTP may receive a commission if you purchase anything through these links.
Amazon USA : geni.us/AmazonUS-TTP
Amazon UK : geni.us/AmazonUK-TTP
Amazon CAN : geni.us/AmazonCAN-TTP
Amazon GER : geni.us/AmazonDE-TTP
Amazon Other : geni.us/TTPAmazonOther
Ending music: youtube.com/watch?v=2N0tmgau5E4
-----------------------
Welcome to the TechTechPotato (c) Dr. Ian Cutress
Ramblings about things related to Technology from an analyst for More Than Moore
#ibm #foundationmodel #sponsored
------------
More Than Moore, as with other research and analyst firms, provides or has provided paid research, analysis, advising, or consulting to many high-tech companies in the industry, which may include advertising on TTP. The companies that fall under this banner include AMD, Armari, Facebook, IBM, Intel, Linode, MediaTek, NordPass, Qualcomm.
Note, this was recorded before chatGPT became huge. I've been playing with it - it could probably write some basic future scripts already. Large Language Models (LLM), and Natural Language Processing (NLP) is crazy. Scary stuff.
But, for certain applications, we have too much data. There are ways to train AI models with lots of unlabelled data, and the result is what we call a Foundation Model. With this super large model, you can fine tune it a little to get a more focused version - chat bot for your onsite support? autocomplete engine for your homework? an easy way to replace low level journalists? In this video, with Dr. Talia Gershon, we cover what a Foundation Model is, and why they're becoming a big thing in modern AI.
Timestamps
[0:08] What is a Foundation Model
[2:30] Is this AI by brute force with unlabeled data?
[3:40] Self-supervised training
[4:22] Trends in Foundation Models
[5:24] Limits
[6:28] Evolution of Models
[8:35] Fine-tuning and prompt-tuning
[9:58] Who can create these models, and how?
[11:00] Interfacing with Foundation Models
[12:44] Cat Tax
IBM Foundation Models: research.ibm.com/topics/foundation-models
IBM Blog: What Are Foundation Models?: research.ibm.com/blog/what-are-foundation-models
IBM AI Hardware Center: research.ibm.com/collaborate/ai-hardware-center
Stanford paper: arxiv.org/abs/2108.07258
-----------------------
more-moore.com : Sign up to the More Than Moore Newsletter!
patreon.com/TechTechPotato : Patreon gets you access to the TTP Discord server!
Follow Ian on Twitter at twitter.com/IanCutress
Follow TechTechPotato on Twitter at twitter.com/TechTechPotato
If you're in the market for something from Amazon, please use the following links. TTP may receive a commission if you purchase anything through these links.
Amazon USA : geni.us/AmazonUS-TTP
Amazon UK : geni.us/AmazonUK-TTP
Amazon CAN : geni.us/AmazonCAN-TTP
Amazon GER : geni.us/AmazonDE-TTP
Amazon Other : geni.us/TTPAmazonOther
Ending music: youtube.com/watch?v=2N0tmgau5E4
-----------------------
Welcome to the TechTechPotato (c) Dr. Ian Cutress
Ramblings about things related to Technology from an analyst for More Than Moore
#ibm #foundationmodel #sponsored
------------
More Than Moore, as with other research and analyst firms, provides or has provided paid research, analysis, advising, or consulting to many high-tech companies in the industry, which may include advertising on TTP. The companies that fall under this banner include AMD, Armari, Facebook, IBM, Intel, Linode, MediaTek, NordPass, Qualcomm.
![[23] Dr. Mukesh Khare, VP Hybrid Cloud, IBM Research](https://i.ytimg.com/vi/Dl6oGdLhCvk/hqdefault.jpg "[23] Dr. Mukesh Khare, VP Hybrid Cloud, IBM Research
My interview time with Dr. Khare during my IBM visit was so good, I decided to release the whole thing! We cover some of IBMs research, and go more into what state-funded semiconductor research will look like.
http://more-moore.com : Sign up to the More Than Moore Newsletter!
[0:00] Intro
[0:07] Disclaimer
[0:29] Where am I?
[1:30] Q1: What research is done by IBM at Albany Nanotech?
- [3:08] Chiplets before 1990
- [4:20] First 28nm pitch EUV in 2020
- [4:55] First 7nm chip, 2015
[5:20] Q2: What goes on a test chip
[7:34] Q3: What is the ASIC Coalition?
[9:15] Q3b: Academic Partnerships and NSTC
[11:33] Q4: IBMs Position in ASIC
[12:29] Q5: Companies not listed in ASIC
[13:43] Q6: Funding Approved, Next Steps
[14:36] Q7: Why should IBM get any CHIPS Act money?
[16:13] Q8: IP/Revenue Model of the NSTC?
[16:52] Q9: Is NSTC aligning with IBMs strategy?
- [21:14] You might have a friend in an Ex-TSMC CTO
[23:30] Q10: Sites for NTSC outside Albany?
http://www.asicoalition.org
https://www.patreon.com/TechTechPotato : Patreon gets you access to the TTP Discord server!
Follow Ian on Twitter at http://twitter.com/IanCutress
Follow TechTechPotato on Twitter at http://twitter.com/TechTechPotato
If youre in the market for something from Amazon, please use the following links. TTP may receive a commission if you purchase anything through these links.
Amazon USA : https://geni.us/AmazonUS-TTP
Amazon UK : https://geni.us/AmazonUK-TTP
Amazon CAN : https://geni.us/AmazonCAN-TTP
Amazon GER : https://geni.us/AmazonDE-TTP
Amazon Other : https://geni.us/TTPAmazonOther
Welcome to the TechTechPotato (c) Dr. Ian Cutress
Ramblings about things related to Technology from an analyst for More Than Moore
#techtechpotato #ibm #chipsact
More Than Moore, as with other research and analyst firms, provides or has provided paid research, analysis, advising, or consulting to many high-tech companies in the industry, which may include advertising on TTP. The companies that fall under this banner include Armari, Facebook, IBM, Linode, MediaTek, NordPass, Qualcomm.")
![[25] Wei-han Lien, Lead CPU Architect, Tenstorrent](https://i.ytimg.com/vi/DnM3J2lAYd0/hqdefault.jpg "[25] Wei-han Lien, Lead CPU Architect, Tenstorrent
Wei-han was also the lead architect of Apples M1 silicon. So he knows about high-performance.
[00:00] Intro
[00:09] Wei-Han Intro
[00:15] Why do you need a CPU architect?
[03:05] Vector vs. Matrix math
[04:10] ARM vs. RISC-V
[05:55] What about RISC-V compatibility?
[06:31] Will you publish the CPU design?
[07:01] Why are you going custom now?
[08:43] What can you tell us about Ascalon?
[10:24] Is Tenstorrent an IP or a chip company?
[10:54] What are your thoughts on chiplets?
[12:49] Tenstorrent mobile designs?
[14:40] Use Tenstorrent for Tenstorrent?
[15:52] Training vs. inference
[16:46] How do you scale networking?
[18:20] 3D integration leadership
[19:24] Hardware vs. software
[21:09] How do you deal with future ML algorithms?
[22:56] What architecture is Tenstorrent?
[24:23] What is beyond AI hardware?
[25:09] Multithreading on RISC-V chips?
[26:02] Outro
Need POTATO merch? Theres a chip for that!
http://merch.techtechpotato.com
http://more-moore.com : Sign up to the More Than Moore Newsletter
https://www.patreon.com/TechTechPotato : Patreon gets you access to the TTP Discord server!
Follow Ian on Twitter at http://twitter.com/IanCutress
Follow TechTechPotato on Twitter at http://twitter.com/TechTechPotato
If youre in the market for something from Amazon, please use the following links. TTP may receive a commission if you purchase anything through these links.
Amazon USA : https://geni.us/AmazonUS-TTP
Amazon UK : https://geni.us/AmazonUK-TTP
Amazon CAN : https://geni.us/AmazonCAN-TTP
Amazon GER : https://geni.us/AmazonDE-TTP
Amazon Other : https://geni.us/TTPAmazonOther
Ending music: https://www.youtube.com/watch?v=2N0tmgau5E4
Welcome to the TechTechPotato (c) Dr. Ian Cutress
Ramblings about things related to Technology from an analyst for More Than Moore
#riscv #weihanlien #tenstorrent
More Than Moore, as with other research and analyst firms, provides or has provided paid research, analysis, advising, or consulting to many high-tech companies in the industry, which may include advertising on TTP. The companies that fall under this banner include AMD, Armari, Facebook, IBM, Infineon, Intel, Lattice Semi, Linode, MediaTek, NordPass, ProteanTecs, Qualcomm, SiFive, Tenstorrent.")
![[7] Was Sundar Correct at Google I/O?](https://i.ytimg.com/vi/FmAYs_5S0OY/hqdefault.jpg "[7] Was Sundar Correct at Google I/O?
Weve been foiled! Google decided to count up the number of times they said AI in the 2024 Google IO keynote and mention it on stage. Are they correct? We did our own analysis.
Need POTATO merch? Theres a chip for that!
http://merch.techtechpotato.com
http://more-moore.com : Sign up to the More Than Moore Newsletter
https://www.patreon.com/TechTechPotato : Patreon gets you access to the TTP Discord server!
Follow Ian on Twitter at http://twitter.com/IanCutress
Follow TechTechPotato on Twitter at http://twitter.com/TechTechPotato
If youre in the market for something from Amazon, please use the following links. TTP may receive a commission if you purchase anything through these links.
Amazon USA : https://geni.us/AmazonUS-TTP
Amazon UK : https://geni.us/AmazonUK-TTP
Amazon CAN : https://geni.us/AmazonCAN-TTP
Amazon GER : https://geni.us/AmazonDE-TTP
Amazon Other : https://geni.us/TTPAmazonOther
Ending music: https://www.youtube.com/watch?v=2N0tmgau5E4
Welcome to the TechTechPotato (c) Dr. Ian Cutress
Ramblings about things related to Technology from an analyst for More Than Moore
#google #ai #megacut
More Than Moore, as with other research and analyst firms, provides or has provided paid research, analysis, advising, or consulting to many high-tech companies in the industry, which may include advertising on TTP. The companies that fall under this banner include AMD, Armari, Baidu, Facebook, IBM, Infineon, Intel, Lattice Semi, Linode, MediaTek, NordPass, ProteanTecs, Qualcomm, SiFive, Supermicro, Tenstorrent, TSMC.")
