Episode Transcript
[00:00:00] Open source AI is catching up. The release of llama 3.1 in a 405 billion parameter model that's for use for any company and their own need and speciality and ecosystem is going to cause major shifts in the industry.
[00:00:18] I think this marks a point where the dominant open source models are now competing directly with the major proprietary ones.
[00:00:27] But are we going to just keep making smarter models?
[00:00:31] Does it ever curve and taper off like previous technological trends, or does it truly go completely vertical?
[00:00:39] As we explore this aggressively evolving and volatile landscape, I want to consider something that challenges some of the ideas we've been discussing recently on the show.
[00:00:50] What if the future of AI is isn't about building ever taller skyscrapers, but it's about creating a more diverse, adaptable and interconnected ecosystem?
[00:01:02] We're going to delve into why the next great leap in AI might not be vertical, but horizontal, and why, if I'm right, it's only open source that can actually satisfy the market.
[00:01:16] I am Guy Swan and it's time for another episode of AI Unchained. Foreign what is up guys? Welcome back to AI Unchained. And this is where we explore the world of open source and sovereign AI. I am Guy Swan and this is where I document my journey into understanding and exploring all of the AI tools and tech that is out there. And honestly, I feel like I'm living in the future.
[00:01:51] There's so many of these tools that don't quite have the implementation at the level that I want them yet, but the way I've been able, I can see like those moments of like, oh my God, this is what it's going to be like. This is how I can utilize this. And this could be the interface, this could be the UX of all of these tools together.
[00:02:16] And what's funny is a lot of this is actually dependent on a lot of the experience that I'm getting out of. This is due to the pairs, the Pear web, the Pear Stack tools wholesale and soon to be Pear Drive Keat. All of these things are actually making it so easy to integrate and connect these devices, all of my devices together.
[00:02:43] Then I can make better use of the open source tools, which is making those open source tools that much more valuable. And I've talked about this recently.
[00:02:53] The episode that we just did had Mark Zuckerberg's article about Llama 3.1 and I had not tried it yet. And that was the big thing is I wanted to, I wanted to say that Llama 3.1 was good, but I didn't really have much input on this, but I am getting to the point. I've just been talking with my team actually behind the scenes because I'd been letting a number of people use my ChatGPT account, especially some other people who hadn't started using it yet and wanted to try it out and see if AI really was of any use at all, for any purpose. And then I noticed that a lot of them have been, a lot of them, one or two of them had actually been using it more frequently.
[00:03:40] But I think I'm getting rid of my ChatGPT subscription. You know, I don't like the company, I don't like the direction they are taking, the fact that they're completely open, excuse me, they are completely closed source and they are not sharing anything about their models and the weights and how they are training and what data they are training them on that they are trying to hide behind the idea of safety in order to keep this closed, when clearly I think this is just, it's purely competitive. And then they have just hired the previous head of the NSA to their staff.
[00:04:18] And I can't just, like there's no other, there's no other interpretation of this.
[00:04:23] Like they're a surveillance company, they're looking at and utilizing every piece of data and every single thing that we plug into this API. And not only do I not want to be more morally associated with a company, but ChatGPT is increasingly not that good. I've been using CLAUDE more often. I first tried Claude specifically because I was just looking for a competitor. But the I think it's Claude 3 now is the one that I'm using.
[00:04:53] Which one is this one? It doesn't have a number on it. I think it's Claude 3 but it's the third version of or it's the latest version of their CLAUDE model. But it's really good. It is fantastic. It helped me parse through a RAID hard drive error that was kind of bugging the ever loving crap out of me on my Linux machine. But honestly, I don't know.
[00:05:16] Like I have no real reason to go back to ChatGPT unless I really wanted those conversations.
[00:05:22] And I don't think it gives.
[00:05:24] Like after doing 100 different tasks on Claude and 100 on ChatGPT there might be some sort of a case to be made that one of them is better than the other claude. Like I've really liked the results that I've gotten from Claude and then there are some times where I've liked the results that I've gotten from ChatGPT.
[00:05:43] But I think the more important thing is that with. There is no.
[00:05:48] It's very hard for me to have some objective statement as to which one is better. And Claude has impressed me enough in certain areas. Then I'm like, why do I need, like, I don't need ChatGPT.
[00:06:00] It's got some fun little features and some extra things. And if I specifically just wanted to speak to, you know, GPT4O in their app or something, like, if that was kind of an interface that I found that I wanted to use often, then maybe there would be kind of a selling point for them. But honestly, after. And it was. It seemed like a huge deal when it first came out, but now that I look at it and the way that I use AI and, you know, kind of going back with the last. The previous episode of how I use AI and kind of going through a lot of those examples I don't like, GPT4O is really kind of a gimmick to me to just have a.
[00:06:45] An AI that's like, talks to me and has emotion. Like, I don't really care, you know, that's. That's not what I'm using AI for. Like, I have no desire for my LLM to speak or sound or behave like a person.
[00:07:02] And even though there are numerous instances where it could be really useful to just be able to talk to my LLM at the exact same time, most of the time I don't need it to do that. And in fact, when I am talking to the LLM or I want it to take in a voice input, I don't need it to talk back to me. I just need it to complete a task or categorize something or transcribe something and put it in a location, which is not at all the interface that they've designed. Like, it doesn't integrate with a lot of the tools or apps or notes and stuff on the computer. It literally is just designed to sound like a person and talk back to you. So all of this is to say, I don't. I don't see why I need ChatGPT anymore. And then this is the real kicker. So not only is there no major difference, can I not come up with any concrete difference between those two models, Claude and ChatGPT? And maybe if I did thousands and thousands and I was like, extremely, you know, detailed and precise about exactly what I wanted out of it, and I had some sort of a solid benchmark, I could conclusively say one was better than the other. But honestly, Going through that amount of time to figure it out with the way that I typically use AI in these LLMs, complete waste of time and resources. And this gets me to a point that I want to bring up in just a minute. So we're at a point where ChatGPT and Claude are not different enough for me to care which one I'm using. And I think there's a key piece of the puzzle there that might even relate to the idea of the intelligence explosion and artificial general intelligence and all of this stuff that we've been talking about leading up to this. I want to represent my alternative way of thinking about how we will get the quote unquote intelligence explosion, how we will get the apparent continuation of these trend lines without the kind of crazy futuristic singularity world sort of thing of artificial general intelligence. And it just running away from us and we having us having no control or no involvement in it, just it suddenly it can just teach itself and it's just over that. We've just completed the study and intelligence of everything and now it's just running and doing everything for us.
[00:09:37] Which that's a little bit of a straw man. That's not really what Aschenbrenner says in his piece, but there is a little bit of that kind of romanticized sci fi sort of it's going to be smarter than any one person. And therefore if you put AI on a task, it will complete it better than any other person that you could possibly put on that task. And having a PhD or being specialized in something will be of no consequence because we will just need a model to do it. I don't think that's the case. And I think there's an alternative way to think about why. Exactly. And the hint of me not caring whether I'm using Claude or ChatGPT may be a little piece of that puzzle. But getting back on track is the open source models are getting are reaching that same zone. You know, Mistral and the mixture of experts from Mistral and, and the Mistral 7B and 22B models, they were good, but they were noticeably different from ChatGPT.
[00:10:43] I would still regularly find times where I was like no, I should just use ChatGPT for this. Especially for coding jobs or trying to build a tiny app.
[00:10:54] It just worked better. And I was going to have far fewer errors or problems at all if I was using ChatGPT because it's just a bigger model. You know, you get the exact same thing if you're using 100 billion parameter model versus a 22 billion parameter model or quantized of the exact same model. You know, if I get the Mixtral 7B or the Mixtral 22B, I'm going to get completely different results now. Maybe not a vast difference, but nonetheless it will be. It will be enough for me to care about which one of those models that I'm using. If I'm using the 7 billion, I'm going to want to use that specifically because I'm limited on resources. If I'm using the 22 is because I want to get the best results that I can get.
[00:11:41] Llama 3.1 has a 405 billion parameter model that they have released. Obviously I cannot run this on my machine. My AI machine is not nearly the AI machine. To run a 405 billion parameter model, I would need what, 10, 20 HP? I don't even know it just it would be.
[00:12:05] You need a beast of a machine. You need massive computation to do this. You need a real server. And note, I do hope to have that at some point in the not too distant future, but because it is open and because anybody can build with it or work with it or use it in their environment.
[00:12:21] Venice AI, which is one of the open platforms like Unleashed Chat. Unleashed Chat doesn't have the llama 3.1 yet. It has all the Mistral models, which is what I've been using it for. And also Unleashed Chat, which we had MVK on the show, by the way. So if you want to listen to that episode, I will have the link in the show notes to the episode we did with MVK about Unleashed Chat. And so there are a lot of really great tools on that one. And one of the really cool things is that you can quote, unquote, talk with Nostr and it will go through NOSTR notes and you can search for stuff, you can interact with different posts, et cetera, et cetera. You can basically use nostr, the NOSTR social web, so to speak, as a database or knowledge base for everything that you're doing with the model, which is a really cool, just really cool idea. And also you don't have to have a subscription.
[00:13:19] You can just pay sats as you go whenever you need to upload a document and have it summarize it or pull information from it, whatever it is that you're trying to work on. But Venice AI, which is the one built by Eric Voorhees, they Now have Llama 3.1, the 405 billion parameter model available.
[00:13:45] I used it for just a little bit and I immediately paid for A yearly pro subscription.
[00:13:53] This model is really good.
[00:13:55] This model is really good.
[00:13:58] Now again, I have done no extensive testing. I haven't looked at benchmarking or anything like that. I am just looking at it from the context of what I use these models for.
[00:14:12] And Venice has a specific. There's one that's just have a conversation obviously with the normal.
[00:14:19] Normal just kind of the raw model. But. And I'm sure this is probably just a system prompt, but then there's one specifically for development or developer engineering or something.
[00:14:31] And for so far for building tiny apps and working with that sort of stuff.
[00:14:39] I have been really impressed with how well the code is and I'm probably going to go back and forth. My next episode of Devs who Can't Code I'm going to be working on today and tomorrow, hopefully enough to get all of the recording and everything done.
[00:14:53] And this will be a great way to show the difference between Llama 3.1 and Claude and see kind of how they both behave back and forth and which one gives me more errors, which one gives me the best fix for the errors, you know, that sort of stuff to just kind of like test and play around with it. But I'm, I'm really kind of blown away. I again, going back to the idea that ChatGPT and Claude, I have no concrete difference between them to the point that I could just get rid of ChatGPT and I'm not, I'm not going to care. I'm never going to be like, man, I really wish I had Chat GPT around.
[00:15:29] No, Claud just does all the same jobs.
[00:15:32] Llama 3.1 does the same thing for me.
[00:15:38] So far I have no concrete evidence or suggestion from any of my interaction with it to suggest I'm losing anything by using Llama instead of Claude or instead of chatgpt.
[00:15:56] That's huge.
[00:15:58] I have never. And granted I've also never used a hundred billion parameter open source model. I've always used just one that I can use on my computer, but a 405 billion parameter model that is entirely open source and that basically any. I could run from any host, from any service provider and I'm running it from Venice AI right now. But this could also be on Unleashed Chat, which they just don't. They haven't implemented it yet.
[00:16:25] Um, but this could be running on any of these and so far I can't tell the difference.
[00:16:33] That's a big deal.
[00:16:35] That's a really big deal. And then to, to kind of see that in my own personal experience and then go back to what Mark Zuckerberg said about how he sees this playing out like the Linux ecosystem and that the ability to fine tune and fork and make low rank adjustments and all of these things and do RLHF on top llama to be able to have this base model to work off of and extend all of this stuff, man, open source is looking better and better. And who would want to.
[00:17:10] When you start to have all of these tools, like the number of tools that we've seen about customizing your data and being able to make better use of fine tuning them on your data set or in your environment and then host them yourself, all of those tools are only going to work with the open source models. I think he has a really good point that the explosion of stuff that we can do to build out this infrastructure is really only going to work and maintain the speed and the growth necessary to be the leaders, the global leaders in AI.
[00:17:51] If we are using open source models, I think being hyper focused on trying to get the biggest and best model that can do every single thing that we could ever think to throw at it and having the hugest, most incomprehensibly large, you know, trillion dollar cluster, it's. This is like a complete and total focus on scaling upward. And one of the things that I think we need to do is scale left and right. We need to scale out. We don't need to just be vertically, we don't just need to push vertical, we need to push horizontal. We need to get as broad and large of an infrastructure scope as possible.
[00:18:32] And this is very much like the Internet. Like we. You don't centralize and curate the Internet. And I genuinely think that these models are like this because like I don't think we're going to get. And this leads me back to Ashton Brenner and the piece on situational awareness and thinking about how the superintelligence explosion and the birth of AI that's just orders of magnitude. First just smarter than humans, but then immediately just hockey sticks to smarter than we can even possibly imagine. At least that's what it seems like the course if we just look at orders of magnitude after we reach human capacity and then enable AI to basically do AI research for us.
[00:19:20] This episode is brought to you by Coin Kite and the ColdCard hardware wallet. CoinKite actually has a ton of fantastic hardware and security devices for Bitcoin. So not only do the do you have the SATS card, which is kind of like cash, like you can put Bitcoin on it and then you can hand it to somebody else and they will literally have the bitcoin. It carries it with it and you don't have the keys exposed and, and you can reuse it. You can actually empty it, basically pull the keys out of it and take the bitcoin off of it and then put more on it again up to 10 times.
[00:19:53] So it's like a dollar bill for bitcoin except that you can change what the denomination is. It's a really, really clever thing if you haven't seen it. And not only that, they have them in a whole bunch of different styles and with different brands and artwork and stuff on them. And they're really fun to kind of just have. I have like a number of them. It's a fun way to hand out bitcoin and to just play around with honestly. Then they have the tap signer which is my favorite on the go hardware wallet because it is just a card. Works with nfc. So you just tap to sign and pay in your nunchuck wallet or whatever wallet that you're using on your phone and you can literally keep it in your wallet. And then of course the cold Card, Cold Card Mark 4 or Cold Card Q, their newest model.
[00:20:40] If you have a multisig, the cold cart should always and forever be a part of your multisig. And if you get just one hardware wallet because you're not stupid or weird like me and I have like 30, I don't know, probably like 12 or 15, I don't know if you get just one, get a cold card. Coinkite has been around since 2011.
[00:21:02] They have been building and focused on building open, secure, reproducible and air gapped bitcoin hardware wallets for longer than anybody else I know. And it is the way to protect your bitcoin. Check them out. And don't forget my discount code. Bitcoin audible link and details are right down in the description.
[00:21:25] So it brings me back to Ash and Brenner and the reason is, is because I've been thinking about this again.
[00:21:33] Uh, it's hard not to get it in my head because every time I start thinking about and playing around with AI, I go back to the artificial general intelligence thing because it just seems like one of those. I can't remember who it was. There's somebody in the audio notes was saying that like we don't need to answer this question. It's just going to eventually we're going to get there in three or four years and we'll just see or know, and that's a really good point. You know, use the tools as they are right now and focus on solving today's problems and, and today's risks or issues and you just kind of set the foundation for those problems having been solved in three to four years.
[00:22:09] But I still can't help but think about four years out. What does artificial general intelligence look like? Do we have superhuman intelligence? Et cetera, et cetera. Now one of the things that actually an analogy that struck me recently and something about going back to the point that I was discussing earlier that I would bring back with chat gbt versus Claude and then Claude versus llama 3.1 is I've reached the point with a lot of these models where I can't tell. They're probably specific tasks like coding certain things and coding, you know, multiple pages versus, you know, 30 lines of code or something like that. Like probably more and more advanced and larger and more thought out projects and processes would be able to show like real, a real difference. Like if I had them all try to write me, you know, 2,000 lines of code for an app, you could easily compare that. Okay, this one actually produced, you know, for a whole page. This one worked until there was an error and it took me five times to go through it to get it to an app that actually completed. Okay, well what about this model? This model never, I could never actually get it to complete and make me an app that worked. And then what about the third model?
[00:23:39] Actually on the very first try, it gave me a huge set of code and it worked. It did what it was supposed to do and in one go it built it. So at that point, when you're looking at something that big and that very, with a very targeted result, you could easily say you could more easily compare larger and far more intelligent models.
[00:24:03] But for general day to day, for the stuff that I'm using it for, I can't tell any difference. I llama 3.1 and Claude and Chatgpt might as well be the same model if you.
[00:24:17] So if you blindfolded me and got me to use them, I'm not 100% sure I could tell you the difference.
[00:24:23] I feel like there are, if I use them all long enough and very specifically there are like subtle tendencies and kind of personalities of the models depending on like, just like a very naive use of them, they will tend to do or favor certain things.
[00:24:43] So I feel like you could probably get to know them almost where you were aware that, oh, this is kind of, this is a Llama Answer. Not a chatgpt answer. But I don't think there's any hard quantitative difference that in my typical day to day use, I could actually make a comparison or assessment with.
[00:25:04] So it's kind of like computer graphics in video games. So everybody who grew up in my era and were playing video games in the 90s and early 2000s, you saw this explosion of graphics go from 2D to 3D. And then suddenly every year, every model of console, the graphics got so much better and so much better, and you're just blown away. And then you watch the Final Fantasy 10 cutscenes and you're like, oh my God, this is a movie. And then they literally make a movie out of it.
[00:25:40] Video game graphics just got so good, so amazing. And every single time it was just this huge leap and then it tapered off and it got so good or it got good enough that it just didn't matter anymore.
[00:25:58] Getting better graphics wasn't the thing that got you further.
[00:26:04] It wasn't. Didn't draw people in anymore. And a couple of times, you know, the, the console shift happened again and they wanted to pioneer and be like, oh, they kept trying to push that. The graphics were this much better. And they would do these like demo video things that just look so good, but you would never actually get game gameplay like that. And more importantly, the big thing is that it became a waste to try to make, became a waste of resources and GPU power and electricity and hard drive space in the device to try to keep making the graphics that much better. And what happened? Well, instead the games got bigger, the games got longer, they got more expansive. They became entire worlds that you could run around in and that it would literally take you all day to run from one side of the world to the other. Like these huge landscapes that would just like, I. I can't tell you, like Zelda and stuff on, on Switch. Like, the Nintendo Switch is crazy small, but one of the few games that I've actually played in the last like 10 years was the new Zelda game Breath of the Wild on Nintendo Switch. And that's a small device. You know, that's not a.
[00:27:20] The computers have gotten incredibly powerful.
[00:27:23] But the amount of that game that, like, there's no, you never go through a door and there's never a loading window. It's just a constant.
[00:27:33] You're in one space and you're. And you're just going. And you can like fly with your, with your little, your little air bender kite thing, whatever the hell it is, and you can Just see the. The distance that you can actually see. And then it loads. And it loads progressively and dynamically like you would. It gets. The closer you get to it, the more detail comes out of it. So there's far more intelligent loading. It's kind of like the video game equivalent of Netflix is that if your connection sucks, it sends it to you in 360p and then you start to buffer and it goes up to 480p, and then buffer a little bit more, it goes up to 720 and it finally goes up to HD. And this all happens while you're watching, so you don't notice it. And it does it just so that you don't have a delay time between pressing the play button and having it actually start. And you don't have buffer zones, you don't have it. So that just stops randomly and starts spinning. What happens is if you're getting close to your buffer window, it literally dynamically gears down the resolution again so that you always stay so you have a streaming experience that trades off resolution for a consistent and basically perfect watching experience or the best that you can get.
[00:28:47] So I bring these up to suggest that having something that intelligence, especially intelligence that we can't even measure.
[00:29:04] You know, basically what we're saying with the idea of super intelligence and artificial general intelligence is we're going to make something that's smarter than us to the point that we can't benchmark it and we can't tell what, we can't test how good it is and where. Only it is going to know if it can produce something better than itself.
[00:29:27] But then it runs into the same problem, right? It can't benchmark or test the next model that's bigger than it or more intelligent than it. Now we're going to have just all of these models that are all smarter than us.
[00:29:42] And it's like, am I gonna give a crap whether I'm using the model that's twice, quote, unquote, twice as smart as me or twice as capable than me than the one that's 10 times capable? Like, if I can't even measure and understand what that would even look like, what's the value?
[00:30:02] How do I understand the value that I'm getting out of it such that I would use one versus the other?
[00:30:08] Like, I just want to complete my task. I just want to do what I'm trying to accomplish. And if both of them do that and they specifically do it to a degree that, like, I can't make a judgment about which one is even Better or better yet, maybe both of them come up with the same solution because there's just an optimal or easiest way to accomplish the task. And being 10 times smarter and having 10 times more compute in a trillion dollar cluster doesn't help.
[00:30:38] Maybe it's just a waste of resources. Maybe it's like graphics and being just more intelligent, more like analytically, computationally intelligent than the last one.
[00:30:53] Isn't that much better that intuition and judgment? There's a lot of other things that aren't strictly just an intelligence decision.
[00:31:05] And there's also the question of can it ever be more intelligent than the information that we give it based on how all the LLMs work now? It can't be.
[00:31:14] It literally can't be. It can be more intelligent with the information that we have given it than we can be. Just like a computer can recall more stuff from a book than we can. Like, it has a more perfect memory, a much better memory, it can understand those things.
[00:31:31] So like, let's say we figure out an LLM to do incredibly deep weighted and thoughtful analysis at like two layers we got. We have like 10 major algorithm improvements and we add two entirely new layers of model weighting.
[00:31:49] I can't remember what they're.
[00:31:51] How they referred to it in the first read that we did on the gentle introduction to LLMs, but the attention score was. I kind of think of that as a new layer of how to.
[00:32:03] After you train the LLM on like the basic probability, well, then the attention score is the next level to understand the language, different pieces of the language and then different types, verbs to nouns and how structures of sentences work.
[00:32:16] So let's say we add two more layers to that and it starts developing reasoning abilities and all this sort of stuff just kind of emerges. Well, it's still not going to be able to come up with something that's not in the math textbook.
[00:32:31] Not unless it's genuinely a different thing than what it is now. Like right now it's a probability machine. And I just feel like no matter how much more of the Internet we give it, there's still this feedback loop, there's still this limiting factor of increasingly AI information being used to train the AI, which means that the only thing that will end up filtering the information for the AI is our judgment of that information.
[00:33:03] And all I can think is that we'll get to a point like, is it better to have a giant.
[00:33:15] What seems to be. What seems to me to be deeply inefficient model, to do everything, to be good at anything and everything versus having a million 1 trillion parameter models that in, you know, two eras of hard drive, excuse me, two more eras or ooms of hardware and compute and energy usage capacity. Like give it four years, five years and you get energy efficiency, you get compute efficiency and you get asics and kind of the next layer of hardware that comes out AI specific.
[00:34:04] Basically everybody can run llama 3.1405 B on prosumer hardware.
[00:34:12] Maybe I'm, maybe that's a stretch, but let's just say the equivalent of that model, which may be smaller now or maybe smaller at that time, but can be run on kind of your high end consumer hardware.
[00:34:24] Would it really be that much better to have a 10 trillion parameter model that is supposedly good at everything rather than having 10100 billion parameter models that are extremely targeted and have extremely in depth information or weightings for something very specific, I go back to the math book example is that you get an AI to learn a math book and it learns calculus, just backward, forward, up and down, sideways, like in every possible way. And you have a 20 billion parameter model that just knows calculus. It just can do all of calculus, it can explain calculus and it's a general model, it's a general open source model. So it can even come up with analogies and comparable real world situations to try to explain calculus because it already knows general language.
[00:35:28] Why does mixture of experts not still apply when you get bigger and bigger?
[00:35:36] Why is it that having one like the biggest and most intelligent thing, why is that so much better? I don't know. Part of me thinks we just put a undo. We're kind of missing 10, 20 other major factors that go into actually like I know plenty of people who are deeply intelligent and live very, very bad lives. Like they don't. They trust the wrong people, they are cowardly, they will not stand up for what is right if they don't think they're going to get a gold star for it. They don't have good judgment, they don't have good values and a lot of common sense is lost on them.
[00:36:27] So my question is when we are training an LLM on certain data we're writing or whatever material or content it is, judgments are being made about what is good and what is bad.
[00:36:43] They are being made on what should be trained heavily, what should be trained very loosely.
[00:36:49] And this is ultimately the same problem of what do we invest in in the economy, what do we invest in? Is it smarter to put more resources toward health care right now? Should we work on just building more houses so that we can solve the homeless problem?
[00:37:07] Should we dedicate resources to fixing the money so that we can solve a lot of the bad incentives in every problem?
[00:37:14] Well, the decisions that you make are based on your exposure of each of those ideas and each of those problems and how you see and understand, intuit what the solutions to each of those things might be.
[00:37:29] Think the overwhelming majority of people still have no idea about the foundations and explanations as to why Bitcoin is the most important thing that we could fix right now. That the money is the biggest problem in society and this is lost on so many people.
[00:37:47] And it's not even, it's not even like super advanced economics. It's basic economics being taught wrong by political institutions that are subsidizing debt. They're doing everything they can to subsidize the over consumption of society, like literally and directly. To issue money out of thin air is to subsidize debt at the cost of savings, is to subsidize destruction at the cost of production. For anybody who isn't aware of that or doesn't know, doesn't have any foundation or grounding in those ideas and then takes, you know, the general economics textbook at face value, written by authoritative academia.
[00:38:28] Well, what happens if you train a model very very intensely and very, very deeply on that?
[00:38:35] It's going to make a bunch of terrible decisions. It's going to make fundamental misunderstandings about how economics, about how the world actually works. It's going to try to quantitative everything. When economics is literally a qualitative science.
[00:38:50] You can't quantify human behavior, human action and decision making.
[00:38:55] It is literally all qualitative, subjective and relative.
[00:39:00] So which is better? A 1 trillion dollar cluster with a 10,050,000,000 parameter model that knows all of mainstream economics, every piece of history, every research paper, all of it backward, forward, up, down, sideways, or a 500 billion parameter model that understands Austrian economic theory?
[00:39:30] And now we're back to the original question judgment.
[00:39:34] And now we're back to a very old and familiar problem. Trying to predict the future of humans and trying to predict humans human choices and behaviors and actions in a market.
[00:39:45] You can't do it.
[00:39:47] It's not a matter of intelligence. You can't just make something smarter and then suddenly it just knows and understands everything.
[00:39:57] You can't model human behavior and subjective value.
[00:40:01] You can't model the universe.
[00:40:03] You can abstract out fundamental laws, but no matter how good the models are, there are too many factors to actually compute. It will always be a map. It will never be the actual place. And there's just so much that there's only so much that you can put in a map. And when you go to the area where the map is, you see the billions of blades of grass that you could not put on the map. You see the actual water, you see the trees, you see the sunlight, you see the incredible infinitely complex ecosystem that is active and existing at every single point in location on that map. And that map barely shows a shadow of an outline of just kind of like what you might see if you were standing a thousand feet away from it.
[00:40:57] The point that I'm getting at with all of this philosophizing is that just being intelligent and just being quote, unquote, super intelligent, if that's really what we end up with in three or four years with these models, doesn't mean you can just invent whatever you want.
[00:41:15] It doesn't mean that it's going to have perfect judgment as to where the winds of the world are blowing and what humans will actually need in the next five to 10 years, and what to invest in and where to target and direct its mental energies, so to speak.
[00:41:39] And going back to our calculus example, it may be very likely that only giving it 25% of the compute necessary to really understand calculus. Up, down, forward, backward, and sideways.
[00:41:56] If you give it a fourth or even a tenth of the compute, it still understands calculus at like 99 and a half percent. Like, it still understands it well enough to write programs that can compute and work in calculus. And that putting in 10 times the amount of compute is just.
[00:42:14] It's just a question of why.
[00:42:16] Why do we need. Why do we need. That's a waste.
[00:42:19] It's actually better to take this base model and then teach it rather than 10 times the amount of compute on calculus, we do 1/10 on 10 different levels of math.
[00:42:33] Algebra, calculus, statistics, geometry, trigonometry, modeling, probability.
[00:42:39] I mean, think about 10 models that know all of those things that well, for the exact same cost of making the one that knows calculus just a little bit better, because you just kind of reached this threshold of it's still just not going to know anything but what you. What was in the textbook.
[00:42:56] And I still have a really hard time deciding or believing that if you taught it calculus that much better and then tried to make a general model that also learned algebra, geometry, calculus, statistics, all of this stuff all in one, that somehow that means it's going to invent the next great thing in math.
[00:43:23] I still have a really hard time with that one. Because even if you go With Aschenbrenner's explanation and the fact that you can train, you know, two GPT2s or 100 GPT2s on the same recipe, and you can tell which one's better, and therefore you can then compute, make it better and scale it up to a GPT4. And thus a GPT4 could train a bunch of GPT2s to be better than could make its next model. Well, that doesn't mean that the model knows or understands or is more intuitive or is more creative than the GPT4.
[00:44:07] It just means that it has better probability of good results from the same information it was given. It's like better graphics, but without a better game to play them with.
[00:44:22] That's why when graphics reached their limit, it became more about changing the game, making the game better with the newer graphics, and it just stopped being about the graphics.
[00:44:36] I kinda wonder if in two, three years, the super intelligence explosion and the way to keep on the trend line, kind of like what happened with Moore's Law, is that we change the direction of AI and we start going wide rather than vertical. We go horizontal rather than vertical. Just like Moore's Law, hit a limit with the size of the transistors and the size of the chips. And what happened is the architectures of the computers started to change basically because. Because the size of the chips were able to get smaller and smaller and smaller so quickly and at such a rapid pace. We basically left a lot of fat. We left a lot of unnecessary material and old systems that weren't super efficient and like, really long distances between different components on the bus and on the motherboard and all of this stuff that was causing lots of different, you know, heat and energy loss and creating enormous bandwidth limitations. And so we went to unified memory and then we started upping the RAM quite a bit because we realized everything could work better if we just focused on random access memory and like, super highly volatile stuff that could, you know, create entire.
[00:45:54] We could open up 20 applications and stuff at the exact same time and then do things like AI. But when we hit the limit of shrinking the computation or shrinking the transistors and the gates, we still basically kept up with Amore's Law. But we did so by moving the paradigm into a different zone where the operating systems got more efficient, we developed better batteries, and we shrank the size of all of the other components and created smartphones. And then we changed the interface, and then all of these components started getting soldered right on top of each other so there were no bandwidth constraints. And we had unified memory attached straight to the CPUs and GPUs of these machines. And then we started creating new HPUs and virtual reality computation and new AI chips. We started getting specialized in the chips so that we made them more asic, like where they were tuned to specific tasks and started having more processors. Even our cords have processors. They have tiny little computers in them. All sorts of stuff is happening to, to organize and sort and delay and correct for information loss and to deal with far more complexity and density in all of our signals. Basically it stopped being about how many gigahertz you have in your machine.
[00:47:21] And 20 other factors started to take over as the things to fix.
[00:47:27] Mostly because the amount of the gigahertz that we, the speed, the incredible increase in speed of the actual processors made it so that it just made sense to put all of our resources there for the first 20 years of this, 30 years of this crazy tech explosion, until we ran to its limits and it started to slow down. And suddenly we realized all of this stupid stuff that we did around it in a very naive fashion. And where we could double the computation by just focusing on the processor and we could just leave the motherboard as is. We just left it. We just left it. And suddenly when it no longer made sense to put it into the CPU because we reached the top of the S curve for that, well then now it made sense to change the design, to separate out chips for different purposes, to put more RAM in the machine, to switch to solid state hard drives. We kept those massive performance increases and we kept the general trend line of all of technology and electronics and we started making everything smart and everything started getting a freaking computer inside of it, even if you didn't want one. But it's not because our chips are a hundredth of the size as they were six years ago.
[00:48:41] Games today aren't better because the graphics are 100 times better than they were 10 years ago.
[00:48:47] And with all of these eras getting shorter and shorter before we shift over to the new thing and existence in general. Like all of the economy and the global ecosystem becoming more and more volatile and more and more disruptive in new eras, I can't help but wonder if intelligence will just kind of reach a limit where it just doesn't make sense to put more into that because there are too many other factors around it that just kind of raw quantitative intelligence doesn't fix.
[00:49:22] I think we've kind of put the idea of more intelligent than humans on this pedestal where if we're looking up at it, that that means it's everything and it's solved. It's, it's all the things. It's somehow just.
[00:49:46] And you know, this might be kind of this arrogance or something that more intelligent than human because we can't qualify it, we can't quantitatively test it, we can't measure just means omnipotent.
[00:50:02] You know, that's just, that's all it is. If it's smarter than us, it's omnipotent.
[00:50:07] It's like we're not that smart. Like we are, but our values suck a lot of time. Our judgment sucks.
[00:50:16] Again, there's so many other factors.
[00:50:18] Like I know stupid people who are way more successful and have way better lives and have way better judgment than very intelligent people that I know. Like it's not like a cheat code for life. Like you can still screw it up and be brilliant.
[00:50:31] Raw intelligence is not a cure all. And it also doesn't solve the problem of the future still being unknowable. It's still simply. It doesn't matter how much raw intelligence you have.
[00:50:45] You can't possibly have access to all of the relevant information because most of the information isn't even quantifiable. It isn't even something that can be measured. It isn't something that's even exposed or visible in any way.
[00:50:59] All we can ever do is look at the map.
[00:51:02] Sometimes on that map you can see that something is shifting. But you'll never understand the whole ecosystem. You'll never have all of the information. There's more information in one tree with all the bugs and the bacteria and all of the stuff. Then the entire map even tries to pull or abstract out of the whole geography of the continent.
[00:51:24] That doesn't change because we have super intelligent models. And I still do agree with the assessment in a lot of ways that the trend lines are going to stay on course, that we're going to get orders of magnitude, that we're going to have an intelligence explosion. But I increasingly think that intelligence explosion is going to be horizontal, not vertical. It won't be about having that much bigger of a model and that much smarter of a model because it won't make. It's just energy, it's wasteful, it's not energy efficient, it's not going to solve a thousand times the problems just because we put a thousand times the compute in it.
[00:52:03] I tend to think it might end up being like computer graphics. And we find out that there's a whole lot more that's important.
[00:52:11] And just like The Internet wasn't about making the biggest possible website to do everything that we could possibly want a website to do.
[00:52:22] And the application and software space wasn't about building one giant application that can do every single thing that we would want an application to do. It was about creating an ecosystem for billions of websites. It was about creating an ecosystem and platforms and languages and app stores and operating system for billions and billions of applications for every possible purpose that we could think of under the sun.
[00:52:53] Why would these be different?
[00:52:56] In fact, it already seems to be the case that it's not different.
[00:53:01] We went in what was it, like three years or four years or something on hugging face. It got a million models and then before another year was out it was at 2 million.
[00:53:13] We're already going horizontal.
[00:53:16] And I think when you look at the progression. So what does all this have to do with open source? Why is this episode called Open Source? AI is catching up and going back to the front. The first thing I talked about that I couldn't tell the difference between Claude Llama and ChatGPT.
[00:53:31] And yet out of those three, only one will ever expand horizontally the way that a market and dynamic and global ecosystem will need it, and that's Llama.
[00:53:44] Because it's open source, it will have millions of forks.
[00:53:51] Not yet.
[00:53:52] Right now it's brand new.
[00:53:54] We're still figuring out the tools for fine tuning, still figuring out how to rethink all of our data collection and curation and judgment and understanding values and what to train it on and how much to train it. We still got algorithmic improvements to make, we've still got systems for fine tuning models on your own data to figure out. We've still got to get, you know, two eras, two, two new generations of like significant hardware improvements and AI targeted chips into machines and figure out algorithmic improvements that make that 405 billion parameter model run like a 40 billion parameter model on a typical prosumer or high level machine with the same results.
[00:54:37] We still have so many different things to solve and improve around so many of the different things. But we will have millions upon millions upon millions of forks and specialized models. I think we will get to the point and this is kind of the dream. This is what everybody thought about immediately when we talked about LLMs, was everybody having their own personalized model. I think we do get there, but we only get there with open source because you're not going to be able to fork ChatGPT if they don't even give you the weights. You can't do it. And they will never be able to build the internal stack with all of the tooling and all of the fine tuning capabilities and the trillion dollar, the deca trillion dollar cluster needed to service everybody's needs. And it is never going to be efficient enough because they're just gonna try to push the model bigger and bigger and bigger. Like PlayStation 5, 6 and 7 is just about better graphics.
[00:55:35] And I think it's just gonna turn out to be a huge waste of resources.
[00:55:39] And that what we actually needed was a model for every person that was llama 3.1, llama 4, llama 5 capable and was fine tuned and guided towards their specificity, their speciality in the market and enabled more general thoughts so that they could apply their speciality to other people's speciality in a more general sense, so that we didn't close off our categories for each other. And you had general wisdom from plumbers to go into your thinking about biochemistry. You had a guide to sort you through technical problems on your machine and a basic software engineer that could solve problems, that could install applications for you so that you could do your job better. And you had exposure and access to more information and more skills generally around everything that you do that's just easier, that's just more accessible and that is built into a software waiting machine, a pattern recognizer and repeater.
[00:56:53] And no chatgpt is ever going to build in all of that functionality. Just like there's never going to be one company that makes all of the websites.
[00:57:02] Imagine, imagine if every single website was being tr. Like every single function of every application that we could think of was just all going to be crammed into itunes. We were just going to have universal tunes and it was just for no reason at all. It was just going to do all of the jobs and it was all going to be in that one application. And there was just going to be one giant development team, one company with 20,000, 100,000, a million developers that are just all designing to figure out where to put in itunes, you know, your, your home security system and your Bitcoin wallet and every other damn thing that didn't happen.
[00:57:40] Nothing about any of the progression and evolution of technology has happened.
[00:57:47] Like the analogous superintelligence to just insane outright vertical explosion, to just a level of intelligence that is incomprehensible and can just do everything and create anything that we need it to.
[00:58:07] And humans are just obsoleted completely.
[00:58:12] I don't know. I can't help but think that it will, that it will turn, that will curve just like so many other things before it.
[00:58:24] And that's the other thing too is it's not hard to come up with an analogy. Like everything that you think about as like a highly, like a fast progressing technology basically did this same thing.
[00:58:38] Like at some point the improvements and the acceleration, it's. Everything continued to accelerate, everything continued to get better.
[00:58:48] We still had orders of magnitude increases in capacity, productivity, output, the value add, all of these things. Like none of these trends stopped globally, like universally across all technology and across all innovation in the economy.
[00:59:07] They stayed in line, but they shifted from place to place, they shifted from layer to layer. You know, Bitcoin progress happened a whole lot faster on layer one from 2009 to 2013.
[00:59:21] Then it moved incredibly slowly and now it's moving at a snail's pace for good reason. And all of that has shifted to layer two and layer three. And this is all just higher layers on top of technology. This is all just higher layers on top of the Internet. And Internet is just a higher layer on top of our hardware and electronics technology with an abstraction of code and pixels and, you know, zeros and ones that turned into programs and functions, down to just the smallest of gates in a literal on, off switch that has been replicated and created into these systems of billions and trillions of little circuits. Like everything's just an abstraction on top of an abstraction. And at each layer it reaches this point in the curve where it doesn't make more sense.
[01:00:16] The output no longer matches the input or the change in the output is no longer noticeably different.
[01:00:25] For it to make sense, to keep putting resources in that place or in towards that direction and we shift into the next thing.
[01:00:34] And I just kind of think that's what we should be looking for in AI in three to four years, when we're talking about the Ooms hitting the point of a sort of singularity of intelligence explosion, is that we should be looking for a horizontal one, not a vertical one, and that there won't be a Manhattan Project, just one giant model that just does everything so well and so completely that we never need another model again, that it just, we just kind of turn it on and it does everything else for us.
[01:01:19] Maybe I'm wrong, maybe I'm wrong, maybe I'm thinking about it wrong, but that's. That framing seems to align, in my opinion, more to reality. And specifically it aligns to open source.
[01:01:35] It basically demands open source for that explosion to happen, for the trend line and the expansion and variety and innovation of the ecosystem to actually stay on course for it to actually work. Open source is the only thing that can fulfill that need.
[01:01:57] And like I said previously, it's also the only thing that can't be replicated.
[01:02:02] You know when we're talking about a vertical model and we're just making a GPT 5 and a GPT 6 and a GPT 7 and a G GPT Everest, whatever, all of that investment is gone like that.
[01:02:16] As soon as the weights leak, soon as somebody with a thumb drive walks out of the building and mails it to somebody in China, it's instantly replicated. It's today's equivalent of trying to make it so that you can't pirate movies. Except that there's only one movie that you have to pirate and everybody wants it.
[01:02:40] Like that's not going to happen. You're not going to be able to stop that from happening. The only thing that can't be replicated is infrastructure, is a million forks, a billion fine tunes, category, industry specialty, specific RLHF combinations of different models, low rank adjustments, all of these things.
[01:03:06] If you make a billion different billion variety and specific things in all of these different companies and for all of these different people and in for individuals and for individual ecosystems and operating systems, if you build all of that, you can't put that on a thumb drive and ship it over to China.
[01:03:24] And all of that will do millions of times more. With millions of times the number of people making judgments and deciding what to do with it and determining what path to take and what to train it on. All of these things that are qualitative decisions, they will produce infinitely more and better superior results Once we run the course on the more I got more gigahertz in my CPU stage and I think that's where we are. My graphics are better than yours, so we'll see. I'll call this a prediction episode because I've gone back and forth on this issue numerous times and I keep finding a lot of new information and then thinking about it and then trying to apply it and fit those contradictions into my worldview about evolution and markets and human psychology and all of that stuff. And so I've drifted around quite a bit in this, in my thinking on this. And then also I've used it a lot, like I've got to use it more and more. And I'm really impressed with Llama 3.1 and the 405 billion parameter model. I'll be sure to link to Venice AI. It's only $50 for which you can pay in Bitcoin. It's only 50 bucks for a yearly subscription and it's 20 bucks a month for both Claude and ChatGPT. So basically for 2ish months for less than a quarter of the year that you get with ChatGPT and Claude you get Llama 3.1 for an entire year. And here's the other thing about that.
[01:05:06] That is only possible because they're not having to train the model because it's an open source model and everybody can build and work together on it.
[01:05:15] I don't know. Something to think about. Let me know your thoughts.
[01:05:19] Boost Send in a comment on Fountain. Hit me up on Noster. You can also try on Twitter. I'm not very good at checking in on that anymore, but I do try and thank you all for supporting the show. Thank you guys for listening. I hope you got something out of this one. And shout out to everybody who boosts and gives value for value on the Fountain app. It makes a huge difference. And of course, thank you to coinkite and the ColdCard hardware wallet for supporting the show. Links, details and discount right in the show. Notes this is AI Unchained. I am Guy Swan and until next time everybody. Take it easy guys.
[01:06:04] Intelligence without wisdom is nothing more than stupidity. That looks smart.
[01:06:12] Craig D. Lonsbrough.
