Can Open Models Keep Up With OpenAI and Anthropic? | Nathan Lambert

When I think about model progress it comes down to different forms of capital And that is compute data and talent you see that playing out in terms of like who s going to win in the next couple of years The Chinese are so good at building models To say that China is anything but equal on talent you re going to have a hard time What they don t have is compute Most people regard distillation as like the main lever for them to do this People distill my products every single day They take it they strip it down to its raw parts music and that s called competition Previous post training was much more reliant on the supervised fine tuning Now this has morphed a bit into environments where it s RL heavy But it s just like hard to map this type of post training which is you get the completions and the score goes up to what is going on in reasoning today The discourse now is just so much fear on like how much the frontier music is just hidden and also breaking away The AI capability plot doesn t necessarily worry me

But the AI capability plot in the pretty much non existing preparation of cross public cybersecurity and all of these things like So Nathan really happy to have you here Thank you for joining in person Super timely given your launching Trillium which I d love to talk about But more sort of on a macro level I think we talked a lot about open source and that s you know a lot of what you care about And what I sort of um or what what we ve been hearing a lot in the public discourse is like Hey we have to pace the frontier you know the frontier this the frontier that And I think an arc of the stuff that I m really trying to ask experts about are is like What is in the frontier What is it even like what are we pacing What s there And then more importantly kind of threading it with open source and open weight models like can they keep up with the frontier And I think this matters very much and this is something that you re an expert on So maybe to to sort of kind of start

on that arc what s gotten to the or sort of how you would break down what s gotten us to the frontier for both closed labs and open source models you know how does it break down in your mind Maybe first at a high level of pre training mid training post training how post training has evolved which is something you ve you know done a great amount of education to the public on When I think about model progress it comes down to like different forms of capital that you put into them And that is like compute data and talent And all of the labs have various amounts of this And I think the the closed frontier labs of OpenAI s and Anthropic s Google s have clearly had the most of these And what we see is like the different abilities of the labs to organize these forms of capital And like making the org chart work for building models is really hard which is why you have companies have to do complete resets from the bottom up And then it actually kind of works sometimes Like company like Meta is like totally reset and

they re like on a positive trajectory again They re like Seriously yeah It s like I think this I m very interested in these human aspects of like why competition drives the frontier and like why the org chart is so important because like we see these things and this comes into play in the open closed dynamic as well where it s like the closed companies the front the true frontier labs which I say if somebody wants to work at the frontier of AI I mostly tell them to work at Anthropic and OpenAI which is the easiest bet I think Um Meta Google um SpaceX AI like I think you still might be there but it s harder to see from the outside And I think these companies have just taken the most ambitious bets the earliest in order to get there And that s why OpenAI has done things like kickstart the reasoning model revolution And I think that kind of breakthrough and like that beautiful plot with inference scaling laws is like peak OpenAI It s like they re really good at getting this plot that just changes how everyone thinks about everything Yeah

And on the other side is like Anthropic has been like very focused on the types of things the models are going to be able to do And then their reward was this Claude code explosion with their focus on coding for so long And there s these subtly different cultures And I think those two companies in particular have been the most forward looking and the most willing to spend egregious amounts to make a step towards a direction that people think is impossible The Chinese companies came into this because they want to have a company that matters And if they heard to build their models and try to sell an API from China nobody would know about them Like nobody would know about Moonshot AI Very few people would know about Jipu and there s a very long list after this It s a very practical decision by them They re following the discourse They know there s going to be a huge market They know there s a huge demand in the US for paying for these things And it s very practical So therefore like releasing open models is a way to shortcut this and

get into the conversation And I I think there s a lot of complexity in like why do they do this Is the government involved And and and not It s like not the case They re just like trying to build a startup And it s not It s not to say that they will not get there Like the Chinese government is does things that would blow people s minds in the US And I think they like That makes me think that the Chinese labs have a much more serious approach to security because like things happen in China that are much more drastic responses to companies doing things in the US And I don t need to go down this rabbit hole But this is just like between the balance of open and closed it s like these companies in China are trying to catch up And this is a very different dynamic than what I said about OpenAI and Anthropic They re trying to create the new paradigms They re trying to like make true RSI happen so that their company wins with the best models forever if you get there first That s these

boom ideas And you just have to fill the market where there is a void And then open weight models are trying to do this And I think over time there will be these tiers of open models where there s the best closed models which you can demand the best margins for And then there s the near frontier models which is a step behind this And those will be open because it s just like some sort of supply and demand there As you get access to a different market that might be less profitable but the for example like if you have private enterprise data or so on And these tiers are kind of natural evolutions of companies that are trying to do things And I think we re talking about the frontier and it s just like if you go back to capital like they have less capital they re going to make less money and like that s okay but I think this drive to catch up is very strong And once you know once Anthropic and OpenAI show you the type of model you need to build it s a lot easier to then

be like Oh we just need to do this thing And Yeah they buy similar data sets It trickles through the data economy where OpenAI and Anthropic spend way more to build out this function of a data industry and then those data companies a year later might sell them to everybody They might sell them to everybody They need to maximize their return on their investment and they had some contractual period And I think there s a lot of dynamics where fast following is going to be very doable And it we ve seen this cycle proceed for a while and like longer than many people would expect I think When ChatGPT was released we talked about consolidation of the industry and predicting it And I think we re just starting to see a much more active mergers and acquisitions market but I thought this was going to happen 2 years ago And the open model companies have lasted much longer than I thought Hey you know there s so many interesting threads to pull on here Um I mean starting with actually your comment uh commentary on the um organizational differences between the top two frontier labs

You know we can maybe table that or or go on through that thread But then of course also kind of explaining or you explaining sort of how the open um Chinese uh models or the open labs in China have really just chosen that as much more of a distribution practical commercialization strategy uh rather than hey you know obviously out of altruism Though it s great that the researchers maybe maybe for better or for worse were were motivated for for you know sharing the beginning but at the end of the day it perpetuated because it was an excellent excellent commercialization uh commercialization strategy Um so the other threat maybe to pull there is sort of you know I think most people regard distillation as sort of like the main lever for them to do this so well and kind of increasingly uh well because um the lag between the open source uh models and the frontier models have only shortened uh with time in the last few years Um now it s like maybe only two to three months Um and this is also pushed the uh frontier labs to maybe want to uh close off

the APIs um close off their best models to do first party services I mean one probably to capture more value cuz it s always you know possible that way but then two also cuz it s harder to distill not impossible but much harder to distill Um maybe on those two fronts cuz they re both very interesting so we ll try to touch on it Um let s start with the with the is the distillation story really more of a caricature and then Chinese labs cuz since you ve actually met with them and you ve studied this a lot more uh is it sort of like they re all rounder like great at pre training also have their own you know good post training uh recipes that they may or may not have revealed um and where do you see the levers kind of accelerating Yeah so the Chinese labs are very technically excellent They re more compute starved I mean Yes Like without a doubt they have a comparable talent pool to draw on I think there might be like a slight edge based on who was where when AI really got going but to