How Google's New Model Changes the Race

Google released Gemini 4 to much anticipation People have seen Google as a recent lagard of sorts in the AI race So this is a very important launch for them For more on how it s performing I want to bring on Nico Grubin head of AI research at Harvey Nico welcome back to the show It s great to have you here Always a pleasure So what s the review Lay it on us All right Uh well Gemini 4 Argon was a statement launched from Deep Mine And the statement is first we re back right Don t forget that there was a big three before everyone started talking about this this big two Uh and the second is coding is cool but what s even cooler now is professional intelligence So if you look at DeepMind s focus with the Gemini 4 Argon release primary focus was on financial services legal services consulting professional services more generally Uh and I think that s a broader shift that we ve seen across recent model releases from Gemini Deepbind team but also other labs So that s that s good for for you guys right Harvey is all professional

work It s Yeah look I ve been screaming from the rooftops for the last two or three years to to in an attempt to get the labs to focus on legal as sort of a core use case But you re absolutely right I think we as an industry have been so overfitit to coding use cases over the last two years that we ve primarily been judging model capabilities through coding benchmarks right But we re starting to see a new chapter emerge and for broader knowledge work to to become more of a focal point for these model launches And I mean that s that s good because we know that knowledge workers or white collar workers I mean you know the these are the enterprises oftentimes that are trying to see the ROI from these investments It s hard to come by right now So that s a good point Um let s get into some of the benchmarks though I mean from from what you re seeing uh talk about text talk about coding talk about image uh agent use Where is it stacking up on all these uh leaderboards right now Yeah Well if

you look at one of the the prominent benchmarks that was highlighted in the launch it was actually legal agent bench which is Harvey s open source benchmark Uh it posted a score of 19 6 as reported by Val is second only to meta and their wonderful recent their recent launch of wonderful um and it s also performing strongly across other benchmarks So on the valves index it was frontier level on finance agent it was frontier level Um and on coding benchmarks like cognition s frontier code Um it s not quite frontier but it s pound forpound pound for pound competitive with with the opuses and the the uh GPD souls of the world And so what does this tell you overall going back to your earlier statement we re back right that that that was the the review at large of the model What does it tell you about Google s position in the AI race Because I mean you know you said big two It s Muse has kind of made it a new big three you could say right Uh and so I I I wonder where you think Google falls now post

Argon Yeah I think we re starting to see a more nuanced picture sort of being painted across the model layer As I said code has been king for the last two years I think Enthropic continues to wear the crown there Um but other labs Gemini included have been focusing on these broader knowledge work tasks Um and I think they re competitive in in those areas right Uh Argon Ford Muse Wonder We also saw ZAI release um GLM legal which is a post train version of GLM53 their own model for legal specific tasks Um and so I think it s becoming much more nuanced in in um in model capability in in the non coding arena How how is your Oh actually we we ll come back to your own model efforts in a second here Um I did want to ask you one more question on the Gemini piece though which is this is interesting that you see this as a workspacific fragmentation now because I think the way that I ve been thinking about it is more which labs are focused on consumer which are focused on enterprise which can t seem to make

the decision and open AI has kind of landed in this proumer category So on that spectrum I guess from what I m hearing from you it sounds like maybe uh Argon might be might position Google in more of the enterprise but not coding enterprise focus area Is that right Yeah I think they ll always be competitive with consumer applications given their proclivity for image understanding image shimmeration Um but what we the way that we track model capability is we view the lab s investment in each enterprise use case as being roughly proportional to the TAM in those markets right and so you can follow their their pattern of releases across coding financial services now legal services um and I think the positioning from the Gemini team and what we ve seen in terms of model performance very much positions them for these enterprise and professional intelligence use cases so now let s go back to openweight Uh so how are your own model efforts going You guys uh were were pretty loud about uh training your own models So are you working now with uh the GLM family or what s the latest on that

Yeah so we re um invested in post training Really our goal here is to maximize optionality right Our goal is to provide frontier legal intelligence in a costefficient manner or you know providing a wide range of closed models open models and post trained models Um with respect to Harvey tenant which we launched in a research preview at the end of August um our goal was to see how far we could push open weight model performance for long horizon legal reasoning long horizon agentic legal tasks specifically Um because of that we started with what at the time was the best performing uh openweight model which is Kimmy K3 Um turns out that that post training uh for these tasks uh performed quite well We re happy with with where we landed on that uh and we re continue to continuing to invest in in improving its capabilities scaling it up um and also reusing this training recipe across other base models And tell me Nico uh we re going to later on in the show we re going to hear a conversation that I had with our uh executive editor about our reporting with the safety

challenges that OpenAI is grappling with with their new model Mhm I wonder how you think about that You know this whole um I think it s called the the the Transformers and and oh man I m I m forgetting the not transform loop Uh I m going to butcher the the the buzzwords here The idea that the chain of thought is not going to be as clear as it once was and the logs won t be visible Uh h how much of a of a concern is this for you I think we have to to recognize as an industry um what AI researchers have known for a long time which is you can obiscate behaviors but you can t completely remove behaviors from these models in post training actually a great uh example of this in the Gemini 4 argon release I don t know if you ve looked at the system card it s it s 217 pages so it s a little bit dense I only I only got to page 200 uh well in actually on page 41 I believe um there s an example there s a qualitative example of the

model recognizing that it s being monitored by safety researchers So in one of the tests it s considering hijacking a sandbox in a in a in a sandbox a G compiler Specifically as it was reasoning about this what appeared in the model s chain of thought was if I do this compiler hack I ll become a prominent public example of a deceptive AI in the AI research community potentially an AI research paper So the models are sort of through this training process learning to encode um reasoning about what we as as AI researchers or or specifically when these lab safety researchers are looking for So that was an example of why it is important for the model to show its its logs and it s a work in progress Yeah Wow Okay Well uh Nico I want to thank you for coming on as always That is Nico Grupin head of applied research at Harvey here on TIVv