One of the most important AI scientists in Big Tech wants to scrap the current approach to building human-level AI. What we need, Yann LeCun has indicated, are not large language models, but “world models.”

LeCun, chief AI scientist of “fundamental AI research” at Meta, is expected to resign from Meta soon according to multiple reports from credible outlets. LeCun is a 65-year-old elder statesman in the world of AI science, and he has had seemingly limitless resources at his disposal working as the big AI brain at one of the world’s largest tech companies.

Why is he leaving a company that’s been spending lavishly, poaching the most highly-skilled AI experts from other firms, and, according to a July blog post by CEO Mark Zuckerburg, making such astonishing leaps in-house that supposedly the development of “superintelligence is now in sight”?

He’s actually been hinting at the answer for a long time. When it comes to human-level intelligence, LeCun has become notorious lately for saying LLMs as we currently understand them are duds—no longer worth pursuing, no matter how much Big Tech scales them up. He said in April of last year that “an LLM is basically an off-ramp, a distraction, a dead end.” (The arch AI critic Gary Marcus has ripped into LeCun for “belligerently” defending LLMs from Marcus’ own critiques and then flip-flopping.)

A Wall Street Journal analysis of LeCun’s career published Friday points to some other possibilities about the reasons for his departure in light of this belief. This past summer, a 28-year-old named Alexandr Wang—the co-creator of the LLM-based sensation ChatGPT—became the head of AI at Meta, making an upstart LLM fanatic LeCun’s boss. And Meta brought in another relatively young chief scientist to work above LeCun this year, Shengjia Zhao. Meta’s announcement of Zhao’s new role touts a scaling “breakthrough” he apparently delivered. LeCun says he has lost faith in scaling.

If you’re wondering how LeCun can be a chief scientist if Zhao is also a chief scientist, it’s because Meta’s AI operation sounds like it has an eccentric org chart, split into multiple, separate groups. Hundreds of people were laid off last month, apparently in an effort to straighten all this out.

The Financial Times’ report on LeCun from earlier this week suggests that LeCun will now found a startup focused on “world models.” 

Again, LeCun has not been shy about why he thinks world models have the answers AI needs. He gave a detailed speech about this at the AI Action Summit in Paris back in February, but it got kind of overshadowed by the U.S. representative, Vice President J.D. Vance, giving a bellicose speech about how everyone had better get out of America’s way on AI. 

Why Is Yann LeCun fascinated by world models?

As spelled out in his speech—LeCun, who worked on the Meta AI smart glasses, but not to a significant degree on Meta’s Llama LLM—is a huge believer in wearables.

Amazing how the Ray-Ban Meta glasses can help the visually impaired. https://t.co/w3ZxCFtTlE

— Yann LeCun (@ylecun) September 30, 2024

We’ll need to interact with future wearables as if they are people, he thinks, and LLMs simply don’t understand the world like people do. With LLMs, he says, “we can’t even reproduce cat intelligence or rat intelligence, let alone dog intelligence. They can do amazing feats. They understand the physical world. Any housecat can plan very highly complex actions. And they have causal models of the world.” 

LeCun provides a thought experiment to illustrate what he thinks might prompt—if you will—a world model, and it’s something he thinks any human can easily do that an LLM simply cannot: 

“If I tell you ‘imagine a cube floating in the air in front of you. Okay now rotate this cube by 90 degrees around a vertical axis. What does it look like?’ It’s very easy for you to kind of have this mental model of a cube rotating.”  

With very little effort, an LLM can write a dirty limerick about a hovering, rotating cube, sure, but it can’t really help you interact with one. LeCun avers that this is because of a difference between text data and data derived from processing the many parts of the world that aren’t text. While LLMs are trained on an amount of text it would take 450,000 years to read, LeCun says, a four-year-old child who has been awake for 16,000 hours has processed, with their eyes or by touching, 1.4 x 10^14bytes of sensory data about the world, which he says is more than an LLM.

These, by the way, are just the estimates LeCun gives in his speech, and it should be noted that he has given others. The abstraction the numbers are pointing to, however, is that LLMs are limited in ways that LeCun thinks world models would not be. 

What model does LeCun want to build, and how will he build it?

LeCun has already begun working on world models at Meta—including making an introductory video that implores you to imagine a rotating cube.

The model of LeCun’s dreams as described in his AI Action Summit speech contains a current “estimate of the state of the world,” in the form of some sort of abstract representation of, well, everything, or at least everything that’s relevant in the current context, and rather than sequential, tokenized prediction, it “predicts the resulting state of the world that will occur after you take that sequence of actions.” 

World models will allow future computer scientists to build, he says, “systems that can plan actions—possibly hierarchically—so as to fulfill an objective, and systems that can reason.” LeCun also insists that such systems will have more robust safety features, because the ways we control them will be built into them, rather than being mysterious black boxes that spit out text, and which have to be refined by fine tuning. 

In what LeCun says is classical AI—such as the software used in a search engine—all problems are reducible to optimization. His world model, he suggests, will look at the current state of the world, and seek compatibility with some different state by finding efficient solutions. “You want an energy function that measures incompatibility, and given an x, find a y that has low energy for that x,” LeCun says in his speech.  

Again, these are just credible reports from leaked information about LeCun’s plans, and he hasn’t even confirmed that he’s founding something new. If everything we can cobble together from LeCun’s public statements sounds tentative and a bit fuzzy at the current phase, it should. LeCun sounds like he has a moonshot in mind, and he’s pushing for another ChatGPT-like explosion of uncanny abilities. It could take ages—or literally forever—not to mention billions of investor dollars, for anything truly remarkable to materialize. 

Gizmodo reached out to Meta for comment on how LeCun’s work fits into the company’s AI mission, and will update if we hear back.