Ilya Sutskever’s SSI Inc raises $1B

Open source LLMs exist and will get better. Is it just that all these companies will vie for a winner-take-all situation where the “best” model will garner the subscription? Doesn’t OpenAI make some substantial part of the revenue for all the AI space? I just don’t see it. But I don’t have VC levels of cash to bet on a 10x or 100x return so what do I know?

By selling to the “dumb(er) money” – if a Softbank / Time / Yahoo appears they can have it, if not you can always find willing buyers in an IPO.

I also don’t think there’s any way the governments of the world let real AGI stay in the hands of private industry. If it happens, governments around the world will go to war to gain control of it. SSI would be nationalized the moment AGI happened and there’s nothing A16Z could do about it.

Increasingly this just seems like fantasy to me. I suspect we will see big changes similar to the way computers changed the economy, but we will not see “capital as we know it become basically worthless” or “the modern economy and society around it collapse overnight”. Property rights will still have value. Manufacturing facilities will still have value. Social media sites will still have value.

If this is a fantasy that will not happen, we really don’t need to reason about the implications of it happening. Consider that in 1968 some people imagined that the world of 2001 would be like the film 2001: A Space Odyssey, when in reality the shuttle program was soon to wind down, with little to replace it for another 20 years.

I see it as capital becoming infinitely more valuable and labor becoming worthless, since capital can be transmuted directly into labor at that point.

Realistically what actually ends up happening imo, we get human level AGI and hit a ceiling there. Agents replace large portions of the current service economy greatly increasing automation / efficiency for companies.

People continue to live their lives, as the idea of having a human level AGI personal assistant becomes normalized and then taken for granted.

> People continue to live their lives

Presumably large numbers of those people no longer have jobs, and therefore no income.

> we get human level AGI and hit a ceiling there

Recently I’ve been wondering if our best chance for a brake on runaway non-hard-takeoff superintelligence would be that the economy would be trashed.

One million Von Neumanns working on that ‘problem’ is not something I’m looking forward to.

We would probably get the ability to generate infinite software, but a lot of stuff, like engineering would still require trial and error. Creating great art would still require inspiration gathered in the real world.

I expect it will bring about a new age of techno-feudalism – since selling intellectual labor will become impossible, only low value-add physical or mixed labor will become viable, which won’t be paid very well. People with capital will still own said capital, but you probably won’t be able to catch up to them by selling your labour, which will recreate the economic situation of the middle ages.

Another analogy I like is gold. If someone invented a way of making gold, it would bring down the price of the metal to next to nothing. In capitalist terms, it would constitute a huge destruction of value.

Same thing with AI – while human intelligence is productive, I’m pretty sure there’s a value in its scarcity – that fancy degree from a top university or any sort of acquired knowledge is somewhat valuable by the nature of its scarcity. Infinite supply would create value, and destroy it, not sure how the total would shake out.

Additionally, it would definitely suck that all the people financing their homes from their intellectual jobs would have to default on their loans, and the people whose services they employ, like construction workers, would go out of business as well.

The play here is to basically invest in all possible players who might reach AGI, because if one of them does, you just hit the infinite money hack.

And maybe with SSI you’ve saved the world too.

Kinda reminds me of the Fallout Toaster situation 🙂

I mean it doesn’t even have to be malicious, it can simply refuse to cooperate.

And we already seeing a ton of value in LLMs. There are lots of companies that are making great use of LLMs and providing a ton of value. One just launched today in fact: https://www.paradigmai.com/ (I’m an investor in that). There are many others (some of which I’ve also invested in).

I too am not rich enough to invest in the foundational models, so I do the next best thing and invest in companies that are taking advantage of the intermediate outputs.

In fact I would say that one of the things that goes to values near zero would be land if AGI exists.

I love this one for an exploration of that question: Charles Stross, Accelerando, 2005

Short answer: stratas or veins of post-AGI worlds evolve semi-independently at different paces. So that for example, human level money still makes sense among humans, even though it might be irrelevant among super-AGIs and their riders or tools. … Kinda exactly like now? Where money means different things depending where you live and in which socio-economic milieu?

Maybe it means a Star Trek utopia of post-scarcity. Maybe it will be more like Elysium or Altered Carbon, where the super rich basically have anything they want at any time and the poor are restricted from access to the post-scarcity tools.

I guess an investment in an AGI moonshot is a hedge against the second possibility?

And maybe with ASI you’ve ruined the world too.

Not a VC, but I’d assume in this case the investors are not investing in a plausible biz plan, but in a group of top talent, especially given how early stage the company is at. The $5B valuation is really the valuation of the elite team in a arguably hyped market.

The lowest hanging fruit aren’t even that pie in the sky. The LLM doesn’t need to be capable of original thought and research to be worth hundreds of billions, they just need to be smart enough to apply logic to analyze existing human text. It’s not only a lot more achievable than a super AI that can control a bunch of lab equipment and run experiments, but also fits the current paradigm of training the LLMs on large text datasets.

The US Code and Code of Federal Regulations are on the order of 100 million tokens each. Court precedent contains at least 1000x as many tokens (1), when the former are already far beyond the ability of any one human to comprehend in a lifetime. Now multiply that by every jurisdiction in the world.

An industry of semi-intelligent agents that can be trusted to do legal research and can be scaled with compute power would be worth hundreds of billions globally just based on legal and regulatory applications alone. Allowing any random employee to ask the bot “Can I legally do X?” is worth a lot of money.

(1) based on the size of the datasets I’ve downloaded from the Caselaw project.

That’s the point of venture capital; making extremely risky bets spread across a wide portfolio in the hopes of hitting the power law lottery with 1-3 winners.

Most funds will not beat the S&P 500, but again, that’s the point. Risk and reward are intrinsically linked.

In fact, due to the diversification effects of uncorrelated assets in a portfolio (see MPT), even if a fund only delivers 5% returns YoY after fees, that can be a great outcome for investors. A 5% return uncorrelated to bonds and public stocks is an extremely valuable financial product.

It’s clear that humans find LLMs valuable. What companies will end up capturing a lot of that value by delivering the most useful products is still unknown. Betting on one of the biggest names in the space is not a stupid idea (given the purpose of VC investment) until it actually proves itself to be in the real world.

Really? Selling goods online (Amazon) is not AGI. It didn’t take a huge leap to think that bookstores on the web could scale. Nobody knew if it would be Amazon to pull it off, sure, but I mean ostensibly why not? (Yes, yes hindsight being what it is…)

Apple — yeah the personal computer nobody fathomed but the immediate business use case for empowering accountants maybe should have been an easy logical next step. Probably why Microsoft scooped the makers of Excel so quickly.

Google? Organizing the world’s data and making it searchable a la the phone book and then (maybe they didn’t think of that maybe Wall Street forced them to) monetizing their platform and all the eyeballs is just an ad play scaled insanely thanks to the internet.

I dunno. I just think AGI is unlike the previous examples so many steps into the future compared to the examples that it truly seems unlikely even if the payoff is basically infinity.

However, I think because the money involved and all of these being forced upon us, one of these companies will get 1000x return. A perfect example is the Canva price hike from yesterday or any and every Google product from here on out. It’s essentially being forced upon everyone that uses internet technology and someone is going to win while everyone else loses (consumers and small businesses).

Imagine organizing the world’s data and knowledge, and integrating it seamlessly into every possible workflow.

Now you’re getting close.

But also remember, this company is not trying to produce AGI (intelligence comparable to the flexibility of human cognition), it’s trying to produce super intelligence (intelligence beyond human cognition). Imagine what that could do for your job, career, dreams, aspirations, moon shots.

I don’t think you remember the dot-com era. Loads of people thought Amazon and Pets.com were hilarious ideas. Cliff Stoll wrote a whole book on how the Internet was going to do nothing useful and we were all going to buy stuff (yes, the books too) at bricks-and-mortar, which was rapturously received and got him into _Newsweek_ (back when everyone read that).

“We’re promised instant catalog shopping — just point and click for great deals. We’ll order airline tickets over the network, make restaurant reservations and negotiate sales contracts. Stores will become obsolete. So how come my local mall does more business in an afternoon than the entire Internet handles in a month?”

If / when AGI happens can we make sure it’s not the Matrix?

I don’t understand this question. How could even average-human-level AGI not be useful in business, and profitable, a million different ways? (you know, just like humans except more so?). Let alone higher-human-level, let alone moderately-super-human level, let alone exponential level if you are among the first? (And see Charles Stross, Accelerando, 2005 for how being first is not the end of the story.)

I can see one way for “not profitable” for most applications – if computing for AGI becomes too expensive, that is, AGI-level is too compute intensive. But even then that only eliminates some applications, and leaves all the many high-potential-profit ones. Starting with plain old finance, continuing with drug development, etc.

Open source LLMs exist. Just like lots of other open source projects – which have rarely prevented commercial projects from making money. And so far they are not even trying for AGI. If anything the open source LLM becomes one of the agent in the private AGI. But presumably 1 billion buys a lot of effort that the open source LLM can’t afford.

A more interesting question is one of tradeoff. Is this the best way to invest 1 billion right now? From a returns point of view? But even this depends on how many billions you can round up and invest.

Minus the urgency, scientific process, well-defined goals, target dates, public ownership, accountability…

The urgency was faked and less true of the Manhattan Project than it is of AGI safety. There was no nuclear weapons race; once it became clear that Germany had no chance of building atomic bombs, several scientists left the MP in protest, saying it was unnecessary and dangerous. However, the race to develop AGI is very real, and we also have no way of knowing how close anyone is to reaching it.

Likewise, the target dates were pretty meaningless. There was no race, and the atomic bombs weren’t necessary to end the war with Japan either. (It can’t be said with certainty one way or the other, but there’s pretty strong evidence that their existence was not the decisive factor in surrender.)

Public ownership and accountability are also pretty odd things to say! Congress didn’t even know about the Manhattan Project. Even Truman didn’t know for a long time. Sure, it was run by employees of the government and funded by the government, but it was a secret project with far less public input than any US-based private AI companies today.

> There was no nuclear weapons race; once it became clear that Germany had no chance of building atomic bombs, several scientists left the MP in protest

You are forgetting Japan in WWII and given casualty numbers from island hopping it was going to be a absolutely huge casualty count with US troops, probably something on the order of Englands losses during WW1. Which for them sent them on a downward trajectory due to essentially an entire generation dying or being extremely traumatized. If the US did not have Nagasaki and Hiroshima we would probably not have the space program and US technical prowess post WWII, so a totally different reality than where we are today.

I’d say they were equal. We were worried about Russia getting nuclear capability once we knew Germany was out of the race. Russia was at best our frenemy. The enemy of my enemy is my friend kind of thing.

Or he will simply shift goalposts, and call some LLM superintelligent.

What evidence can you provide to back up the statement of this “significant possibility”? Human brains use neural networks…

Modern ANN architectures are not actually capable of long-term learning in the same way animals are, even stodgy old dogs that don’t learn new tricks. ANNs are not a plausible model for the brain, even if they emulate certain parts of the brain (the cerebellum, but not the cortex)

I will add that transformers are not capable of recursion, so it’s impossible for them to realistically emulate a pigeon’s brain. (you would need millions of layers that “unlink chains of thought” purely by exhaustion)

even if we bought this negative result as somehow “proving impossibility”, i’m not convinced plasticity is necessary for intelligence

huge respect for richard sutton though

More specifically: it is highly implausible that an AI system could learn to improve itself beyond human capability if it does not have long-term plasticity: how would it be able to reflect upon and extend its discoveries if it’s not able to learn new things during its operation?

(That said, I agree plasticity is key to the most powerful systems. A human race with anterograde amnesia would have long ago gone extinct.)

But there is no reason the company can’t come up with a different paradigm.

But I suppose you could say we don’t know 100% since we don’t fully understand how the brain learns.

(I did AI and Psychology at degree level, I understand there are definitely also big differences too, like hormones and biological neurones being very async)

1. Either you are correct and the neural networks humans have are exactly the same or very similar to the programs in the LLMs. Then it will be relatively easy to verify this – just scale one LLN to the human brain neuron count and supposedly it will acquire consciousness and start rapidly learning and creating on its own without prompts.

2. Or what we call neural networks in the computer programs is radically different and or insufficient to create AI.

I’m leaning to the second option, just from the very high level and rudimentary reading about current projects. Can be wrong of course. But I have yet to see any paper that refutes option 2, so it means that it is still possible.

If you wanted to reduce it down, I would say there are two possibilities:

1. Our understanding of Neurel Nets is currently sufficient to recreate intelligence, consciousness, or what have you

2. We’re lacking some understanding critical to intelligence/conciousness.

Given that with a mediocre math education and a week you could pretty completely understand all of the math that goes into these neurel nets, I really hope there’s some understand we don’t yet have

MLPs and transformers are ultimately theoretically equivalent. That means there is an MLP that represent the any function a given transformer can. However, that MLP is hard to identify and train.

Also the transformer contains MLPs as well…

1. Humans have general intelligence.
2. Human brains use biological neurons.
3. Human biological neurons give rise to human general intelligence.
4. Artificial neural networks (ANNs) are similar to human brains.
5. Therefore an ANN could give rise to artificial general intelligence.

Many people are objecting to #4 here. However in writing this out, I think #3 is suspect as well: many animals who do not have general intelligence have biologically identical neurons, and although they have clear structural differences with humans, we don’t know how that leads to general intelligence.

We could also criticize #1 as well, since human brains are pretty bad at certain things like memorization or calculation. Therefore if we built an ANN with only human capabilities it should also have those weaknesses.

Edit: actually I’m not sure if AIXItl is technically galactic or just terribly inefficient, but there’s been trouble making it faster and more compact.

In any case anyone who is completely sure that we can/can’t achieve AGI is delusional.

The fact is many things we’ve tried to develop for decades still don’t exist. Nothing is guaranteed

Basically, unless you can show humans calculating a non-Turing computable function, the notion that intelligence requires a biological system is an absolutely extraordinary claim.

If you were to argue about conscience or subjective experience or something equally woolly, you might have a stronger point, and this does not at all suggest that current-architecture LLMs will necessarily achieve it.

1. There is a chemical-level nature to intelligence which prevents other elements like silicon from being used as a substrate for intelligence

2. There is a non material aspect to intelligence that cannot be replicated except by humans

To my knowledge, there is no scientific evidence that either are true and there is already a large body of evidence that implies that intelligence happens at a higher level of abstraction than the individual chemical reactions of synapses, ie. the neural network, which does not rely on the existence of any specific chemicals in the system except in as much as they perform certain functions that seemingly could be performed by other materials. If anything, this is more like speculating that there is a way to create energy from sunlight using plants as an existence proof of the possibility of doing so. More specifically, this is a bet that an existing physical phenomenon can be replicated using a different substrate.

A very cynical take is that this is an extreme version of ‘we plan to spend all money on growth and figure out monetization later’ model that many social media companies with a burn rate of billions of $$, but no business model, have used.

i read the article but I am not sure how they know when this condition will be true.

Is this obvious to ppl reading this article? is it emperor has no clothes type situation ?

There’s plenty of players going for the same goal. R&D is wildly expensive. No guarantee they’ll reach the goal, first or even at all.

How can investment like this not transform a company’s mission into eventually paying back Billions and making Billions of dollars?

> Indeed, one should be sophisticated themselves when negotiating investment to not be unduly encumbered by the unsophisticated. But let us not get too far off topic and risk subthread detachment.

Edit: @jgalt212: Indeed, one should be sophisticated themselves when negotiating investment to not be unduly encumbered by shades of the unsophisticated or potentially folks not optimizing for aligned interests. But let us not get too far off topic and risk subthread detachment. Feel free to cut a new thread for further discussion on the subject.

Regardless, point is moot, money is money, and a16z’s money isn’t their money but other people’s money

On a serious note I would love to bet on him at this valuation. I think many others would as well. I guess if he wanted more money he would easily get it but probably he values small circle of easy to live investors instead.

Related to this, DAO’s (decentralized autonomous organizations which do not have human shareholders) are intrinsically dangerous, because they can benefit their fiduciary duty even if it involves causing all humans to die. E.g., if the machine faction in The Matrix were to exist within the framework of US laws, it would probably be a DAO.

We can’t build critical software without huge security holes and bugs (see crowdstrike) but we think we will be able to contain something smarter than us? It would only take one vulnerability.

enterprises, corps, banks, governments will want to buy “safe” AI, to push liability for mistakes on someone who proclaimed them “safe”.

I guess the “mountain” is the key. “Safe” alone is far from being a product. As for the current LLM, Id even question how valuable “safe” can be.

We’ll be able to generate most of Chat GPT4o’s capabilities locally on affordable hardware including “unsafe” and “unaligned” data as the noise-to-qubits is drastically reduced meaning smaller quantized models that can run on good enough hardware.

We’ll see a huge reduction in price and inference times within two years and whatever SSI is trained on won’t be economically viable to recoup that $1B investment guaranteed.

all depends on GPT-5’s performance. Right now Sonnet 3.5 is the best but theres nothing really ground breaking. SSI’s success will depend on how much uplift it can provide over GPT-5 which already isn’t expected to be significant leap beyond GPT4

Before Tandem, computers used to fail regularly. Tandem changed that forever (with a massive reward for their investors).

Similarly, LLMs are known to fail regularly. Until someone figures out a way for them not to hallucinate anymore. Which is exactly what Ilya is after.

I read that it cost Google ~$190 million to train Gemini, not even including staff salaries. So feels like a billion gives you about 3 “from scratch” comparable training runs.

On one hand, I think it’s great that investors are willing to throw big chunks of money at hard (or at least expensive) problems. I’m pretty sure all the investors putting money in will do just fine even if their investment goes to zero, so this feels exactly what VC funding should be doing, rather than some other common “how can we get people more digitally addicted to sell ads?” play.

On the other hand, I’m kind of baffled that we’re still talking about “AGI” in the context of LLMs. While I find LLMs to be amazing, and an incredibly useful tool (if used with a good understanding of their flaws), the more I use them, the more that it becomes clear to me that they’re not going to get us anywhere close to “general intelligence”. That is, the more I have to work around hallucinations, the more that it becomes clear that LLMs really are just “fancy autocomplete”, even if it’s really really fancy autocomplete. I see lots of errors that make sense if you understand an LLM is just a statistical model of word/token frequency, but you would expect to never see these kinds of errors in a system that had a true understanding of underlying concepts. And while I’m not in the field so I may have no right to comment, there are leaders in the field, like LeCun, who have expressed basically the same idea.

So my question is, has Sutskever et al provided any acknowledgement of how they intend to “cross the chasm” from where we are now with LLMs to a model of understanding, or has it been mainly “look what we did before, you should take a chance on us to make discontinuous breakthroughs in the future”?

That said, my personal hypothesis is that AGI will emerge from video generation models rather than text generation models. A model that takes an arbitrary real-time video input feed and must predict the next, say, 60 seconds of video would have to have a deep understanding of the universe, humanity, language, culture, physics, humor, laughter, problem solving, etc. This pushes the fidelity of both input and output far beyond anything that can be expressed in text, but also creates extraordinarily high computational barriers.

And what I’m saying is that I find that argument to be incredibly weak. I’ve seen it time and time again, and honestly at this point just feels like a “humans should be a hundred feet tall based on on their rate of change in their early years” argument.

While I’ve also been amazed at the past progress in LLMs, I don’t see any reason to expect that rate will continue in the future. What I do see the more and more I use the SOTA models is fundamental limitations in what LLMs are capable of.

I.e. the real breakthrough that allowed such rapid progress was transformers in 2017. Since that time, the vast majority of the progress has simply been to throw more data at the problem, and to make the models bigger (and to emphasize, transformers really made that scale possible in the first place). I don’t mean to denigrate this approach – if anything, OpenAI deserves tons of praise for really making that bet that spending hundreds of millions on model training would give discontinuous results.

However, there are loads of reasons to believe that “more scale” is going to give diminishing returns, and a lot of very smart people in the field have been making this argument (at least quietly). Even more specifically, there are good reasons to believe that more scale is not going to go anywhere close to solving the types of problems that have become evident in LLMs since when they have had massive scale.

So the big thing I’m questioning is that I see a sizable subset of both AI researchers (and more importantly VC types) believing that, essentially, more scale will lead to AGI. I think the smart money believes that there is something fundamentally different about how humans approach intelligence (and this difference leads to important capabilities that aren’t possible from LLMs).

I feel it is fair to say that neither of these were natural extrapolations from prior successful models directly. There is no indication we are anywhere near another nonlinearity, if we even knew how to look for that.

Blind faith in extrapolation is a finance regime, not an engineering regime. Engineers encounter nonlinearities regularly. Financiers are used to compound interest.

Video is not necessarily less information dense than text, because when considered in its entirety it contains text and language generation as special cases. Video generation includes predicting continuations of complex verbal human conversations as well as continuations of videos of text exchanges, someone flipping through notes or a book, someone taking a university exam through their perspective, etc.

On one hand, I understand what he’s saying, and that’s why I have been frustrated in the past when I’ve heard people say “it’s just fancy autocomplete” without emphasizing the awesome capabilities that can give you. While I haven’t seen this video by Sutskever before, I have seen a very similar argument by Hinton: in order to get really good at next token prediction, the model needs to “discover” the underlying rules that make that prediction possible.

All that said, I find his argument wholly unconvincing (and again, I may be waaaaay stupider than Sutskever, but there are other people much smarter than I who agree). And the reason for this is because every now and then I’ll see a particular type of hallucination where it’s pretty obvious that the LLM is confusing similar token strings even when their underlying meaning is very different. That is, the underlying “pattern matching” of LLMs becomes apparent in these situations.

As I said originally, I’m really glad VCs are pouring money into this, but I’d easily make a bet that in 5 years that LLMs will be nowhere near human-level intelligence on some tasks, especially where novel discovery is required.

He puts a lot of emphasis on the fact that ‘to generate the next token you must understand how’, when thats precisely the parlor trick that is making people lose their minds (myself included) with how effective current LLMs are. The fact that it can simulate some low-fidelity reality with _no higher-level understanding of the world_, using purely linguistic/statistical analysis, is mind-blowing. To say “all you have to do is then extrapolate” is the ultimate “draw the rest of the owl” argument.

Without some raw reasoning (maybe Neuro-symbolic is the answer maybe not) capacity, LLM won’t be enough. Reasoning is super tough because its not as easy as predicting the next most likely token.

I wouldn’t. There are some extraordinarily stupid humans out there.
Worse, making humans dumber is a proven and well-known technology.

So? One of the most frustrating parts of these discussions is that for some bizzare reason, a lot of people have a standard of reasoning (for machines) that only exists in fiction or their own imaginations.

Humans have a long list of cognitive shortcomings. We find them interesting and give them all sorts of names like cognitive dissonance or optical illusions. But we don’t currently make silly conclusions like humans don’t reason.

The general reasoning engine that makes neither mistake nor contradiction or confusion in output or process does not exist in real life whether you believe Humans are the only intelligent species on the planet or are gracious enough to extend the capability to some of our animal friends.

So the LLM confuses tokens every now and then. So what ?

> Humans have a long list of cognitive shortcomings. We find them interesting and give them all sorts of names like cognitive dissonance or optical illusions. But we don’t currently make silly conclusions like humans don’t reason.

Exactly! In fact, things like illusions are actually excellent windows into how the mind really works. Most visual illusions are a fundamental artifact of how the brain needs to turn a 2D image into a 3D, real-world model, and illusions give clues into how it does that, and how the contours of the natural world guided the evolution of the visual system (I think Steven Pinker’s “How the Mind Works” gives excellent examples of this).

So I am not at all saying that what LLMs do isn’t extremely interesting, or useful. What I am saying is that the types of errors you get give a window into how an LLM works, and these hint at some fundamental limitations at what an LLM is capable of, particularly around novel discovery and development of new ideas and theories that aren’t just “rearrangements” of existing ideas.

ANN architectures are not like brains. They don’t come pre-baked with all sorts of evolutionary steps and tweaking. They’re far more blank slate and the transformer is one of the most blank slate there is.

Mostly at best, maybe some failure mode in GPT-N gives insight to how some concept is understood by GPT-N. It rarely will say anything about language modelling or Transformers.
GPT-2 had some wildly different failure modes than 3, which itself has some wildly different failure modes to 4.

All a transformer’s training objective asks it to do is spit out a token. How it should do so is left for transformer to figure along the way and everything is fair game.

And confusing words with wildly different meanings but with some similarity in some other way is something that happens to humans as well. Transformers don’t see words or letters(but tokens). So just because it doesn’t seem to you like two tokens should be confused doesn’t mean there isn’t a valid point of confusion there.

For example, you’ll have little luck achieving AGI with decision trees no matter what’s their training objective.

Doesn’t really imply let’s just do more LLMs.

I also don’t really see AGI emerging from LLMs any time soon, but it could be argued that human intelligence is also just ‘fancy autocomplete’.

But that’s my point – in some ways it’s obvious that humans are not just doing “fancy autocomplete” because humans generally don’t make the types of hallucination errors that LLMs make. That is, the hallucination errors do make sense if you think of how an LLM is just a statistical relationship between tokens.

One thing to emphasize, I’m not saying the “understanding” that humans seem to possess isn’t just some lower level statistical process – I’m not “invoking a soul”. But I am saying it appears to be fundamentally different, and in many cases more useful, than what an LLM can do.

They do though – I’ve noticed myself and others saying things in conversation that sound kind of right, and are based on correct things they’ve learned previously, but because memory of those things is only partial and mixed with other related information things are often said that are quite incorrect or combine two topics in a way that doesn’t make sense.

I’m not. Lots of people and companies have been sinking money into these ventures and they need to keep the hype alive by framing this as being some sort of race to AGI. I am aware that the older I get the more cynical I become, but I bucket all discussions about AGI (including the very popular ‘open letters’ about AI safety and Skynet) in the context of LLMs into the ‘snake oil’ bucket.

Why Tel Aviv in Israel ?

If we say that half of innovations came from Alphabet/Google, then most of them (transformers, LLMs, tensorflow) came from Google Research and not Deep Mind.

Israel is a leading AI and software development hub in the world.

A couple years??

Nobody even knew what OpenAI was up to when they were gathering training data – they got away with a lot. Now there is precedent and people are paying more attention. Data that was previously free/open now has a clause that it can’t be used for AI training. OpenAI didn’t have to deal with any of that.

Also OpenAI used cheap labor in Africa to tag training data which was also controversial. If someone did it now it would they’d be the ones to pay. OpenAI can always say “we stopped” like Nike said with sweat shops.

A lot has changed.

Companies are willing to pay a lot for clean training data, and my bet is there will be a growing pile of training sets for sale on a non-exclusive basis as well.

A lot of this data – what I’ve seen anyway, is far cleaner than anything you’ll find on the open web, with significant data on human preferences, validation, cited sources, and in the case of e.g. coding with verification that the code runs and works correctly.

Very interesting, thanks for sharing that detail. As someone who has tinkered with tokenizing/training I quickly found out this must be the case. Some people on HN don’t know this. I’ve argued here with otherwise smart people who think there is no data preprocessing for LLMs, that they don’t need it because “vectors”, failing to realize the semantic depth and quality of embeddings depends on the quality of training data.

>SSI says it plans to partner with cloud providers and chip companies to fund its computing power needs but hasn’t yet decided which firms it will work with.

1bn in cash is crazy…. usually they get cloud compute credits (which they count as funding)

I mean, I’d like at least a brief blurb about their entire premise of safety. Maybe a definition or indication of a public consultation or… something.. otherwise the insinuation is that these three dudes are gonna sit around defining it on instinct, as if it’s not a ludicrously hard human problem.

(I think it’s branding, yes. A kind of “we don’t care about aesthetics, we care about superintelligence” message)

To me this sounds like maybe they won’t be doing transformers. But perhaps they just mean “we will have safety in mind as we scale, unlike everyone else.”

Idk about this claim.

I think if you take the multi-verse view wrt quantum mechanics + a veil of ignorance (you don’t know which entity your conciousness will be), you pretty quickly get morality.

ie: don’t build the Torment Nexus because you don’t know whether you’ll end up experincing the Torment Nexus.

“Super” intelligence typically refers to being better than humans in achieving goals, not to being better than humans in knowing good from evil.

An alternative sequence of reasoning places less emphasis on Ilya specifically and uses Ilya as an indicator of research health. Repeat (1), (2), and (3) above, but replace “Ilya” with something like “strong and healthy fundamental research group”. In this version, Ilya’s departure is taken as indication that OpenAI no longer has a strong and healthy fundamental research group but that the company is “compromised” by relentless feature roadmaps for current products and their variations. That does not mean OpenAI will fail, but in this perspective it might mean that OpenAI is not well positioned to capture future research breakthroughs and the products that they will generate.

From my perspective, it’s just about impossible to know how true these premises really are. And that’s what makes it a bet or gamble rather than anything with any degree of assurance. To me, just as likely is the scenario where it’s revealed that Ilya is highly ineffective as a generalist leader and that research without healthy tension from the business goes nowhere.

Do YOU want to miss out being a share holder on this new line that will bring immeasurable wealth ?? 😉

Simultaneously too wealthy to imagine and never wealthy enough. Capitalism is quite the drug.

In this case, everyone knows it takes hundreds of millions to train models. So I’m investors are essentially rolling the dice on an extremely well-regarded team. And if it takes about a billion just to get off the ground, the valuation would need to at least be in the couple billion range to make it worth it for employees to work there.

That feels very different than say selling a company where founders are cashing out. In that case, the business should expect to meaningful contribute to revenue, and quickly.

How is this going to ever pay the investors back? How is it going to raise more money at such an insane valuation?

I just dont see how you justify such a crazy valuation from day 1 financially.

Amazon is one of the most famous successes of the era. Bezos quit his job, launched the business out of his garage, with seed money being $10K of his own savings, and was doing $20K/week in sales just 30 days later. And I believe their only VC round before going public was an $8 investment from Kleiner Perkins. But they were a company who proved their viability early on, had a real product with rapid revenue growth before getting any VC $$.

I’d say this SSI round is more similar to Webvan, who went public with a valuation of $4.8 billion, and at that time had done a grand total of $395K in sales, with losses over $50 million.

I’m sure there are good investments out there for AI companies that are doing R&D and advancing the state of the art. However, a $1 billion investment at a $5 billion valuation, for a company with zero product or revenue, just an idea, that’s nuts IMO, and extremely similar to the type of insanity we saw during the dot com bubble. Even more so given that SSI seemingly don’t even want to be a business – direct quote from Ilya:

> This company is special in that its first product will be the safe superintelligence, and it will not do anything else up until then … It will be fully insulated from the outside pressures of having to deal with a large and complicated product and having to be stuck in a competitive rat race.

This doesn’t sound to me like someone who wants to build a business, it sounds like someone who wants to hack on AI with no oversight or proof of financial viability. Kinda wild to give him $1 billion to do that IMO.

>Agreed, the car bubble is very, very real. Not that the internal combustion carriage is all hype, it’s certainly impressive with useful applications, but car manufacturers are getting insane valuations with zero proof they’re viable businesses.

> If you are seeking capital for a startup with a product, you have to sell the startup on realities (ie how much revenue you are making). If you are seeking capital for a startup with no product, you can sell the startup on dreams, which is much much easier but also way riskier for investors.

Since these guys don’t have a product yet, they 100% sold it on big dreams combined with Ilya’s track record at OpenAI.

There’s no liquidity until they are making money.

It means that AI startups are actually a really poor value proposition compared to traditional tech companies, because your multiplier is limited. First round $50M valuation leaves a lot more opportunity to get rich.

This kind of structure isn’t as unusual for capital intensive businesses.

Using the definition from the article:

> AI safety, which refers to preventing AI from causing harm, is a hot topic amid fears that rogue AI could act against the interests of humanity or even cause human extinction.

If the purpose of a state is to ensure its continued existence, then they should be able to make >=$1 in profit.

… however, I’m on the camp that believes it’s not going to be hyper-profitable for only one (or a few) single commercial entities.

AGI will not be a product like the iPhone where one company can “own” it and milk it for as long as they want. AGI feels more like “the internet”, which will definitely create massive wealth overall but somehow distributed among millions of actors.

We’ve seen it with LLMs, they’ve been revolutionary and yet, one year after a major release, free to use “commodity” LLMs are already in the market. The future will not be Skynet controlling everything, it will be uncountable temu-tier AIs embedded into everything around you. Even @sama stated recently they’re working on “intelligence so cheap that measuring its use becomes irrelevant”.

/opinion

> It may look—on the surface—that we are just learning statistical correlations in text. But it turns out that to ‘just learn’ the statistical correlations in text, to compress them really well, what the neural network learns is some representation of the process that produced the text. This text is actually a projection of the world.

(https://www.youtube.com/watch?v=NT9sP4mAWEg – sadly the only transcripts I could find were on AI grifter websites that shouldn’t be linked to)

This is transparently false – newer LLMs appear to be great at arithmetic, but they still fail basic counting tests. Computers can memorize a bunch of symbolic times tables without the slightest bit of quantitative reasoning. Transformer networks are dramatically dumber than lizards, and multimodal LLMs based on transformers are not capable of understanding what numbers are. (And if Claude/GPT/Llama aren’t capable of understanding the concept of “three,” it is hard to believe they are capable of understanding anything.)

Sutskever is not actually as stupid as that quote suggests, and I am assuming he has since changed his mind…. but maybe not. For a long time I thought OpenAI was pathologically dishonest and didn’t consider that in many cases they aren’t “lying,” they blinded by arrogance and high on their own marketing.

This is pretty sloppy thinking.

The neural network learns some representation of a process that COULD HAVE produced the text. (this isn’t some bold assertion, it’s just the literal definition of a statistical model).

There is no guarantee it is the same as the actual process. A lot of the “bow down before machine God” crowd is guity of this same sloppy confusion.

1. An Octopus and a Raven have wildly different brains. Both are intelligent. So just the idea that there is some “one true system” that the NN must discover or converge on is suspect. Even basic arithmetic has numerous methods.

2. In the limit of training on a diverse dataset (ie as val loss continues to go down), it will converge on the process (whatever that means) or a process sufficiently robust. What gets the job done gets the job done. There is no way an increasingly competent predictor will not learn representations of the concepts in text, whether that looks like how humans do it or not.

No amount of training will cause a transformer to magically sprout feedback paths or internal memory, or an ability to alter it’s own weights, etc.

Architecture matters. The best you can hope for an LLM is that training will converge on the best LLM generating process it can be, which can be great for in-distribution prediction, but lousy for novel reasoning tasks beyond the capability of the architecture.

Go back a few evolutionary steps and sure you can. Most ANN architectures basically have relatively little to no biases baked in and the Transformer might be the most blank slate we’ve built yet.

>No amount of training will cause a transformer to magically sprout feedback paths or internal memory, or an ability to alter it’s own weights, etc.

A transformer can perform any computation it likes in a forward pass and you can arbitrarily increase inference compute time with the token length. Feedback paths? Sure. Compute inefficient? Perhaps. Some extra programming around the Model to facilitate this ? Maybe but the architecture certainly isn’t stopping you.

Even if it couldn’t, limited =/ trivial. The Human Brain is not Turing complete.

Internal Memory ?
Did you miss the memo ? Recurrency is overrated. Attention is all you need.

That said, there are already state keeping language model architectures around.

Altering weights ?
Can a transformer continuously train ? Sure. It’s not really compute efficient but architecture certainly doesn’t prohibit it.

>Architecture matters

Compute Efficiency? Sure. What it is capable of learning? Not so much

No it can’t.

A transformer has a fixed number of layers – call it N. It performs N sequential steps of computation to derive it’s output.

If a computation requires > N steps, then a transformer most certainly can not perform it in a forward pass.

FYI, “attention is all you need” has the implicit context of “if all you want to build is a language model”. Attention is not all you need if what you actually want to build is a cognitive architecture.

https://arxiv.org/abs/2310.02226

And again, human brains are clearly limited in the number of steps it can compute without writing something down.
Limited =/ Trivial

>FYI, “attention is all you need” has the implicit context of “if all you want to build is a language model”.

Great. Do you know what a “language model” is capable of in the limit ? No

These top research labs aren’t only working on Transformers as they currently exist but it doesn’t make much sense to abandon a golden goose before it has hit a wall.

No – there is a loop between the cortex and thalamus, feeding the outputs of the cortex back in as inputs. Our brain can iterate for as long as it likes before initiating any motor output, if any, such as writing something down.

In practice, the cortex-thalamus loop allows for some degree of internal iteration, but the brain cannot endlessly iterate without some form of external aid (e.g., writing something down) to offload information and prevent cognitive overload.

I’m not telling you anything here you don’t experience in your everyday life. Try indefinitely iterating on any computation you like and see how well that works for you.

The input sequence is processed in parallel, regardless of length, so number of tokens has no impact on number of sequential compute steps which is always N=layers.

> Do you know what a “language model” is capable of in the limit ?

Well, yeah, if the language model is an N-layer transformer …

Then increase N (N is almost always increased when a model is scaled up) and train or write things down and continue.

A limitless iteration machine (without external aid) is currently an idea of fiction. Brains can’t do it so I’m not particularly worried if machines can’t either.

I think it’s fair to call one process that can imitate a more complex one a representation of that process. Especially when in the very next sentence he describes it as a “projection”, which has the mathematical sense of a representation that loses some dimensions.

  5 x 3 = 15

without learning

  *****    ****     *******
  ***** =  *****  = *******
  *****    ******   *

And this generalizes to almost every sentence an LLM can regurgitate.

Here’s GPT-4 Turbo in April botching a test almost all preschoolers could solve easily: https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_pr…

I have not used LLMs since 2023, when GPT-4 routinely failed almost every counting problem I could think of. I am sure the performance has improved since then, though “write an essay with 250 words” still seems unsolved.

The real problem is that LLM providers have to play a stupid game of whack-a-mole where an enormous number of trivial variations on a counting problem need to be specifically taught to the system. If the system was capable of true quantitative reasoning that wouldn’t be necessary for basic problems.

There is also a deception is that “chain of thought” prompting makes LLMs much better at counting. But that’s cheating: if the LLM had quantitative reasoning it wouldn’t need a human to indicate which problems were amenable to step-by-step thinking. (And this only works for O(n) counting problems, like “count the number of words in the sentence.” CoT prompting fails to solve O(nm) counting problems like “count the number of words in this sentence which contain the letter ‘e'” For this you need a more specific prompt, like “First, go step-by-step and select the words which contain ‘e.’ Then go step-by-step to count the selected words.” It is worth emphasizing over and over that rats are not nearly this stupid, they can combine tasks to solve complex problems without a human holding their hand.)

I don’t know what you mean by “10 years ago” other than a desire to make an ad hominem attack about me being “stuck.” My point is that these “capabilities” don’t include “understands what a number is in the same way that rats and toddlers understand what numbers are.” I suspect that level of AI is decades away.

Beyond that LLMs don’t see words or letters (tokens are neither) so some counting issues are expected.

But it’s not very surprising you’ve been giving tests that make no sense.

But the company name specifically says “superintelligence”

The company isn’t named “as smart as the average redditor, Inc”

I get the impression that they really do believe scale is all you need, other than perhaps some post-training changes to encourage longer horizon reasoning.
Maybe Ilya is in this camp, although frankly it does seem a bit naive to discount all the architectural and operational shortcomings of pre-trained Transformers, or assume they can be mitigated by wrapping the base LLM in an agent that provides what’s missing.

I suspect there’s a big corporate market for LLMs with very predictable behaviour in terms of what the LLM knows from its training data, vs what it knows from RAG or its context window.

If you’re making a chatbot for Hertz Car Hire, you want it to answer based on Hertz policy documents, even if the training data contained policy documents for Avis and Enterprise and Budget and Thrifty car hire.

Avoiding incorrect answers and hallucinations (when appropriate) is a type of AI safety.

The PayPal mafia includes Elon Musk, Peter Thiel, etc. They now parlayed that capital into more platforms and can easily arrange investments. Heck Peter Thiel even works with governments (Palantir) and got J D Vance on Trump’s ticket, while Elon might be in his admin.

Kolomoisky got Zelensky elected in Ukraine, by launching a show about an unlikely guy who wins the presidency and named the party after the show. They call them oligarchs over there but it’s same thing.

The first guy to 1 million followers on Twitter was Ashton Kutcher. He had already starred in sitcoms and movies for years.

This idea that you can just get huge audiences and investments due to raw talent, keeps a lot of people coming to Hollywood and Silicon Valley to “make it” and living on ramen. But even just coming there proves the point — a talented rando elsewhere in the world wouldn’t even have access to the capital and big boys networks.

They all even banked at the same bank! It’s all extremely centralized: https://community.intercoin.app/t/in-defense-of-decentralize…

Clearly there are reasons why opportunities are gated.

> This idea that you can just get huge audiences and investments due to raw talent, keeps a lot of people coming to Hollywood and Silicon Valley to “make it” and living on ramen. But even just coming there proves the point — a talented rando elsewhere in the world wouldn’t even have access to the capital and big boys networks.

All those people start somewhere though. Excluding nepotism, which is tangential point, all those people started somewhere and then grew through execution and further opening of opportunity. But it’s not like they all got to where they are in one-shot. Taking your Ashton Kutcher example – yes he had a head start on twitter followers, but that’s because he executed for years before on his career. Why would it make sense for some rando to rack up a million followers before he did?

Talent will earn you opportunities, but it’s not going to open the highest door until you’ve put in the time and work.

Of course, it’s not to say inequity or unequal access to opportunities doesn’t exist in the world. Of course it does. But even in an ideal, perfectly equitable world, not everyone would have the same access to opportunities.

So yes, it makes perfect sense that someone would give Ilya $1B instead of some smart 18 year old, even if that 18 year old was Ilya from the past.

The analogy would be if the private could become a major overnight because they knew a guy.

What we see with ilya is not dissimilar. I don’t see why it’s bad that people are more hesitant to give a talented 18 year old $1B than the guy who’s been at the forefront of AI innovation.

And sometimes not even necessary. Paris Hilton got a music distribution deal overnight cause of her dad’s capital!

It’s not personally my goal to amass immense wealth and start giant companies (I would rather work minimally and live hedonically) but I am impressed by those that do so.

Overwhelmingly talent isnt sufficient. For most startups, the old boys network gets to choose who gets millions. And the next rounds a few people choose who will get billions.

Ilya proved himself as a leader, scientist, and engineer over the past decade with OpenAI for creating break-through after break-through that no one else had.

He’s raised enough to compete at the level of Grok, Claude, et al.

He’s offering investors a pure play AGI investment, possibly one of the only organizations available to do so.

Who else would you give $1B to pursue that?

That’s how investors think. There are macro trends, ambitious possibilities on the through line, and the rare people who might actually deliver.

A $5B valuation is standard dilation, no crazy ZIRP style round here.

If you haven’t seen investing at this scale in person it’s hard to appreciate that capital allocation just happens with a certain number of zeros behind it & some people specialize in making the 9 zero decisions.

Yes, it’s predicated on his company being worth more than $500B at some point 10 years down the line.

If they build AGI, that is a very cheap valuation.

Think how ubiquitous Siri, Alexa, chatGPT are and how terrible/not useful/wrong they’ve been.

There’s not a significant amount of demand or distribution risk here. Building the infrastructure to use smarter AI is the tech world’s obsession globally.

If AGI works, in any capacity or at any level, it will have a lot of big customers.

AGI assumes exponential, preferably infinite and continuous improvement, something unseen before in business or nature.

Neither siri nor Alexa were sold as AGI and neither alone come close to a $1B product. gpt and other LLMs has quickly become a commodity, with AI companies racing to the bottom for inference costs.

I don’t really see the plan, product wise.

Moreover you say:
> Ilya proved himself as a leader, scientist, and engineer over the past decade with OpenAI for creating break-through after break-through that no one else had.

Which is absolutely true, but that doesn’t imply more breakthroughs are just around the corner, nor does the current technology suggest AGI is coming.

VCs are willing to take a $1B bet on exponential growth with a 500B upside.

Us regular folk see that and are dumbfounded because AI is obviously not going to improve exponentially forever (literally nothing in the observed universe does) and you can already see the logarithmic improvement curve.
That’s where the dismissive attitude comes from.

There are many things on earth that don’t exist anywhere else in the universe (as far as we know). Life is one of them. Just think how unfathomably complex human brains are compared to what’s out there in space.

Just because something doesn’t exist anywhere in the universe doesn’t mean that humans can’t create it (or humans can’t create a machine that creates something that doesn’t exist anywhere else) even if it might seem unimaginably complex.

Obviously, there are plenty of investors who don’t fall into this situation. But lets not pretend that just because someone has a lot of money or invests a lot of money that it means they know what they are doing.

They also have the warchest to afford a $1B gamble.

If the math worked out for me too, I’d probably invest even if I didn’t personally believe in it.

Also investors aren’t super geniuses, they’re just people.

I mean look at SoftBank and Adam Neuman… investors can get swept up in hype and swindled too.

This is wrong. The models may end up cheaply available or even free. The business cost will be in hosting and integration.

the whole LLM race seems deaccelerate, and all the hard problems about LLMs seems not do have had that much progress the last couple of years (?)

In my naaive view I think a guy like David Silver the creator/co-lead of Alpha-Zero deserves more praise, atleast as a leader/scientist.
He even have lectures about Deep RL after doing AlphaGo: https://www.davidsilver.uk/teaching/

He has no LinkedIn and came straight from the game-dev industry before learning about RL.

I would put my money on him.

Even assuming the public breakthroughs are the only ones that happened, the fact that openai was able to make an llm pipeline from data to training to production at their scale before anyone else is a feat of research and engineering (and loads of cash)

One of the smartest computer science researchers is taking a stab at the most important problem of our lifetimes, we should be cheering him on.

That doesn’t sound like the hype is dissipating to me.

That’s the theoretical basis and path for achieving AGI (if it’s even possible). I’m tired of all the “we stick enough data in the magic black box blender and ta-da! AGI!”

Every giant technological break-through throughout history has had a massive underpinning of understanding before ever achieving it. And yet, with the AI bubble somehow we’re just about to secretly achieve it, but we can’t tell you how.

I feel like this could be a path to GI without “truly” understanding the human brain: make a large enough simulation of the brain and turn it on. I actually do think we understand enough about the nuts and bolts of neuron interaction to achieve this. What we don’t understand is where neurons firing turns into consciousness. It seems like it is probably just an emergent property of a complex enough neuron graph.

This doesn’t make any sense.