People going on about sentient machines, making humans extinct... nah. This is the real nightmare, and it is more than plausible.

The first people accessing an ASI will be tempted in a way no human has been tempted before. Not using that power to alleviate human suffering may be less dangerous than using it as you see fit, regardless of your intentions. Power corrupts, and this will be a power like no other.

I agree, but I can't imagine how those events will play out. Even in the most optimistic scenarios, millions of people will face a miserable experience, regardless of their willingness to find jobs, re-train, and so on.

What the other guy said, try to cash in. I guess the only safe jobs for a while are those where the human touch is needed. But yes we are fucked.

Mandatory chart to explain the psych effect of linear vs exp

Hey I'm very interested. Do you mean the epilogue which is at the end? I can't find a prologue.

edit: wait, it's the "prelude" right?

I agree some things may be physically impossible, but there are no physical impossibilities for fusion, nowadays it's "only" an engineering problem, we don't even need new materials. Quantum computing I don't know, but you get the gist of the idea.

Last week I'd have said 15 years but have you seen deepmind's Sparrow? My god...

Trying to think in current and future exponential progress, which is very hard, I'm now closer to 10 years. The amount of groundbreaking discoveries that are waiting by only using the information we already have... Yeah.

I don't think it'd be released, knowledge is power, and we are talking about super-human knowledge. Maybe an open source version, but I feel we'll be close to extinction or in some other unimaginable crazy state by that point. Wouldn't Google or whoever has it use it to solve all their engineering challenges, file 50.000 amazing patents, and sell all the magic-like products they can now make to become even richer?

I can't say I agree. Sure, physical construction is kinda slow, but that's just one "outlet" of the wealth of information ASI will bring.

Right now we depend on some human geniuses to bring us intelligent, movie-like robots (which have a software problem, not a hardware one). Imagine having 100 super-human super-geniouses that need 5 minutes to figure out how to do anything. And that's far from only one field that could instantly benefit from the right knowledge

We can manufacture virtually any compound, we just don't know what compounds would magically cure cancer, spinal cord injuries, aging, etc and what compounds would kill you. Formula for antibiotics that can't be beaten? What genes to modify to eradicate all diseases? Imagine if we could have any software imaginable? How to build dream-like VR goggles?

Even in your example, if a company knew for certain how to build a fusion reactor that magically works, you don't think they'd raise 10B$ in a month and hire 20.000 wokers to build it in 6 months?

Yes! Exactly, the plot was going so well... and they stopped it all with a mortar and a small force of old mercenaries led by a lost girl...?

It is a great movie indeed, but it has this sci-fi flavor where the machine has agency, conciousness etc like an individual. I don't think that's the AGIs we'll see (at least first).

I believe the first AGI/ASI moels will behave like Deepmind's Sparrow. You ask questions and get straightforward answers. I would ask for so many scientific miracles...

Observation is done with your senses and brain, not with you mouth or fingers. It was all a judgement, and only that, without questions or counterarguments, nothing. Now you know the difference.

This is exactly the god of the gaps thing.

LaMDA engineer enters the chat

Then how about you suspend your judgement until you know about the thing you are talking about? It's an essential tenet of science.

Can you imagine how the massive shitstorm will break when these models actually become widely used?
I'm really excited about the potential for intelligent chatbots to effectively end loneliness for good, but... It's going to get nuts.

Imagine Q on steroids at an industrial scale.

Sometimes I wonder if I should even bother trying to make those points; it seems pointless.

>AI creation would do on its very first day of sentience

Sentience, intelligence, agency, etc. are all different things. One doesn't automatically give you the other.