Or everything just stays mostly normal. Don’t lose your mind over this until it happens at your place of work

For the specific tasks you mentionned, I doubt we'd have a LLM beating human experts or even anyone that knows a little about the topic. LLMs are not good enough for that kind of touchy, precise stuff yet !

Youtube (Minecraft x Copilot Demo): https://www.youtube.com/watch?v=6bxfGPmL2BQ

Forbes article: https://www.forbes.com/sites/paultassi/2023/02/17/microsoft-is-testing-an-ai-powered-minecraft-where-you-tell-it-what-to-build/?sh=6c3e06ef2aa7

They are actively working on various projects in AI. Just one example: Recently two different teams flew an F-16 autonomously in lots of combat scenarios. They beat the humans mostly because of precision, and lack of self-preservation. The human pilots said the computers were "too aggressive".

But for LLMs like Bing and ChatGPT, yeah that would be interesting and powerful like you say. The military moves slowly in terms of budgets and projects, but I'm sure they have at least a small team on it dreaming and investigating.

10 years is about right to go from “this will take $100,000 a day to run” to “this can run on my machine”.

Microsoft is similar to Google; both like to experiment and make cool stuff, but Microsoft doesn't cut the fat and likes to put out products which are effectively trash under the guise of open beta. Heck, even their hardware is sometimes like that, while Google's products are typically solid, even if they have a short lifespan.

Going back to New Bing, it's genuinely innovative. It just sucks. That's not paradoxical, because a lot of new stuff does suck. We just rarely see it, because companies like Google are generally disciplined enough.

Most "deep" innovations are developed over decades. That development could be secretive (military tech), or open (SpaceX, Tesla), but it takes time nonetheless. Microsoft leans towards the latter, Google the former.

The latter is generally more efficient, if your audience is results-focused, not emotions-focused. AI is pretty emotionally charged, so maybe the former method is better.

Dig it. I have a similar background and have had conversations with interns at ai firms like Palantir that have been doing the shit you described for years. I agree. It’s too expensive to train ai’s for every specific use case. That’s what I meant by “general”.

I think the most fascinating part of this current trend is seeing the general populations reaction to these tools being publicly released. And that’s what’s at the heart of my question…if the tech is unreliable, expensive, and generally not scalable …why is MS doing this?

I mean obviously they are generating data on user interactions to retrain the model but I can’t imagine that being the silver bullet.

Google implemented plenty of ai tech in their search engine but nobody raises an eyebrow, but now all this? I’m rambling at this point but it’s just not adding up in my brain ¯_(ツ)_/¯

Classical computing / engineering advances are good at repetitive actions. A human can never put in a screw 10,000x times with 0.01mm precision or calculate 5000 graphs by hand without quitting. But it's bad at actions that require flexibility and adaptation, like what chefs, dry cleaners, or software engineers do.

LLM and AI attempt to bridge that gap, by allowing for computers to be flexible and adapt. The issue is that we don't know how much they're actually capable of adapting, and how fast. We know humans have a limit; nobody in the world fluently speaks & reads & writes in more than 10 languages (probably not even >5). Do computers have a limit? How expensive is that limit? Because materials, manufacturing, and energy are finite resources.

What do you define as general use cases? Receptionist calls? (already done, one actually fooled me into thinking it was a human) Making a cup of coffee?

Anything repetitive will be automated, if it's economical to do so. You probably still make tea by hand, because it's a waste of money to buy a $100 tea maker (and they probably dont even exist because of how easy it is to make tea). But you probably have a blender, because it's a huge waste of time and energy to chop stuff yourself.

I think humans (on this subreddit especially) tend to underestimate how much finances & logistics play into tech. We've had flying cars since the 90s, yet they'll never "transform transportation" like sci-fi said, because it's dumb to have a car-plane hybrid.

We might get an impressive AGI in the next few years, but it might be so expensive that it's just used the same way we use robots: you get the cutting-edge stuff you'll never see cause it's in some factory, the entertaining stuff like the cruise ship robo-bartenders, and the consumer-grade crap like Roombas. AGI might also kill millions of humans but I know nothing about that side of AI so I won't comment.

Btw, I'm not an expert, I'm just a software engineer that likes talking to AI engineers.

Because they don't host any pornographic content in their servers, just links. That gives responsibility to the user.

Isn't bing somewhat known as the porn search engine? If they cared that much why wouldn't they strip that functionality? Why even offer to turn safe search off?

I see. Even at a 5x reduction in parameter size, that's still not enough to run on consumer hardware (we're talking 10b vs. 500m) , but I recognize what you're trying to say.

Maybe the AI has crossed the singularity into the 5th dimension of alternate parallel universes and these hallucinations are actual realities elsewhere -- possible scenarios that almost occurred here, but didn't. 🤷‍♀️

It's just going to make it mad. And then we'll really be in trouble.

isnt this kinda similar to Li-Fi? what a powerful combination it would be to combine the two

chatgpt says it wrote the above. While this can be considered technically possible, in reality there are many things that have to be figured out, like processing power needs homogeneity in p2p and distributed training systems, that means different gpus/cpus and pc configurations won't work well if at all together, a DAO cannot do that yet, how about their training sets? How about latency and bandwidth? With current blockchain speeds and confirmations that would take centuries. We are not there yet, when we figure out how to P2P train an LLM we are going to do it without the use of a blockchain. This looks to me as an overambitious project by ill-informed people or a scam.

I meant that IMHO, gpt3 level LLMs will have fewer parameters in the future.

⭐️ Refreshing to see someone who knows their shit on this sub. Where do you see this tech going for general use cases? Everything I read tells me it just isn’t ready. What is MS’s endgame for implementing all this?

Ten years ago none of this existed Ten years for efficiencies to improve to consumer level seems out of step with agreed upon tech progressions.

Thanks for the idea!

Which part? LLM-capable hardware getting really really cheap, or useful LLMs not growing hugely in parameter size?

The thing is that in LLM training, memory and IO bandwidth are the big bottlenecks. If every GPU has to communicate via the internet, and wait for the previous person to be done first (because pipelined model parallel is still sequential, despite the name), it's gonna finish in like 100 years. Another slowdown is breaking up each layer into pieces that individual GPUs can handle. Currently they're being spread out to 2000-3000 huge GPUs and there's already significant latency. What happens if there's 20,000 small-sized GPUs? Each layer is gonna be spread out so thin the latency is gonna be enormous. The final nail in the coffin is that neural network architecture changes a lot, and each time the hardware has to be reconfigured too.

Crypto mining didn't have these problems because 1. bandwidth was important, but not the big bottleneck, 2. "layers" could fit on single GPUs, and if they couldn't (on a 1050ti for example), it was very slow, and 3. the architecture didn't really change, you just did the same thing over and over.

Cerebras is trying to make a huge chip that disaggregates memory from compute, and also bundles compute into a single chip, saving energy and time. The cost for the CS-2 system is around $3-10 million for the hardware alone. It's pretty easy for a medium-sized startup to offer some custom LLM. I mean there's already dozens, if not hundreds of startups starting to do that right now. It's expensive. All complex computing is expensive, we can't really get around that, we can only slowly make improvements.

Yes, you must think long and hard about what you've done.

I disagree

fuck man i'm so worried about the incoming AI sentience movement from all you dumbasses anthromoprhizing language models

LLMs will not be getting smaller. Getting better ≠ getting smaller.

Now, will really small models be run on some RTX 6090 ti in the future? Probably. Think GPT-2. But none of the actually useful models (X-Large, XXL, 10XL, etc) will be accessible at home.