I know that autoregressive models hallucinate, but training them on a enormous clean corpus of probably several trillions tokens and images, and the fact that GPT 4 may be two magnitude orders bigger than GPT 3 didn't change the problem. The model still hallucinates

There are tons of mistakes in these models, many are more subtle. Which is why people should stop hyping it so much. It's not just math..

Man, ChatRNN
The stars would be pouring over the repo if you named it ChatRNN. People love an antagonist, and "going back to the old days" and proving that was better

first vey is not vey ! :)

Yes, it's well-known that current language models are pretty bad at math

Pytorch 2.0 is really impressing everyday

What's wrong with Arveycavey ;).

Could someone please recommend a machine learning algorithm to create a "Flood Risk Matrix"? Additionally, any article or video tutorial on this subject that elaborates on methodology is highly desired.

in case you didn't follow https://www.reuters.com/technology/chinese-search-giant-baidu-introduces-ernie-bot-2023-03-16/ but nothing open source AFAICT.

Smh smh carbon footprint

Wow!!! This is fantastic!

I definitely hope someone proves me wrong and I wish all the people attempting the challenge the best.

>Samsung Galaxy S22 Ultra.

can you link to the samsung galaxy post? that sounds great

Interesting. However I find some merges in SD to be terrible. But I have no doubt the open source community will make something amazing

Exactly.

I'm just eager to see what fine tunes are going to be made on LLaMA now, and how model merging effects them. The combination of those two techniques has lead to some crazy advancements in the Stable Diffusion world. No idea if merging will work with LLMs as it does for diffusion models. (has anyone even tried yet?)

Everyone with their own private oracle in their hands. Pretty cool tbh

Is it possible to split the model and do inference across multiple lower VRAM GPUs or does a single card have to have the minimum 16gig VRAM?

> There are solutions that cost less than GPT-4, and they don't require integration of a black box that is gatekept by a single provider.

Management has a different perspective on costs than you and me. The way cost-benefit is analyzed in a company is whether by increasing the input cost X% can the profit then be increased by a corresponding Y% due to an increase in scale (number of contracts). They are also shit scared of the new guy on the block and losing existing business to the 100 or so startups that will come up over the next week flashing the shiny new thing in front of customers. They also don't have the same perspective on open as us, where they see black boxes as a partnership opportunity.

I'm not saying you're wrong, in fact I agree with your sentiment and it's the same as mine, and I've tried to put forth some of these arguments to my boss for why we should still be building products in-house instead of GPT-everything. What I realised is when you talk to somebody on the business side you'd get a very different response to the ironclad defense that works perfectly in your head.

First, this is not a "small" model so size DOES matter. It may not be hundreds billion parameters but it's definitely not small imo.

Second, it always has been (about data) astronaut pointing gun meme.

sure, but its main focus is to try to help many people in the academic community out there can do “pretraining phase” by themeself for fast, cheap and reproducible research experiments.

This is great! It just needs a name that's as great as the work

RWKV is a tongue twister. How about Ruckus?

I don't get this anymore if it's not the model size nor the transformer architecture then what is it?

Models were just not trained enough / not on the right data?

stolen from vitalik

>70 years is the time between the first computer and modern smart watches.

>70 years is more than the time between the first heavier-than-air flight and landing on the moon.

>70 years is 1.5x the time between the invention of public key cryptorgaphy and modern general-purpose ZK-SNARKs.

Wen paper?

Especially now that OpenAI have stopped publishing details of what goes into their black box. GPT-4 is the first time they haven't revealed details of their training architecture or dataset generation in the technical report.