Right. And even once you have enough VRAM, memory bandwidth limits the speed more than tensor core bandwidth.

They could pack more tensor cores in there if they wanted to, they just wouldn't be able to fill them with data fast enough.

"what does a cow drink?" "Milk"

VRAM is the limiting factor to run these things though, not tensor cores

In theory, Open Assistant should at least match the best corporate models if enough people start accessing the language project and contribute at least a little bit each week to creating prompts, sorting prompts, etc.

In theory, if 10,000 people do this work every month, that's a much greater number of people than any AI team in large corporations. The issue is the quality of the work.

Not expecting a magic bullet solution. Been in the field long enough to know that.

However, any written record of the intelligent ways you mentioned are valuable and worth going through.

One of the reasons it gets asked a lot is because image quality analysis doesn't seem to get enough air time. There are only few papers and sone as old as 2016. They font reflect the trend since 'all you need is attention '

Idk, sounds like a cute diminutive this way ,😂😂

That or just use the cloud until Nvidia releases a 48gb gpu (which will happen sooner than one would think. Games are getting limited by VRAM)

Just price. They have the same amount of VRAM. The 4090 is faster of course.

Any reason, beside price, to buy 3090s instead of 4090s?

“How many months are there in the year?” “There are 365 days in the year”

Got a ways to go I guess, nice to see this stuff moving tho, I remember writing my first chat bot in 1999 and even the worst of the current models are brilliant in comparison

It's mentioned as open source. So it means I can get the model weights and run it locally if I want to right?

Tom's Hardware has an article on that: https://www.tomshardware.com/news/running-your-own-chatbot-on-a-single-gpu

LoRA is how you train llama into alpaca on consumer hardware

Yeah, I wouldn't buy AMD either. It's a shame that NVIDIA is basically a monopoly in a AI, but it is what it is.

yea no problem, happy to chat more if you are doing research in this space. you can always reach out to me through email

I'm hoping that non-Vonn-Neumann chips will scale up in the next few years. There's some you can buy today but they're small:

>NDP200 is designed natively run deep neural networks (DNN) on a variety of architectures, such as CNN, RNN, and fully connected networks, and it performs vision processing with highly accurate inference at under 1mW.

>Up to 896k neural parameters in 8bit mode, 1.6M parameters in 4bit mode, and 7M+ In 1bit mode

An arduino idles at about 10mw, for comparison.

The idea is that if you're not shuffling the entire network weights across the memory bus every inference cycle, you save ludicrous amounts of time and energy. Someday, we'll use this kind of tech to run LLMs on our phones.

Those are actually super compelling problems. I'll keep an eye out. Again thank you, you contribute so much.

actually, i'm keeping an eye on Hyena! there are however a number of issues i still have with the paper (i'm not going to play reviewer 2, as it is not my place nor is reddit a good forum for that), but i intend to reserve judgement and try it out on few difficult problems like genomics and EEG later this year. proof is in the pudding.

hello, I've recently run the alpaca.cpp on my laptop, but I want to give it a context window so that it can remember conversations, and make it voice activated using python. Can someone guide me on this?

Better light a candle rather than buy an AMD GC for anything close to cutting edge.

What are the input length of the Llama model? Can't find it anywhere.

Honestly, they already cost more than I can afford to spend on a side project.

I'm just gonna have to wait and hope that AMD gets their act together on AI support.

I keep seeing this getting asked like people are expecting a magic bullet solution.

​

In general you can only get out something within the realm of what you put in.

There are intelligent ways to structure training and models, but you can't fill in expected gaps without training with a reference or a close approximation of what those gaps are.

My best suggestion is to limit your input data or muxed model to specific high resolution subsets.

ex. You can train a LoRa on a small focused subset of data.

Since you're the OG with this. Can I pick your brain? You don't see value in hyena hierachrcy. Inference with 64k context window but 100x more efficient than flash attention. I notice on github, you plan on implementing flash attention on all your transformer based models? HH perplexity actually scales with parameter count scaling. Thoughts?

Data is really hard to get because of privacy regulations too.

There are millions of brain MRI scans sitting in hospital databases but nobody can use them without individually asking each patient. Most published datasets are only a couple dozen scans, and plenty are N=1.