Get a power supply and 3090. Cheaper than keeping up. Making one now.

Whenever I try to connect to a GPU on Colab, I get a GPU limit message. This has been the case for the last 3 weeks

I was able to run the 7B on a v100 on lambda labs. Didnt try the other ones.

OP's use case though is just looking for a cheap gpu to dabble into. If you have the money for the 3090 then go ahead. However the cloud and Colab is a lot cheaper at the moment until Google decides to screw everyone over in the future.

I have dual 3090 + 3060 setup running on 850 W PSU. 3090 is about 4x speed of 3060

power supplies are quite cheap now, probably around $125 for 850 W.

Thanks to you, I found that the RTX 3060 has 12GB VRAM. I'll consider it.

I suggest using Colab free. The resources are more than most people need and use the cloud when you got a serious work load like a business or research.

If you want to do gaming with that then try the rtx 3060 instead. More VRAM let’s you do more than rtx 3070 ironically.

Either paths will eventually lead you to the cloud to be remotely competitive in serious workloads.

3060 is better. The vram let’s you get more stuff done

Thank you for your kind comments. I'm seriously considering buying the RTX3070, and I'll probably keep it within this week.

Thank you for your kind comments.

I can find it, but installing the RTX3090 requires changing the power supply as well, so I'm trying to compromise with the 3070.

can you find an used RTX3090 ?

Colab became expensive since they introduced compute units. I was using pro+ and running code for 2-3 hours a day. They give you 500 units per month and a premium gpu that costs 13-15 units per hour. I’d rather get a 3070 and run all I want.

Unless everything you want to try out fits in 3070 memory(8gb), if recommend colab.

For inference?yes. Look at eleutherai transformer math page. Also others are trying out llama rn so check them out

https://twitter.com/ericjang11/status/1627818245406461952?s=20

I really appreciate this response.

I'm not planning on using any of our data or touching the infrastructure yet, but for some reason I never considered using the cloud to determine hardware configuration.

Thanks again. Exactly what I needed!

Thanks for the response. Do you recall where you read the "only 200 people" bit? I'll take a look around for it as well; seems like the context could have found itself surrounded by interesting conversation.

P2P is not so much of a limitation so long as you can fit the entire model / pipeline into a single cards VRAM though, correct?

So for example, if you have a 7B Param model at FP16 and its around 14GB, presumably you should be safe with 24GB VRAM?

Thanks again for your time.

Take a step back:

• Start on a cloud -- renting GPUs or TPUs -- with nonsensitive data.

I know you said "but bottom line the data running through our platform is all back-office, highly sensitive business information, and many have agreements explicitly restricting the movement of data to or from any cloud services".

You shouldn't be touching such information during development anyway.

Make or find a non-sensitive dataset of similar scale for development.

Don't buy hardware up front until you have almost the entire data pipeline working well on rented servers. Rent them hourly on any of the big cloud platforms, and you'll quickly be able to quantify most of your hardware requirements. How much RAM you need in GPUs/TPUs. How much RAM you need on CPUs. How fast a storage layer you'll need.

Only after you have an at-scale dev/qa environment working on a cloud, will you have any idea what physical hardware you'd want to buy.

What kind of statistical models?

Forget nn for regression. I would consider statistical models for it .

Flan t5 xl is a very good transformer finetuned on instructions. T5 is also good. You might want to tweak around some generation parameters to get good results.

Padding has zero attention hence you don't lose context.

The first question is kinda difficult. Deep speed, zero, Megatron all play into it. There's a reason somebody recently said that there are only 200 people on the world atm that can pull it off.

For the second question ,

4090s just won't cut it. Nvidia fused off P2P this generation so unless you have an embarrassingly parallel pipeline ( which current llms aren't) they are not useful. Problem is ada a6000 was restricted severely P2P wise.

If you're doing llms at billion scale you gotta get v,a,h100s