Is there another torrent? I cant seem to download it

I’d buy used anyways, kinda a hot take but I refuse to give NVIDIA money :P

I’m thinking of going with 3090 for 900$ or 3080 for 650$ (I can get FE for 750$ which would be pretty epic)

Got any advice? I don’t like that the 3080 only has 10GB vram. But 3080 is already pretty much overkill for anything I’d use it for other than deep learning. Kinda on the fence here tbh

Consider used if you really want to maximize what you can do on a budget. It's very likely going to give you identical performance to new and you can get a used 3090 for the cost of a new 3070 ti and open so many doors for what you can do memory-wise.

the hero we needed

Thanks, god bless

You should be able to do your trade off analysis after reading this:

https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/

I am worried of getting a mined card that might die the next day

2.9x tensor cores , 2.8x cuda cores.

I used mix precision training, should have been largely fp16. But you can input as float32. Pytorch amp will auto cast to fp16. I only get 2x speed more with 3090.

[deleted]

yeah, got it.. thank you!

This is when they were running individually on full 16x PCIE 4.0, can be expected with TFLOPS (3x) as well. (i.e. I have compared times when I had only 3060 vs 3090 on the same slot, running model on a single GPU each time)

I don't do much training on 3060 now, just connected to monitors etc.

I have changed the batch sizes to suit 24 GB anyway as I am working with CV data. Could be bit different with other types of models.

3060 = FP32 (float) 12.74 TFLOPS (https://www.techpowerup.com/gpu-specs/geforce-rtx-3060.c3682)
3090 = FP32 (float) 35.58 TFLOPS (https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622)

I must say 3060 is a wonderful card and helped me a lot until I found this ex-mining 3090. Really worth for the price with 12 GB VRAM.

Go to the used market. Buying brand new cards will be more expensive because those are scalpers. Second hand is much more reasonable.

Only 2×more than 3060. May be you are more power limited or CPU bottle necked when using both GPUs, or PCEi bandwidth limited.

You need to buy credit now to use colab.

Colab is not free anymore except for very short time.

Buy 3060 12GB. 3070 8GB vram has more limitations. Colab is largely not free now. It is fine you are willing to pay for the service. You can also use vast.ai and lambda labs for cloud GPU.

in my country 4070 ti is like 1000$ and 3090 is 1600-1700$ i can’t afford that expensive GPU

You might get a better deal by getting and rtx 3090. Double the VRAM for around the same price. It’s way to easy to hit the VRAM limit on a gpu.

I will be buying a 4070 Ti this summer, and wanted to use Colab till then. But no luck getting a GPU

I suggest to get the 3070 (or similar) for prototyping/testing your ideas. You can reduce VRAM usage by scaling down your data or training with smaller batch size etc. to see if your concept makes sense.

At some point you may run into VRAM or other hardware limitation issues. (you can't train with larger images for example). If that happens you can run training on colab or some other high performance hardware offer.

Develop on your PC first then just use it for a train job. If you have to use it longer than the timeout, the just buy a cloud instance. I have been using Colab for years and never got those issues. Use what it’s meant for and you won’t run into problems.

For the first question , we are moving towards not having to design such pipelines , ideally we will have a library which will do the model sharding or parallel computation for us. Look at parallelformers which worked for some big models(11B) i tried. Why i think this is going to happen is , 3 years back distributed training used to be a big black box, horovod, pytorch distributed training and TPUs are the only solution but right now no one designs such peipelines anymore ,everyone uses deepspeed. It has implementations of all known techniques(zero , cpu offloading etc). So if you are not one of these computation/data engineers , i suggest to watch out for such libraries.

Actually runpod looks like a better alternative than colab for cloud GPU rentals.

3070 should be much much faster than Colab and you have the added bonus of working with full debugging capabilities (PyCharm/Spyder...etc)

Even my 2nd hand 3050 is much faster than Colab...but it is always helpful to have a 2nd machine...so 3070 AND Colab