"They also have the tendency to hallucinate frequently unless parameters are made more restrictive."

I am not really understand this point in term of technical

Probably good to display some sort of prominent * TherapistGPT and it’s creators do not practice medicine, TherapistGPT is not an alternative for actual Therapeutic care administered by professionals, etc

[removed]

Not sure I understand. Is it better? Depends on what you're trying to do. I can say that alpaca-7B and alpaca-13B operate as better and more consistent chatbots than llama-7B and llama-13B. That's what standard alpaca has been fine-tuned to do.

Is it bigger? No, alpaca-7B and 13B are the same size as llama-7B and 13B.

There's also the tenstorrent chips coming out to public which are vastly more efficient than nvidia stuff

Llamma.cpp uses the neural engine, so does StableDiffusion. And the speed is not that far off from VRAM, actually.

>Memory bandwidth is increased to 800GB/s, more than 10x the latest PC desktop chip, and M1 Ultra can be configured with 128GB of unified memory.

By comparison, the Nvidia 4090 is clocking in at ~1000GB/s

Apple is clearly positioning their devices for AI.

I will assume you are much more knowledgeable than I am in this space.. have few basic questions that have been bothering me since all the craze started around GPT and LLM recently.

I managed to get Alpaca working on my end using the above link and get very good result. LLaMa biggest takeaway was it is able to reproduce quality comparable to GPT and much lower compute size. If this is the case, why is the ouput much shorter on LLaMa than what I get on OpenGPT? I would imagine the OpenGPT reponse is much longer because ... it is just bigger? What is the limiting factor to not for us to get longer generated response comparable to GPT?

ggml-alpaca-7b-q4.bin is only 4 gigabyt - I guess this what it means by 4bit and 7 billion parameter. Not sure if rumor or fact, GPT3 model is 128B, does it mean if we get trained model of GPT, and manage to run 128B locally, will it give us the same results? Will it be possible to retrofit GPT model within Alpaca.cpp with minor enhancement to get output JUST like OpenGPT? I have read to fit 128B, it requires muliple Nvidia A100.

Last question, inference means that it gets output from a trained model. Meta/OpenAI/Stability.ai have the resources to train a model. If my understanding is correct, Alpaca.cpp or https://github.com/ggerganov/llama.cpp are a sort of 'front-end' for these model. They allow us to provide an input to get an output by inference with the model. The question I am trying to ask is, what is so great about llama.cpp? Is it because it's in C? I know there is Rust version of it out, but it uses llama.cpp behind the scene. Is there any advantage of an inference to be written in Go or Python?

Technically the truth?

Have people tried doing "textual inversion" for language models? (i.e not in a context of StableDiffusion)

alpaca-30B > llama-30B ?

> if you're not shuffling the entire network weights across the memory bus every inference cycle

Isn't this common though?

Good to hear. Thanks!

depends on the age of the cow I suppose.

You can run llama-30B on a CPU using llama.cpp, it's just slow. The alpaca models I've seen are the same size as the llama model they are trained on, so I would expect running the alpaca-30B models will be possible on any system capable of running llama-30B.

You can run it on your CPU. My old i7 6700k spits out 13B words a little slower than I could read them out loud. I'll test the 30B tonight on my 5600X

> 128Gb of Uniform RAM which can be used by CPU, GPU or Neural Engine.

But it doesn't have the same bandwidth as the VRAM on the GPU card iirc.

Otherwise every integrated GPGPU would be better due to available ram.

The neural engine on M1 and M2 is usable IIRC only with apple libraries, that may not be used by notable models yet.

Untuned 30B LLaMA, you're saying? It's excellent and adept at storywriting, chatting, and so on, and it can output faster than ChatGPT at 4-bit precision. While I'm not into this myself, I understand that there is a very large RP community at subs like CharacterAI and Pygmalion, and the 30B model is genuinely great for feeling like talking to a real person. I'm using it with text-generation-webui and custom parameters and not the llama.cpp implementation.

For assistant tasks, I've been using either the ChatLLaMA 13B LoRA or the Alpaca 7B LoRA, both of which are very good as well. ChatLLaMA, for instance, was able to answer a reasoning question correctly that GPT-3.5 failed, but it has drawbacks in other areas.

The limitations so far are that none of these models can answer programming questions competently yet, and a finetune for that will be needed. They also have the tendency to hallucinate frequently unless parameters are made more restrictive.

Active learning deals with using a small subset of representative images that should perform as well as a larger set of uncurated images. You can consider looking into that.

Hey, recently I was thinking if Apple Silicon Macs may be best thing for AI in the future. Most powerful Mac Studio has 128Gb of Uniform RAM which can be used by CPU, GPU or Neural Engine. If only memory size is considered, even A100, let alone any consumer oriented model, can't match. With this amount of memory you could run GPT3 Davinci size model in 4bit mode.

[removed]

Thanks. So bit confused here. Ot mentions needing an A100 to train. Am I able to run this off a 3090?

Am I looking in the wrong place? It seems like the torch 2.0 code still requires training==False in order to use FlashAttention:

https://github.com/pytorch/pytorch/blob/663e7c9eeb66fb049b8487a6a5a7ea4311fb53d3/torch/nn/modules/activation.py#L1139

Time to see if I can get this running on Dalai

Which takes years

Niche supercar gamers start up the industry which then will lead into realistic VR which will then lead into consumer high quality stuff?