How open are they? I mean having open models is an improvment, but the training methods should be open too. And if we croud source data that should be accessible too.

> and uses a different base model and claims it’s a big innovation

Huh? My read of their blog was that they wanted to highlight the fact that you can fine-tune a ~2yo LLM and still get decent results. I don't think they've claimed this is innovative, or that the innovation is theirs to boast...

I've played with GPT-neo (non X) and GPT-J when they were released, and the results were rough. You had to do a ton of prompt engineering work and exploration to find useful cases. This shows that even smaller, older models can be fine-tuned with the method proposed in Alpaca.

The demo was up for a couple of days. The first hours of it being online were rough (80-200 people in queue). It got better the following day, and better still the 3'rd day. I believe they removed the demo ~1week later. IMO they've proven a point - the demo was extremely impressive for a 7b model.

Wow so we can hook it up with cargo --check and it will generate perfect Rust code.

I have not seen a way in the GPT-4 UI by OpenAI to submit an image? How do you do it?

Thanks Obama

Knowing how a lot of text data from Reddit comments ends up in these huge text datasets only for them to make it completely closed source rubs me the wrong way.

IMHO, I think that cancer and aging are necessary for complex organism. It is more likely that we solve cloning or build the first in vitro womb, than we are at deafeating cancer or aging.

I wish you were right, but people are calling for investment in AGI to cease altogether:

> There is no way for humans to adapt for alien intelligence. The idea of developing general AI is insanely horrifying from the beginning.

One of the parent comments.

Such absolutist comments leave no room whatsoever for venturing into AGI.

I have a theory that the main reason OpenAI decided to start keeping it's training and architectural details private is because through minor modification in training data and data augmentation they were able to gain significant improvements in the qualitative output of GPT.

Thus any competitor could replicate the pipeline with ease and reproduce the improvements, so they decided to keep it as a trade secret.

Glad more research like this is being done and shared to the rest of the community.

In my understanding, if you have text, it's not a challenge to train on next word prediction. Just keep the learning rate low. The reason there's a focus on the instruction based fine tuning is because that data is harder to come by.

My only experience is I've done this with a sentence embedding model (using sbert) and I just trained on my new text and the original training data 50/50 and it both got better at embedding my text and didn't forget how to do what it was originally trained on

Interesting methodology/technology. I realize it's GPT-4+ a refining process but even so, 88% is ~64% fewer errors than 67%, which proves it's a powerful technique even when the underlying model is already fairly capable.

If an intelligence explosion happens, there's really no telling what's possible. Maybe these problems are trivial to a 1 million IQ machine, maybe not. The only question really is if the explosion will happen. Two years ago I would have said 1% in the next ten years, now I'm up to 10%. Maybe in two more years it'll look like 30%.

The use of neural nets (ReLU + LayerNorms) layered between each attention step counts as a brain, no? I know the attention mechanism is what gets the most ... attention, but there's still traditional neural nets sandwiched between and in some cases the transformer is just a neck feeding into more traditional modules. ReLU is Turing complete so I can always tune a neural net to have the same response pattern of electrical outputs as any neuron in your brain.

The million dollar question according to David Chalmers is, would you agree that slowly replacing each neuron with a perfect copy one at a time will never cause you to go from completely fine to instantly blacked out? If you answered yes, then it can be shown (sections 3&4) that you must accept that neural nets can be conscious, since by contradiction if there was a gradual phasing out of conscious experience rather than sudden disappearance, that would necessarily require the artificial neurons to at some point begin behaving differently than the original neurons would (we would be aware of the dulling of our sensation).

Considering we lose brain cells all the time and don't get immediately knocked out, I think you can at least agree that most people would find these assumptions reasonable. It would be pretty weird to have such a drastic effect for such a small disturbance.

Because the recent release of LLMs has been too vigorous, I organized recent notable models from the news. Some may find the diagram useful, so please allow me to distribute it.

Please let me know if there is anything I should change or add so that I can learn. Thank you very much.

If you want to edit or create an issue, please use this repo.

---------EDIT 20230326

Thank you for your responses, I've learnt a lot. I have updated the chart:

• https://github.com/michaelthwan/llm_family_chart/blob/master/LLMfamily2023Mar.drawio.png
• (Look like I cannot edit the post)

Changes 20230326:

• Added: OpenChatKit, Dolly and their predecessors
• More high-res

To learn:

• RWKV/ChatRWKV related, PaLM-rlhf-pytorch

Models that not considered (yet)

• Models that is <= 2022 (e.g. T5 (2022May). This post is created to help people quickly gather information about new models)
• Models that is not fully released yet (e.g. Bard, under limited review)

Yeah, I think you're on the money there. It's very hard for us to not anthropomorphize this behavior, especially because we literally used RLHF in order to make it more human-like.

Sadly that is what academia came to. I am doing my phd and 80% od my papers is just padding. And if you don t follow the “template” you can t publish anything

Model weights though, are, I assume, not copyrightable.

Is there actually a law giving Stanford any special rights to the weights?

This is what I love about this community.

True, but with AI more computing power/data means better models. With medicine, things move slower. If we get a cure for one or two cancer this decade, it would be a massive achievement.

Oh my gosh. Wolfram plugin!

I just want humans to stop dying of cancer!

Monkey's paw curls. The humans all die of being shot by drones instead

This is not an alien intelligence yet. We understand how it works how it thinks. But eventually this version can generate an AI that is harder for us to understand, and that version can generate another ai. At some point it will become alien to us because we may not understand the math behind jt,

Thanks! If that is really the TL;DR, I have never seen an abstract that beats about the bush so much

What do you mean by "frame"? How many images do you think GPT-4 would need to get a cursor where it needs to go? I'd estimate four or five should be plenty.