Okay, tankie. Keep it about machine learning.

Not knowing and not being allowed to know are radically different things. If you want to conflate the two, you are acting in bad faith.

If OpenAI wants people to respect their IP they should take the word “open” out of their name. They scraped our data to train their models after all, it’s not like OpenAI themselves aren’t pushing the boundaries of what’s acceptable when it comes to copyright law.

Legally it’s questionable, but ethically speaking I think it’s a fine idea.

You can get llama running on consumer grade hardware. There's 4 and 8 bit quantization for it i believe where it fits in a normal gpu's vram, i saw floating around here

[removed]

Thanks!

[removed]

I think the new model gets most of its knowledge from its original model and the training is mostly about how to act like a RLHF model.

Pretty sure whoever knows what happened to Jimmy Hoffa made sure they kept their trap shut in public… ;)

A proper port to the neural engine would be especially interesting. There was one by Apple for Stable Diffusion.

The results for LLaMA-33B quantised to 3bit are rather interesting. That would be an extremely potent LLM capable of running on consumer hardware. Pity that there are no test results for the 2bit version.

Have you tried GPT4? It’s been quite resilient in my testing

[removed]

[removed]

Careful. That MIT license won't work, I think, thanks to ClosedAIs licences

You know ChatGPT and GPT4 licenses forbid using their output data for training competing AI models. What Stanford did was to show proof of concept for their paper, not to open-source the model, at all.

Just saw https://www.reddit.com/r/MachineLearning/comments/11v261n/p_d_datasetgpt_a_commandline_tool_to_generate/

seems very similar

Yeah I'm wrong it seems. Read a few articles using bootstrapping in the definition I used so I assumed that was generally it.

That's not what bootstrapping is, it is a resampling technique used to create multiple datasets of the same size from the original dataset using random sampling with replacement. It is done to get the estimate of the standard deviation of a desired variable.

Here's the link to the ISLR textbook. The bootstrap chapter will verify what it is.

I was talking to GPT 4 about this and it said that it seems plausible and can dramatically bring down costs.

It called it "knowledge distillation"

It also mentioned that if we had access to the weights from open ai you can use a process called model compression to scale down the hardware and put it on less powerful gpus or distributed gpus (like how render farms work)

This also explains why open ai is so cagey on releasing weights - the initial training cost is where the money sink is but once weights are out their is ways to make it run on cheaper hardware.

But Im wondering does this mean the smaller model can ONLY respond to the questions your generating or will it have latent knowledge outside of just the knowledge transfer? Like would say the smaller model thats trained off this approach also be able to answer questions on topics that are "restricted" in open ais view that you couldnt ask it or do you absolutely must need to get an initial answer for such restricted content for it to be able to produce a responce?

Talking about things like writing malicious code or what not. I dont plan on doing that obviously but Im curious on if this means that these smaller models will basically be totally unrestricted now or if its just trained on say tons of python code it can just create said malicious code from scratch without actually being exposed with examples of "how" to make it (since it has a greater knowledge of the ubderlying principals of python)

Edit: Okay guess it can per GPT 4.

Damn these things are fascinating.

>Yes, the same concerns can apply to a smaller model being trained from a larger one via knowledge distillation. Knowledge distillation is a technique where the smaller model learns to mimic the larger model's behavior by training on a dataset generated using the larger model's outputs. The smaller model effectively learns from the larger model's knowledge and understanding of language patterns and concepts.

>As a result, the smaller model can also gain latent knowledge about various topics and domains, even if it hasn't been explicitly exposed to specific examples during training. This means that the smaller model could potentially generate undesirable content based on its understanding of the relationships between words and concepts, similar to the larger model.

ChatGLM is really good. I sometimes have a hard time distinguishing its Chinese outputs from those of chatgpt.

Sadly its English could use some improvement as it usually use Chinese adjectives when similar words are lacking in English.

I tried googling it, it's is a nonexistent terminology in the realm of statistics. I know what bootstrapping is, but not this version of it.

It's better to ask the GitHub authors about this to make sure they're not just spitting out pseudostatistical terminology.

Addendum: another guy did query the authors regarding this terminology in the issues tab, they did not respond.

This makes perfect sense thank you. I’m going to think through this further. If you have any suggestions for verification/sanity testing for any of the components listed above please let me know.

[deleted]

I mean, even the Taiwan flag emoji is banned on Chinese phones lmao