Good papers don't need to do that. If papers are self contained, no need for gibberish.

From my experience, it comes from when the paper is not planned before being written, or when results/methodology is either not refined or not interesting enough.

We already know of complex organisms that essentially don't age, and also others that are cancer-free or close to it. In any case, "prevent any and all aging and cancer before it happens" is a stupid goalpost. "Be able to quickly and affordably detect, identify and treat arbitrary strains of cancer and/or symptoms of aging" is essentially "just as good", and frankly seems like it could well already be within the reach of current models if they had the adequate "bioengineering I/O" infrastructure, and fast & accurate bioengineering simulations to train on.

ML could plausibly help in getting those online sooner, but unless you take the philosophical stance that "if we just made AGI they'd be able to solve every problem we have, so everything is effectively an ML problem", it doesn't seem like it'd be fair to say the bottlenecks to solving either of those are even related to ML in the first place. It's essentially all a matter of bioengineering coming up with the tools required.

Well cloning and artificial wombs are basically done or very close, we just haven't applied it to humans due to ethical reasons. Six years ago there was already a very premature lamb kept alive in an artificial womb for four weeks.

As for cancer and aging...it seems increasingly clear that part of the process is just that genes necessary for development get dysregulated later on in life. I think the fact that we can rejuvenate our own cells by making sperm and eggs points to the fact that the dysregulation should be fixable, and recent advances in aging research seem to show that this is true. The issue is, of course, pushing that process too far and ending up with cells dedifferentiating or becoming cancerous, but I think it's possible if we're careful.

This is actually a very good PR material, as it will save engineers' time. Just opened it and referenced your comment. https://github.com/noahshinn024/reflexion-human-eval/pull/1

Sounds like we need a LLM to generate padding for the academia and LLM to write the tldr for the readers. World is dumb.

Well if you have connections you would’ve seen they made a good amount of posts.

That's fair. But you commented out of context, on a post that linked to the blog and not the WSJ article. That's on you.

It seems that contrary to conventional wisdom, models with more parameters learn more efficiently. My personal ‘hunch’ is that training large models and then some form of distillation may become the standard thing to do.

Lmao it seems everyone used chatGPT for a grand total of 20 minutes and threw their hands up saying "this is the end!". I have always wondered how the public would react once this tech finally became good enough for the public to notice, can't say this was too far from what I envisioned. "What if it's conscious and we don't even know it!" Cmon give me a break

Well if you ever worked with marketing or communication teams you would’ve known that DB co-authored the WSJ article. My point is that the democratization is an achievement of the Stanford Alpaca team, not DB. DB marketed it like they did the major work which is untrue.

And yes the plan is to offer distributed training! As you can imagine there are about a million things we want/need to add! If you would like to get involved in the project and help out, please do

Hey! Yes, there are similarities to PBT, but there are a few differences here. Firstly, the mutations implemented with AgileRL are much more dynamic. Rather than only mutating hyperparameters, we’re allowing any part of the algorithm/model to mutate - HPs, network architecture (layers and nodes), activation functions and network weights themselves. We also train the population in one go, and offer efficient learning by sharing experience within the population.

[removed]

Nice

> https://www.databricks.com/blog/2023/03/24/hello-dolly-democratizing-magic-chatgpt-open-models.html

This is the blog post that I've read. I can't comment on the WSJ article, and your original message implied a bunch of things that, IMO, were not found in the blog post. If you don't like the WSJ angle, your grief should be with them, not databricks. shrug

From the actual blog:

> We show that anyone can take a dated off-the-shelf open source large language model (LLM) and give it magical ChatGPT-like instruction following ability by training it in 30 minutes on one machine, using high-quality training data.

> Acknowledgments > > This work owes much to the efforts and insights of many incredible organizations. This would have been impossible without EleutherAI open sourcing and training GPT-J. We are inspired by the incredible ideas and data from the Stanford Center for Research on Foundation Models and specifically the team behind Alpaca. The core idea behind the outsized power of small dataset is thanks to the original paper on Self-Instruct. We are also thankful to Hugging Face for hosting, open sourcing, and maintaining countless models and libraries; their contribution to the state of the art cannot be overstated.

More to the point of your original message, I searched for "innovative" "innovation" "inovate" and found 0 results in the blog post. I stand by my initial take, the blog post was fair, informative and pretty transparent in what they've done, how, and why.

Awesome! I’m still being ghosted :(

[removed]

I’ve been experimenting with Alpaca and able to fine-tune it using the dataset provided in 40 minutes with 8 A100s, spot instances. It actually works well.

I second this. I was able to extract fairly useful results from Neo but it took a huge amount of prompt trial and error, eventually getting decent/stable results but not in the same ballpark as GPT3+. The dolly training results here seem good, if not expected. I'm now ready to move to a superior model like LLaMA/Alpaca though. What are you running?

WSJ:

“Databricks Launches ‘Dolly,’ Another ChatGPT Rival The data-management startup introduced an open-source language model for developers to build their own AI-powered chatbot apps” (Apparently DB paid them)

DB’s blog:

“Democratizing the magic of ChatGPT with open models”

Introduced? ChatGPT rival? Didn’t you just follow Stanford’s approach? You used Stanford’s dataset which was generated by GPT right? huh? This is Stanford’s achievement not DB’s. DB went too far on marketing.

I'm a little surprised at the seeming lack of any backlash, tbh. I'm sure it's coming though.

tell me about the one with the microprocessor

exactly - no one wants to be the first one to say it for some reason. If it were me I'd be grabbing that place in history with both hands

Thanks for sharing! Just a couple of questions. Is the evolutionary algorithm you use similar to PBT or fundamentally different in some way? And is there a plan to implement distributed training and HPO (similar to Ray RLlib with PBT from Tune)?

Classy, especially when they are breezing past any copyright of the datasets they are training off of. I wonder if they can legally enforce that without creating a potentially bad precedent for themselves. Or if it could be worked around if the training was indirect through something like Alpaca.