How ChatGPT drives open source AI development



summary
Summary

Stanford’s Alpaca was just the beginning: In recent weeks, several AI models have been unveiled using training data generated by ChatGPT.

In mid-March, Stanford researchers unveiled the Alpaca language model, a variant of Meta’s LLaMA 7B that was fine-tuned with AI-generated data. The team trained the LLaMA model with 52,000 example statements generated by OpenAI’s GPT-3.5 (text-davinci-003).

Alpaca achieved partially comparable results in the team’s benchmarks, and was so cheap at $600 that alignment researcher Eliezer Yudkowsky, for example, sees it as a real challenge for companies like OpenAI.

Alpaca formula is open source, but may not be used commercially

However, the LLaMA model used for Alpaca is not released for commercial use, and the OpenAI GPT-3.5 terms of use prohibit using the model to develop AI models that compete with OpenAI. Stanford has therefore not yet released the model, only the training data and the code to generate the data and fine-tune the model.

ad

Shortly after Alpaca, several open-source models were released that build directly on or are heavily inspired by this work. Alpaca-LoRA uses the resource-efficient low-rank adaptation (LoRA) method, also widely used in Stable Diffusion, with Meta’s LLaMA to achieve results comparable to Alpaca.

Startup Nomic AI released GPT4Alla LLaMA variant trained with 430,000 GPT-3.5-turbo outputs selected from a dataset of one million outputs in total.

ChatDoctor, on the other hand, is a LLaMA model specialized for medical chats. The authors of the scientific paper trained LLaMA first with the 52,000 Alpaca training examples and then with 5,000 real conversations between medical professionals and patients.

The startup Databricks linked on EleutherAI’s GPT-J-6B instead of LLaMA for its chatbot Dolly, which also used the Alpaca training dataset. “We find that even years-old open source models with much earlier architectures exhibit striking behaviors when fine tuned on a small corpus of instruction training data,” Databricks said of training with the GPT-generated data.

ChatGPT clones and data gold

Stanford has provided a recipe for better open-source models with Alpaca, usually based on the powerful LLaMA models with special datasets generated by ChatGPT. Now that the larger LLaMA models have been leaked, we can expect much better models – but without a commercial license for now.

Recommendation

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top