Skip to main content

free-willy-2

Model Name: FreeWilly2
Developer: Stability AI

Overview​

FreeWilly2 is an auto-regressive language model that is based on the Llama2 70B architecture. It's primarily designed for research use and is capable of generating text in English. Its responses are generated through prompts which have been pre-configured to a specific format.

Base Model​

The Llama2 70B model, a renowned transformer model known for its ability to produce relevant and context-aware text, serves as the foundation for FreeWilly2. This model is then refined or 'fine-tuned' to meet the specific needs of the tasks it is intended to handle.

Unique Features​

What sets FreeWilly2 apart is the use of an Orca-style dataset for its fine-tuning. The term 'Orca-style' here refers to the manner in which the data used to train the model is structured and organized. Similar to how the Orca, a type of whale, communicates using complex and layered sounds, this dataset is layered with various types of data and complex structures to mimic real-world language scenarios. This helps the model to better understand context and generate more appropriate responses. However, it is important to remember that while the model's output may be sophisticated, it should not replace human discretion and judgment.

Training Method​

FreeWilly2 was trained in two phases using a method known as supervised fine-tuning. This process involved teaching the model using a meticulously labeled dataset, allowing the model to learn and improve its performance. Both phases used a type of precision known as BF16 and were optimized using a method called AdamW. The specifics of the two phases varied slightly in terms of batch size, learning rate, and other parameters. It should be noted, however, that this technical information may not directly affect the user's experience with the model.

Commercial License​

The licensing of FreeWilly2 falls under the Non-Commercial Creative Commons (CC BY-NC-4.0), meaning it is available for non-commercial use, such as research. For more information, inquiries, or comments about the model, Stability AI can be reached at lm@stability.ai.