Meta's Llama 2 Shakes Up the AI Landscape with 70 Billion Parameters

 

 Meta recently unveiled Llama 2, a new large language model boasting up to 70 billion parameters. This move has caught the attention of OpenAI, as Meta rarely divulges details about its AI models. Llama 2's release was accompanied by a significant partnership with Microsoft, who has demonstrated support for the model in its Azure and Windows platforms. Additionally, Qualcomm has also joined the Llama 2 fray by announcing plans to bring the model to smartphones.

One point of contention lies in Meta's claim that Llama 2 is open source, which has been promoted by both Meta and Microsoft. However, some open-source developers disagree with this classification. Nevertheless, Llama 2's license provides developers and researchers the flexibility to fine-tune the model for their specific requirements.

Llama 2 has already showcased its capabilities through platforms like Perplexity.ai, where multiple Llama 2 models were demonstrated to generate natural and coherent text. The model performed well in various tasks, including generating code, solving equations, and providing commonly understood facts. Llama 2's academic benchmarks, such as MMLU and GSM8K, put it on par with OpenAI's GPT 3.5.

Meta's researchers achieved this level of performance through techniques like supervised fine-tuning, reinforcement learning with human feedback, and a novel method called "Ghost Attention" (GAtt). Ghost Attention allows Llama 2 to generate desired results within specific constraints, making it useful for scenarios like simulating historical figures or producing responses related to particular topics like architecture.

Llama 2 offers various models with different parameter sizes, such as Llama 2 70B, Llama 2 7B, and Llama 2 13B. The larger model performs exceptionally well in benchmarks, but the smaller variants are optimized for running on less powerful devices, including smartphones. In collaboration with Qualcomm, Meta plans to have Llama 2 running locally on Qualcomm-powered smartphones from 2024, thereby saving energy and enabling offline usage.

One distinctive aspect of Llama 2 is its open-source license, which allows unlimited, free commercial, and academic use. While it does not meet all standards of the Open Source Initiative, it still empowers developers to explore and enhance the model to suit their needs. The open-source community has already embraced Llama 2, with models based on it already appearing on HuggingFace's Open LLM leaderboard.

This open-source nature is seen as a force multiplier by experts like Aravind Srinivas, as it enables collaborative improvements and accelerated progress. Developers can start forks of Llama 2 and focus on various aspects such as quantization, low-rank fine-tuning, and distillation of larger models into smaller ones. This versatility is particularly advantageous for edge devices like smartphones.

Overall, Llama 2's arrival marks a significant expansion in the capability and reach of open-source AI models. As developers continue to build upon Llama 2, it is expected to see further improvements and rise in popularity in the open-source community.