Exabits, a decentralized cloud compute platform, has partnered with MyShell to dramatically reduce the cost of training large language models (LLMs) from billions of dollars to under $100,000. The JetMoE-8B model, developed by MyShell and inspired by the sparse activation architecture of ModuleFormer, has outperformed LLaMA2-7B from Meta AI while costing a mere $0.1 million to train.
The JetMoE-8B, with its 8 billion parameters and sophisticated structure of 24 blocks, each housing two Mixture of Experts (MoE) layers, showcases advanced efficiency and computational intelligence. The model’s selective activation of 2 out of 8 experts per input token demonstrates a refined utilization of the Sparse Mixture of Experts (SMoE) framework, enhancing the model’s responsiveness and resource management. This efficiency, with only 2.2 billion activation parameters, has significantly lowered the training costs while delivering robust performance.
The effectiveness of the JetMoE-8B is evident in its state-of-the-art results across five categories on eight evaluation benchmarks, outperforming competitors like LLaMA-13B, LLaMA2-7B, and DeepseekMoE-16B. On the MT-Bench benchmark, the JetMoE-8B scored 6.681, surpassing models with larger capacities, such as LLaMA2 and Vicuna, which possess 13 billion parameters.
Exabits’ contribution to this breakthrough is its accelerated and stabilized cluster of 12 H100 GPU nodes (96 GPUs), which played a pivotal role in powering the JetMoE model. Exabits’ platform has demonstrated that decentralized GPU platforms can be suitable for LLM training, overcoming the skepticism surrounding their capabilities.
Exabits, a decentralized cloud compute platform, has aggregated, accelerated, and stabilized consumer-grade GPUs to match enterprise-grade GPU performance. This approach taps into a vast, yet largely idle reserve of consumer GPUs, easing the GPU shortage crisis. Exabits’ extensive experience in the data center sector has also provided access to coveted enterprise-grade H100 and A100 GPUs, further advancing the democratization of AI development.
The success of the JetMoE-8B underlines the feasibility of Exabits’ platform in executing high-end model training, paving the way for more sustainable and inclusive advancements in AI research and development. Exabits is not just a technological platform; it is a beacon for the future of LLM training, embodying affordability, accessibility, and environmental consciousness.
This news story relied on a press release distributed by BlockchainWire. Blockchain Registration, Verification & Enhancement provided by NewsRamp™. The source URL for this press release is Exabits and MyShell’s Breakthrough: From Billions to $100K in LLM Training Costs.