In 2023, language models are taking center stage. Many new models have been developed from scratch, and thousands of variations of these models have been fine-tuned on domain sets and made available to the public. Generative AI is a hot topic, and more and more companies are interested in creating their own models. One way to do this is by fine-tuning one of the existing open models, such as BLOOM, LLaMA, LLaMA 2, or Mistral, or by training a new model from scratch. However, training a new model from scratch can be expensive. To get an idea of how much computational resources are required, we have gathered information about the training time of the LLM models.
2023 年，语言模型将成为中心舞台。许多新模型都是从头开始开发的，这些模型的数千种变体已经在域集上进行了微调并向公众提供。生成式人工智能是一个热门话题，越来越多的公司对创建自己的模型感兴趣。一种方法是微调现有的开放模型之一，例如 BLOOM、LLaMA、LLaMA 2 或 Mistral，或者从头开始训练新模型。但是，从头开始训练新模型可能很昂贵。为了了解需要多少计算资源，我们收集了有关LLM模型训练时间的信息。
The chart below shows the total GPU training time for each model. If a model was trained on 100 GPUs for 2 hours, then the total GPU time is counted as 100*2=200 hours.
下图显示了每个模型的总 GPU 训练时间。如果一个模型在 100 个 GPU 上训练了 2 小时，则 GPU 总时间计为 100*2=200 小时。
The chart includes only the models for which information about their training time was publicly available. It does not contain information about ChatGPT&zhida_source=entity" target="_blank" style="text-decoration-line: none; color: rgb(9, 64, 142); cursor: pointer;">ChatGPT, GPT-4, or Mistral. If you know of any sources that describe the training time of another model, please share them in the comments.
该图表仅包括有关其训练时间信息公开的模型。它不包含有关 ChatGPT、GPT-4 或 Mistral 的信息。如果您知道任何描述另一个模型训练时间的来源，请在评论中分享。
BLOOM (2022, July) BLOOM（2022 年 7 月）
384 (NVIDIA A100 80GB GPUs) [1] * 3.5 (months) = 999,936 GPU hours
384 （NVIDIA A100 80GB GPU） [1] * 3.5（个月）= 999,936 GPU 小时
PaLM (2023, March) PaLM（2023 年 3 月）
6144 (v4 TPU)[1] * 1200 (hours)[1] * 1.2 (TPU/GPU)[2] = 8,847,360 GPU hours
6144 （v4 TPU）[1] * 1200（小时）[1] * 1.2 （TPU/GPU）[2] = 8,847,360 GPU 小时
https://www.nocode.ai/what-it-takes-to-train-a-foundation-model/
https://www.datanami.com/2023/04/05/google-claims-its-tpu-v4-outperforms-nvidia-a100/
LLaMa 1 (2023, February) LLaMa 1（2023 年 2 月）
LLaMA-7B — 82,432 GPU hours [1]
LLaMA-7B — 82,432 GPU 小时 [1]LLaMA-13B — 135,168 GPU hours [1]
LLaMA-13B — 135,168 个 GPU 小时 [1]LLaMA-33B — 530,432 GPU hours [1]
LLaMA-33B — 530,432 个 GPU 小时 [1]LLaMA-65B — 1,022,362 GPU hours [1]
LLaMA-65B — 1,022,362 GPU 小时 [1]
BloomberGPT (2023, May) BloomberGPT（2023 年 5 月）
512 (A100 GB GPU 40GB)[2] * 49 (days)[2] = 1,300,00 GPU hours
512 （A100 GB GPU 40GB）[2] * 49（天）[2] = 1,300,00 GPU 小时
LLaMa 2 (2023, July) LLaMa 2（2023 年 7 月）
Llama 2 7B — 184,320 GPU hours [1]
Llama 2 7B — 184,320 GPU 小时 [1]Llama 2 13B — 368,640 GPU hours [1]
Llama 2 13B — 368,640 GPU 小时 [1]Llama 2 34B — 1,038,336 GPU hours [1]
Llama 2 34B — 1,038,336 GPU 小时 [1]Llama 2 70B — 1,720,320 GPU hours [1]
Llama 2 70B — 1,720,320 GPU 小时 [1]
推荐本站淘宝优惠价购买喜欢的宝贝:
本文链接：https://hqyman.cn/post/9553.html 非本站原创文章欢迎转载，原创文章需保留本站地址！
休息一下~~
萌ICP备20248119号
