BLOOM

A group of AI experts have produced a language model that is larger than GPT-3 and they are offering it for free.

The development of the corpus and datasets began some time before the training began on March 11, 2022. It has taken them four months to create this model.

It has been quite a challenge: using 384 powerful graphic cards, BLOOM has 176 billion parameters, 70 layers, 112 attention heads per layer, a hidden dimensionality of 14336, and a sequence length of 2048 tokens. It also uses ALiBi positional embeddings and GeLU activation functions.

The training was open to everyone and it has been conducted in various languages, including English, Spanish and Italian, as well as programming codes. All the resources are available and well-documented.

Big Science

Big Science is an open collaboration driven by HuggingFace, GENCI and IDRIS. This research workshop brings together researchers from many backgrounds in AI, NLP, social science, law, ethics and public policy.