/docs/ai/gpt/lamda/ Directory Listing

Google LaMDA is a large 137b-parameter dense neural network model, announced by Google in May 2021⁠, as a followup to ⁠; it is most similar to OpenAI’s May 2020 (175b) in both design and capabilities. This parity may be due to higher-quality training data used for LaMDA, particularly the large dialogue training dataset inherited from Meena.

LaMDA is one of the standard testbeds for Google scaling research and examining the many surprising capabilities scaled-up models turn out to have, and many papers have been published about it. Mysteriously, Googlers are not allowed to name LaMDA in those papers, or even to confirm or deny whether it is LaMDA when asked; instead, the papers will vaguely allude to a series of large Transformers (eg. “we used pre-trained dense decoder-only Transformer language models, ranging in size from 2 million to 137 billion parameters. These models were pre-trained on web documents and dialog data”).

This index collates papers I infer make use of LaMDA: typically, if a Google paper uses a model size <20b, then it is probably a bidirectional Transformer; >200b-parameters, it is actually a mixture-of-experts model (eg. ); if a >150b-parameter model is specified to be dense, then it may be a different model like DeepMind’s 280b-parameter ⁠.

Miscellaneous