Gemini 1.5 Pro has a massive context window
Google DeepMind recently released a report on Gemini 1.5 Pro.
Perhaps the most significant advance over Gemini 1.0 is the context window, which can now span up to 10 million tokens of text.
Timestamps:
00:00 - Gemini 1.5 Pro has a massive context window
00:48 - Needle-in-haystack analysis
03:40 - Model architecture (sparse mixture-of-expert Transformer)
05:58 - Training infrastructure
06:53 - Long-context evaluation (Jax codebase, Les Miserables etc.)
09:15 - Core Capability Evaluations (MMLU, HumanEval etc.)
10:08 - Responsible Deployment
The paper can be found here: https://storage.googleapis.com/deepmi...
Topics: #gemini #llm #google
For related content:
Twitter: / samuelalbanie
personal webpage: https://samuelalbanie.com/