🐥 I'm building my own LLM
As I’ve joined the course based on the book "Understanding Large Language Models".
Course: skool.com/launchai
The first week is to grasp what is tokenization.
Think of it like chopping up a sentence into smaller pieces.
We turn words into numbers or token IDs, and add special tokens to help the model understand better ( like 'endoftext').
Cool website to play around:
tiktokenizer.vercel.app
GPT-3 uses the byte pair encoding (BPE) tokenizer, which is like grouping common letters together.
Also, we lack a sense of the token's position in a sequence.
That’s why GPT models use absolute positional embeddings (like adding a number to each word to tell) and those are optimized too during the training.
Stay tuned for the week 2!