Interactive Lesson: How Do AI Language Models Predict Text?
Have you ever wondered how language models generate text? In this interactive lesson, you'll explore how this works for simple probabilistic language models, such as those used for predictive text, and for large language models (LLMs), such as those used for AI chatbots. You will examine how a model’s training corpus shapes its output, break sentences into tokens, and generate text using unigram, bigram, and trigram models. You will then investigate how large language models represent words as positions in an embedding space, how self-attention helps them represent words based on context, and how bias in training data influences what a model learns.
Topics
- probabilistic language models
- simple language model
- large language models
- LLM
- language models
- tokenization
- word tokenization
- subword tokenization
- n-grams
- unigram
- bag of words
- bigram
- trigram
- backing off
- word embeddings
- embedding space
- vector
- vector math
- probability
- statistical patterns
- training data
- corpus
- token
- generative AI
- AI chatbots
- text prediction
- semantic map
- predictive text
- machine learning
- artificial intelligence
- static embedding
- self-attention