Introduction¶

Text Generation Methodologies¶

• \(n\)-grams
• Bigrams
• Trigrams
• Bag of words
• Bag of tokens; token = subwords
  • Byte pair encoding

Tokenization causes issues - LLM cannot spell words - LLM cannot perform simple string processing tasks, such as reversing a string - LLM performs worse in non-English languages - LLM is bad at simple arithmetic - LLM prefers YAML over JSON with LLMs - LLM breaks due to special/unstable tokens - <|endoftext|> - trailing whitespace - SolidGoldMagikarp - special tokens - LLM is not end-to-end language modelling

Architectures¶

• MLP
• RNN
• GRU
• LSTM
• Transformers

Comments