Paper List

Machine Learning & Mathematics

• Intuition for entropy & KL divergence

AI

Introductory:

• A visual proof that neural nets can compute any function

Transformers:

• Attention is All You Need
• An Introduction to Transformers
• Formal Algorithms for Transfomers

LLMs:

• Let's build GPT: from scratch, in code, spelled out.
• Language Models are Unsupervised Multitask Learners (GPT-2)
• Language Models are Few-Shot Learners (GPT-3)
• Scaling Laws for Neural Language Models

Fine-Tuning:

• LoRA: Low-Rank Adaptation of Large Language Models
• Training language models to follow instructions with human feedback

Mixture of Experts (MoE):

• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Mixture-of-Depths: Dynamically allocating compute in transformer-based language models

Augmented LLMs:

• Augmented Language Models: a Survey
• Retrieval-Augmented Generation for Large Language Models: A Survey
• ReAct: Synergizing Reasoning and Acting in Language Models

OpenAI o1 & Test-Time Compute:

• Let's Verify Step by Step
• Qwen2.5-Math Technical Report

Model Deployment:

• Distributed Training of Deep Learning Models: A Taxonomic Perspective
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Transformer Inference Arithmetic
• LLM Inference Performance Engineering

Quantization:

• Survey of Quantization Methods
• GPTQ Paper
• AWQ Paper
• GGUF

Embedding Models:

• Text and Code Embeddings by Contrastive Pre-Training

Evaluation:

• RAGAS: Automated Evaluation of Retrieval Augmented Generation

Diffusion:

• Scalable Diffusion Models with Transformers

State Space Models:

• Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Distributed Systems

• Spanner: Google’s Globally-Distributed Database
• MapReduce: Simplified Data Processing on Large Clusters
• ZooKeeper: Wait-free coordination for Internet-scale systems