[0.0] - Prerequisites

Colab: exercises | solutions

Please send any problems / bugs on the #errata channel in the Slack group, and ask any questions on the dedicated channels for this chapter of material.

If you want to change to dark mode, you can do this by clicking the three horizontal lines in the top-right, then navigating to Settings → Theme.

Links to all other chapters: (0) Fundamentals, (1) Transformer Interpretability, (2) RL.

Introduction

This page contains important prerequisites for the rest of the material in this chapter. The first page Core Concepts / Knowledge is a long list of important concepts and libraries that you should be familiar with before starting the rest of the material. The second page Einops, Einsum & Tensor Manipulation provides some coding exercises to help you get familiar with the einops and einsum libraries, as well as some exercises on indexing and other aspects of tensor manipulation. We anticipate adding more exercises here in the future, in the style of the second page.

See the homepage for more instructions on how to set up your environment, and install all the dependencies you need to run the exercises.

Content & Learning Objectives

1️⃣ Core Concepts / Knowledge

This section doesn't have exercises; it goes through a list of important concepts with associated reading material and questions to check your understanding. We cover all the core material that you'll need for the rest of the course such as neural networks, linear algebra, probability, and calculus.

Learning Objectives
  • Understand the structure and function of neural networks
  • Learn essential linear algebra concepts like matrix operations and transformations
  • Understand core principles of probability and statistics, including expected value and variance
  • Learn how calculus concepts (particularly differentiation) are applied in optimization tasks
  • Cover some foundational information theory concepts, such as entropy and KL divergence
  • Enhance Python programming skills, focusing on NumPy and PyTorch basics

2️⃣ Einops, Einsum & Tensor Manipulation

These sections will introduce you to a third party library which you'll find very useful for the rest of this course, and will also require you to engage with various tensor operations (something else that will come up repeatedly during this course).

Learning Objectives
  • Understand the basics of Einstein summation convention
  • Learn how to use einops to perform basic tensor rearrangement, and einsum to to perform standard linear algebra operations on tensors