Probability Theory

Introduction

Probability theory is the mathematical language for reasoning about uncertainty. It provides a rigorous framework for describing random experiments, assigning probabilities to events, modelling random quantities, updating information, and studying long-run behavior. In university mathematics, statistics, computer science, engineering, economics, data science, physics, and many other fields, probability is both a theoretical subject and a practical toolkit.

You will begin with probability spaces and the axioms of probability, then move through counting, conditional probability, independence, Bayes' theorem, random variables, distributions, expectation, variance, joint distributions, covariance, and correlation. You will then study two fundamental limit results—the law of large numbers and the central limit theorem—and connect them to simulation, statistical inference, and stochastic modelling.

The diagram above represents a probability space: a collection of events is mapped to numerical probabilities. This abstraction is the central mathematical structure of the subject.

The MIT OpenCourseWare lecture above introduces probability models and axioms at university level. While watching, identify the roles played by outcomes, events, and probability laws.

Learning Goals

By the end of this aiMOOC, you should be able to explain and apply the axioms of probability; construct finite and countable probability models; use combinatorial reasoning; calculate conditional probabilities; apply Bayes' theorem; distinguish independence from mutual exclusivity; work with discrete and continuous random variables; compute expectation and variance; analyze joint distributions, covariance, and correlation; recognize common probability distributions; use the law of large numbers and the central limit theorem appropriately; and design simulations to investigate probabilistic claims.

Why Probability Matters

Probability does not claim that every individual random event is predictable. Instead, it describes structured uncertainty. A single coin toss cannot be predicted with certainty in the standard model, but the distribution of many tosses can be analyzed precisely. A single measurement may contain noise, yet repeated measurements can reveal stable statistical patterns.

Probability is also the mathematical foundation of much of statistics. Statistical inference uses observed data to reason about unknown populations or mechanisms, while probability specifies how data could arise under a model. In machine learning, probability quantifies predictive uncertainty; in reliability engineering, it models component failure; in finance, it models risk; in genetics, it models inheritance; and in communication systems, it models noise and information.

Foundations of Probability

Random Experiments, Outcomes, and Sample Spaces

A random experiment is a process whose exact outcome is uncertain before observation, even though the set of possible outcomes can often be described. An outcome is one possible result, and the sample space is the set of all possible outcomes, commonly denoted by $Ω$ .

For a single six-sided die, one natural sample space is $Ω = {1, 2, 3, 4, 5, 6}$ . For two coin tosses, one may use $Ω = {H H, H T, T H, T T}$ . A sample space can also be uncountable. If you measure a waiting time, a model may use $Ω = [0, \infty)$ .

An event is a set of outcomes. If a die is rolled, the event “an even number appears” is $A = {2, 4, 6}$ . Events can be combined by set operations: intersection corresponds to “and,” union corresponds to “or,” and complement corresponds to “not.”

This Venn-style picture helps you interpret an event as a region inside the full sample space.

Probability Measures and the Kolmogorov Axioms

A probability model is commonly formalized as a triple $(Ω, ℱ, P)$ . Here $Ω$ is the sample space, $ℱ$ is a collection of measurable events, and $P$ is a probability measure.

The modern axiomatic framework requires:

Non-negativity: For every event $A$ , $P (A) \geq 0$ .
Normalization: $P (Ω) = 1$ .
Countable additivity: If $A_{1}, A_{2}, \dots$ are pairwise disjoint events, then $P (⋃_{i} A_{i}) = \sum_{i} P (A_{i})$ .

Many familiar rules follow from these axioms. For example, $P (\emptyset) = 0$ , $P (A^{c}) = 1 - P (A)$ , and $P (A \cup B) = P (A) + P (B) - P (A \cap B)$ .

In finite sample spaces with equally likely outcomes, probability can often be computed by counting: $P (A) = \frac{| A |}{| Ω |}$ . This shortcut is valid only when the outcomes in the chosen sample space really are equally likely.

Counting Techniques

Counting is essential when a probability problem reduces to equally likely finite outcomes. The multiplication principle states that if one stage can occur in $m$ ways and a second stage in $n$ ways for each first-stage choice, then there are $m n$ ordered two-stage possibilities.

For $n$ distinct objects, the number of permutations is $n!$ . The number of ways to choose $k$ objects from $n$ without regard to order is the binomial coefficient $(\binom{n}{k}) = \frac{n!}{k! (n - k)!}$ .

A standard application is the probability of exactly $k$ successes in $n$ independent Bernoulli trials with success probability $p$ : $(\binom{n}{k}) p^{k} (1 - p)^{n - k}$ .

Conditional Probability and Dependence

Conditional Probability

Conditional probability answers the question: how should a probability change when you learn that another event has occurred? For events $A$ and $B$ with $P (B) > 0$ , $P (A ∣ B) = \frac{P (A \cap B)}{P (B)}$ .

The denominator restricts attention to the new relevant universe, namely event $B$ . The numerator keeps only those outcomes that satisfy both $A$ and $B$ .

The diagram illustrates how conditional probability focuses attention on a restricted part of the sample space.

Use the Khan Academy example above to compare a direct calculation with the formal definition of conditional probability.

The multiplication rule follows immediately: $P (A \cap B) = P (A ∣ B) P (B)$ . More generally, a chain of events can be expanded as $P (A_{1} \cap \dots \cap A_{n}) = P (A_{1}) P (A_{2} ∣ A_{1}) \dots P (A_{n} ∣ A_{1} \cap \dots \cap A_{n - 1})$ , provided the conditional probabilities are defined.

Law of Total Probability

Suppose $B_{1}, \dots, B_{n}$ form a partition of the sample space: they are pairwise disjoint, cover $Ω$ , and have positive probabilities. Then for any event $A$ , $P (A) = \sum_{i = 1}^{n} P (A ∣ B_{i}) P (B_{i})$ .

This formula is useful when an event can happen through several mutually exclusive routes. For example, if a product can come from several factories, the total probability of a defect can be computed by conditioning on the factory of origin.

Bayes' Theorem

Bayes' theorem reverses a conditional probability: $P (B_{j} ∣ A) = \frac{P (A ∣ B_{j}) P (B_{j})}{\sum_{i} P (A ∣ B_{i}) P (B_{i})}$ .

In applications, $P (B_{j})$ is often called a prior probability, $P (A ∣ B_{j})$ a likelihood for the observed evidence, and $P (B_{j} ∣ A)$ a posterior probability. This language is especially important in Bayesian statistics.

The visualization above shows Bayes' theorem through proportions. It is useful for understanding why a highly accurate test can still produce a non-obvious posterior probability when the underlying condition is rare.

The 3Blue1Brown video above gives a compact geometric proof of Bayes' theorem. Try to translate each geometric area in the video into a probability expression.

Independence and Mutual Exclusivity

Events $A$ and $B$ are independent when $P (A \cap B) = P (A) P (B)$ . If $P (B) > 0$ , this is equivalent to $P (A ∣ B) = P (A)$ .

Independence means that learning whether one event occurred does not change the probability assigned to the other. By contrast, mutually exclusive events cannot occur together, so $P (A \cap B) = 0$ . Two nontrivial mutually exclusive events with positive probabilities are therefore not independent.

Pairwise independence is weaker than mutual independence. Three events can be independent in every pair while still failing the stronger condition required for all combinations. This distinction matters in advanced models and in carefully designed counterexamples.

Random Variables and Distributions

Random Variables

A random variable is a measurable function from the sample space to the real numbers. It assigns a numerical value to each outcome. The randomness comes from which outcome occurs, not from the function itself.

A random variable is discrete if it takes values in a finite or countable set. It is continuous when it is modeled using a density over an interval or another uncountable domain. Some random variables are mixed, containing both point masses and continuous components.

The cumulative distribution function of any real-valued random variable $X$ is $F_{X} (x) = P (X \leq x)$ . Every CDF is nondecreasing, right-continuous, approaches 0 as $x \to - \infty$ , and approaches 1 as $x \to \infty$ .

Discrete Distributions and Probability Mass Functions

For a discrete random variable $X$ , the probability mass function is $p_{X} (x) = P (X = x)$ . It satisfies $p_{X} (x) \geq 0$ and $\sum_{x} p_{X} (x) = 1$ .

The graph above is an example of a probability mass function: probability is concentrated at specific points rather than spread continuously.

Important discrete distributions include:

Bernoulli distribution: Models a single success-failure trial with success probability $p$ .
Binomial distribution: Counts successes in a fixed number of independent Bernoulli trials.
Geometric distribution: Models the number of trials until the first success under a common convention.
Poisson distribution: Models counts of events under assumptions such as a constant event rate and independent increments in a Poisson process.

Continuous Distributions and Densities

A continuous random variable may have a probability density function $f_{X}$ such that $P (a \leq X \leq b) = \int_{a}^{b} f_{X} (x) d x$ . A density is not itself a probability at a point. For an absolutely continuous random variable, $P (X = x) = 0$ for every fixed $x$ , even though intervals can have positive probability.

The CDF and density are related by $F_{X} (x) = \int_{- \infty}^{x} f_{X} (t) d t$ , and where differentiable, $f_{X} (x) = {F_{X}}^{'} (x)$ .

Common continuous distributions include the uniform, exponential, normal, gamma, and beta distributions.

This MIT OpenCourseWare lecture develops continuous random variables and densities in a formal probability framework.

The Normal Distribution

A normal random variable with mean $μ$ and variance $σ^{2}$ has density $f (x) = \frac{1}{σ \sqrt{2 π}} \exp (- \frac{(x - μ)^{2}}{2 σ^{2}})$ .

Standardization transforms $X \sim N (μ, σ^{2})$ into $Z = \frac{X - μ}{σ} \sim N (0, 1)$ . The normal distribution is central to probability and statistics because of its mathematical properties and its role in the central limit theorem.

The image visualizes familiar probability regions around the mean of a normal distribution.

The StatQuest video above provides an intuitive interpretation of the normal distribution and its parameters.

Expectation, Variability, and Dependence

Expected Value

The expected value is a probability-weighted average and represents the long-run center of a random variable under repeated sampling, when the relevant expectation exists.

For a discrete random variable, $E [X] = \sum_{x} x p_{X} (x)$ . For a continuous random variable with density, $E [X] = \int_{- \infty}^{\infty} x f_{X} (x) d x$ .

Expectation is linear: $E [a X + b Y + c] = a E [X] + b E [Y] + c$ , whenever the expectations exist. Independence is not required for linearity.

More generally, if $g$ is a suitable function, then $E [g (X)] = \sum_{x} g (x) p_{X} (x)$ in the discrete case or $E [g (X)] = \int g (x) f_{X} (x) d x$ in the continuous case.

Variance and Standard Deviation

Variance measures squared deviation from the mean: $Var (X) = E [(X - E [X])^{2}]$ . An equivalent computational identity is $Var (X) = E [X^{2}] - (E [X])^{2}$ .

The standard deviation is $SD (X) = \sqrt{Var (X)}$ , so it has the same physical units as $X$ .

For constants $a$ and $b$ , $Var (a X + b) = a^{2} Var (X)$ . Adding a constant shifts a random variable without changing its variance.

Joint Distributions, Covariance, and Correlation

When several random variables are studied together, their joint distribution specifies how combinations of values occur. From a joint distribution you can obtain marginal distributions by summing or integrating out other variables.

The covariance of $X$ and $Y$ is $Cov (X, Y) = E [(X - E [X]) (Y - E [Y])]$ . It can also be written as $E [X Y] - E [X] E [Y]$ .

The correlation coefficient is $ρ_{X, Y} = \frac{Cov (X, Y)}{σ_{X} σ_{Y}}$ , when both standard deviations are positive. Correlation lies between -1 and 1 and measures linear association after scale normalization.

Independence implies zero covariance when the relevant moments exist, but zero covariance does not generally imply independence. An important exception occurs for jointly normal random variables, where zero covariance does imply independence.

Conditional Expectation

Conditional expectation extends conditional probability to random quantities. The value $E [X ∣ Y]$ is itself a random variable determined by $Y$ . It represents the best mean-square prediction of $X$ based on the information contained in $Y$ under standard conditions.

A central identity is the law of total expectation: $E [X] = E [E [X ∣ Y]]$ . The law of total variance is $Var (X) = E [Var (X ∣ Y)] + Var (E [X ∣ Y])$ . These formulas decompose overall behavior into within-group and between-group components.

Limit Theorems and Long-Run Behavior

Law of Large Numbers

The law of large numbers explains why empirical averages stabilize. In one common weak form, if $X_{1}, X_{2}, \dots$ are independent and identically distributed with finite mean $μ$ , then the sample mean ${\overline{X}}_{n} = \frac{1}{n} \sum_{i = 1}^{n} X_{i}$ converges in probability to $μ$ .

The theorem does not say that short runs must look balanced, nor does it say that deviations are “due” to reverse themselves. It says that, under its assumptions, the average becomes increasingly concentrated near the expected value as the sample size grows.

The plot shows the running average of repeated die rolls approaching the theoretical mean of 3.5 as the number of rolls grows.

Central Limit Theorem

The central limit theorem describes the shape of fluctuations of sums and averages. A common version states that if $X_{1}, \dots, X_{n}$ are independent and identically distributed with mean $μ$ and finite positive variance $σ^{2}$ , then $\frac{\sum_{i = 1}^{n} X_{i} - n μ}{σ \sqrt{n}}$ converges in distribution to a standard normal random variable as $n \to \infty$ .

Equivalently, for large $n$ , the sample mean is approximately normal with mean $μ$ and variance $σ^{2} / n$ . The theorem has many variants, and its assumptions should always be checked before applying a normal approximation.

The sequence of dice-sum distributions above becomes progressively more bell-shaped after suitable centering and scaling.

The StatQuest video above gives an accessible explanation of the central limit theorem. After watching, explain in your own words the difference between the distribution of individual observations and the sampling distribution of their mean.

LLN Versus CLT

The law of large numbers and the central limit theorem answer different questions. The law of large numbers tells you where the sample mean goes: toward the population mean. The central limit theorem describes how the properly scaled error around that mean is distributed asymptotically.

This difference matters in applications. The law of large numbers supports consistency of averages, while the central limit theorem supports approximate confidence intervals, normal approximations, and error quantification.

Modelling, Simulation, and Applications

Building a Probability Model

A useful probability model requires choices. You must specify the possible outcomes, decide which events are relevant, assign probabilities or a stochastic mechanism, and check whether assumptions such as independence, identical distribution, stationarity, or a particular parametric family are defensible.

A model is not the same thing as reality. It is a mathematical representation designed for a purpose. Good modelling therefore includes sensitivity analysis: ask how conclusions change when assumptions are modified.

Monte Carlo Simulation

Monte Carlo simulation uses repeated random sampling to approximate probabilities, expectations, integrals, or distributions that may be difficult to calculate analytically. A basic simulation workflow is:

Define the stochastic model and the quantity of interest.
Generate pseudo-random samples according to the model.
Compute the target statistic for each run.
Aggregate the results.
Quantify Monte Carlo uncertainty and check convergence.

For example, you can approximate $π$ by sampling points uniformly from a square and estimating the fraction that fall inside an inscribed quarter-circle. The estimate improves with more samples, although random error remains finite for any fixed simulation size.

Stochastic Processes

A stochastic process is a collection of random variables indexed by time, space, or another parameter. Examples include random walks, Poisson processes, Markov chains, Brownian motion, queue lengths, asset-price models, and time-dependent reliability states.

A Markov chain has the property that, conditional on the present state, the future evolution does not depend on the past history. Markov models are used in algorithms, population models, operations research, statistical physics, economics, and many other areas.

At a first university level, you should recognize that probability theory extends naturally from single random variables to entire random systems evolving through time.

Probability in Data Science and Scientific Reasoning

Probability helps distinguish variability inherent in a system from uncertainty caused by limited information. In data science, probabilistic models underpin regression, classification, Bayesian inference, graphical models, generative models, and uncertainty quantification.

When applying probability to real data, you should distinguish mathematical randomness from practical sources of uncertainty such as measurement error, missing data, sampling bias, model misspecification, and changing environments. A mathematically correct calculation can still be misleading if its assumptions do not match the data-generating process.

Common Reasoning Errors

Several errors recur in probabilistic reasoning. The gambler's fallacy is the mistaken belief that independent random events must quickly compensate for recent imbalance. The base-rate neglect error occurs when prior probabilities are ignored while interpreting evidence. Confusing correlation with causation can also lead to unsupported conclusions, because probability models describe association unless additional causal assumptions are justified.

A useful habit is to ask: What is the sample space? What exactly is conditioned on? Are events independent? What is the relevant base rate? Is the probability a model assumption, a theoretical consequence, or an empirical estimate?

Interactive Tasks

Quiz: Test Your Knowledge

Which statement is one of the basic probability axioms? (The probability of the sample space is 1) (!Every event has probability greater than 1) (!All events are independent) (!Every sample space is finite)

What does conditional probability P of A given B describe? (The probability of A after restricting attention to B) (!The probability that A and B are mutually exclusive) (!The probability of A before any information is known) (!The variance of A inside B)

When are two events independent? (Their joint probability equals the product of their probabilities) (!They can never occur together) (!They have equal probabilities) (!Their union has probability zero)

What is the purpose of Bayes theorem? (To reverse a conditional probability using prior information) (!To prove that all events are equally likely) (!To calculate only unconditional averages) (!To replace probability with deterministic prediction)

Which object assigns a numerical value to each outcome? (A random variable) (!A sample space) (!A sigma algebra) (!A confidence interval)

What must the probabilities in a discrete probability mass function do? (Sum to 1) (!Sum to 0) (!All equal the same value) (!All exceed 1)

What does variance measure? (The average squared deviation from the mean) (!The number of possible outcomes) (!The probability of the sample space) (!The median of every distribution)

Which statement about covariance is generally true? (Independence implies zero covariance when moments exist) (!Zero covariance always implies independence) (!Covariance must lie between minus 1 and 1) (!Covariance is unchanged by every rescaling)

What does the law of large numbers describe? (The stabilization of sample averages near the expected value) (!The exact outcome of the next trial) (!The equality of all probability distributions) (!The impossibility of rare events)

What does a common central limit theorem approximate for large samples? (The standardized sum by a normal distribution) (!Every individual observation by a uniform distribution) (!Every probability by exactly one half) (!Every sample space by a finite set)

Memory Game

Sample space	Set of all possible outcomes in a model
Event	Set of outcomes to which a probability may be assigned
Independence	Property that one event does not change the probability of another
Expectation	Probability-weighted average of a random variable
Variance	Expected squared deviation from the mean
Likelihood	Probability model for observed evidence under a specified hypothesis
Posterior	Updated probability after incorporating evidence
Convergence	Mathematical notion describing approach of random quantities or distributions toward a limit

Drag and Drop

Match the correct terms.	Topic
Conditional probability	Updates the probability of an event given information
Bayes theorem	Reverses conditioning by combining likelihood and prior probability
Expected value	Describes the mean of a random variable
Law of large numbers	Explains stabilization of sample averages
Central limit theorem	Explains asymptotic normality of standardized sums

Crossword Puzzle

Outcome	What is a single possible result of a random experiment?
Independence	What property means that learning one event does not change the probability of another?
Expectation	What term names the probability-weighted mean of a random variable?
Variance	What quantity measures expected squared deviation from the mean?
Bayesian	What adjective describes inference that updates prior probabilities with evidence?
Convergence	What term describes approaching a mathematical limit?

LearningApps

Cloze Text

A probability model begins with a

containing all possible outcomes. An event is assigned a value by a

. Conditional probability restricts attention to information that is already

. Bayes theorem combines a likelihood with a

. A discrete random variable is described by a probability

. A continuous random variable may be described by a probability

. The expected value represents a probability-weighted

. Variance measures squared dispersion around the

. The law of large numbers explains why sample averages tend to

. The central limit theorem explains why suitably standardized sums often approach a

.

Open-Ended Tasks

Easy

Coin-toss experiment: Toss a coin at least 50 times, record the running proportion of heads, create a graph, and explain how the result relates to the law of large numbers.
Probability vocabulary poster: Create an image or one-page infographic that connects sample space, outcome, event, complement, union, intersection, and probability using one consistent example.
Conditional probability interview: Interview a classmate about a real situation involving updated information, translate the situation into two events, and explain which conditional probability is relevant.
Simulation video: Produce a short video showing a simple random experiment and explain the difference between theoretical probability and empirical relative frequency.

Standard

Bayes theorem case study: Choose a screening, spam-filtering, quality-control, or diagnostic example, define prior probability and likelihood clearly, compute a posterior probability, and interpret the result.
Distribution comparison: Simulate one binomial and one Poisson distribution, visualize both, and write a short explanation of when each model is appropriate.
Expected value project: Design a simple game with numerical payoffs, calculate its expected value and variance, simulate repeated play, and compare theory with simulation.
Probability in practice visit: Visit a laboratory, analytics team, insurance office, quality-control setting, weather service, or another relevant place in person or virtually and document how uncertainty is quantified.

Advanced

Dependence investigation: Construct or find a data set with two variables, estimate covariance and correlation, discuss whether independence is plausible, and explain why correlation alone does not establish causation.
Central limit theorem experiment: Simulate sample means from at least two non-normal distributions for several sample sizes, create comparative plots, and analyze when the normal approximation becomes useful.
Model criticism report: Select a published probabilistic claim, reconstruct the underlying assumptions, identify possible violations such as selection bias or dependence, and propose a more defensible model.
Stochastic process project: Build a simulation of a random walk or finite Markov chain, produce a written or video explanation of its transition mechanism, and investigate a long-run property.

Learning Assessment

Probability model construction: Given an unfamiliar random experiment, define a defensible sample space and event structure, justify the probability assignments, and derive at least two nontrivial event probabilities.
Bayesian reasoning assessment: Analyze a low-base-rate detection problem, compute the posterior probability, and explain how changing the base rate affects the conclusion.
Distribution selection assessment: Compare binomial, Poisson, exponential, and normal models for four applied scenarios and justify each selection using assumptions rather than surface resemblance.
Expectation and variance transfer: Derive the expectation and variance of a transformed random variable and explain how the transformation changes location and spread.
Dependence reasoning assessment: Determine whether a supplied joint distribution implies independence, compute covariance, and explain why the two conclusions are or are not equivalent.
Limit theorem application: Decide whether the law of large numbers, the central limit theorem, both, or neither supports a proposed approximation and defend your choice by checking assumptions.
Simulation validation: Design a Monte Carlo experiment for a probability that is difficult to compute directly, estimate its uncertainty, and compare the result with any available analytical benchmark.

Evidence of Learning

Evidence of learning should show more than correct formulas. It should demonstrate that you can connect formal definitions with modelling choices and applications.

Knowledge: Accurate understanding of probability spaces, axioms, conditional probability, Bayes theorem, independence, random variables, distributions, expectation, variance, dependence, and limit theorems.
Skills: Ability to translate verbal problems into mathematical events, derive probabilities, compute expectations and variances, condition on information, analyze joint distributions, and justify approximations.
Products: Completed simulations, graphs, written derivations, visual explanations, case studies, data analyses, and a stochastic-process project.
Reasoning: Clear distinction between mutual exclusivity and independence, density and probability, zero covariance and independence, empirical frequency and theoretical probability, and LLN versus CLT.
Transfer: Ability to recognize probabilistic structures in new contexts such as reliability, machine learning, finance, epidemiology, engineering, quality control, and scientific measurement.
Reflection: Ability to identify modelling assumptions, uncertainty sources, potential biases, and limits of interpretation.

OERs on the Topic

You can deepen your study with these open university resources:

MIT OpenCourseWare: Introduction to Probability: University-level lectures, notes, and exercises on probability models, conditioning, random variables, and limit theorems.
MIT OpenCourseWare: Probability and Random Variables: Undergraduate mathematics materials covering counting, distributions, expectation, variance, conditioning, and major limit results.