Linear Algebra

Introduction

Linear algebra studies vectors, vector spaces, matrices, linear transformations, systems of linear equations, and structures such as eigenvalues and singular values. It is a foundational language for Mathematics, Physics, Engineering, Computer science, Data science, Economics, statistics, optimization, and machine learning.

In this university-level aiMOOC, you will connect symbolic calculations with geometry and applications. You will learn not only how to carry out procedures such as Gaussian elimination, but also why those procedures reveal subspaces, rank, invertibility, projection, and change of coordinates.

The image above visualizes vector addition. Geometrically, placing vectors head-to-tail or completing a parallelogram gives the same sum.

The video introduces vectors from several complementary viewpoints. As you study, keep moving between coordinates, geometric arrows, and abstract vector-space elements.

Learning Goals

After completing this aiMOOC, you should be able to explain and use vector spaces, linear combinations, span, linear independence, bases, coordinates, matrices, linear transformations, determinants, orthogonality, projections, eigenvalues, eigenvectors, diagonalization, least squares, and the singular value decomposition. You should also be able to connect these ideas to data analysis, dynamical systems, networks, and numerical computation.

Prerequisites and Notation

You should be comfortable with algebraic manipulation, functions, coordinate geometry, and basic summation notation. Calculus is helpful for some applications but is not required for the central theory.

A vector in real coordinate space may be written as $𝐱 = (x_{1}, \dots, x_{n})^{T}$ . A matrix $A \in ℝ^{m \times n}$ has $m$ rows and $n$ columns. The transpose is $A^{T}$ . The identity matrix is $I$ . The zero vector is $𝟎$ . Throughout the course, you should check dimensions before multiplying matrices or applying formulas.

Vectors and Vector Spaces

Vectors, Scalars, and Linear Combinations

A vector can represent displacement, force, velocity, a signal, a polynomial, an image, or a collection of measured features. A scalar is an element of the underlying field, usually $ℝ$ or $ℂ$ . If $𝐯_{1}, \dots, 𝐯_{k}$ are vectors and $c_{1}, \dots, c_{k}$ are scalars, then $c_{1} 𝐯_{1} + \dots + c_{k} 𝐯_{k}$ is a linear combination.

The set of all linear combinations of given vectors is their span. For example, two nonparallel vectors in $ℝ^{2}$ span the whole plane, while two parallel nonzero vectors span only a line through the origin.

Subspaces

A subset $W$ of a vector space $V$ is a subspace if it contains the zero vector and is closed under vector addition and scalar multiplication. Equivalently, every linear combination of vectors in $W$ remains in $W$ .

Important examples include solution sets of homogeneous systems, polynomial spaces of bounded degree, and spaces of functions satisfying linear constraints. A line in $ℝ^{2}$ is a subspace only when it passes through the origin.

Linear Independence, Basis, and Dimension

Vectors $𝐯_{1}, \dots, 𝐯_{k}$ are linearly independent if $c_{1} 𝐯_{1} + \dots + c_{k} 𝐯_{k} = 𝟎$ implies $c_{1} = \dots = c_{k} = 0$ . A basis of a vector space is a linearly independent spanning set. Once a basis is chosen, every vector has a unique coordinate representation in that basis.

The number of vectors in any basis of a finite-dimensional vector space is its dimension. For example, the vectors $(1, 0, 1)^{T}$ and $(0, 1, 1)^{T}$ are independent, but adding $(1, 1, 2)^{T}$ creates a dependent set because the third vector is the sum of the first two.

Matrices and Systems of Linear Equations

Matrices as Data and as Operators

A matrix can store coefficients, data, or the coordinates of a linear map. Addition and scalar multiplication operate entrywise. Matrix multiplication is different: if $A$ is $m \times n$ and $B$ is $n \times p$ , then $A B$ is $m \times p$ . Its entries are $(A B)_{i j} = \sum_{k = 1}^{n} a_{i k} b_{k j} .$

Matrix multiplication represents composition of linear transformations, which explains why order matters. In general, $A B \neq B A$ .

Solving Ax = b by Elimination

A linear system can be written compactly as $A 𝐱 = 𝐛$ . Elementary row operations preserve the solution set. Gaussian elimination transforms the augmented matrix into row-echelon form, while Gauss-Jordan elimination continues to reduced row-echelon form.

For example, $\begin{matrix} x + 2 y & = 5 \\ 3 x + 4 y & = 11 \end{matrix}$ has augmented matrix $[\begin{matrix} 1 & 2 & 5 \\ 3 & 4 & 11 \end{matrix}] .$ Replacing row two by row two minus three times row one gives $[0, - 2 | - 4]$ , so $y = 2$ and then $x = 1$ .

A system may have one solution, infinitely many solutions, or no solution. Pivot positions identify basic variables. Free variables parameterize families of solutions.

Rank, Column Space, and Null Space

The column space of $A$ is the span of its columns and equals the set of all possible outputs $A 𝐱$ . The null space or kernel is $𝒩 (A) = {𝐱 : A 𝐱 = 𝟎} .$

The rank of $A$ is the dimension of its column space. If $A$ has $n$ columns, the rank-nullity theorem states $rank (A) + nullity (A) = n .$

This theorem connects the number of independent output directions with the number of input directions that are collapsed to zero.

Inverses and Factorizations

A square matrix $A$ is invertible if there exists $A^{- 1}$ with $A^{- 1} A = A A^{- 1} = I$ . For square matrices, invertibility is equivalent to full rank, a trivial null space, a nonzero determinant, and the existence of a unique solution to $A 𝐱 = 𝐛$ for every $𝐛$ .

In practical computation, you normally solve a system using a factorization rather than explicitly forming the inverse. For example, elimination leads to an $L U$ factorization, while orthogonalization leads to a $Q R$ factorization.

Linear Transformations and Coordinates

What Makes a Transformation Linear?

A transformation $T : V \to W$ is linear when $T (𝐮 + 𝐯) = T (𝐮) + T (𝐯)$ and $T (c 𝐯) = c T (𝐯)$ for all appropriate vectors and scalars. These two conditions are equivalent to preservation of every linear combination.

Once bases are chosen, every finite-dimensional linear transformation can be represented by a matrix. The columns of that matrix are the coordinate vectors of the transformed basis vectors.

The figure includes several familiar matrix actions such as scaling, rotation, reflection, and shear. Translation is shown in homogeneous coordinates; ordinary translation in $ℝ^{2}$ is affine rather than linear because it does not keep the origin fixed.

Kernel, Image, and Rank-Nullity

For a linear map $T : V \to W$ , the kernel is the set of inputs mapped to zero, while the image is the set of outputs that can actually be reached. If $V$ is finite-dimensional, then $\dim (\ker T) + \dim (im T) = \dim V .$

A linear transformation is one-to-one exactly when its kernel contains only the zero vector. It is onto exactly when its image equals the whole codomain.

Change of Basis

Coordinates depend on a basis, but the underlying vector does not. If the columns of an invertible matrix $P$ are a new basis written in old coordinates, then $[𝐯]_{old} = P [𝐯]_{new} .$ For a linear operator represented by $A$ in the old basis, its matrix in the new basis is $P^{- 1} A P .$ This similarity transformation changes coordinates without changing intrinsic properties such as determinant, trace, or eigenvalues.

Determinants

Geometric Meaning

For a square matrix, the determinant measures signed volume scaling. In two dimensions, $| det A |$ is the area-scaling factor; in three dimensions, it is the volume-scaling factor. A negative determinant reverses orientation, while a zero determinant means that the transformation collapses space into a lower-dimensional set.

Algebraic Properties

For square matrices $A$ and $B$ , $det (A B) = det (A) det (B) .$ Swapping two rows changes the sign of the determinant, multiplying a row by a scalar multiplies the determinant by that scalar, and adding a multiple of one row to another leaves the determinant unchanged. A matrix is invertible exactly when its determinant is nonzero.

For a $2 \times 2$ matrix, $det (\begin{matrix} a & b \\ c & d \end{matrix}) = a d - b c .$ For larger matrices, elimination is usually more efficient than a full cofactor expansion.

Inner Products, Orthogonality, and Least Squares

Dot Products and Orthogonality

In $ℝ^{n}$ , the standard inner product is $𝐱^{T} 𝐲 = \sum_{i = 1}^{n} x_{i} y_{i} .$ The induced norm is $‖ 𝐱 ‖_{2} = \sqrt{𝐱^{T} 𝐱}$ . Two vectors are orthogonal when their inner product is zero.

An orthogonal matrix $Q$ satisfies $Q^{T} Q = I$ . Its columns form an orthonormal basis, and it preserves lengths and angles.

Orthogonal Projection

The projection of $𝐛$ onto the line spanned by a nonzero vector $𝐚$ is ${proj}_{𝐚} 𝐛 = \frac{𝐚^{T} 𝐛}{𝐚^{T} 𝐚} 𝐚 .$ For a subspace with orthonormal basis columns collected in $Q$ , the projection matrix is $Q Q^{T}$ .

Projection is characterized by an orthogonality condition: the error between the original vector and its projection is perpendicular to the target subspace.

Gram-Schmidt and QR Factorization

The Gram-Schmidt process converts a linearly independent set into an orthonormal set spanning the same subspace. Applied to the columns of a full-column-rank matrix $A$ , it produces $A = Q R,$ where $Q$ has orthonormal columns and $R$ is upper triangular. Numerically stable implementations typically use modified Gram-Schmidt or Householder reflections.

Least Squares

When $A 𝐱 = 𝐛$ is inconsistent, least squares chooses $\hat{𝐱}$ to minimize $‖ A 𝐱 - 𝐛 ‖_{2} .$ At the optimum, the residual is orthogonal to the column space of $A$ , giving the normal equations $A^{T} A \hat{𝐱} = A^{T} 𝐛 .$

The normal equations clarify the geometry, but directly forming $A^{T} A$ can worsen numerical conditioning. QR factorization or SVD is generally preferred for robust numerical least-squares computation.

Eigenvalues, Eigenvectors, and Diagonalization

Eigenvectors and Eigenvalues

For a square matrix $A$ , a nonzero vector $𝐯$ is an eigenvector with eigenvalue $λ$ if $A 𝐯 = λ 𝐯 .$ The transformation may stretch, shrink, reverse, or preserve the direction of an eigenvector, but it does not move that vector away from its eigenvector line.

Eigenvalues are found from the characteristic equation $det (A - λ I) = 0.$

For example, $A = (\begin{matrix} 2 & 1 \\ 1 & 2 \end{matrix})$ has eigenvalue $3$ with eigenvector direction $(1, 1)^{T}$ and eigenvalue $1$ with eigenvector direction $(1, - 1)^{T}$ .

Diagonalization and the Spectral Theorem

If a matrix has a basis of eigenvectors, then it can be diagonalized: $A = P D P^{- 1},$ where the columns of $P$ are eigenvectors and $D$ contains the corresponding eigenvalues. Then $A^{k} = P D^{k} P^{- 1},$ which makes repeated application easy to analyze.

Every real symmetric matrix has an orthonormal basis of real eigenvectors. Therefore, the spectral theorem gives $A = Q Λ Q^{T}$ for a real symmetric matrix $A$ , with orthogonal $Q$ and real diagonal $Λ$ .

Dynamical Systems and Stability

In a discrete system $𝐱_{k + 1} = A 𝐱_{k}$ , eigenvalues help determine long-term behavior. Components along eigenvectors are multiplied repeatedly by powers of their eigenvalues. In the differential equation $𝐱^{'} = A 𝐱$ , eigenvalues determine exponential modes through terms such as $e^{λ t}$ .

You should interpret these statements carefully when a matrix is not diagonalizable or is highly nonnormal, because eigenvalues alone may not capture every transient behavior.

Singular Value Decomposition

The SVD of Any Matrix

Every real $m \times n$ matrix has a singular value decomposition $A = U Σ V^{T},$ where the columns of $U$ and $V$ are orthonormal and $Σ$ is diagonal in the rectangular sense with nonnegative singular values.

Geometrically, the SVD decomposes a linear map into an orthogonal change of coordinates, axis-aligned scaling, and another orthogonal change of coordinates. The nonzero singular values are the square roots of the nonzero eigenvalues of $A^{T} A$ .

Rank, Compression, Pseudoinverse, and Conditioning

The number of nonzero singular values equals the rank of $A$ . Keeping only the largest singular values gives a low-rank approximation, a key idea in compression, denoising, latent semantic analysis, and principal component methods.

The Moore-Penrose pseudoinverse uses reciprocal nonzero singular values and extends inverse-like solving to rectangular or rank-deficient matrices. In the Euclidean norm, the ratio of largest to smallest nonzero singular value measures sensitivity through a condition number. A large condition number warns that small input perturbations may cause much larger changes in a computed solution.

Applications and Connections

Data Science and Principal Components

A data matrix places observations and features into rows and columns. After centering the data, principal component analysis identifies orthogonal directions of maximal variance. Computationally, these directions can be obtained from eigenvectors of a covariance matrix or from the SVD of the centered data matrix.

This application shows why basis choice matters: changing to a principal-component basis can concentrate information into fewer coordinates.

Networks and Graphs

Matrices encode networks through adjacency, incidence, and Laplacian matrices. Null spaces can represent conservation laws or connected components, while eigenvalues reveal structural and dynamical properties. These ideas connect Graph theory with linear algebra.

Markov Chains

A finite-state Markov chain can be represented by a stochastic matrix. Under a consistent row- or column-vector convention, repeated matrix multiplication evolves probability distributions. Stationary distributions correspond to eigenvectors associated with eigenvalue one, subject to normalization and suitable assumptions about the chain.

Differential Equations and Control

Systems of linear differential equations use matrix dynamics. Eigenvalues and invariant subspaces help you understand modes, stability, resonance, and long-term behavior. In control theory, matrices describe state evolution, inputs, outputs, controllability, and observability.

Computer Graphics and Geometry

Rotations, reflections, scalings, projections, and coordinate changes are linear transformations represented by matrices. Graphics pipelines often use homogeneous coordinates so that affine operations such as translations can be combined with linear operations through matrix multiplication.

Problem-Solving Strategy

A Conceptual Workflow

When you face a new problem, first identify the spaces, dimensions, and map involved. Then ask which structure matters: a system of equations, a subspace, an orthogonality condition, an invariant direction, or a low-rank approximation. Choose a basis or factorization that exposes that structure. Finally, check the result geometrically, algebraically, and numerically when possible.

A useful habit is to distinguish three layers:

Geometric interpretation: What does the transformation do to directions, lengths, areas, or subspaces?
Algebraic representation: Which matrix equation or factorization represents the structure?
Numerical computation: Which method is stable and efficient for the size and conditioning of the problem?

Common Misconceptions

A matrix is more than a table of numbers. It usually represents a linear map once bases are chosen.

Row operations do not preserve every matrix property. They preserve the solution set of a system, but they may change eigenvalues, eigenvectors, and the determinant unless you track the effects.

The determinant is not merely a formula. Its geometric meaning is signed volume scaling.

Eigenvectors need not form a basis. Diagonalization requires enough linearly independent eigenvectors.

The normal equations are not always the best numerical algorithm. QR or SVD methods are often more stable.

Zero dot product and zero matrix product are different ideas. Orthogonality concerns an inner product, while a zero matrix product may arise for other structural reasons.

Interactive Tasks

Quiz: Test Your Knowledge

What does the span of a set of vectors contain? (All linear combinations of those vectors) (!Only unit vectors) (!Only orthogonal vectors) (!Only the original vectors)

What property must a basis have? (It must be linearly independent and spanning) (!It must contain only eigenvectors) (!It must have determinant one) (!It must contain infinitely many vectors)

What does the rank of a matrix measure? (The dimension of its column space) (!The number of all matrix entries) (!The size of its determinant) (!The number of zero columns only)

When is a square matrix invertible? (When it has full rank) (!When every entry is positive) (!When it is symmetric) (!When its trace is zero)

What does a zero determinant indicate geometrically? (The transformation collapses dimension) (!The transformation preserves every length) (!The transformation is always a rotation) (!The transformation has no matrix representation)

What condition defines orthogonal vectors? (Their inner product is zero) (!Their lengths are equal) (!Their coordinates are positive) (!Their spans are identical)

What does least squares minimize? (The Euclidean norm of the residual) (!The determinant of the data matrix) (!The number of variables) (!The trace of the coefficient matrix)

What equation defines an eigenvector direction? (The matrix maps it to a scalar multiple of itself) (!The matrix maps it to the zero vector in every case) (!The vector has every coordinate equal) (!The vector must be orthogonal to every other vector)

What is guaranteed for every real matrix? (A singular value decomposition) (!A real eigenbasis) (!A nonzero determinant) (!A unique inverse)

Why are QR or SVD methods often preferred for numerical least squares? (They can be more numerically stable) (!They always avoid matrix multiplication) (!They require no arithmetic) (!They make every system exactly solvable)

Memory Game

Vector space	A set closed under vector addition and scalar multiplication
Span	The collection of all linear combinations generated by given vectors
Basis	An independent spanning set that gives unique coordinates
Rank	The dimension of the reachable output space of a matrix
Kernel	The set of inputs mapped to the zero vector
Orthogonal	Having inner product equal to zero
Eigenvector	A nonzero direction preserved up to scaling by a linear operator
Singular value	A nonnegative scale factor appearing in the SVD

Drag and Drop

Match the correct terms.	Topic
Linear independence	No nontrivial linear combination produces the zero vector
Projection	Closest-point representation in a subspace under an inner product
Determinant	Signed volume scaling of a square linear transformation
Diagonalization	Representation using an eigenvector basis and a diagonal matrix
Pseudoinverse	Generalized inverse used for rectangular or rank-deficient problems

...

Crossword Puzzle

Vector	What object can be added and scaled in a vector space?
Basis	What independent spanning set gives unique coordinates?
Kernel	What set contains all inputs mapped to zero?
Rank	What is the dimension of the column space called?
Eigenvector	What nonzero direction is preserved up to scaling by an operator?
Determinant	What scalar measures signed volume scaling?

LearningApps

Cloze Text

A set closed under vector addition and scalar multiplication is a

. The set of all linear combinations of given vectors is their

. An independent spanning set is called a

. The dimension of the column space of a matrix is its

. The inputs mapped to zero form the

. A square matrix is invertible exactly when its determinant is

. Vectors with zero inner product are

. Least squares chooses a solution that minimizes the norm of the

. A preserved direction under a linear map is represented by an

. Every real matrix admits a

. The number of nonzero singular values equals the matrix

. A large condition number indicates possible numerical

.

Open-Ended Tasks

Easy

Vector visualization: Draw two vectors and their sum using both the head-to-tail method and the parallelogram method, then explain why the constructions agree.
Basis experiment: Choose two different bases of the plane, represent the same vector in both bases, and create a labeled image showing how the coordinates change.
Matrix transformation video: Make a short video showing how one two-dimensional matrix transforms a grid, and identify the images of the standard basis vectors.
Linear algebra interview: Interview a student, researcher, engineer, or data professional about one place where matrices or vectors appear in their work, then summarize the example.

Standard

Gaussian elimination project: Create and solve a three-variable linear system, document every row operation, and interpret rank, pivots, and free variables.
Least squares investigation: Collect a small real dataset, fit a line using least squares, plot the residuals, and explain the orthogonality condition behind the fit.
Orthogonality experiment: Implement Gram-Schmidt on three independent vectors, verify the resulting dot products, and discuss the effect of rounding errors.
Campus transformation hunt: Visit or photograph a campus, workplace, or public location where rotations, reflections, projections, grids, or coordinate transformations are visible, then model one example with a matrix.

Advanced

Eigenvalue dynamics study: Compare two discrete dynamical systems with different eigenvalue patterns, simulate repeated matrix powers, and explain the observed long-term behavior.
PCA data project: Center a multivariable dataset, compute principal directions using an eigenvalue method or SVD, and create a visual explanation of dimensionality reduction.
SVD compression experiment: Apply a truncated SVD to a grayscale image or numerical data matrix, compare several retained ranks, and evaluate the trade-off between compression and reconstruction error.
Numerical stability investigation: Construct or find matrices with contrasting condition numbers, solve related systems with two computational approaches, and write a technical report explaining sensitivity and algorithm choice.

Learning Assessment

Model selection assessment: Given several real situations, decide whether each is best modeled by a linear system, projection, eigenvalue problem, or low-rank approximation, and justify every choice.
Rank-nullity assessment: Analyze a linear map from its matrix, determine bases for its kernel and image, verify rank-nullity, and explain what the dimensions mean geometrically.
Transformation assessment: Given a matrix transformation, predict its effect on a basis, area, orientation, and invertibility before computing, then compare prediction with calculation.
Least squares assessment: Derive the orthogonality condition for a least-squares fit and compare solving with normal equations and QR from the perspectives of geometry and numerical reliability.
Spectral assessment: Determine when a matrix can be diagonalized, use the resulting form to analyze repeated powers, and explain how the conclusion changes if a full eigenbasis is unavailable.
SVD transfer assessment: Explain how one SVD can simultaneously reveal rank, important directions, a pseudoinverse, conditioning information, and a low-rank approximation in an applied example.

Evidence of Learning

Evidence of successful learning should show more than procedural speed. You should be able to demonstrate the following:

Knowledge: Explain central definitions, theorems, equivalences, and geometric meanings, including span, basis, rank, determinant, orthogonality, eigenstructure, and SVD.
Skills: Solve systems, compute and interpret subspaces, change coordinates, project vectors, use matrix factorizations, and check dimensions and assumptions.
Products: Produce clear mathematical explanations, annotated calculations, visualizations, computational notebooks or reports, and data-driven models.
Reasoning: Connect algebraic formulas with geometric structure, justify method choices, detect invalid assumptions, and distinguish exact mathematics from numerical approximation.
Transfer: Apply linear algebra to unfamiliar problems in data science, engineering, physics, networks, economics, graphics, or other disciplines.

OERs on the Topic

You can deepen your study with MIT OpenCourseWare: Linear Algebra, which provides lecture videos, notes, problems, and solutions. You can also use 3Blue1Brown linear algebra resources for visual intuition.