Die Druckversion wird nicht mehr unterstützt und kann Darstellungsfehler aufweisen. Bitte aktualisiere deine Browser-Lesezeichen und verwende stattdessen die Standard-Druckfunktion des Browsers.

Deep Learning

Introduction

Deep learning is a branch of machine learning that uses multilayer artificial neural networks to learn useful representations directly from data. Instead of asking a programmer to hand-code every relevant feature, a deep model can learn successive transformations: early layers may detect simple patterns, while later layers combine them into more task-specific representations. Deep learning is especially influential in computer vision, natural language processing, speech technology, scientific modeling, robotics, and generative AI.

At university level, you should understand deep learning as more than a collection of powerful software tools. It is a mathematical and computational framework involving linear algebra, multivariable calculus, probability, optimization, statistical learning, software engineering, and responsible model evaluation. A model can achieve a high benchmark score and still fail when the data distribution changes, when labels encode bias, or when the deployment context differs from the training environment.

The image above shows the layered structure that gives deep learning its name. Depth increases the number of successive transformations that the model can learn, but useful depth depends on architecture, data, optimization, and the task.

This MIT lecture provides a current university-level introduction to deep learning foundations. Use it to compare the mathematical ideas in this course with a full lecture treatment.

Learning Goals

By the end of this aiMOOC, you should be able to explain how deep neural networks compute predictions, derive the role of gradients in learning, distinguish major architecture families, diagnose common training and generalization problems, evaluate models with appropriate metrics, and reason about responsible deployment. You should also be able to design and document a small deep-learning experiment in a reproducible way.

From Neurons to Deep Networks

The Artificial Neuron

A basic artificial neuron receives an input vector $𝐱$ , combines it with a weight vector $𝐰$ and a bias $b$ , and then applies an activation function $ϕ$ :

$z = 𝐰^{T} 𝐱 + b$

$a = ϕ (z)$

The weights determine how strongly each input influences the neuron. The bias shifts the activation threshold. The activation function introduces nonlinearity. Without nonlinear activations, stacking many linear layers would still be equivalent to a single linear transformation.

Common activation functions include ReLU, sigmoid, hyperbolic tangent, GELU, and variants designed for particular architectures. ReLU, defined as $max (0, z)$ , became important because it is simple and often supports more effective gradient-based training than saturating activations in deep feed-forward networks.

The video develops an intuitive and mathematical view of neurons, layers, weights, biases, activations, and learned representations.

Layers and Representation Learning

A layer transforms one vector or tensor into another. In a fully connected layer, every output unit can depend on every input unit. A deep network composes many functions:

$f (𝐱) = f_{L} (f_{L - 1} (\dots f_{2} (f_{1} (𝐱)) \dots))$

This composition lets the network learn hierarchical representations. For an image task, early layers may respond to local edges or textures, intermediate layers may combine them into motifs or parts, and later layers may encode task-relevant concepts. These descriptions are useful intuitions, not guarantees that each hidden unit will correspond to a human-readable concept.

A central idea is representation learning: the model learns internal features that are useful for its objective. This contrasts with pipelines that depend mainly on manually designed features. However, representation learning does not eliminate human choices. Researchers still choose the dataset, target, loss, architecture, preprocessing, augmentation, evaluation protocol, and deployment constraints.

Forward Pass, Loss, and Learning Objective

During the forward pass, input data move through the network to produce predictions. Training then compares predictions with targets using a loss function. For multi-class classification, cross-entropy is common. If the true class is $y$ and the model assigns it probability $p_{y}$ , the single-example loss is:

$L = - \log p_{y}$

For regression, mean squared error is often used when its assumptions fit the problem. Other tasks require other objectives: contrastive learning uses similarity-based objectives, language models often optimize next-token likelihood, and generative models may combine several losses.

The training objective is usually an empirical average over a dataset, sometimes with a regularization term:

$J (θ) = \frac{1}{N} \sum_{i = 1}^{N} L (f_{θ} (x_{i}), y_{i}) + λ R (θ)$

Here $θ$ denotes all trainable parameters. The objective is not the same as the real-world goal. A well-designed project therefore checks whether the mathematical objective, evaluation metric, and actual use case are aligned.

Backpropagation and Optimization

Gradients and the Chain Rule

Training usually requires the gradient of the loss with respect to millions or billions of parameters. Backpropagation computes these derivatives efficiently by applying the chain rule through the computational graph. If a quantity depends on another through a sequence of differentiable operations, derivatives can be propagated backward from the loss toward earlier parameters.

Backpropagation tells you which direction changes the loss locally. An optimizer decides how to use that information to update parameters. The simplest update is gradient descent:

$θ_{t + 1} = θ_{t} - η \nabla_{θ} J (θ_{t})$

The learning rate $η$ controls the step size. Too large a value can make training unstable; too small a value can make learning extremely slow.

Stochastic Optimization

Computing the exact gradient over a huge dataset after every update is usually inefficient. Mini-batch training estimates the gradient from a subset of examples. This makes each update cheaper and introduces noise that can sometimes help optimization.

Widely used optimizers include stochastic gradient descent with momentum, Adam, and AdamW. Their behavior depends on learning-rate schedules, batch size, normalization, weight decay, gradient clipping, initialization, and model scale. There is no universally best optimizer for every task.

When you train a model, inspect learning curves rather than treating training as a black box. A falling training loss with a rising validation loss suggests overfitting. Highly erratic loss may suggest an excessive learning rate, problematic data, unstable numerical behavior, or an implementation error.

Generalization, Regularization, and Data

Deep learning aims to perform well on unseen examples, not merely memorize the training set. Generalization is therefore central. A model with high capacity can fit complex patterns, including noise and accidental correlations.

Useful regularization strategies include weight decay, dropout, data augmentation, early stopping, label smoothing, stochastic depth, and architectural constraints. The right choice depends on the model and data. Increasing model size does not automatically reduce overfitting, and modern deep learning can show behavior that is more complex than the classical bias-variance picture alone suggests.

Data quality is equally important. A dataset may contain duplicate examples, mislabeled samples, hidden shortcuts, demographic imbalance, leakage from the test set, or a collection process that fails to represent deployment conditions. A strong experimental design separates training, validation, and test data and prevents information from the test set from shaping repeated tuning decisions.

This diagram emphasizes an essential limitation: a sophisticated network cannot turn unreliable evidence into infallible conclusions. Model performance must always be interpreted together with data provenance, uncertainty, and deployment context.

Major Deep-Learning Architectures

Multilayer Perceptrons

A multilayer perceptron uses stacked fully connected layers. MLPs are conceptually simple and useful for studying the mathematics of deep networks. They are also components inside many larger architectures. Their limitation is that dense connectivity does not exploit domain structure such as spatial locality in images or sequence order in text.

Convolutional Neural Networks

Convolutional neural networks use learnable filters that are applied across local regions. Parameter sharing makes them efficient for grid-like data and gives them an inductive bias toward local pattern detection. Pooling, striding, normalization, residual connections, and increasingly sophisticated blocks can be combined to build powerful vision models.

Convolutions remain important in computer vision, medical imaging, audio, scientific data, and hybrid architectures. Modern vision systems also use transformers, but CNNs remain a valuable example of how architecture can encode assumptions about the structure of data.

Recurrent Networks and Sequence Models

Recurrent neural networks process sequences while maintaining a hidden state. LSTM and GRU architectures were designed to improve learning over longer dependencies. Recurrent models remain useful in some settings, especially when streaming or stateful computation matters, although transformers have replaced them in many large-scale language and multimodal systems.

Transformers and Attention

A transformer uses attention mechanisms to model relationships among elements in a sequence or set. In self-attention, each token is transformed into query, key, and value representations. Similarity between queries and keys determines how strongly values contribute to the updated representation.

A simplified scaled dot-product attention rule is:

$A t t e n t i o n (Q, K, V) = s o f t m a x (\frac{Q K^{T}}{\sqrt{d_{k}}}) V$

Multi-head attention learns several such interaction patterns in parallel. Positional information is added because pure self-attention does not by itself encode sequence order.

Transformers are now central to large language models and are also used in vision, audio, biology, robotics, and multimodal learning. Their strengths come with costs: training and inference can require substantial compute and memory, and learned outputs can reproduce errors or biases present in training data.

Autoencoders and Generative Models

An autoencoder learns to encode an input into a latent representation and reconstruct the input from that representation. Variational autoencoders introduce a probabilistic latent space. GANs train a generator and discriminator in competition. Diffusion models learn to reverse a progressive noising process. Autoregressive models learn a probability factorization by predicting the next element conditioned on previous context.

These families illustrate a key distinction: discriminative models predict targets from inputs, while generative models learn enough structure to produce or model data distributions. Many modern systems combine discriminative and generative objectives.

Training Deep Networks in Practice

Initialization, Normalization, and Residual Connections

Deep networks can be difficult to optimize because signals and gradients may shrink, grow, or become poorly conditioned across many layers. Careful parameter initialization helps keep activations in workable ranges. Batch normalization and layer normalization stabilize internal statistics in different ways. Residual connections allow a block to learn a change relative to its input and create short paths through very deep networks.

These techniques do not remove the need for diagnosis. You should still inspect gradient norms, activation distributions, training speed, validation behavior, and numerical precision.

Batch Size, Learning Rate, and Schedules

Batch size affects memory use, throughput, gradient noise, and sometimes final generalization. Learning-rate schedules may include warm-up, step decay, cosine decay, or adaptive strategies. Larger batches often require corresponding changes in the learning rate and optimizer configuration.

A practical workflow begins with a small model and a small data subset. First verify that the network can overfit a tiny sample; if it cannot, there may be a bug or an optimization problem. Then scale up gradually while tracking reproducible experiments.

Transfer Learning and Fine-Tuning

Transfer learning reuses representations learned on one task for another. A pretrained network may be frozen as a feature extractor, partially fine-tuned, or fully adapted. In language models, parameter-efficient methods can update only a small subset of additional parameters.

Transfer learning can reduce data and compute requirements, but it also transfers assumptions and biases from the source model. You should document model provenance, licenses, training data information when available, and the limits of the target evaluation.

Evaluation and Experimental Design

A single metric is rarely sufficient. For classification, accuracy may be misleading when classes are imbalanced. Precision, recall, F1 score, ROC curves, precision-recall curves, calibration, and subgroup metrics may reveal different properties. For regression, mean absolute error and mean squared error emphasize different kinds of mistakes. Generative systems require additional evaluation because open-ended outputs may not have one correct target.

Your test set should represent the conditions in which the model will be used. If the deployment distribution changes, benchmark performance can become a poor guide. Robust evaluation may include distribution shifts, corrupted inputs, adversarial cases, uncertainty estimates, ablations, and error analysis.

Statistical reliability matters. When feasible, report confidence intervals, repeat experiments with multiple random seeds, and compare against strong baselines. A small improvement is not automatically meaningful if the variance is large or the evaluation set has been repeatedly tuned against.

This StatQuest video offers a complementary explanation of the main ideas behind neural networks. Compare its simplified examples with the more formal training framework used in this course.

Interpretability, Robustness, and Responsible Use

Deep neural networks can be difficult to interpret because behavior emerges from many interacting parameters. Feature visualization, saliency methods, probing, attribution techniques, mechanistic analysis, and example-based explanations can help, but every explanation method has assumptions and failure modes.

Robustness asks whether the system continues to behave acceptably when inputs are noisy, shifted, manipulated, or outside the training distribution. Security matters because adversarial inputs, data poisoning, model extraction, and prompt-based attacks can exploit weaknesses in machine-learning systems.

Responsible practice also includes fairness, privacy, accessibility, intellectual property, labor impacts, environmental cost, and accountability. These are not separate from technical design. Dataset construction, objective choice, threshold selection, interface design, and monitoring all influence who benefits and who bears risk.

Deep Learning as a Scientific and Engineering Process

A successful deep-learning project is not just a model checkpoint. It is a chain of evidence: a well-defined problem, suitable data, a justified baseline, a transparent training procedure, appropriate metrics, documented experiments, analysis of failures, and a deployment plan.

A reproducible project should record software versions, random seeds, data splits, preprocessing, hyperparameters, hardware assumptions, and evaluation code. Model cards and data documentation can make assumptions and limitations easier to audit.

When results are surprising, seek alternative explanations. Ask whether information leakage, shortcuts, duplicated data, preprocessing artifacts, or evaluation mistakes could explain the apparent gain. Deep learning rewards careful skepticism because highly flexible models can exploit patterns that humans did not intend.

Interactive Tasks

Quiz: Test Your Knowledge

Why are nonlinear activation functions important in deep feed-forward networks? (They allow stacked layers to represent nonlinear functions) (!They guarantee zero training loss) (!They remove the need for weights) (!They make every network interpretable)

What is the main computational purpose of backpropagation? (To compute gradients of the loss with respect to parameters efficiently) (!To select the training examples randomly) (!To convert labels into input features) (!To guarantee a global optimum)

What does the learning rate primarily control during gradient-based optimization? (The size of parameter updates) (!The number of classes in the dataset) (!The dimensionality of the input) (!The number of test examples)

Which practice most directly helps detect overfitting during training? (Comparing training and validation performance) (!Removing the validation set) (!Training only on the easiest examples) (!Evaluating only the final batch)

What architectural idea makes convolutional networks efficient for images? (Local connectivity with shared filters) (!A separate parameter for every pixel pair) (!Removing all nonlinear activations) (!Using only recurrent connections)

What does self-attention in a transformer primarily model? (Relationships among elements of the same input sequence) (!Only the final class label) (!The physical location of the training server) (!A fixed convolution kernel)

What is transfer learning? (Reusing learned representations from one task for another) (!Deleting all pretrained parameters before training) (!Training without any data) (!Replacing evaluation with intuition)

Why can accuracy be misleading on an imbalanced classification problem? (A model can score highly by favoring the majority class) (!Accuracy always equals recall) (!Accuracy requires unlabeled data) (!Accuracy cannot be computed from predictions)

What is a major purpose of a held-out test set? (To estimate performance on data not used for model selection) (!To provide gradients during every update) (!To replace the training set) (!To guarantee fairness in deployment)

Which statement best describes responsible deep-learning evaluation? (It combines technical metrics with analysis of data limits and deployment risks) (!It uses a single benchmark score in every context) (!It ignores subgroup performance when overall accuracy is high) (!It assumes a larger model is always safer)

Memory Game

Backpropagation	Efficient gradient computation through a computational graph
ReLU	Nonlinear activation that returns the positive part of its input
Dropout	Regularization method that randomly deactivates units during training
Convolution	Local operation with shared learnable filters
Attention	Mechanism that weights relationships among representations
Embedding	Dense vector representation of a discrete or structured item
Fine-tuning	Adaptation of a pretrained model to a target task
Calibration	Agreement between predicted confidence and observed frequency

Drag and Drop

Match the correct terms.	Topic
Gradient computation	Backpropagation
Local feature extraction	Convolutional network
Sequence interaction weighting	Self-attention
Reuse of pretrained features	Transfer learning
Protection against memorization	Regularization

...

Crossword Puzzle

Gradient	What vector of partial derivatives guides many optimization updates?
Convolution	What local shared operation is central to classic CNNs?
Embedding	What dense representation often maps tokens into continuous vector space?
Attention	What mechanism weights relationships among elements in a transformer?
Dropout	What regularization method randomly disables units during training?
Backpropagation	What algorithm efficiently applies the chain rule through a network?

LearningApps

Cloze Text

A deep neural network learns a sequence of parameterized

from data. A neuron first forms a weighted sum and then usually applies a nonlinear

. During training, predictions are compared with targets through a

function. Backpropagation uses the chain rule to compute the required

. An optimizer updates model parameters using those gradient signals and a chosen

. A model that performs well on training data but poorly on unseen data may be

. Convolutional networks use shared filters to exploit local

. Transformers use self-attention to model relationships among input

. Transfer learning adapts knowledge from a

model. Reliable deployment also requires evaluation of robustness, fairness, uncertainty, and data

.

Open-Ended Tasks

Easy

Activation function comparison: Plot ReLU, sigmoid, and hyperbolic tangent over the same input range, then explain how their shapes could affect gradient-based learning.
Neural network concept map: Create a one-page concept map linking neurons, layers, weights, activations, loss, gradients, backpropagation, and optimization.
Model error diary: Collect ten incorrect predictions from a public demonstration model and classify the errors into meaningful categories without claiming causes you cannot verify.
Deep learning explainer video: Produce a two-minute video that explains forward propagation and loss to first-year university students using one concrete example.

Standard

Small classifier experiment: Train a compact neural classifier on an openly licensed dataset, report train and validation curves, and document every preprocessing decision.
Regularization study: Compare two regularization strategies on the same architecture and analyze their effects on training loss, validation loss, and generalization.
CNN feature investigation: Train or reuse a small convolutional network and create visual evidence showing how selected internal activations respond to different images.
Practitioner interview: Interview a researcher, engineer, or advanced student about how they detect data leakage, overfitting, and unreliable evaluation in real projects.

Advanced

Optimizer ablation study: Compare at least two optimizers under a controlled protocol, justify the learning-rate choices, repeat runs with multiple seeds, and discuss variance.
Transformer attention investigation: Implement or inspect a small transformer, visualize selected attention patterns, and critically evaluate what those patterns do and do not prove about model reasoning.
Robustness audit: Design a stress test involving distribution shift, corruption, or adversarially chosen inputs and propose measurable criteria for acceptable degradation.
Responsible deployment dossier: Build a technical dossier for a hypothetical high-impact deep-learning application covering data provenance, subgroup evaluation, uncertainty, privacy, security, monitoring, and human oversight.

Learning Assessment

Architecture justification: Given three data modalities and resource constraints, select a suitable neural architecture for each and justify the inductive bias, expected bottlenecks, and evaluation plan.
Training diagnosis: Interpret a set of training and validation curves, propose at least three plausible failure mechanisms, and design experiments that could distinguish among them.
Backpropagation reasoning: Derive the gradient for a small two-layer network and explain how the chain rule connects the loss to an early-layer parameter.
Evaluation redesign: Replace an inadequate accuracy-only evaluation for an imbalanced application with a metric suite and decision protocol that better reflects deployment costs.
Transfer learning decision: Compare training from scratch with fine-tuning a pretrained model under limited data and compute, including possible benefits, biases, and licensing constraints.
Model governance case: Analyze a hypothetical model failure after deployment and assign technical and organizational controls that could reduce the chance or impact of recurrence.

Evidence of Learning

Knowledge: You can explain neurons, activations, losses, gradients, backpropagation, optimization, generalization, regularization, convolution, recurrence, attention, transfer learning, and evaluation in mathematically informed language.

Skills: You can design controlled experiments, read learning curves, choose metrics, implement or configure a small neural model, diagnose common failure patterns, and communicate uncertainty.

Products: Strong evidence includes reproducible notebooks or code, experiment logs, model cards, visualizations, error analyses, short technical reports, and clearly attributed media or presentations.

Transfer achievements: You can apply deep-learning principles to a new dataset or domain, justify architectural and evaluation choices, compare alternatives under constraints, and identify risks that appear only when a model moves from a benchmark into a real system.

OERs on the Topic

Useful openly accessible study paths include Artificial neural network, Machine learning, Backpropagation, Convolutional neural network, transformers, Representation learning, and explainable AI. For further university-level study, compare textbook treatments with current lecture courses and reproduce small experiments rather than relying on passive viewing alone.