Zum Inhalt springen

English:Mathematical Statistics

Aus MOOCsWiki Staging
aiMOOC-Siegel

Mathematical Statistics



Introduction

Mathematical statistics develops the mathematical foundations of statistical reasoning. It uses probability theory, analysis, linear algebra, optimization, and computation to turn data into statements about unknown features of a population or data-generating process. In this aiMOOC, you will move from probability models to estimators, sampling distributions, confidence intervals, hypothesis tests, asymptotic theory, resampling, and Bayesian ideas.

The central question is not merely "What number did the sample produce?" but "What can you justify about an unknown quantity when your evidence is random?" Mathematical statistics makes that question precise by specifying a model, identifying a target parameter or estimand, choosing a statistical procedure, and quantifying the procedure's uncertainty and long-run performance.

The normal density is a useful reference model throughout statistical theory, but mathematical statistics is not limited to normal data. Discrete models, heavy-tailed models, exponential families, nonparametric methods, and simulation-based procedures all belong to the subject.

The MIT OpenCourseWare lecture above introduces the role of mathematical theory in understanding statistical procedures.


Intended Learners and Prerequisites

This course is designed for university students who have completed introductory calculus and probability. Familiarity with derivatives, integrals, sums, limits, basic matrix algebra, and common probability distributions will help you work through the derivations. Some sections also use the language of convergence of random variables.

By the end of the course, you should be able to:

  1. Statistical model: Translate a data problem into a probabilistic model with clearly identified assumptions.
  2. Point estimation: Construct estimators and compare bias, variance, mean squared error, consistency, sufficiency, and efficiency.
  3. Likelihood function: Derive and analyze likelihoods, score functions, Fisher information, and maximum likelihood estimators.
  4. Sampling distribution: Derive or approximate the distribution of a statistic under repeated sampling.
  5. Confidence interval: Construct and interpret interval estimators using pivots, asymptotics, or resampling.
  6. Statistical hypothesis testing: Formulate null and alternative hypotheses and analyze Type I error, Type II error, power, and p-values.
  7. Asymptotic theory: Apply the law of large numbers, central limit theorem, Slutsky's theorem, and the delta method.
  8. Bayesian inference: Contrast frequentist procedures with posterior inference and decision-theoretic reasoning.


Probability Models and Statistical Inference

A statistical model is a family of probability distributions indexed by an unknown parameter. Write the model as 𝒫={Pθ:θΘ}, where Θ is the parameter space. If the observations have density or probability mass function f(x;θ), then the unknown θ determines which member of the model generated the data.

A parameter is a fixed but unknown feature of the model, such as a population mean μ, variance σ2, event probability p, or regression coefficient. A statistic is any function of the observed sample that does not depend on unknown parameters. An estimator is a statistic used to estimate a parameter, while an estimate is the numerical value obtained after the data have been observed.


Random Samples and Independence

A standard mathematical idealization is an independent and identically distributed sample,

X1,,XniidPθ.

Independence makes the joint density factorize, while identical distribution means that the same model applies to each observation. These assumptions are powerful, but you should not adopt them automatically. Time series, clustered samples, repeated measurements, spatial data, and adaptive experiments often require dependence structures.

For an iid sample, the joint density is

fθ(x1,,xn)=i=1nf(xi;θ).

This simple product is the starting point for likelihood-based inference.


Distribution Functions, Expectation, and Variability

A distribution can be summarized by its cumulative distribution function F(x)=P(Xx). When densities exist, F(x)=f(x). Expectations such as Eθ[X] and variances such as Varθ(X) describe model properties, while sample means and sample variances are random statistics whose distributions depend on the sampling process.

For iid data with finite variance,

E[X¯]=μ,Var(X¯)=σ2n.

The factor 1/n explains why averages become more stable as sample size grows.


Sampling Distributions and Large-Sample Theory

A sampling distribution is the probability distribution of a statistic over hypothetical repetitions of the sampling process. It is the bridge between probability theory and statistical inference. When you report a standard error, confidence interval, or p-value, you are using information about a sampling distribution, either exactly or approximately.


Law of Large Numbers

Under standard conditions, the law of large numbers states that the sample mean approaches the population mean:

X¯nPμ.

This is a statement of consistency for the sample mean. It says that large samples increasingly concentrate around the target, not that any finite sample must be close to the truth.


Central Limit Theorem

If X1,,Xn are iid with mean μ and finite positive variance σ2, then the classical central limit theorem gives

n(X¯nμ)σdN(0,1).

This result explains why normal approximations occur throughout statistics even when the individual observations are not normally distributed.

The image illustrates how sums of independent dice become increasingly bell-shaped after centering and rescaling.

The central limit theorem is an asymptotic result. Its practical accuracy depends on the underlying distribution, the statistic, and the sample size; highly skewed or heavy-tailed data can require much larger samples than nearly symmetric light-tailed data.


Slutsky's Theorem and the Delta Method

Slutsky's theorem allows consistent estimators to replace unknown constants in many limiting distributions. For example, if SnPσ, then

n(X¯nμ)SndN(0,1).

The delta method transfers asymptotic normality through a differentiable function. If

n(Tnθ)dN(0,V)

and g is differentiable at θ, then

n(g(Tn)g(θ))dN(0,[g(θ)]2V).

This is useful for transformed parameters such as log odds, risk ratios, or nonlinear functions of estimated coefficients.


Point Estimation

A point estimator compresses the sample into a proposed value for an unknown parameter. Good estimators are not judged only by one observed value; they are judged as random procedures.


Bias, Variance, and Mean Squared Error

For an estimator θ̂,

Biasθ(θ̂)=Eθ[θ̂]θ.

Its mean squared error is

MSEθ(θ̂)=Eθ[(θ̂θ)2]=Varθ(θ̂)+Biasθ(θ̂)2.

A small bias can sometimes be acceptable if it yields a much larger reduction in variance. This bias-variance tradeoff is a central idea in modern statistics.


Consistency and Efficiency

An estimator is consistent if θ̂nPθ. Consistency is asymptotic: it concerns what happens as the sample size tends to infinity. An estimator can be unbiased but noisy, biased but consistent, or both unbiased and consistent.

Within a class of estimators, an efficient estimator has relatively small variance or asymptotic variance. Under regularity conditions, the Cramér–Rao bound gives

Varθ(θ̂)1In(θ)

for unbiased estimators, where In(θ) is the Fisher information in the sample.


Sufficiency

A statistic T(X) is sufficient for θ if, once T is known, the remaining sample contains no additional information about θ within the model. The Fisher-Neyman factorization criterion states that T is sufficient when the joint density can be written as

fθ(x)=gθ(T(x))h(x).

Sufficiency is especially important in exponential families, where low-dimensional sufficient statistics often summarize all likelihood information about the parameter.


Likelihood and Maximum Likelihood Estimation

For observed data x=(x1,,xn), the likelihood function views the joint density as a function of the parameter:

L(θ;x)=i=1nf(xi;θ).

The data are fixed in this expression; θ varies. The log-likelihood

(θ)=logL(θ;x)

turns products into sums and is usually easier to differentiate and optimize.

The score function is U(θ)=(θ)/θ. The Fisher information can be written, under regularity conditions, as

In(θ)=Eθ[U(θ)2]=Eθ[(θ)].


Maximum Likelihood Estimator

A maximum likelihood estimator is any parameter value that maximizes the likelihood:

θ̂MLEargmaxθΘL(θ;x).

When the maximum is in the interior of the parameter space and the likelihood is smooth, solving the score equation U(θ)=0 often identifies candidates, but boundary solutions and multiple local maxima must also be checked.

This Poisson example shows a likelihood curve whose maximum identifies the maximum likelihood estimate.

Maximum likelihood has attractive large-sample properties under regularity conditions: consistency, asymptotic normality, and asymptotic efficiency. Those claims are not automatic; non-identifiability, parameters on boundaries, model misspecification, or irregular likelihoods can invalidate standard results.


Method of Moments

The method of moments equates sample moments to theoretical moments. If Eθ[X]=m(θ), a one-parameter moment estimator may solve

X¯=m(θ).

For multi-parameter models, several moments can be matched. Method-of-moments estimators are often easy to compute and can provide useful starting values for likelihood optimization, although they need not be efficient.


Worked Example: Bernoulli Model

Suppose X1,,XniidBernoulli(p), where p(0,1). Let S=iXi. Then

L(p)=pS(1p)nS.

The log-likelihood is

(p)=Slogp+(nS)log(1p).

Differentiating gives

(p)=SpnS1p.

For 0<S<n, setting the score to zero gives

p̂MLE=Sn=X¯.

The estimator is unbiased, because E[X¯]=p, and

Var(p̂)=p(1p)n.

The Fisher information is

In(p)=np(1p),

so p̂ attains the Cramér-Rao bound in this model. The central limit theorem also gives the approximation

n(p̂p)N(0,p(1p))

for large samples.

This example demonstrates how likelihood, unbiasedness, information, efficiency, and asymptotic normality can describe the same estimator from different viewpoints.


Confidence Intervals

A confidence interval is a random interval constructed by a procedure with a specified long-run coverage probability. A 95% frequentist confidence procedure is designed so that, under repeated sampling from the assumed model, approximately 95% of intervals produced by the procedure contain the fixed true parameter.

It is important to distinguish this repeated-sampling claim from a probability statement about a fixed parameter after a particular frequentist interval has been observed.

This image uses 50% intervals to make repeated-sampling coverage visually easy to see: some random intervals cover the true mean and some do not.


Pivotal Quantities

A pivot is a function of the data and parameter whose sampling distribution does not depend on unknown parameters. Pivots can often be inverted to produce exact confidence intervals.

For normal data with unknown mean and variance,

T=X¯μS/ntn1.

Therefore an exact 100(1α)% confidence interval for μ is

X¯±t1α/2,n1Sn.

The Student t family has heavier tails than the standard normal distribution for small degrees of freedom and converges toward the standard normal as the degrees of freedom increase.


Confidence Interval for a Normal Variance

If the data are iid normal, then

(n1)S2σ2χn12.

This pivot yields an exact interval for σ2:

[(n1)S2χ1α/2,n12,(n1)S2χα/2,n12].

The result depends strongly on the normality assumption.


Hypothesis Testing

A statistical test converts data into a decision rule about competing claims. A null hypothesis H0 specifies the default model or parameter set, while an alternative H1 describes departures of interest.

A Type I error occurs when you reject H0 even though it is true. The significance level α controls the probability of this error under the null. A Type II error occurs when you fail to reject H0 for a parameter value in the alternative. The power of a test at a particular alternative is the probability that the test rejects H0 when that alternative is true.


p-Values

A p-value is computed under the null hypothesis. It is the probability, assuming H0 and the specified model are true, of obtaining a test statistic at least as incompatible with H0 as the observed statistic, according to the chosen test. It is not the probability that H0 is true and it is not, by itself, a measure of practical importance.

Statistical significance should be interpreted together with effect sizes, uncertainty intervals, study design, model assumptions, and the costs of false positive and false negative decisions.


Neyman-Pearson Lemma

For testing a simple null H0:θ=θ0 against a simple alternative H1:θ=θ1, the Neyman–Pearson lemma states that a likelihood-ratio rejection region gives the most powerful test of a specified size. This result motivates many likelihood-based tests and clarifies why likelihood ratios compare how well two parameter values explain the observed data.


Likelihood-Ratio Tests

For a null parameter set Θ0Θ, define

Λ(x)=supθΘ0L(θ;x)supθΘL(θ;x).

Small values of Λ indicate that restricting the model to the null greatly reduces the best attainable likelihood. Under regularity conditions, Wilks' theorem gives an asymptotic chi-square distribution for

2logΛ.

The image illustrates a right-tailed chi-square test in which the observed test statistic lies beyond the critical value.


Worked Example: Normal Mean and Variance

Suppose X1,,XniidN(μ,σ2), with both parameters unknown. The maximum likelihood estimator of the mean is

μ̂=X¯.

The maximum likelihood estimator of the variance is

σ̂MLE2=1ni=1n(XiX¯)2.

This variance estimator is biased downward. The unbiased sample variance uses denominator n1:

S2=1n1i=1n(XiX¯)2.

The pair (iXi,iXi2) is sufficient for (μ,σ2). Normal theory also provides exact pivots:

X¯μS/ntn1,(n1)S2σ2χn12.

These identities explain why Student t and chi-square distributions are fundamental to classical inference for normal samples.


Resampling and the Bootstrap

When exact sampling distributions are unavailable, you can approximate uncertainty computationally. The nonparametric bootstrap treats the empirical distribution of the observed sample as an estimate of the population distribution.

A basic bootstrap workflow is:

  1. Draw a sample of size n with replacement from the observed data.
  2. Compute the statistic of interest on that resample.
  3. Repeat the resampling many times to approximate the statistic's sampling distribution.
  4. Use the resulting bootstrap distribution to estimate standard errors, bias, or confidence intervals.

Bootstrap methods are powerful but not automatic. They can fail or require modification for very small samples, extreme-value statistics, strong dependence, parameters on boundaries, or estimators with irregular asymptotic behavior. Time series and clustered data require resampling schemes that respect dependence.


Bayesian Inference and Decision Theory

In Bayesian inference, the parameter is assigned a prior distribution π(θ). The likelihood updates the prior through Bayes' theorem:

π(θx)=L(θ;x)π(θ)L(u;x)π(u)du.

The resulting posterior distribution quantifies uncertainty about the parameter conditional on the model, prior, and observed data.

A conjugate example arises for Bernoulli data. If pBeta(a,b) and the sample contains S successes and nS failures, then

pxBeta(a+S,b+nS).

Bayesian credible intervals and frequentist confidence intervals answer different probability questions. A posterior credible interval assigns posterior probability to parameter values under the chosen prior and likelihood. A frequentist confidence interval is calibrated by repeated sampling under a statistical model.

Decision theory adds a loss function L(θ,a) for action a. A Bayesian decision minimizes posterior expected loss. Frequentist decision theory instead evaluates procedures through risk functions over repeated samples. These frameworks help make the consequences of statistical decisions explicit.


Model Assumptions, Diagnostics, and Responsible Inference

A mathematically elegant procedure can still give misleading conclusions if its assumptions do not match the data-generating process. Before interpreting inferential results, ask whether the sampling mechanism, independence assumptions, distributional form, parameter identifiability, and measurement process are plausible.

You should also separate statistical significance from scientific or practical importance. A tiny effect can be statistically detectable in a very large sample, while an important effect can remain uncertain in a small sample. Report effect sizes and uncertainty, inspect sensitivity to modeling choices, and document exclusions or transformations.

When data come from observational studies, statistical association alone does not establish causation. Causal claims require additional design or identification assumptions, such as randomization, valid instruments, or carefully justified causal models.


A Strategy for Solving Mathematical Statistics Problems

When you face a new inference problem, use the following reasoning sequence:

  1. Model specification: State the sample space, probability model, parameter space, and assumptions.
  2. Target parameter: Identify exactly what quantity you want to learn.
  3. Statistic construction: Choose an estimator, test statistic, pivot, or posterior quantity.
  4. Sampling behavior: Derive an exact distribution or justify an approximation.
  5. Uncertainty quantification: Compute a standard error, confidence interval, credible interval, or power function.
  6. Assumption check: Identify conditions under which the result is valid and discuss sensitivity.
  7. Interpretation: Translate the mathematical result back to the substantive question without overstating what the procedure proves.


Interactive Tasks


Quiz: Test Your Knowledge

Which statement best defines a statistic? (A function of the observed sample that contains no unknown parameter) (!A fixed unknown feature of the population) (!A probability assigned to a hypothesis after observing data) (!A rule that must always be unbiased)




What does consistency of an estimator mean? (The estimator converges in probability to the target parameter as sample size grows) (!The estimator is exactly correct for every sample) (!The estimator has zero variance for every sample size) (!The estimator has a normal distribution for every model)




What does maximum likelihood estimation choose? (A parameter value that maximizes the likelihood of the observed data) (!A sample that minimizes the population variance) (!A prior distribution with the largest mean) (!A test with zero Type I error)




Which quantity combines variance and squared bias? (Mean squared error) (!Fisher information) (!Statistical power) (!Likelihood ratio)




What does the central limit theorem typically approximate? (The standardized distribution of a sample mean for large samples) (!The exact distribution of every statistic for every sample size) (!The posterior distribution of any parameter) (!The probability that a null hypothesis is true)




What is the usual frequentist meaning of a confidence level? (The long run coverage rate of the interval procedure under the model) (!The probability that the fixed parameter changes between samples) (!The proportion of observations that lie inside every interval) (!The probability that the statistical model is correct)




What is a Type I error? (Rejecting a true null hypothesis) (!Failing to reject a false null hypothesis) (!Estimating a parameter with positive bias) (!Using a small sample in a normal model)




What does statistical power measure? (The probability of rejecting the null at a specified alternative) (!The probability that the null hypothesis is true) (!The numerical size of an estimated effect) (!The number of parameters in a model)




What does a sufficient statistic preserve within a model? (All sample information about the parameter represented in the likelihood) (!Every raw observation in its original order) (!Only the sample size and nothing else) (!The prior probability of each parameter value)




What is the defining operation in a basic nonparametric bootstrap? (Resampling observed data with replacement) (!Resampling observed data without replacement only once) (!Replacing the likelihood with a normal density) (!Discarding observations farther than two standard deviations away)





Memory Game

Parameter Fixed but unknown feature of a statistical model
Statistic Function of observed data containing no unknown parameter
Bias Difference between an estimator's expectation and its target
Consistency Convergence of an estimator toward its target as sample size grows
Sufficiency Property of retaining all model information about a parameter
Power Probability that a test rejects the null at a specified alternative





Drag and Drop

Match the correct terms. Topic
Maximum likelihood estimator Parameter value that maximizes the observed-data likelihood
Central limit theorem Large-sample normal approximation for standardized sums or means
Confidence interval Random interval procedure calibrated by repeated-sampling coverage
Likelihood-ratio test Test based on comparing restricted and unrestricted maximum likelihoods
Bootstrap distribution Empirical distribution of a statistic across resampled datasets





Crossword Puzzle

Likelihood What function treats the observed data as fixed and varies the model parameter?
Estimator What statistic is designed to approximate an unknown parameter?
Variance What quantity measures squared dispersion around an expectation?
Sufficiency What property means a statistic retains all likelihood information about a parameter?
Bootstrap What resampling method repeatedly samples observed data with replacement?
Power What is the probability that a test rejects the null at a specified alternative?





LearningApps


Cloze Text

Complete the text.
A statistical model is indexed by an unknown

. A function of the observed sample that contains no unknown parameter is a

. The likelihood is maximized by the

. An estimator that converges in probability to its target is called

. The central limit theorem often produces an approximate

sampling distribution after standardization. A frequentist confidence procedure is calibrated through repeated-sampling

. A Type I error occurs when a true null hypothesis is

. The probability of rejection at a specified alternative is the test's

. The nonparametric bootstrap resamples observed data with

. In Bayesian inference, prior information is updated to form a

distribution.




Open-Ended Tasks


Easy

  1. Sampling Distribution Simulation: Simulate repeated samples from a skewed distribution, plot the sample means for several sample sizes, and explain how the shape and spread change.
  2. Estimator Comparison Poster: Create a one-page visual explanation of bias, variance, and mean squared error using two hypothetical estimators.
  3. Likelihood Sketch: Choose a simple Bernoulli or Poisson sample, compute its likelihood on a grid of parameter values, and mark the maximizing value.
  4. Confidence Interval Explanation: Write a 300-word explanation for a first-year university student that distinguishes a confidence interval from a probability statement about a fixed parameter.


Standard

  1. Bernoulli Inference Notebook: Build a small computational notebook that derives the Bernoulli maximum likelihood estimator, verifies its bias and variance by simulation, and compares theoretical and simulated standard errors.
  2. Central Limit Theorem Video: Produce a short video that demonstrates the central limit theorem for at least two non-normal parent distributions and discusses when convergence appears slow.
  3. Hypothesis Test Investigation: Design a simulation that estimates Type I error and power for a one-sample test across several effect sizes and sample sizes, then interpret the resulting power curves.
  4. Bootstrap Confidence Interval Study: Compare an asymptotic confidence interval with a bootstrap interval for a statistic such as a median, and describe conditions under which the two methods disagree.


Advanced

  1. Sufficiency Proof Project: Choose an exponential-family model, identify a sufficient statistic using the factorization criterion, and explain what information reduction the statistic achieves.
  2. Fisher Information Investigation: Derive the Fisher information and Cramér-Rao bound for a one-parameter model, then compare the bound with the variance of a natural estimator.
  3. Likelihood-Ratio Research Note: Derive a likelihood-ratio test for a simple parametric problem, justify its reference distribution, and evaluate its finite-sample behavior by simulation.
  4. Statistical Interview and Case Analysis: Interview a researcher who uses inferential statistics, reconstruct one of their analyses as a model-estimator-uncertainty pipeline, and critically assess assumptions, interpretation, and possible sources of error.



Learning Assessment

  1. Model-to-Inference Assessment: Given a new dataset description, specify a plausible statistical model, identify the parameter of interest, choose an estimator, and justify how you would quantify its uncertainty.
  2. Estimator Risk Assessment: Compare two estimators with different bias and variance patterns and determine which you would prefer under squared-error loss for several parameter values.
  3. Asymptotic Reasoning Assessment: Starting from a central limit theorem result, use Slutsky's theorem or the delta method to derive an approximate distribution for a transformed estimator and explain every assumption.
  4. Testing and Power Assessment: Design a level-controlled hypothesis test for a stated scientific question, derive or approximate its power, and explain how sample size and effect size influence the result.
  5. Likelihood Assessment: Derive the likelihood, score, maximum likelihood estimator, and observed or expected information for a one-parameter model, then discuss regularity conditions needed for asymptotic normality.
  6. Interval Interpretation Assessment: Critique several statements about confidence intervals, p-values, and credible intervals, correcting any statement that confuses repeated-sampling and posterior probability.
  7. Resampling Assessment: Decide whether an ordinary bootstrap is appropriate for a given dataset with dependence or extreme statistics, and propose a more suitable resampling strategy when necessary.




Evidence of Learning

Important evidence of learning includes:

  1. Knowledge: You can define models, parameters, statistics, estimators, likelihood, sufficiency, Fisher information, confidence procedures, hypothesis tests, power, asymptotic distributions, and posterior distributions.
  2. Mathematical skills: You can derive likelihoods, differentiate log-likelihoods, compute expectations and variances, manipulate pivots, and justify limiting distributions.
  3. Computational skills: You can simulate sampling distributions, verify theoretical approximations, implement bootstrap procedures, and inspect the numerical behavior of estimators and tests.
  4. Products: You can produce derivations, simulation reports, visualizations, reproducible notebooks, short explanatory videos, and research-style statistical analyses.
  5. Transfer achievements: You can take an unfamiliar applied problem, identify the inferential target, select a defensible method, check assumptions, quantify uncertainty, and communicate conclusions without confusing statistical evidence with certainty or causation.




OERs on the Topic

The English Wikipedia article provides a broad overview of the field:

For deeper university study, you can also use the openly available MIT OpenCourseWare materials: MIT OpenCourseWare: Mathematical Statistics
MIT OpenCourseWare: Statistics for Applications
NIST/SEMATECH Engineering Statistics Handbook



Linked Learning Areas


aiMOOC Projects

MOOCwiki · Deutsch

Nach dem Lernen ist vor dem Lernen

Entdecke direkt den nächsten Lernkurs. Weitere Inhalte erscheinen, wenn Du weiter nach unten scrollst.

Zur MOOCwiki-Hauptseite

Mediathek

Mediathek

Inhalte werden geladen ...

Mediathek wird aus dem Wiki geladen ...