Statistics for Data Science

Introduction

Statistics gives data science a language for describing variation, reasoning under uncertainty, estimating unknown quantities, testing claims, and judging how well models generalize. In a data-science project, you rarely observe the whole population or every future case. You work with samples, measurements, incomplete information, and models. Statistical thinking helps you decide what the data support, what remains uncertain, and what assumptions make an analysis credible.

This university-level aiMOOC connects statistics with data science. You will move from exploratory data analysis to probability, sampling, estimation, hypothesis testing, regression, and model validation. The emphasis is not only on calculating statistics but also on interpreting them, checking assumptions, visualizing data, and communicating uncertainty.

By the end of the course, you should be able to choose suitable summaries and visualizations, reason with probability distributions, explain sampling variability, construct and interpret interval estimates, evaluate hypotheses, fit and diagnose simple statistical models, and distinguish statistical inference from predictive validation.

Statistics in the Data-Science Workflow

From a Question to Evidence

A strong statistical analysis begins with a question, not with a test. You first define the target population, the variables, and the outcome you want to understand or predict. You then examine how the data were generated. This matters because a large data set can still be biased if its sampling or measurement process is poor.

A useful workflow is to formulate a question, inspect data provenance, clean and visualize the data, select a statistical model, quantify uncertainty, validate the model, and communicate results with limitations. Each stage can change what conclusions are justified.

Population means the full set of cases relevant to the question. A sample is the subset you actually observe. A parameter is a numerical feature of a population, such as a population mean, while a statistic is calculated from a sample, such as a sample mean. Statistical inference uses sample statistics to learn about unknown population parameters.

Data Types, Measurement, and Bias

Variables may be numerical or categorical. Numerical variables can be continuous or discrete. Categorical variables can be nominal, where categories have no natural order, or ordinal, where order matters. The measurement scale affects which summaries and models make sense.

Missing values deserve explicit attention. Missingness can arise from data-entry problems, sensor failures, dropout, nonresponse, or rules in the data-generating process. Simply deleting incomplete rows can distort an analysis when the missingness is related to the outcome or predictors.

Bias is a systematic tendency for an estimate or conclusion to depart from the target. Examples include selection bias, measurement bias, survivorship bias, and data leakage. Random sampling helps control selection bias, while random assignment in experiments helps support causal comparisons. Neither technique is a substitute for careful measurement.

Exploratory Data Analysis and Descriptive Statistics

Center, Spread, and Shape

Exploratory data analysis, or EDA, helps you understand distributions before fitting models. For a numerical variable, common measures of center are the mean and median. The sample mean is

$\bar{x} = \frac{1}{n} \sum_{i = 1}^{n} x_{i} .$

The mean uses every observation and is sensitive to extreme values. The median is the middle ordered value and is more robust to outliers. For strongly skewed distributions, reporting the median and interquartile range can be more informative than reporting only the mean and standard deviation.

The sample variance and sample standard deviation measure spread:

$s^{2} = \frac{1}{n - 1} \sum_{i = 1}^{n} (x_{i} - \bar{x})^{2}, s = \sqrt{s^{2}} .$

The interquartile range is $I Q R = Q_{3} - Q_{1}$ . It summarizes the middle half of the observations and is useful when distributions are skewed or contain outliers.

A histogram displays how numerical observations are distributed across intervals. The apparent shape depends on bin width, so you should examine more than one reasonable binning and compare the graph with numerical summaries.

A box plot summarizes the median, quartiles, and potential outliers. It is especially useful for comparing distributions across groups, but it hides details such as multimodality that a histogram or density plot may reveal.

Robust Summaries and Standardization

A summary is robust if a small number of extreme observations has limited influence on it. The median and interquartile range are more robust than the mean and standard deviation. Robustness is important in data science because real data often contain recording errors, heavy tails, or rare but valid cases.

A z-score standardizes a value relative to a mean and standard deviation:

$z = \frac{x - μ}{σ} .$

Standardization places variables on a comparable scale, but it does not make a non-normal distribution normal. It also does not remove outliers or solve measurement problems.

Probability and Random Variables

Probability as a Model of Uncertainty

Probability provides a mathematical framework for uncertainty. A random variable maps outcomes of a random process to numerical values. A probability distribution describes how probability is assigned to possible values or ranges of values.

For events $A$ and $B$ , conditional probability is

$P (A ∣ B) = \frac{P (A \cap B)}{P (B)}$

when $P (B) > 0$ . Two events are independent when learning that one occurred does not change the probability of the other, equivalently $P (A \cap B) = P (A) P (B)$ .

Conditional reasoning is essential in classification, medical testing, anomaly detection, recommendation systems, and risk analysis because observed evidence often changes the probability of a hypothesis or class.

Bayes' Theorem

Bayes' theorem reverses a conditional probability:

$P (A ∣ B) = \frac{P (B ∣ A) P (A)}{P (B)} .$

In data science, $P (A)$ can represent a prior probability, $P (B ∣ A)$ a likelihood contribution, and $P (A ∣ B)$ an updated probability after observing evidence. The base rate matters: a highly accurate classifier can still produce many false positives when the positive class is very rare.

Probability Distributions

Discrete and Continuous Models

A discrete random variable has probabilities attached to countable values. The Bernoulli distribution models a single binary trial, and the binomial distribution models the number of successes in a fixed number of independent Bernoulli trials with constant success probability.

A continuous random variable is described by a probability density function. Probabilities correspond to areas under the density curve over intervals. The probability at one exact point is zero for a continuous distribution, even though intervals can have positive probability.

The expected value $E [X]$ describes the long-run center of a random variable, while $V a r (X)$ describes its dispersion. These population quantities are not the same as sample summaries, although sample summaries are often used to estimate them.

The Normal Distribution

The normal distribution is a continuous, symmetric distribution determined by its mean $μ$ and variance $σ^{2}$ . Changing $μ$ shifts its center; changing $σ$ changes its spread.

Normal models are useful for measurement error and for many sampling distributions, but real data do not become normal simply because a normal model is convenient. Always inspect the data and the model assumptions.

Sampling, Sampling Distributions, and the Central Limit Theorem

Sampling Variability

A statistic changes from sample to sample. Its sampling distribution is the distribution of the statistic across repeated samples generated under the same sampling process. This idea separates the variability of individual observations from the uncertainty of an estimator.

For independent observations with population standard deviation $σ$ , the standard error of the sample mean is

$S E (\bar{X}) = \frac{σ}{\sqrt{n}} .$

In practice, $σ$ is often unknown and is estimated by the sample standard deviation. The square-root relationship is important: multiplying the sample size by four approximately halves the standard error, all else equal.

Central Limit Theorem

The central limit theorem states, under suitable conditions, that the standardized sampling distribution of a sum or mean approaches a normal distribution as sample size grows. It does not say that the raw data become normal. The quality of the approximation depends on the population distribution, dependence structure, and sample size.

The theorem explains why normal-based confidence intervals and tests often work for means when sample sizes are sufficiently large and assumptions are reasonable.

Estimation and Confidence Intervals

Point Estimates and Uncertainty

A point estimate gives one best estimate of a parameter, but it does not communicate precision. An interval estimate combines a point estimate with uncertainty. For a simple large-sample mean problem, an approximate interval has the form

$estimate \pm critical value \times standard error .$

A 95% frequentist confidence procedure is designed so that, over many repetitions of the full sampling process, about 95% of the constructed intervals contain the true parameter when the model assumptions hold. For one realized interval, the parameter is treated as fixed; the interval is the random object generated by the procedure.

Narrow intervals are more precise, but precision is not the same as validity. A very narrow interval around a biased estimate can still be misleading.

Bootstrap Estimation

The bootstrap estimates sampling variability by repeatedly resampling observations with replacement from the observed sample and recalculating a statistic. It is useful for statistics whose analytic standard errors are difficult to derive.

A bootstrap workflow includes choosing the statistic, drawing many resamples of size $n$ with replacement, computing the statistic for each resample, examining the bootstrap distribution, and constructing an interval using a method appropriate to the problem.

Bootstrap methods depend on the observed sample being informative about the target population. Resampling cannot repair severe selection bias, data leakage, or a fundamentally unrepresentative sample.

Hypothesis Testing and Statistical Significance

Null and Alternative Hypotheses

A hypothesis test compares observed data with what would be expected under a null model. The null hypothesis usually represents a benchmark such as no difference, no association, or a specified parameter value. The alternative represents departures of scientific or practical interest.

A test statistic summarizes how far the observed result lies from the null expectation relative to its variability. A p-value is the probability, assuming the null hypothesis and the test model are correct, of obtaining a result at least as incompatible with the null as the observed result according to the chosen test statistic.

A p-value is not the probability that the null hypothesis is true. It is also not an effect size, a measure of practical importance, or the probability that a result will replicate.

Errors, Power, and Multiple Testing

A Type I error is a false positive: rejecting a true null hypothesis. A Type II error is a false negative: failing to reject a false null hypothesis. Statistical power is the probability of rejecting the null when a specified alternative is true. Power depends on effect size, sample size, variability, significance threshold, and study design.

When many hypotheses are tested, the chance of at least one false positive increases. Multiple-testing procedures such as Bonferroni correction or false discovery rate control adjust decision rules to address this problem. In exploratory data science, you should distinguish hypotheses generated after inspecting the data from hypotheses specified before analysis.

Effect sizes and confidence intervals should usually accompany significance tests. With very large data sets, tiny effects can be statistically significant while remaining practically unimportant.

Association, Correlation, and Regression

Correlation Is Not Causation

Correlation measures association, not causation. Pearson correlation summarizes linear association and ranges from negative one to positive one. A value near zero means little linear association, but a strong nonlinear relationship may still be present.

Different data sets can share the same means, variances, correlation, and fitted regression line while having very different structures. Anscombe's quartet makes this point visually.

This is why visualization should accompany numerical summaries. Outliers, clusters, nonlinear patterns, and restricted ranges can strongly affect correlation.

Simple Linear Regression

Simple linear regression models the conditional mean of an outcome $Y$ as a linear function of a predictor $X$ :

$Y_{i} = β_{0} + β_{1} X_{i} + ε_{i} .$

The intercept $β_{0}$ is the expected outcome when $X = 0$ if that value is meaningful. The slope $β_{1}$ is the expected change in $Y$ associated with a one-unit increase in $X$ , under the model.

Ordinary least squares chooses coefficients that minimize the sum of squared residuals. Residuals are observed outcomes minus fitted outcomes. You should examine residual plots to assess nonlinearity, nonconstant variance, influential observations, and other departures from assumptions.

The coefficient of determination, $R^{2}$ , describes the fraction of outcome variation explained by the fitted model relative to a mean-only baseline in the sample. It is not, by itself, evidence of causality, good out-of-sample prediction, or correct model specification.

Prediction, Validation, and Generalization

Training Error Is Not Test Error

A model can fit training data extremely well and still perform poorly on new cases. This is overfitting. Data science therefore separates model fitting from model evaluation.

A test set should remain untouched until final evaluation whenever feasible. If you repeatedly use the test set to choose features, tune hyperparameters, or compare models, information from the test set leaks into model development and the reported performance becomes optimistic.

For regression, common predictive metrics include mean absolute error and root mean squared error. For classification, accuracy can be misleading with imbalanced classes, so precision, recall, specificity, ROC curves, and calibration may also matter depending on the decision problem.

Cross-Validation

Cross-validation estimates generalization performance by repeatedly fitting a model on training folds and evaluating it on held-out folds. In k-fold cross-validation, the data are divided into $k$ parts; each part serves as a validation fold once.

Cross-validation must respect the data-generating structure. Time-series data often require forward-looking splits rather than random folds. Grouped data may require keeping all observations from the same person, device, household, or site in one fold. Preprocessing steps that learn from data, such as scaling, imputation, and feature selection, should be fitted inside each training fold to avoid leakage.

Statistical Thinking for Responsible Data Science

Association, Prediction, and Causation

A predictive model answers a question such as “How accurately can we predict the outcome for new cases?” A causal analysis asks “What would happen to the outcome if we intervened on a variable?” These are different goals. High predictive accuracy does not automatically identify a causal effect.

Confounding occurs when a third variable influences both the predictor and outcome and creates or distorts an association. Randomized experiments can balance confounders in expectation, while observational studies require stronger assumptions and design strategies to support causal claims.

Reproducibility and Communication

A reproducible analysis records how raw data become final results. Keep data-cleaning rules, model choices, random seeds, software versions, and evaluation procedures documented. Use version control and scripted workflows where possible.

Communicate uncertainty in language appropriate to the evidence. Report effect sizes, uncertainty intervals, validation performance, and important assumptions. Avoid selective reporting of only favorable models or statistically significant findings. A statistically sophisticated analysis can still be misleading if the research question, measurement process, or reporting choices are weak.

Worked Data-Science Example

Suppose you want to predict apartment rent from floor area and neighborhood. A responsible statistical workflow begins by defining the prediction target and unit of observation. You inspect missing rents, duplicate listings, impossible floor areas, and whether the sample represents the geographic market of interest.

Next, visualize rent and floor area with histograms and scatterplots. Compare medians and interquartile ranges across neighborhoods. If rent is heavily right-skewed, consider whether a logarithmic transformation is scientifically and practically meaningful.

Fit a baseline linear model and examine residuals. Use cross-validation that reflects how the model will be deployed. If the model will predict future listings, a time-aware split can be more realistic than a random split. Report out-of-sample error alongside uncertainty and explain where the model performs poorly.

Finally, distinguish prediction from causation. A positive coefficient for floor area does not prove that increasing the size of an existing apartment would cause its rent to rise by the estimated amount. Neighborhood, building quality, age, amenities, and market conditions may confound simple associations.

Interactive Tasks

Quiz: Test Your Knowledge

What does a sampling distribution describe? (The values of a statistic across repeated samples) (!The raw values in one observed data set) (!The order in which variables were collected) (!The labels used for categorical variables)

Which summary is generally more robust to extreme values? (The median) (!The arithmetic mean) (!The variance) (!The standard deviation)

What does a p-value quantify? (Compatibility of the data with the null model) (!The probability that the null hypothesis is true) (!The practical importance of an effect) (!The probability that the study will replicate)

What does the central limit theorem concern? (The sampling distribution of sums or means) (!The raw data becoming normally distributed) (!Every population having a bell shape) (!All variables becoming independent)

What is a Type I error? (Rejecting a true null hypothesis) (!Failing to reject a false null hypothesis) (!Using a sample that is too large) (!Reporting a confidence interval)

Which statement about correlation is correct? (It measures association rather than causation) (!It proves a causal relationship) (!It detects every nonlinear relationship) (!It is unaffected by extreme observations)

What is the main purpose of a held-out test set? (To estimate performance on unseen data) (!To choose every model hyperparameter) (!To increase the training sample repeatedly) (!To eliminate all sources of bias)

What does a bootstrap resample do? (It samples observed cases with replacement) (!It samples only the largest observations) (!It removes all outliers before estimation) (!It guarantees an unbiased estimator)

Why should preprocessing be fitted inside cross-validation folds? (To reduce information leakage) (!To make every variable normally distributed) (!To guarantee causal interpretation) (!To eliminate the need for a test set)

What does a narrower confidence interval usually indicate when the method is valid? (Greater precision) (!Greater practical importance) (!Proof of no bias) (!Proof of a causal effect)

Memory Game

Parameter	Numerical feature of a population
Statistic	Numerical summary calculated from a sample
Residual	Observed outcome minus fitted outcome
Bootstrap	Resampling method using replacement
Power	Probability of detecting a specified effect under an alternative
Confounder	Variable related to both an exposure and an outcome

Drag and Drop

Match the correct terms.	Topic
Exploratory data analysis	Inspect distributions patterns and unusual observations
Confidence interval	Quantify uncertainty around an estimated parameter
Hypothesis test	Compare observed evidence with a null model
Regression	Model an outcome as a function of predictors
Cross-validation	Estimate generalization performance using held-out folds

...

Crossword Puzzle

Sampling	What process selects observations from a population?
Variance	What quantity measures squared dispersion around a mean?
Quantile	What value divides an ordered distribution at a chosen cumulative probability?
Residual	What is the difference between an observed and fitted outcome?
Bootstrap	What resampling method repeatedly samples observed cases with replacement?
Likelihood	What function describes how plausible parameter values are given observed data?

LearningApps

Cloze Text

A sample is used to learn about a larger

. The sample mean is a

calculated from observed values. The spread of a sampling distribution is summarized by a

. Under suitable conditions, the central limit theorem explains why many sampling distributions become approximately

. A confidence interval combines an estimate with its

. A p-value is calculated under a specified

. A small p-value does not measure the practical size of an

. Repeated hypothesis tests can increase the risk of

. Correlation measures statistical

rather than causation. In regression, the difference between an observed and fitted value is a

. Cross-validation estimates performance on

. Preprocessing must be learned within training folds to prevent

.

Open-Ended Tasks

Easy

Distribution profile: Choose one numerical variable from an open data set, create a histogram and box plot, and explain its center, spread, skewness, and possible outliers in 200 words.
Summary comparison: Calculate the mean, median, standard deviation, and interquartile range for one variable, then explain which summaries you would report and why.
Probability simulation: Simulate repeated coin tosses or dice rolls in a spreadsheet, Python, or R and create a short visual explanation of how relative frequency behaves as the number of trials grows.
Statistics explainer: Produce a two-minute video or one-page infographic that explains the difference between a population, sample, parameter, and statistic using one original example.

Standard

Sampling experiment: Draw many random samples from a known simulated population, plot the distribution of sample means for several sample sizes, and explain what changes in center, spread, and shape.
Confidence interval investigation: Simulate repeated samples, construct 95 percent confidence intervals for each sample, visualize which intervals cover the true parameter, and explain the long-run interpretation.
Regression case study: Find a public data set with two meaningful numerical variables, fit a simple linear regression, inspect residuals, report uncertainty, and discuss whether a causal interpretation is justified.
Data interview: Interview a researcher, analyst, or data professional about one real statistical decision they make at work, then compare their workflow with the principles in this course.

Advanced

Bootstrap project: Compare an analytic standard error with a bootstrap estimate for a statistic of your choice, investigate when the two approaches agree or differ, and justify your conclusions.
Multiple testing study: Simulate many null hypotheses, compare unadjusted testing with Bonferroni or false discovery rate control, and present the trade-off between false positives and power.
Validation audit: Reproduce a small predictive modeling workflow, deliberately introduce one form of data leakage, quantify its effect on estimated performance, and redesign the workflow correctly.
Statistical consulting report: Visit a university lab, civic data office, company, or public data portal and develop a complete analysis proposal covering sampling, measurement, EDA, inference, validation, ethics, and communication.

Learning Assessment

Analysis design: Given a research question and a messy data description, identify the target population, sampling unit, variables, likely biases, and a defensible analysis strategy before selecting any statistical test.
Uncertainty reasoning: Compare two estimates with different sample sizes and standard errors, explain why their precision differs, and determine what additional information is needed before deciding which estimate is more credible.
Inference critique: Evaluate a report that treats a p-value as the probability that the null hypothesis is true, correct the interpretation, and propose an improved results statement using effect size and uncertainty.
Model diagnosis: Inspect a hypothetical scatterplot and residual plot, identify violations of a simple linear model, and recommend transformations, alternative models, or data checks with justification.
Generalization plan: Design a train-validation-test or cross-validation scheme for grouped or time-dependent data and explain how your design prevents leakage and matches deployment conditions.
Causal versus predictive reasoning: Given a high-performing observational prediction model, explain why predictive accuracy alone does not establish causal effects and propose evidence that would strengthen a causal claim.

Evidence of Learning

Evidence of learning should show both statistical knowledge and disciplined data-science practice.

Knowledge: You can explain sampling variability, probability distributions, estimation, confidence intervals, hypothesis testing, regression, and model validation in your own words.
Skills: You can visualize distributions, calculate and interpret core statistics, simulate sampling processes, fit simple models, diagnose residuals, and evaluate out-of-sample performance.
Reasoning: You can distinguish parameters from statistics, uncertainty from variability, association from causation, statistical significance from practical importance, and training fit from generalization.
Products: You can produce reproducible notebooks or scripts, annotated visualizations, statistical reports, simulation studies, and validation plans.
Transfer: You can adapt statistical methods to a new domain, question data provenance, identify leakage or bias, choose evaluation methods that match deployment, and communicate limitations to a non-specialist audience.