Zum Inhalt springen

English:Dataset preparation and avoiding data leakage - AI Year 7

Aus MOOCsWiki Staging
aiMOOC-Siegel aiMOOC

Dataset preparation and avoiding data leakage - AI Year 7

QR-Code



Introduction

Simple and flexible boundaries around contrasting example points
A good-looking fit is not enough. This illustration of overfitting is separate from the original delivery-data audit below.

Audit a fictional dispatch dataset, expose three information-flow risks and build a transparent correction log. Compare a weak but available baseline with a perfect-looking after-event shortcut, then transfer the audit questions to a new context.


Define the task and audit fictional dispatch data


Begin with a prediction moment and a data dictionary

Imagine a fictional parcel desk. At dispatch, you want to flag a delivery that might arrive later than its promised duration. A feature is information supplied to a prediction method; a label is the outcome used to check or learn from past examples. Provenance records where data came from and what its use permits. These are invented records, not a real delivery company's data or a system for judging workers.

CodeAI: AI, What is Machine Learning? Use the verified introduction to distinguish learning from examples from simply writing a rule. Our program below fits one preparation value, the training median, but uses a handwritten distance rule as its prediction baseline. It does not train a neural network or operate a delivery service.

The eight fields are row identifier, delivery identifier, dispatch day, distance in kilometres, promised duration in minutes, actual duration in minutes, outcome label and provenance. Row identifiers track copies; delivery identifiers track the underlying event. A day here is an ordered fictional dispatch index, not a date or measurement from the current world. Distances can be missing. In this exercise promised and actual durations must be positive numbers.

Row Delivery Dispatch day Distance km Promised minutes Actual minutes Outcome label Provenance
R1 D1 1 2 30 25 on_time original
R2 D2 2 4 30 35 LATE original
R3 D3 3 missing 30 20 on_time original
R4 D4 4 6 30 45 late original
R5 D5 5 3 30 28 on_time original
R6 D6 6 5 30 40 late original
R7 D4 4 6 30 45 late original
R8 D7 7 missing 30 50 late original
R9 D8 8 9 30 24 on_time original
R10 D9 9 9 30 55 late missing

At dispatch, the stated example makes distance and promised duration available. Actual duration is known only afterwards. The outcome label is late when actual duration is greater than promised duration, otherwise on_time. Keep that later outcome for evaluation; do not feed it into a dispatch-time predictor. Record identifiers and provenance are audit fields, not automatically useful prediction features.


Inspect five review locations without inventing observations

R2 uses the approved spelling variant LATE; normalise it to late while preserving the original. R3 and R8 have missing distances; preserve that fact instead of pretending someone measured a replacement. R7 is another copy of delivery D4 with the same declared event fields, so retain the first copy and log the duplicate. R10 has no resolved provenance, so keep it out of the usable dataset pending review. Do not invent a source, licence or permission to make the row usable.

After these decisions there are eight distinct usable deliveries: D1 through D8. Ten raw rows are not ten independent observations. A larger table can contain duplicated, inappropriate or poorly described data. Our log explains each action and preserves the raw list; it does not certify a representative survey, a complete rights audit or true measurements.


In predictive modelling, leakage lets information influence preparation, fitting or evaluation in a way that would not be available for the intended prediction. It can make an evaluation misleading. This use of leakage is different from a security incident that exposes private records. Missing provenance is a reason for review even when it is not itself evidence of target leakage.

Three concrete risks in our exercise are an after-delivery feature, a duplicate delivery crossing the training/test boundary, and fitting a replacement value using later test records. Name the mechanism, not just the score. A high result is a reason to investigate; it is not proof of leakage by itself, and a lower or unchanged result does not prove the process sound.



Separate later cases fit training transformations and run the program


Build a correction log and split by the intended use

The native exercise uses the following audit decisions. A spelling change is different from an invented outcome, and a missing value remains missing in the retained clean table. Conflicting copies of the same delivery stop the example for review rather than being merged silently.

Row Actual audit decision
R2 approved label spelling normalised
R3 distance remains missing; do not guess
R7 duplicate observation; original retained
R8 distance remains missing; do not guess
R10 provenance unresolved; excluded pending review

Choose the split before fitting data-dependent transformations. Here the stated purpose is to predict later dispatches from earlier examples: days one through six are training and later days are test. That gives training deliveries D1 to D6 and test deliveries D7 and D8. This is one purpose-specific chronological split, not a universal prescription for every project. Other settings can need grouping, additional validation data or different designs.

If you naively take the first six raw rows as training and the rest as test, D4 appears on both sides because of R7. Distinct row identifiers do not prevent this overlap. The proper audit resolves the exact copy before the chronological split; the event identifier then remains disjoint. Related observations in other projects may need an explicit group rule rather than simple duplicate removal.


Fit the replacement from training observations only

The five known training distances are 2, 4, 6, 3 and 5 kilometres. Sorting gives 2, 3, 4, 5, 6, so the middle value is 4. Preserve the fact that D3's distance was missing, and use 4 only as a declared model-input replacement. The later missing D7 input uses the same fitted value. It is not a new observation of either delivery's true distance.

For an intentionally incorrect comparison, include the known later test distance 9 in the fit. The six known values become 2, 3, 4, 5, 6, 9, whose two middle values average to 4.5. The test data has now changed a preparation parameter. The program labels that comparison as leaky and does not use it for the proper baseline. A pipeline must keep the fitted training transformation and apply it consistently; it must not refit it on the test set.

Hint: decide which records may affect a learned preparation value before calculating it. Partial step: D7 and D8 are later test events, so exclude them from fitting. Worked conclusion: the correct training-only replacement is 4; the all-clean-row comparison is 4.5 and is unsuitable for this evaluation. This does not mean a training median is always the best handling of missing data.


Run and inspect the complete original baseline

Use an approved Python environment or trace the supplied values on paper. The standard-library program audits declared fields, preserves missing distances, separates days, fits the training median and applies a fixed rule: predict late for a distance at least four, otherwise on_time. The threshold is set for this classroom demonstration, not fitted to the test outcomes or claimed as a useful real delivery model.

"""Fictional dispatch records; a handwritten baseline, not a deployed model."""
from copy import deepcopy
from statistics import median
import json

RAW = [
    ["R1", "D1", 1, 2, 30, 25, "on_time", "original"],
    ["R2", "D2", 2, 4, 30, 35, "LATE", "original"],
    ["R3", "D3", 3, None, 30, 20, "on_time", "original"],
    ["R4", "D4", 4, 6, 30, 45, "late", "original"],
    ["R5", "D5", 5, 3, 30, 28, "on_time", "original"],
    ["R6", "D6", 6, 5, 30, 40, "late", "original"],
    ["R7", "D4", 4, 6, 30, 45, "late", "original"],
    ["R8", "D7", 7, None, 30, 50, "late", "original"],
    ["R9", "D8", 8, 9, 30, 24, "on_time", "original"],
    ["R10", "D9", 9, 9, 30, 55, "late", None],
]


def positive_number(value):
    return type(value) in [int, float] and value > 0 and float("inf") > value


def audit(raw):
    if type(raw) is not list or not raw:
        raise ValueError("A nonempty original record list is required")
    unique, log, held, row_ids = dict(), [], [], set()
    for item in raw:
        if type(item) is not list or len(item) != 8:
            raise ValueError("Use exactly eight declared fields")
        rid, did, day, distance, promised, actual, label, source = item
        if any(type(x) is not str or not x for x in [rid, did]):
            raise ValueError("Record and delivery identifiers are required")
        if rid in row_ids:
            raise ValueError("Row identifiers must be unique")
        row_ids.add(rid)
        if type(day) is not int or day < 1:
            raise ValueError("Dispatch day must be a positive integer")
        if distance is not None and not positive_number(distance):
            raise ValueError("Distance is positive kilometres or missing")
        if not positive_number(promised) or not positive_number(actual):
            raise ValueError("The exercise uses positive minute values")
        labels = dict(late="late", LATE="late", on_time="on_time")
        if type(label) is not str or label not in labels:
            raise ValueError("Use only the declared label spellings")
        normal_label = labels[label]
        expected = "late" if actual > promised else "on_time"
        if normal_label != expected:
            raise ValueError("The invented outcome and label conflict")
        if source != "original":
            held.append(rid)
            log.append([rid, "provenance unresolved; excluded pending review"])
            continue
        row = [rid, did, day, distance, promised, actual, normal_label, source]
        if did in unique:
            if unique[did][1:] != row[1:]:
                raise ValueError("Conflicting delivery copies require review")
            log.append([rid, "duplicate observation; original retained"])
            continue
        unique[did] = row
        if label != normal_label:
            log.append([rid, "approved label spelling normalised"])
        if distance is None:
            log.append([rid, "distance remains missing; do not guess"])
    return sorted(unique.values(), key=lambda r: (r[2], r[1])), log, held


def split_by_day(cleaned, cutoff):
    if type(cutoff) is not int or cutoff < 1:
        raise ValueError("Use a positive integer cutoff")
    train = deepcopy([r for r in cleaned if r[2] <= cutoff])
    test = deepcopy([r for r in cleaned if r[2] > cutoff])
    if not train or not test:
        raise ValueError("Both chronological subsets are required")
    if set(r[1] for r in train) & set(r[1] for r in test):
        raise ValueError("Delivery identities must not overlap")
    return train, test


def fit_distance_fill(training_rows):
    known = [r[3] for r in training_rows if r[3] is not None]
    if not known:
        raise ValueError("No observed training distance; request review")
    return median(known)


def dispatch_predict(distance, fill):
    if not positive_number(fill):
        raise ValueError("A valid training-derived fill is required")
    if distance is not None and not positive_number(distance):
        raise ValueError("Invalid distance")
    used = fill if distance is None else distance
    return "late" if used >= 4 else "on_time"


def demonstrate():
    clean, log, held = audit(RAW)
    train, test = split_by_day(clean, 6)
    fill = fit_distance_fill(train)
    expected = [r[6] for r in test]
    predicted = [dispatch_predict(r[3], fill) for r in test]
    unavailable_comparison = ["late" if r[5] > r[4] else "on_time" for r in test]
    naive_overlap = sorted(set(r[1] for r in RAW[:6]) & set(r[1] for r in RAW[6:]))
    return dict(clean=clean, log=log, held=held,
                training_ids=[r[1] for r in train], test_ids=[r[1] for r in test],
                training_only_fill=fill,
                leaky_all_rows_fill_for_comparison_only=fit_distance_fill(clean),
                test_expected=expected, dispatch_predictions=predicted,
                dispatch_matches=sum(a == p for a, p in zip(expected, predicted)),
                forbidden_after_delivery_comparison=unavailable_comparison,
                naive_row_split_overlap=naive_overlap)


if __name__ == "__main__":
    print(json.dumps(demonstrate(), indent=2))


Read the actual two-case result without hiding the failure

For D7, distance is missing, so the fitted input is 4 and the rule outputs late; its recorded outcome is late. For D8, the distance is 9, so the rule outputs late, but its recorded outcome is on_time. The actual run matches one of two test outcomes. That tiny result does not establish real-world performance or a reliable estimate for future deliveries.

The deliberately forbidden comparison uses actual duration after delivery. It matches both outcomes because it is applying the outcome definition itself. Two matches cannot make this a valid dispatch-time prediction: the required information does not yet exist then. Keeping the complete code and its weak proper result makes the contrast inspectable. The leaky 4.5 replacement happens not to change these two fixed baseline outputs; a pipeline error can exist without a visible score change in this tiny test.



Test information boundaries and transfer the pipeline


Inspect an illustration without confusing failure types

The introductory Commons diagram contrasts a flexible boundary with a simpler boundary around example points. Treat it as an illustration of overfitting, not a chart of our deliveries or evidence of leakage in that particular image. Overfitting and leakage are related evaluation concerns but are not identical: one describes fitting patterns too narrowly, while the other concerns inappropriate information flow.

CodeAI: AI, Training Data & Bias. Ask whether example selection and coverage support an intended task. Our made-up data does not measure real delivery bias, and holding an unknown-provenance row does not make the remaining eight events representative. Keep data quality, rights, leakage and performance as separate questions that can all matter.


Test boundaries and the information flow

With cutoff day six, day six belongs to training and day seven to test. Test those neighbouring cases explicitly. A split with no training or no test must stop in this example. Distances zero, negative values, booleans and non-finite numbers are invalid under the stated positive-distance contract; None means missing. If every training distance is missing, there is no observed training median, so the example requests review instead of inventing zero.

Try the prediction rule just below four, exactly four and just above four. Those outputs test the declared threshold, not a physical law. Next change a later test distance while leaving training unchanged. The proper training-only fill must remain 4, although an all-row fill may move. Add an exact event copy and then a conflicting copy. The exact copy is logged; the conflict must stop for review. Do not erase a real conflict just to improve a score.

The audit checks each row's declared shape and label consistency before use. It is an inspectable teaching example with stated input contracts, not a complete validation library for arbitrary files. Retain expected and actual outcomes and distinguish an actual program execution from a paper trace. A successful boundary check is limited evidence about that contract.


Write a defensible pipeline and keep test evidence separate

For this exercise, first agree the task and prediction moment. Preserve raw records and document fields, units and provenance. Resolve exact copies, conflicts and approved spelling variants under recorded rules; hold uncertain sources. Choose the chronological/group boundary. Fit any data-dependent replacement using training only, then apply the frozen transformation to later inputs. Keep features separate from outcomes and identifiers. Evaluate the predeclared baseline and report the mismatch as well as the match.

This early audit applies fixed, declared checks; it does not choose a preprocessing parameter by inspecting test outcomes. In larger work, keep validation separate for development choices and reserve final test evidence. Repeatedly changing a method until it fits the final test turns that test into development feedback. The correct response to an invalid evaluation is to repair the method and obtain suitable independent evidence, not simply rename the split.


Transfer to a fictional greenhouse request

Suppose a fictional greenhouse wants to predict at the start of a day whether a tray will need watering by evening. A morning moisture reading may be available; the evening reading and the final watering decision are later outcomes. Repeated copies of the same tray/day should not become independent examples on opposite sides of a split. Source permissions and measurement units still need a dictionary and audit log.

Reuse the questions about provenance, timing, grouping, missingness and fitted transformations. Reconsider the fields, target, split and human check for this new purpose. Do not reuse the delivery threshold four or assume our median is biologically appropriate. No real plant-care recommendation or greenhouse trial is performed here. Your transfer product is a justified plan unless you actually implement and test an approved fictional version.



Interactive Tasks


Quiz: Test Your Knowledge

Which field is unavailable at the stated dispatch prediction time? (Actual delivery duration) (!The declared promised duration) (!The already supplied distance) (!The dispatch day)



Why does R7 need review before evaluation? (It repeats delivery D4 under another row identifier) (!Every longer table is more representative) (!Its row identifier is a prediction outcome) (!Every repeated number proves a duplicate delivery)



How is the missing-provenance row handled? (Held out pending source review) (!Assigned a plausible licence automatically) (!Copied into both splits) (!Labelled original because its values look useful)



Which deliveries form the proper later test set? (D7 and D8) (!D1 and D2) (!D4 and its raw copy) (!All deliveries used to fit the median)



What is the training-only distance median? (Four kilometres) (!Four and a half kilometres) (!Nine kilometres) (!Zero because one distance is missing)



What does the replacement value represent? (A declared input transformation rather than an observed distance) (!A measured correction to the real delivery) (!The known outcome label) (!A proof that missingness cannot matter)



Why is the after-delivery comparison invalid as a dispatch predictor? (It uses information available only after the prediction moment) (!It has too few perfect predictions to be allowed) (!It omits the row identifier from its inputs) (!It uses the stated outcome definition incorrectly)



What is the actual proper baseline result on the two later cases? (One matching outcome and one mismatch) (!Two matches proving universal accuracy) (!No predictions because every distance is unknown) (!A trained neural network with certified performance)



What should happen if all training distances are missing? (Stop and request review rather than invent a median) (!Treat every unknown as an observed zero) (!Fit the replacement on later test distances) (!Drop the label and call the score perfect)



Which transfer choice fits the greenhouse example? (Recheck timing fields grouping and missing-value handling) (!Use the final evening outcome as a morning input) (!Reuse the delivery threshold without justification) (!Treat repeated tray records as independent by default)





Memory Game

Provenance The documented source and permitted use of data
Feature Information supplied to a prediction method
Label An outcome used for learning or evaluation
Duplicate Another copy of the same underlying observation
Imputation A declared replacement strategy for missing input values
Leakage Inappropriate information flow affecting a predictive evaluation





Drag and Drop

Describe fields units and what is available at dispatch Data dictionary
Preserve the raw copies and record decisions Audit log
Keep earlier and later event identities separate Chronological split
Calculate the replacement using earlier observations only Training-only fitting
Compare unchanged predictions with later outcomes Held-out evaluation





Crossword Puzzle

FEATURE What information is supplied to a prediction method?
LABEL What outcome is used for learning or checking?
MEDIAN What middle statistic supplies the demonstrated replacement?
DUPLICATE What is another copy of the same observation?
PROVENANCE What describes source and permitted use?
LEAKAGE What is inappropriate information flow in evaluation?





LearningApps

Optional extra practice. Choose an appropriate activity with your teacher. If the external collection is unavailable, use the native activities and independent tasks in this course.


Cloze Text

Complete the explanation.
The intended prediction occurs at

. Actual duration is a later

. The spelling variant LATE is

. R7 repeats an underlying

. R10 has unresolved

. Missing distances remain marked as

. Earlier events form the

subset. Later events form the

subset. The training-only median equals

. A replacement is not a measured

. The proper two-case baseline has one

. A new domain needs new

.



Open Tasks


Easy

  1. Mark field roles: Annotate all eight declared fields as feature candidate, outcome, identifier or provenance, and state what is available at dispatch.
  2. Build the audit log: Find the approved spelling change, duplicate, two missing distances and unresolved source, then justify each recorded action.
  3. Calculate the training median: Write the five known training distances in order, locate the middle and distinguish replacement from a measured value.
  4. Trace the two outcomes: Predict D7 and D8 using the stated rule, compare with recorded outcomes and retain the mismatch.


Standard

  1. Repair a naive split: Show why first-six-raw-rows versus the rest overlaps on D4, then reconstruct the audited chronological split.
  2. Expose a future-information feature: Compare the actual-duration shortcut with the dispatch-only baseline and explain why the higher comparison score is invalid.
  3. Test fitting isolation: Change a later test distance, keep training fixed and check whether the training-only fill changes; contrast the intentional all-row comparison.
  4. Exercise the boundaries: Test cutoff six and its neighbours, the threshold near four, all-missing training distances and one conflicting duplicate; record expectations and actual results.


Advanced

  1. Design a documented preparation pipeline: Create a task statement, data dictionary, provenance and correction log, split rule, fitted transformation and separate evaluation report for this example.
  2. Review a misleading score claim: Explain why a useful-looking feature or score does not establish availability, rights, valid evaluation or real-world generalisation; use three mechanisms from the original data.
  3. Transfer to a greenhouse: Design original fictional fields and outcomes, a timing/group boundary and a human review route without importing the parcel threshold.
  4. Plan an independent follow-up: State what further fictional or properly authorised evidence a revised method needs, keep validation and final test roles distinct and explain which claims remain untested.



Learning Check

  1. Identify three information-flow risks: Name and correct three distinct risks in the original task.
  2. Reconstruct the corrected data: What remains after the declared audit and split?
  3. Calculate and interpret the fill: Why are the correct and intentionally leaky medians different?
  4. Explain the actual test result: What does the proper baseline show compared with the forbidden shortcut?
  5. Interpret failure boundaries: How should the example handle a cutoff with an empty subset, all-missing training distances and conflicting copies?
  6. Transfer the audit principles: What changes for a start-of-day greenhouse prediction?


Worked checks and assessment guidance

  1. Actual duration and the derived label are unavailable at dispatch, so retain them as outcomes rather than prediction inputs. R7 repeats D4 and can cross a naive row split; audit the event identity before the purpose-specific split. Fitting the replacement using later test distances changes the median from 4 to 4.5; fit on training only and freeze the transformation. Missing R10 provenance adds a separate reason for a hold, not proof of target leakage by itself.
  2. Normalise only the approved LATE spelling, preserve R3 and R8 missing distances, retain the first D4 copy and log R7, and hold unresolved R10. Eight distinct usable deliveries remain. Days one through six give D1 to D6 for training; D7 and D8 form the later test set. Row identifiers alone do not establish independent observations.
  3. Known training distances sort to 2, 3, 4, 5, 6, whose median is 4. Including the later known 9 gives 2, 3, 4, 5, 6, 9 and median 4.5. Later test information must not influence the fitted replacement for this evaluation. Neither value is a measured repair of a missing distance, and neither is claimed universally best.
  4. D7 uses the training fill 4 and is predicted late, matching its outcome. D8 has distance 9 and is also predicted late, but its outcome is on_time. One of two matches is a limited result for this handwritten rule. Actual duration reproduces both outcome labels but is unavailable at dispatch, so its two matches are not valid predictive performance. The leaky median happens not to change these fixed outputs, which does not make its fitting procedure sound.
  5. An empty chronological training/test subset stops. No observed training distance means no median can be fitted, so review is required instead of using later test data or inventing zero. Identical event copies are logged while conflicting copies stop for review. The threshold includes exactly four in late; this checks a declared classroom rule, not a real transport law.
  6. Define the new target, prediction moment, field units, source rights and tray/day grouping. Keep evening readings or final decisions out of morning prediction inputs, fit transformations on the appropriate earlier training data and evaluate independently. Reconsider the missing-value policy and human fallback. Do not reuse a parcel threshold or claim a real greenhouse experiment or observed learner competence.


Learning evidence

Retain an annotated eight-field dictionary, raw records and justified five-location audit log, a purpose-specific identity-disjoint chronological split, training-only median calculation and preserved missingness, predicted and actual test outputs including the mismatch, the rejected after-event comparison, expected and actual boundary checks or a clearly labelled paper trace, and a justified greenhouse transfer plan. Distinguish technical execution, source permissions, evaluation validity, real-world performance and actual learner work. No independent expert certification or complete curriculum coverage is implied.


OER and Wikipedia

DfE Computing, Key Stage 3 includes creative computational projects. This Year 7 AI/data application, fictional task and assessments are editorial enrichment. scikit-learn: Common pitfalls, inconsistent preprocessing and data leakage explains training-only fitting and applying the retained transformation to test data. IBM: Data leakage in machine learning provides further context for future-information and train/test contamination. Primary relevant sections read 15 November 2026. No external example code or dataset is copied.

Overfitting.svg, Chabacano, CC BY-SA 4.0. Original diagram unchanged. It illustrates overfitting, not the deliveries, an actual leakage experiment or this lesson's measured outputs.

https://www.youtube.com/watch?v=OeU5m6vRyCk — CodeAI: AI, What is Machine Learning? https://www.youtube.com/watch?v=x2mRoFNm22g — CodeAI: AI, Training Data & Bias. Actual publisher, metadata, embedding status and preview files checked; recordings embedded unchanged. No copied transcript, endorsement or relicensing implied.

All delivery records, labels, classroom contracts, code and greenhouse transfer are original fictional material. The complete standard-library Python was actually executed and independently checked; it fits a preparation median and applies a handwritten rule, not a trained predictive model or local generative LLM. No real workers, learner records, greenhouse experiment, paid API or deployment is used.



Linked learning areas


aiMOOC Projects



Schulfach+




aiMOOCs



aiMOOC Projekte









MOOCwiki · Deutsch

Nach dem Lernen ist vor dem Lernen

Entdecke direkt den nächsten Lernkurs. Weitere Inhalte erscheinen, wenn Du weiter nach unten scrollst.

Zur MOOCwiki-Hauptseite

Mediathek

Suche verfeinern

Dein Lernplan bestimmt, was oben steht. Alle anderen Inhalte bleiben erreichbar. Seitenaufrufe beeinflussen die Auswahl nicht.

Inhalte werden geladen ...

Mediathek wird aus dem Wiki geladen ...