Zum Inhalt springen

English:Data pipelines, provenance and temporal leakage - AI Year 9

Aus MOOCsWiki Staging
aiMOOC-Siegel aiMOOC

Data pipelines, provenance and temporal leakage - AI Year 9

QR-Code



Introduction

An ETL job diagram with source filtering and several output branches
Hansth's generic ETL job illustrates a sequence of data-processing stages. It is not our delivery fixture or proof of a working prediction service.

A model can look excellent by using information from the future. Audit an original fictional delivery dataset, separate features from outcomes and build a reproducible time-aware preparation plan. Handle duplicates, conflicting records and missing values openly. The complete program is an offline synthetic audit; it does not connect to a delivery service or train a classifier.


Ask when each field becomes available

The fictional task is to predict at dispatch whether a delivery will take longer than its planned duration. Every dispatch in our fixture has a distance in kilometres and a planned duration of 30 minutes. Those two feature values are declared available at dispatch. Actual duration and the late label are known afterwards. A late label is true when actual minutes exceed planned minutes; equality would mean not late under this declared rule.

Actual duration might give a perfect-looking shortcut: compare it with the plan and you recover the label exactly. But the answer is not known when the prediction is required. Keeping a high-scoring future field changes the task into reporting an outcome after it happened. That is target leakage, not a successful advance forecast. A field's name or correlation is not enough; ask how and when it was produced.

Field Meaning Availability and role
id and source Stable fictional record identity and snapshot version Used for provenance and duplicate checks rather than predictive scoring dispatch_at UTC time when a forecast would be requested Defines the prediction event and chronological split distance_km Planned route distance in kilometres Declared available at dispatch; permitted input planned_minutes Planned duration in minutes Declared available at dispatch; permitted input actual_minutes Completed duration in minutes Outcome information; forbidden as an advance-prediction feature late Whether actual exceeds planned duration Reference label rather than an input feature label_available_at Time the label is recorded as available Controls which past outcomes could be used at a training cutoff

Provenance records where a value came from, which version was used and what transformations occurred. Here the source is synthetic-delivery-v1, every record is authored and UTC timestamps end in Z. The program records a SHA-256 fingerprint of the ordered input snapshot. A fingerprint helps detect a changed input; it does not prove that a source is accurate, licensed, representative or truthful. A timestamp written in a row is also an assertion that needs evidence in a real system.

Draw the pipeline as immutable source snapshot → schema and consistency checks → duplicate and conflict review → time-aware split → training-only preprocessing → proposed prediction → later evaluation. Mark the boundary between outcome data used to check a prediction and features that would have existed beforehand. Keep a data dictionary with units, missing-value meaning, identity rules, label definition and responsible reviewer. A collection of functions without this context is not a reproducible scientific claim.

CodeAI introduces machine learning. Ask what counts as evidence at prediction time rather than merely evidence present in a later spreadsheet. Our program does not train a model. The authored distance threshold used later is declared in advance to expose the leakage shortcut, not selected because it scored well on the test rows.

Attempt an availability audit: identify two permitted inputs, one reference label and one forbidden future feature. Hint: the decision must be made at dispatch. Partial step: actual duration is a result, however convenient it looks in the final table. Check after attempting that your diagram distinguishes event time, feature availability and label availability rather than calling every timestamp simply the date.


Preserve difficult records and split by time

Our raw fixture contains twelve rows for ten distinct fictional IDs. The first eight describe deliveries A01 through A08 on 1–8 January 2026, each dispatched at 08:00 UTC. There is an identical extra copy of A02, a conflicting second version of A03, A09 with missing distance and A10 with a label inconsistent with its own durations. These deliberate faults are visible teaching cases, not errors discovered in a real company's data.

Identical A02 rows represent one delivery, so the audit retains one and records one removed copy. Both A03 versions are internally plausible but disagree about the outcome: one says 40 minutes and late, the other 20 minutes and not late. The program quarantines both versions instead of selecting whichever improves a score. A09 remains unresolved because its required distance is missing; no zero or guessed distance is substituted. A10 says 45 minutes against a 30-minute plan but late=false, so its inconsistent label is quarantined for review. None of these original rows is silently rewritten.

Audit decision IDs or rows Reason
Retain seven distinct records A01 A02 A04 A05 A06 A07 A08 Pass the declared checks Remove one identical extra copy A02 Same identity and all values Quarantine two versions A03 Contradictory outcomes need source review Quarantine one row A09 Required distance is unanswered Quarantine one row A10 Label disagrees with the declared duration rule

The chosen training cutoff is 5 January at 08:00 UTC. For a record to enter training, its dispatch must be strictly before that instant and its label must already be available by the cutoff. A01 and A02 qualify. A04 was dispatched on 4 January but its label is not available until 6 January at 10:00, so it enters a pending-label group. It must not be used for training merely because its delivery date is in the past. A05 is dispatched exactly at the cutoff and belongs to the test group, along with A06–A08.

Group at the cutoff IDs Purpose
Training-ready A01 A02 Earlier dispatches with available outcomes Past but label pending A04 Wait or investigate; do not pretend its outcome was known Later evaluation cases A05 A06 A07 A08 Forecast events at or after the cutoff

We inspect test outcomes only retrospectively, after all labels have become available in the fictional snapshot. That does not make those outcomes available at the earlier prediction event. The assumed future labels must remain outside model fitting and tuning. In a real reconstruction, version histories must establish what was actually known at each time; a final spreadsheet alone may not be enough.

A random row split can mix future and past events, and a duplicated event can appear on both sides of a split if identity is ignored. Chronology addresses this specific forward-in-time question, but it is not a universal guarantee. Related entities, revised records, delayed outcomes and overlapping prediction windows may require additional separation. Scikit-learn's TimeSeriesSplit documentation explains time-ordered evaluation and its assumptions; our custom cutoff audit is not an implementation of that library's cross-validation method.

Try a boundary case: move a label-availability time from one second before the cutoff to exactly the cutoff and then one second after it. Hint: our policy permits availability by the cutoff, including equality, while requiring dispatch strictly before it. After attempting, record which group changes and why. Do not resolve disagreement by dropping the timestamp or rerunning random splits until the result looks better.


Freeze transformations and report honest evidence

A preprocessing parameter estimated from data belongs to the training process. The two training-ready distances are two and six kilometres, giving mean four. Centre later distances by subtracting that frozen value: test distances two, six, three and seven become minus two, two, minus one and three. The mean of all seven retained distances is 30/7, not four. Using it would let future and label-pending examples influence the transformation. No test labels are required for this kind of information leakage to occur.

The demonstration also compares two predeclared calculations on four later labels. An authored rule predicts late when distance is at least five kilometres. The forbidden shortcut predicts late by comparing actual with planned minutes. The shortcut matches all four labels because it repeats their definition. The permitted rule matches only two of four. This does not show that the rule is useful or that a real model would achieve 50 percent accuracy; the dataset is tiny, synthetic and deliberately constructed.

Test ID Distance km Actual minutes known later Late reference Distance-rule prediction
A05 2 45 True False
A06 6 25 False True
A07 3 20 False False
A08 7 50 True True

The four-out-of-four shortcut is not an acceptable candidate for an advance forecast. A high score cannot repair unavailable information. The two-out-of-four result is an arithmetic description of the selected fictional cases, not a deployment recommendation. Quarantined and pending cases still matter: report their counts and reasons because excluding difficult records can change coverage and bias an evaluation. Any authorised future service would need relevant independent data, useful baselines, error consequences, monitoring and responsible review.

CodeAI discusses training data and bias. Relate it to the missing-distance and conflicting-outcome cases: what kinds of events might disappear if preparation keeps only easy rows? More rows are not automatically better data. A repeatable pipeline can consistently repeat an unsuitable assumption, so reproducibility and validity need separate attention.

The complete standard-library program below was actually executed. Its input construction is included, so the twelve rows and every audit decision can be reproduced without a private file or account. It produces a quarantine report rather than modifying the source, freezes a training-only mean and calculates the two descriptive matches. It performs no network request, fits no classifier and uses no live delivery or learner data.

"""An offline synthetic audit. No delivery service, model training or live data."""
from collections import defaultdict
from datetime import datetime, timedelta
from fractions import Fraction
import hashlib
import json

FIELDS = set(["id", "source", "dispatch_at", "label_available_at",
              "distance_km", "planned_minutes", "actual_minutes", "late"])
SOURCE = "synthetic-delivery-v1"
CUTOFF = "2026-01-05T08:00:00Z"


def timestamp(value):
    if not isinstance(value, str) or not value.endswith("Z"):
        raise ValueError("Expected an explicit UTC timestamp")
    result = datetime.fromisoformat(value.replace("Z", "+00:00"))
    if result.tzinfo is None or result.utcoffset() != timedelta(0):
        raise ValueError("Expected a timezone-aware UTC timestamp")
    return result


def record(day, distance, actual, delay_days=0):
    dispatch = timestamp(f"2026-01-{day:02d}T08:00:00Z")
    available = dispatch + timedelta(hours=2, days=delay_days)
    return dict(id=f"A{day:02d}", source=SOURCE,
                dispatch_at=dispatch.isoformat().replace("+00:00", "Z"),
                label_available_at=available.isoformat().replace("+00:00", "Z"),
                distance_km=distance, planned_minutes=30,
                actual_minutes=actual, late=actual > 30)


def fixture():
    distances = [2, 6, 3, 4, 2, 6, 3, 7]
    actuals = [20, 40, 40, 35, 45, 25, 20, 50]
    rows = [record(i + 1, d, a, 2 if i == 3 else 0)
            for i, (d, a) in enumerate(zip(distances, actuals))]
    rows.append(dict(rows[1]))  # One identical copy, not another delivery.
    rows.append(record(3, 3, 20))  # Conflicts with the original A03 outcome.
    missing = record(9, 2, 25)
    missing["distance_km"] = None
    rows.append(missing)
    inconsistent = record(10, 5, 45)
    inconsistent["late"] = False
    rows.append(inconsistent)
    return rows


def problem(row):
    if not isinstance(row, dict) or set(row) != FIELDS:
        return "schema"
    if row["source"] != SOURCE or not isinstance(row["id"], str) or not row["id"]:
        return "identity_or_source"
    if type(row["distance_km"]) is not int or row["distance_km"] not in range(101):
        return "missing_or_invalid_distance"
    for field in ["planned_minutes", "actual_minutes"]:
        if type(row[field]) is not int or row[field] not in range(1, 601):
            return "invalid_minutes"
    if type(row["late"]) is not bool:
        return "invalid_label_type"
    if row["late"] != (row["actual_minutes"] > row["planned_minutes"]):
        return "inconsistent_label"
    try:
        if timestamp(row["dispatch_at"]) > timestamp(row["label_available_at"]):
            return "label_before_dispatch"
    except (ValueError, TypeError):
        return "invalid_timestamp"
    return None


def prepare(rows):
    if not isinstance(rows, list):
        raise ValueError("Expected a list of authored records")
    groups, retained, quarantined = defaultdict(list), [], []
    duplicates = 0
    for row in rows:
        reason = problem(row)
        if reason:
            name = row.get("id", "unknown") if isinstance(row, dict) else "unknown"
            if not isinstance(name, str) or not name:
                name = "unknown"
            quarantined.append(dict(id=name, reason=reason, copies=1))
        else:
            groups[row["id"]].append(row)
    for name in sorted(groups):
        copies = groups[name]
        versions = set(json.dumps(r, sort_keys=True) for r in copies)
        if len(versions) != 1:
            quarantined.append(dict(id=name, reason="conflicting_versions", copies=len(copies)))
        else:
            retained.append(json.loads(next(iter(versions))))
            duplicates += len(copies) - 1
    quarantined.sort(key=lambda r: (r["id"], r["reason"]))
    return dict(retained=retained, quarantined=quarantined,
                identical_copies_removed=duplicates)


def split_asof(rows, cutoff_text):
    cutoff = timestamp(cutoff_text)
    train, pending, test = [], [], []
    for row in rows:
        if cutoff > timestamp(row["dispatch_at"]):
            target = train if cutoff >= timestamp(row["label_available_at"]) else pending
            target.append(row)
        else:
            test.append(row)
    return dict(train=train, pending_label=pending, test=test)


def demonstrate():
    raw = fixture()
    audit = prepare(raw)
    split = split_asof(audit["retained"], CUTOFF)
    train, test = split["train"], split["test"]
    train_mean = Fraction(sum(r["distance_km"] for r in train), len(train))
    all_mean = Fraction(sum(r["distance_km"] for r in audit["retained"]),
                        len(audit["retained"]))
    baseline_correct = sum((r["distance_km"] >= 5) == r["late"] for r in test)
    leaked_correct = sum((r["actual_minutes"] > r["planned_minutes"]) == r["late"]
                         for r in test)
    return dict(raw_rows=len(raw), input_snapshot_sha256=hashlib.sha256(
                    json.dumps(raw, sort_keys=True).encode()).hexdigest(),
                retained_ids=[r["id"] for r in audit["retained"]],
                quarantine=audit["quarantined"],
                identical_copies_removed=audit["identical_copies_removed"],
                split_ids=dict((k, [r["id"] for r in v]) for k, v in split.items()),
                train_distance_mean=train_mean, inappropriate_all_row_mean=all_mean,
                centred_test_distances=[r["distance_km"] - train_mean for r in test],
                predeclared_rule_correct=baseline_correct, test_cases=len(test),
                leaked_shortcut_correct=leaked_correct,
                classifier_trained=False, live_delivery_data_used=False)


if __name__ == "__main__":
    print(json.dumps(demonstrate(), default=str, indent=2))

Keep a test log with source version and fingerprint, field definitions, audit counts, excluded IDs and reasons, cutoff and equality policy, available-label checks, permitted features, frozen transformation, expected and actual outputs and remaining uncertainty. Name a reviewer for conflicting labels and an owner for monitoring availability assumptions. The program's format checks do not verify a real source's ownership, permission, timing truth or representativeness.

Transfer workshop: adapt the plan to a fictional equipment-loan return forecast. Replace delivery-specific fields with a loan-time prediction event, planned return, later actual return and label-availability record. Revisit repeated borrower or equipment identities, changing policies, units, permissions and missing data. Use invented records, not pupil borrowing histories. Explain which checks transfer and which need a new definition. The source's KS3 creative-project objective supports this editorial Year 9 data project; it does not prescribe this particular pipeline or certify real-world forecasting competence.


Interactive Tasks


Quiz: Test Your Knowledge

Which field is forbidden for an advance lateness forecast at dispatch? (The later actual duration) (!The distance declared available at dispatch) (!The planned duration declared available at dispatch) (!The prediction-event time)



Why does the actual-duration shortcut match every valid reference label? (It repeats the outcome definition using future information) (!It proves useful advance forecasting) (!It was trained on millions of real deliveries) (!A random split removed all temporal risks)



How is the identical extra A02 copy handled? (Retain one delivery and record one removed copy) (!Treat it as a second independent delivery) (!Put a copy in each split) (!Change its ID until both copies look unique)



How are the conflicting A03 versions handled? (Quarantine both for source review) (!Select whichever raises the test score) (!Average the two Boolean labels) (!Delete the conflict history)



Why does A04 stay out of training at the cutoff? (Its label was not available by the cutoff) (!Its distance is necessarily false) (!All records from January are invalid) (!Every past dispatch must be in the test group)



Where does a dispatch exactly at the chosen cutoff belong? (The test group) (!The training-ready group automatically) (!The duplicate group) (!A group selected by the higher score)



What is the frozen mean of training-ready distances? (Four kilometres) (!Thirty sevenths kilometres) (!Five kilometres) (!Zero kilometres)



What is wrong with estimating the transformation from all retained rows? (Later and pending examples influence the preparation) (!All transformations are forbidden) (!The arithmetic sum is impossible) (!It automatically proves label accuracy)



What does the predeclared distance rule score on the four test cases? (Two matching labels out of four) (!Four matching labels out of four) (!A guaranteed future success probability) (!An independent deployment certification)



What does a snapshot fingerprint establish? (A way to detect changed input bytes or representation) (!Proof of source truth permission and representativeness) (!That every feature existed at prediction time) (!That all missing values should be replaced by zero)





Memory Game

Provenance The origin version and processing history of data
Feature Information permitted at the prediction event
Label The reference outcome used to check a prediction
Leakage Information entering a model or evaluation when it should be unavailable
Cutoff A declared time boundary for the preparation
Quarantine Unresolved records retained outside use pending review





Drag and Drop

Keep a future actual duration out of input features Prevent target leakage
Keep an outcome unavailable at the cutoff out of training Check label availability
Record one retained event from identical copies Deduplicate with identity
Hold both contradictory versions for review Preserve a conflict
Estimate a transformation only on the permitted training subset Freeze preprocessing





Crossword Puzzle

SOURCE Where a data record originates
LABEL A reference outcome
FEATURE A permitted input value
CUTOFF A chosen time boundary
LEAKAGE Use of information that should not be available
VERSION An identifier for a retained data state





LearningApps

Optional extra practice. Choose an appropriate activity with your teacher. If the external collection is unavailable, use the native activities and independent tasks in this course.


Cloze Text

Complete the explanation.
The forecast is required at

. Actual duration is future outcome

. The reference outcome is a

. An identical event copy is a

. Contradictory versions go to

. A04 has a label still

. The preparation uses a time

. Only training data estimate the frozen

. A fingerprint helps detect a changed

. A high score cannot justify data

.



Open Tasks


Easy

  1. Write the data dictionary: Name each field meaning unit source and availability. Hint: actual duration and a late label have different roles from distance and plan. Explain why an ID supports provenance without being used as an arbitrary predictive shortcut.
  2. Spot the future shortcut: Explain why comparing actual and planned minutes is not a forecast at dispatch. Hint: a perfect-looking score can repeat the label definition. State the prediction event and the missing future information.
  3. Count the preparation decisions: Reconcile twelve raw rows with seven retained records one identical extra copy and four quarantined rows. Hint: A03 contributes two conflicting versions. Preserve IDs and reasons rather than calling all excluded records unimportant.
  4. Separate the time fields: Explain dispatch time label availability and the training cutoff. Hint: a past event can have a late-recorded outcome. Use A04 to show why a delivery date alone is insufficient.


Standard

  1. Reproduce the split: List training pending and test IDs at the chosen UTC cutoff. Hint: dispatch is strictly before the cutoff for training and label availability includes equality. Explain the exact-cutoff A05 boundary.
  2. Test availability boundaries: Check a label one second before exactly at and one second after the cutoff for a past dispatch. Hint: change one assumption at a time. Record expected and actual group assignments without changing the policy afterwards.
  3. Freeze the transformation: Calculate the training mean and centred test distances and compare with the all-row mean. Hint: training distances are two and six. Explain how future features can leak into preprocessing even without using future labels.
  4. Audit the two scores: Recalculate both descriptive matches on the four test rows. Hint: the actual-duration shortcut repeats the label rule. Explain why neither four-out-of-four leakage nor two-out-of-four on a tiny fixture is deployment evidence.


Advanced

  1. Challenge random splitting: Give a concrete future-to-past or duplicate-across-splits risk and an appropriate separation plan. Hint: chronology addresses one task assumption but does not solve related-entity or overlapping-window leakage automatically.
  2. Document missing and conflicting data: Propose a review process for A03 A09 and A10 with responsible roles evidence and decisions that preserve originals. Hint: no outcome or distance should be invented merely to complete a table. Report the effect of exclusions on coverage.
  3. Build a reproducible audit record: Include snapshot fingerprint units availability provenance IDs cutoff frozen parameters expected actual results and limitations. Hint: a hash detects a change but does not certify source quality permission or truth.
  4. Transfer to equipment returns: Design a preparation and split plan for a fictional equipment-loan return forecast. Hint: define prediction and label availability times again and consider repeated entities. Use invented records and state which checks need new policies before actual use.



Learning Check

  1. Identify three pipeline risks: Name three distinct risks and their checks.
  2. Reconcile the source audit: How are twelve raw rows accounted for?
  3. Apply the time rules: Which IDs enter each group and why?
  4. Check preprocessing arithmetic: What values should be frozen and why?
  5. Explain the misleading high score: Compare the two descriptive matches without overclaiming.
  6. Specify a responsible transfer: What must an equipment-loan version revisit?


Worked checks and assessment guidance

  1. Future actual duration leaks the target and must remain outside advance features. Duplicated or contradictory identities require recorded deduplication or quarantine. Past dispatches with unavailable labels must not enter training. Missing values inconsistent labels future-informed preprocessing and related entities are additional checks whose policies need documentation.
  2. Seven distinct records A01 A02 A04 A05 A06 A07 A08 are retained. One identical extra A02 copy is recorded as removed. Two A03 versions are quarantined for conflict plus A09 for missing distance and A10 for its inconsistent late label. Seven plus one plus four equals twelve and the original fixture remains unchanged.
  3. At 5 January 2026 08:00 UTC A01 A02 are training-ready. A04 was dispatched earlier but its label is available only 6 January 10:00 so remains pending. A05 is exactly at the cutoff and enters test with A06 A07 A08. Retrospective evaluation may inspect later labels but cannot pretend they were known at dispatch.
  4. The training distances two and six have mean four. Later distances two six three seven centre to minus two two minus one three. All retained distances have mean thirty sevenths which would use future and pending examples. Fit preparation only on permitted training data and apply that fixed transformation consistently.
  5. The authored distance-at-least-five rule matches A07 and A08 giving two of four. Comparing actual with planned duration matches all four by repeating the outcome definition. The latter is unavailable at prediction time and invalid for an advance forecast. Neither tiny synthetic result establishes a real model success rate or useful deployment.
  6. Define the new prediction event fields units identities planned and actual outcomes availability delays missing-value and conflict policies and relevant permissions. Keep source versions expected and actual tests frozen preparation exclusions and review owners. Check related entities and time assumptions afresh rather than copying labels or claiming a real pupil-data experiment.


Learning evidence

Retain an availability-aware data dictionary, a pipeline diagram with trust boundaries, a twelve-row reconciliation and quarantine reasons, exact time-aware split IDs, boundary tests and a frozen training mean. Include expected and actual score calculations a source fingerprint and a new-domain plan. Distinguish a deterministic synthetic audit from model fitting live forecasting source certification or observed learner competence.


OER and Wikipedia

scikit-learn: Common pitfalls and recommended practices, data-leakage section, explains why unavailable information and test-informed preparation can give misleading performance. scikit-learn: TimeSeriesSplit documents time-ordered evaluation and its assumptions. Relevant primary sections inspected 15 November 2026. Our original one-cutoff standard-library audit is not a reproduction of the library's cross-validation API.

DfE: Computing programmes of study, KS3 creative-project content, supplies the broad source anchor. The specific Year 9 delivery-pipeline example and assessment decomposition are editorial, not a statutory requirement for this particular forecasting project.

ETL jobb.jpg, Hansth, public domain. Original generic ETL-job illustration embedded unchanged. Its pictured software diagram does not depict the authored delivery dataset or certify a live pipeline.

https://www.youtube.com/watch?v=OeU5m6vRyCk — CodeAI: AI: What is Machine Learning? https://www.youtube.com/watch?v=x2mRoFNm22g — CodeAI: AI: Training Data & Bias. Actual metadata playability embedding and preview bytes verified; previews visually inspected. Original recordings embedded unchanged without transcript copying endorsement redistribution or relicensing implied. Complete written explanations are provided independently of playback.

All delivery dates IDs distances durations labels faults availability delays and the equipment-loan scenario are original fictional teaching constructions. The complete offline audit was actually executed; no classifier was fitted, no live service or learner data was accessed, and no source permission or operational reliability was certified. No local generative LLM or paid API was used.


Linked learning areas


aiMOOC Projects



Schulfach+




aiMOOCs



aiMOOC Projekte









MOOCwiki · Deutsch

Nach dem Lernen ist vor dem Lernen

Entdecke direkt den nächsten Lernkurs. Weitere Inhalte erscheinen, wenn Du weiter nach unten scrollst.

Zur MOOCwiki-Hauptseite

Mediathek

Suche verfeinern

Dein Lernplan bestimmt, was oben steht. Alle anderen Inhalte bleiben erreichbar. Seitenaufrufe beeinflussen die Auswahl nicht.

Inhalte werden geladen ...

Mediathek wird aus dem Wiki geladen ...