Zum Inhalt springen

English:Training, validation, testing and generalisation - AI Year 6

Aus MOOCsWiki Staging
aiMOOC-Siegel aiMOOC

Training, validation, testing and generalisation - AI Year 6

QR-Code



Introduction

Smooth and winding classification boundaries with red blue and black-outlined points
Compare fitting a sample with checking new evidence. This original source graphic is separate from our synthetic allocation and prediction logs.

Fit, choose, then test: three different jobs need clearly separated evidence. Plan one hundred records, compare two prepared candidates and audit a frozen final test without hiding mistakes. All records are fictional; no actual model training or real messages are claimed.


Plan three roles and protect source independence


Three subsets have three different jobs

Training examples are used to fit a model. Validation examples help compare development choices, such as a model family or threshold. A final test set is kept out of those choices and used to evaluate the frozen method. These roles matter more than memorising a percentage. An example that repeatedly influences model choices is no longer an untouched final test.

The introductory graphic contrasts a very winding boundary with a smoother one on its own red and blue points. The black-outlined points illustrate new cases in the source image. Examine why following every bend of one sample may fail on another. The picture's points are not the one hundred records in our separate classroom plan, and a visual impression alone is not a measured performance guarantee.

Our fictional task routes message cards to adult review: one means REVIEW and zero KEEP under an explicit teacher-supplied target definition. KEEP is not a safety guarantee and REVIEW is not a judgement of a real person. No emails, addresses or actual messages are used. We use original prepared prediction logs to examine evaluation decisions; no actual training run or external AI service generated them.


Plan one hundred independent source records before tuning

Imagine one hundred original synthetic source records, R001 to R100, each initially linked to a different source ID, S001 to S100. Assign R001–R070 to training, R071–R085 to validation and R086–R100 to test: seventy, fifteen and fifteen. Every record has exactly one role and all one hundred are accounted for.

Subset Record IDs Count Permitted role
Training R001 to R070 70 Fit model parameters
Validation R071 to R085 15 Guide declared development choices
Test R086 to R100 15 Evaluate the frozen selected method

This ordered allocation is a transparent planning illustration, not a claim that sorting real records by ID makes a representative random split. Actual time order, related sources and intended use may require a suitable grouped or time-aware design. Seventy/fifteen/fifteen is our worked allocation, not a universal requirement or proof that fifteen tests are enough for a real deployment.

CodeAI: AI: What is Machine Learning? With teacher support, identify fitting from examples and the need to inspect different cases. The publisher's broader unit is for older grades; our Year 6 activity is a concrete paper planning/evaluation exercise. It needs no live pupil AI account or real inbox.


Keep source identity separate from a row name

Now consider a separate faulty copy of the split plan: R100 is discovered to be another copy of source S010, already represented by R010 in training. Different row names did not make the underlying source independent. Testing on the copied source can reward remembering a training example. Keep this fault separate from the clean plan and do not advertise the faulty one as one hundred independent sources.

Repair the source/group policy before fitting and reserve genuinely independent final cases. Removing the copied record changes the available counts; grouping related records can also change subset sizes. Do not invent a new independent observation or silently keep the old seventy/fifteen/fifteen claim after a repair. Document the changed inventory and reassess whether a fresh test is needed.



Use validation for an explicit development choice


Declare how selection will be made

Suppose a developer offers two candidate methods, A and B. Their following validation outputs are deliberately prepared original teaching records, not measured outputs from a real trained system. The seventy training IDs show where fitting belongs in the plan; this lesson does not fabricate their features, training history or a claim that a model was actually fitted on them.

For our main classroom comparison, decide in advance to select the candidate with more correct validation predictions. Candidate A comes first if the counts tie. Cover the test log while making this choice. Keep both validation prediction lists; a headline score alone makes errors hard to inspect.

Validation ID Target Candidate A Candidate B
R071 1 0 1
R072 1 1 0
R073 1 1 0
R074 1 1 1
R075 1 1 1
R076 1 1 1
R077 1 1 1
R078 0 1 0
R079 0 1 0
R080 0 0 0
R081 0 0 0
R082 0 0 0
R083 0 0 0
R084 0 0 0
R085 0 0 0

A matches twelve of fifteen. It misses one actual REVIEW case and makes two unnecessary review suggestions. B matches thirteen of fifteen, but misses two actual REVIEW cases and makes no unnecessary review suggestions. Under our declared correct-count criterion, select B and freeze the decision before looking at the test outcomes.


A different error cost can change the choice

Consider a second, explicitly hypothetical task where a missed REVIEW costs three classroom penalty units and an unnecessary REVIEW costs one. The units are invented, not financial damages or a universal moral scale. Declare that criterion before its comparison too. A has one miss and two unnecessary suggestions, cost five. B has two misses and no unnecessary suggestions, cost six. The lower-cost choice for that different criterion is A.

This contrast explains why accuracy alone cannot settle suitability. It is not permission to inspect final test results and then invent whichever criterion makes a chosen model look best. Define the intended use and relevant error consequences with responsible people first, then retain the chosen criterion and reference version in the evidence log.


Check the complete allocation and prediction lists

This complete Python reference counts the one hundred planned IDs and the two fifteen-case logs. It was actually executed during course preparation. It does not train a model or access a service. Trace the lists on paper or use an approved environment; integer one/zero are the two target categories, not probabilities.

def counts(truth, predicted):
    if not truth or len(truth) != len(predicted):
        raise ValueError("nonempty matched outcome lists required")
    if any(type(x) is not int or x not in (0, 1)
           for x in list(truth) + list(predicted)):
        raise ValueError("outcomes must be integer zero or one")
    return {
        "TP": sum(y == 1 and p == 1 for y, p in zip(truth, predicted)),
        "TN": sum(y == 0 and p == 0 for y, p in zip(truth, predicted)),
        "FP": sum(y == 0 and p == 1 for y, p in zip(truth, predicted)),
        "FN": sum(y == 1 and p == 0 for y, p in zip(truth, predicted)),
    }


split = {"training": list(range(1, 71)),
         "validation": list(range(71, 86)),
         "test": list(range(86, 101))}
print("split_sizes", {name: len(ids) for name, ids in split.items()})
validation_truth = [1] * 7 + [0] * 8
a = validation_truth.copy()
b = validation_truth.copy()
for index in (0, 7, 8):
    a[index] = 1 - a[index]
for index in (1, 2):
    b[index] = 1 - b[index]
print("validation_A", counts(validation_truth, a))
print("validation_B", counts(validation_truth, b))
test_truth = [1] * 6 + [0] * 9
test_b = [1] * 4 + [0] * 2 + [1] * 3 + [0] * 6
print("frozen_B_test", counts(test_truth, test_b))

Expected subset sizes are training seventy, validation fifteen, test fifteen. Validation A has TP six, TN six, FP two, FN one. Validation B has TP five, TN eight, FP zero, FN two. The frozen B test counts are TP four, TN six, FP three, FN two. The actual program's dictionary output uses these abbreviations in that order. It rejects unequal lists or invalid outcome values instead of silently truncating them with zip.


Two further leakage checks

First, do not use test labels to choose the candidate, threshold, features or explanation you report as selected. Calling a file test does not make it independent if its results guided development. After changing a method based on the final test, treat that test as development evidence and plan a genuinely separate final evaluation.

Second, any fitted preprocessing information must come from the permitted training data. For example, if a future method estimates a missing-value replacement from data, do not calculate it using the entire dataset before splitting, including future test records. Retain the training-only procedure and apply the frozen transformation consistently. Our prepared log has no such replacement step; this is a proposed pipeline risk, not a claim that an imputation experiment was performed.



Audit the frozen test and limit conclusions


Inspect the frozen final test rather than retuning on it

For the main correct-count selection, B has already been chosen on validation. Only now uncover the separate fifteen-case test log. Record its results even if they are less attractive than the validation score. Keep the tested version and criterion in the record; do not quietly substitute A after seeing these results.

Test ID Target Frozen B output
R086 1 1
R087 1 1
R088 1 1
R089 1 1
R090 1 0
R091 1 0
R092 0 1
R093 0 1
R094 0 1
R095 0 0
R096 0 0
R097 0 0
R098 0 0
R099 0 0
R100 0 0
Confusion matrix with predicted classes in rows and actual classes in columns
Define positive as REVIEW, then place every test case in one of the four cells. This source diagram does not supply our original prediction values.

R086–R089 are four true positives; R090–R091 are two false negatives. R092–R094 are three false positives and R095–R100 six true negatives. B therefore has ten correct and five wrong among fifteen test records. An always-KEEP baseline matches the nine zero targets and misses all six one targets. Reporting only ten versus nine correct would conceal the different kinds of errors and the need for human review.


Interpret generalisation with an honest evidence boundary

Generalisation concerns how a learned method behaves beyond its fitting examples. A held-out test is useful evidence under its actual conditions, not a guarantee for every future setting. Here even the prediction lists are prepared teaching records: they demonstrate evaluation arithmetic and decisions, not a measured real system's generalisation or fairness.

The smaller final-test match count does not by itself prove a particular cause such as overfitting. The samples are small and invented. In a real investigation one would inspect the data, distribution, uncertainty and development history before diagnosing why results differ. A single aggregate score cannot establish the experience of every subgroup or cover every kind of input.

CodeAI: AI: Training Data & Bias. Discuss which people, sources or conditions may be absent from an evaluation. Our logs contain no demographic attributes, so they cannot prove equal outcomes between real groups. A meaningful reviewer needs examples and authority to challenge an error, not only a reassuring percentage.


Attempt a small audit before reading the worked checks

Verify that the planned subset counts sum to one hundred and that no row ID is shared. Then explain why the separate S010 copy still leaks a source despite unique row IDs. Finally compare A and B using the main criterion and the alternate penalty criterion, naming which was used. Partial step: two false negatives at three units each cost six; finish A's cost independently.

For a boundary case, compare the last training row R070 with first validation row R071, and the last validation row R085 with first test row R086. Off-by-one mistakes can silently alter the advertised allocation. For the evaluation function, an empty list, a missing prediction or the word one is invalid; do not silently count a shorter paired list as the whole test.


Transfer and preserve the unresolved questions

A fictional environmental project might have several readings from the same sensor or location. Splitting rows at random may place closely related sources in both development and test data. Decide what independence means for the intended future use; keep relevant source groups together or use an appropriate time-based evaluation. Recheck target definitions and error costs instead of copying the message-card criteria unchanged.

Keep the one hundred-ID plan, clean/faulty source versions, selection rule, validation lists, frozen test log, error counts, baseline and human fallback. State two limitations: small prepared samples and incomplete coverage of future conditions. Propose a new independent source/period trial but label it unperformed. No actual pupil data, live service, model training or deployment authorisation is claimed. This is editorial AI Year 6 enrichment linked to KS2 data evaluation.



Interactive Tasks


Quiz: Test Your Knowledge

Which subset is used to fit model parameters? (Training) (!The final test used repeatedly for choices) (!Only the page title) (!A media credit list)



Which planned record begins validation? (R071) (!R070) (!R085) (!R086)



Which allocation is used in this worked plan? (Seventy training fifteen validation fifteen test) (!Fifty training fifty validation fifteen test) (!One hundred test records and no training) (!A universal rule required for every project)



Why does the faulty R100 copy of S010 cause a problem? (Its underlying source already appears in training) (!All different row IDs guarantee independence) (!It creates a new independent observation) (!It proves the final model is fair)



Which candidate wins the declared validation correct-count comparison? (B with thirteen correct) (!A with thirteen correct) (!B because the final test was already tuned) (!Always-KEEP with fifteen correct)



Which candidate has lower cost under the separate three-to-one penalty rule? (A with cost five) (!B with cost five) (!A with cost six) (!Neither because counts cannot be compared)



How many correct predictions does frozen B have in the final test log? (Ten of fifteen) (!Thirteen of fifteen) (!Fifteen of fifteen) (!Nine of fifteen)



Which final-test count represents unnecessary REVIEW suggestions? (Three false positives) (!Two false negatives) (!Six true negatives) (!Four true positives)



What changes if final test results guide a new model choice? (That test has become development evidence) (!Its independence is automatically restored) (!All previous results must be hidden) (!The model is guaranteed to improve)



What was actually executed in this lesson? (A program checking prepared allocation and outcome logs) (!A real message-filter training run on seventy emails) (!A pupil account upload) (!An independent fairness certification)





Memory Game

Training set Examples used to fit model parameters
Validation set Examples guiding development choices
Test set Reserved final evaluation of the frozen method
Leakage Information crosses an intended evidence boundary
Generalisation Behaviour beyond the examples used for fitting
Source group Related records that may share underlying information





Drag and Drop

A copied training source in the test set Reconcile related sources before fitting and recheck counts
Repeated choices guided by final test outcomes Treat the used test as development evidence
A data-fitted replacement computed on all records Fit that procedure on permitted training data only
A different cost for missed REVIEW cases Declare and evaluate an appropriate criterion before selection
A lower final-test match count Retain errors and investigate without inventing a cause





Crossword Puzzle

TRAINING Which subset fits model parameters?
VALIDATION Which subset guides development choices?
TEST Which subset is reserved for final evaluation?
LEAKAGE What violates an intended information boundary?
SOURCE What underlying identity matters beyond a row name?
FROZEN How should the selected method be kept for final evaluation?





LearningApps

Optional extra practice. Choose an appropriate activity with your teacher. If the external collection is unavailable, use the native activities and independent tasks in this course.


Cloze Text

Complete the explanation.
Use

to fit,

to guide choices and

for final evaluation. Our plan has

training records and fifteen in each other subset. Main validation selection chooses

. Frozen B has

correct test cases. A copied source can cause

. Reusing test outcomes for choices turns them into

evidence. The program checks prepared

, not a real trained service.



Open Tasks


Easy

  1. Give three roles: Distinguish fitting, development choices and final evaluation.
  2. Check the boundaries: Trace R070/R071 and R085/R086 into their planned subsets.
  3. Read a reference class: Define positive before reading the confusion diagram.
  4. Name the evidence limit: Explain why prepared predictions are not a measured live model.


Standard

  1. Audit one hundred IDs: Account for seventy/fifteen/fifteen and check row/source overlap separately.
  2. Compare validation candidates: Count A/B matches and explain the main selection.
  3. Trace fifteen final tests: Classify all four outcomes for frozen B without retuning.
  4. Correct two leakage risks: Address repeated source copies and using final test results for development.


Advanced

  1. Change the error criterion: Compute the hypothetical three-to-one penalty and explain why its declared choice differs.
  2. Audit preprocessing: Explain why estimating a transformation using future test data leaks information.
  3. Transfer to environmental readings: Define source/time independence, targets and consequences for a different use.
  4. Plan a new independent trial: State limitations and a future source/period test, retaining human authority and the original results.



Learning Check

  1. Three roles and one hundred records: Verify the planned allocation.
  2. Source leakage and preprocessing: Correct two different risks.
  3. Validation choice and alternate cost: Compare both declared criteria.
  4. Final test and baseline: Read the original fifteen-case log.
  5. An independent test can wear out: Explain repeated tuning and a boundary.
  6. Transfer and future evidence: Design a different source-aware evaluation.


Worked checks and assessment guidance

  1. R001–R070 gives seventy training records, R071–R085 fifteen validation records, and R086–R100 fifteen tests. Training fits, validation guides declared choices and test evaluates a frozen method. This ordered illustration is not proof of random or representative sampling and the percentages are not universal.
  2. A faulty R100 sharing source S010 with training R010 is not independent despite a new row ID. Reconcile groups and counts before fitting; do not invent a replacement. Fit preprocessing only on permitted training data, then apply the frozen procedure. Do not estimate a replacement using future test records.
  3. A has TP6 TN6 FP2 FN1: twelve correct. B has TP5 TN8 FP0 FN2: thirteen correct, so main selection is B. In the separate hypothetical cost task, A costs three plus two equals five; B costs six. That different criterion chooses A; it is not a post-test excuse to change the main result.
  4. Frozen B has TP4 TN6 FP3 FN2, ten correct and five wrong. Always-KEEP has nine true negatives and six false negatives. Count all records and error types; a small prepared log cannot establish live generalisation, fairness or safety.
  5. Using test outcomes to choose features, thresholds or models turns them into development evidence. Keep that history and plan a genuinely separate final evaluation. Check R085 remains validation and R086 starts test. A fresh filename alone does not create new independent evidence.
  6. Related readings may share a sensor, place or time period. Define the intended future use and choose appropriate grouping or time-based evaluation. Recheck targets and error costs, retain errors and reviewer authority, and mark a proposed new-period trial unperformed. The program in this lesson checks prepared logs; no real model was trained.


Learning evidence

Retain the one-hundred-ID plan, clean/faulty source versions, declared criteria, two validation logs, frozen B test and all four outcome counts. Explain two leakage risks, a baseline, the alternate penalty choice, two limitations and a genuinely unperformed new-source/period trial with human review. This documents evaluation reasoning, not actual model training or deployment certification.


OER and Wikipedia

DfE Computing supplies KS2 data-evaluation context; this annual AI pathway is editorial. Google ML Crash Course: dividing datasets was consulted for the three roles and independence risks. Google: classification metrics supports error-type/criterion distinctions. All ID allocation, prediction logs, penalty units and program are original; publisher examples and exercise solutions are not copied. Reviewed 15 November 2026.

Overfitting.svg, Chabacano, CC BY-SA 4.0. Unchanged source diagram; contextual illustration, not our original one-hundred-record or prediction data.

ConfusionMatrix.png, Jackverr, CC BY-SA 3.0. Unchanged source diagram; contextual illustration, not our original one-hundred-record or prediction data.

Overfitting.svg: the current file history also credits DancingPhilosopher for outlined new-example points. This course embeds that source file unchanged.

https://www.youtube.com/watch?v=OeU5m6vRyCk — CodeAI: AI: What is Machine Learning? Original recording embedded unchanged.

https://www.youtube.com/watch?v=x2mRoFNm22g — CodeAI: AI: Training Data & Bias. Original recording embedded unchanged. Teacher-supported context; no live pupil AI account required.



Linked learning areas


aiMOOC Projects



Schulfach+




aiMOOCs



aiMOOC Projekte









MOOCwiki · Deutsch

Nach dem Lernen ist vor dem Lernen

Entdecke direkt den nächsten Lernkurs. Weitere Inhalte erscheinen, wenn Du weiter nach unten scrollst.

Zur MOOCwiki-Hauptseite

Mediathek

Suche verfeinern

Dein Lernplan bestimmt, was oben steht. Alle anderen Inhalte bleiben erreichbar. Seitenaufrufe beeinflussen die Auswahl nicht.

Inhalte werden geladen ...

Mediathek wird aus dem Wiki geladen ...