English:Machine learning workflow and evaluation design - AI Year 10
Introduction

Define a prediction task, document its records, split them before fitting, compare against a baseline, select using validation data and reserve test data for the final report. Execute a small original threshold-learning experiment on one hundred synthetic records. The example really fits a numeric threshold, but uses no local generative language model, external AI service or pupil data. Its deliberately authored data is not evidence of real-world model performance.
Define the target and protect the split
A supervised workflow begins with a clear target. Our invented task predicts whether a fictional item should be flagged for additional inspection. The input feature x is a dimensionless synthetic reading from zero through ninety-nine. The target y is zero or one. A one means the authored inspection flag is present; a zero means it is absent. Predicting that flag is not deciding whether a real object is safe, and neither the feature nor the label is an actual laboratory measurement.
Each record contains an opaque key, a source-group marker, x, y and a role. The key identifies a record; the group indicates related observations that must not cross roles in this experiment. Neither key nor group is a model feature. The role is train, validation or test. A source marker and a declared role document an intended workflow; they are not authentication, a guarantee of independence or a substitute for checking how real observations were collected.
The full source example uses one hundred labelled records split into seventy training, fifteen validation and fifteen test examples before model selection. Training data supplies fitted parameters. Validation data compares the predeclared candidates. Test data supplies a final report after the choice is fixed. A ratio is not a universal standard: data quantity, class distribution, grouping, time order and the intended use can require another design.
Our reproducible allocation is deliberately deterministic. Within each block of twenty readings, remainders zero through thirteen go to training, fourteen through sixteen to validation, and seventeen through nineteen to test. Five blocks give seventy, fifteen and fifteen. This is not a random or statistically representative sampling claim. The complete fixture is visible for teaching, so the exercise is not a blinded trial; its code nevertheless keeps test rows out of fitting and selection.
| Role | Original record count | Permitted use in this workflow |
|---|---|---|
| Train | 70 | Fit the majority baseline and the threshold parameter |
| Validation | 15 | Choose between the two already fitted candidates |
| Test | 15 | Report the selected candidate after the choice is fixed |
The invented target is initially one for readings at least fifty, with six explicitly authored exceptions at readings seven, 52, fifteen, 75, eighteen and 78. An exception changes the target to its opposite. These are properties of a manufactured teaching fixture, not discovered human-labelling mistakes or a reason to rewrite observed labels. The program must learn from the permitted training labels rather than being told that the generating boundary is fifty.
CodeAI's AI: What is Machine Learning? supplies general context for learning from examples. Our own program below shows a small fitted model and reports exactly what was executed. It is not a neural network or a claim that a simple classroom pattern will generalise to production data. All required methods remain in the written course.
Data leakage occurs when information that should not be available for the intended prediction or independent evaluation affects development. Related views of one source object placed in both training and test can make evaluation misleading. Repeatedly choosing changes after observing test scores turns those test results into development information. Even a preprocessing step, such as learning a scaling mean from all records, can bring held-out information into fitting. Our current program uses raw readings and fits no scaler; a future fitted transformation would have to learn its parameters from training data only.
Plan first: draw target definition, dictionary, split, fitting, validation choice and final evaluation in order. Mark which records may influence each stage. Hint: the final test report is not an instruction to tune the same candidate until its test score looks better. Partial check: seventy plus fifteen plus fifteen accounts for all one hundred records, but correct counts alone do not prove independent groups or absence of leakage.
Fit candidates and choose with validation evidence
A baseline makes a comparison meaningful. The original training set has thirty-eight zero targets and thirty-two one targets. The majority baseline therefore predicts zero for every reading. Its majority value is fitted from training labels; a tie is defined to choose zero. A constant baseline is simple, transparent and sometimes hard to beat when one class is common, but it can miss every positive case.
The second candidate is a single threshold model. It predicts one when x is at least cut and zero otherwise. During fitting, try the 101 integer cuts from zero through one hundred and count mistakes on the seventy training rows. Choose the cut with the fewest training mistakes, breaking ties toward the smaller cut. This enumerated fitting method is actual supervised parameter selection from training labels. It is not an external AI call, an LLM or a manually supplied final threshold.
For the executed fixture, fitting returns cut fifty with two training mistakes, so sixty-eight of seventy training predictions are correct. The baseline has thirty-eight of seventy correct. Training performance describes how a candidate matches its fitting data; it is not an independent estimate of future performance. A more flexible method could memorise training examples and still fail elsewhere, so fitting success alone is not a reason to publish or deploy a model.
Now compare the two predeclared fitted candidates on validation rows. The baseline has six correct out of fifteen. The learned threshold has thirteen correct out of fifteen. The selection rule chooses the candidate with the larger correct count; if equal, choose the earlier predeclared candidate. This simple rule is appropriate only for the exercise's declared equal-error comparison. A real task with unequal consequences may need another predeclared selection measure and suitable validation evidence.
| Fitted candidate | Training correct | Validation correct | Decision |
|---|---|---|---|
| Majority baseline, always zero | 38 of 70 | 6 of 15 | Retain for comparison |
| Threshold cut fifty | 68 of 70 | 13 of 15 | Select under the declared validation rule |
Freeze the candidate choice before computing the final test report. The test score in this authored fixture happens to equal the validation correct count, but equality is not required and gives no extra proof of generalisation. Choosing a threshold, feature, data cleaning rule or report after studying test labels would change how the test evidence should be interpreted. If test results motivate redevelopment, document that use and arrange a genuinely new appropriate final evaluation rather than reusing the old score as untouched evidence.
The program demonstrates two specific blocked mistakes. First, it copies the fixture and assigns a test row the source-group marker of a training row. The split validator rejects that cross-role group overlap. The correction is to investigate the source relationship and keep related observations together under an appropriate split, not silently change the marker to make a check pass. In our synthetic exercise the original separate groups are known because we authored them; real group independence needs evidence.
Second, a deliberately attempted call passes test rows to the model-selection function. That function requires the validation role and raises a rejection. The correct workflow passes the validation rows, fixes the choice, then evaluates test rows. A role check is a semantic programming safeguard, not a security boundary: someone could falsely relabel data or manually tune after seeing the output. Appropriate access, records and human review remain necessary in real work.
CodeAI's AI: Training Data & Bias connects data choices with the limitations of a model. Ask whether the label, coverage, collection process and intended use are appropriate, rather than treating a neat split or a high score as proof of fairness. The program records no protected personal attributes and cannot establish demographic fairness from this invented numeric fixture.
Try before checking: identify the information route in each blocked mistake and give a correction that preserves source evidence. Hint: counts can still be seventy fifteen fifteen while related groups cross roles. Partial check: passing test rows into selection is a role error; changing a real source-group name without evidence would conceal rather than repair the first problem.
Execute the workflow and interpret each error
The complete original program below was executed. The two models receive training rows only, candidate selection receives validation rows only, and the chosen candidate is then evaluated on the fifteen test rows. The code also executes and catches the two deliberately invalid demonstrations. All identifiers, readings, targets and source groups are authored; no actual item-inspection dataset or school account is involved.
"""Original synthetic threshold fitting with separate selection and test roles."""
import json
from fractions import Fraction
def dataset():
rows = []
for x in range(100):
role = "train" if x % 20 <= 13 else "validation" if x % 20 <= 16 else "test"
y = int(x >= 50)
if x in [7,52,15,75,18,78]:
y = 1-y
rows.append(dict(key="R"+str(x).zfill(3), group="G"+str(x).zfill(3), x=x, y=y, role=role))
return rows
def checked_rows(rows, role=None):
if not isinstance(rows, list) or not 1 <= len(rows) <= 200:
raise ValueError("Expected a nonempty bounded record list")
for r in rows:
if not isinstance(r, dict) or set(r) != {"key","group","x","y","role"}:
raise ValueError("Unexpected record schema")
if any(not isinstance(r[k],str) or not r[k] for k in ["key","group"]):
raise ValueError("Expected opaque nonempty identifiers")
if type(r["x"]) is not int or not 0 <= r["x"] <= 99 or type(r["y"]) is not int or r["y"] not in [0,1]:
raise ValueError("Invalid feature or target")
if r["role"] not in ["train","validation","test"] or role is not None and r["role"] != role:
raise ValueError("Wrong data role")
def validate_split(rows):
checked_rows(rows)
keys, groups = set(), dict()
split = dict(train=[], validation=[], test=[])
for r in rows:
if r["key"] in keys:
raise ValueError("A record key appears twice")
keys.add(r["key"])
if r["group"] in groups and groups[r["group"]] != r["role"]:
raise ValueError("Related source group crosses roles")
groups[r["group"]] = r["role"]
split[r["role"]].append(dict(r))
if any(not part for part in split.values()):
raise ValueError("All three roles need data")
return split
def fit_models(train):
checked_rows(train,"train")
positive = sum(r["y"] for r in train)
majority = int(2*positive > len(train))
losses = [sum(int(r["x"] >= cut) != r["y"] for r in train) for cut in range(101)]
cut = min(range(101),key=lambda c:(losses[c],c))
return [dict(kind="constant",value=majority),dict(kind="threshold",cut=cut)]
def predict(model,x):
if type(x) is not int or not 0 <= x <= 99 or not isinstance(model,dict):
raise ValueError("Invalid prediction request")
if set(model) == {"kind","value"} and model["kind"] == "constant" and type(model["value"]) is int and model["value"] in [0,1]:
return model["value"]
if set(model) == {"kind","cut"} and model["kind"] == "threshold" and type(model["cut"]) is int and 0 <= model["cut"] <= 100:
return int(x >= model["cut"])
raise ValueError("Invalid model contract")
def evaluate(model,rows,role):
checked_rows(rows,role)
tp=tn=fp=fn=0
for r in rows:
p=predict(model,r["x"])
tp += int(p==1 and r["y"]==1)
tn += int(p==0 and r["y"]==0)
fp += int(p==1 and r["y"]==0)
fn += int(p==0 and r["y"]==1)
return dict(tp=tp,tn=tn,fp=fp,fn=fn,correct=tp+tn,total=len(rows),
accuracy=str(Fraction(tp+tn,len(rows))),
precision=str(Fraction(tp,tp+fp)) if tp+fp else None,
recall=str(Fraction(tp,tp+fn)) if tp+fn else None)
def select_model(models,validation):
checked_rows(validation,"validation")
if not isinstance(models,list) or len(models)!=2:
raise ValueError("Expected two predeclared candidates")
reports=[evaluate(model,validation,"validation") for model in models]
selected=max(range(2),key=lambda i:(reports[i]["correct"],-i))
return dict(selected=dict(models[selected]),validation_reports=reports)
def demonstrate():
rows=dataset()
split=validate_split(rows)
models=fit_models(split["train"])
choice=select_model(models,split["validation"])
final=evaluate(choice["selected"],split["test"],"test")
faulty=[dict(r) for r in rows]
faulty[17]["group"]=faulty[0]["group"]
overlap_rejected=False
test_selection_rejected=False
try:
validate_split(faulty)
except ValueError:
overlap_rejected=True
try:
select_model(models,split["test"])
except ValueError:
test_selection_rejected=True
return dict(split_counts={k:len(v) for k,v in split.items()},models=models,
training_reports=[evaluate(model,split["train"],"train") for model in models],
choice=choice,final_test=final,overlap_rejected=overlap_rejected,
test_selection_rejected=test_selection_rejected,
fixture_is_authored=True,real_world_performance_estimated=False,
test_labels_used_in_selection=False,external_ai_used=False)
if __name__ == "__main__":
print(json.dumps(demonstrate(),indent=2))The final selected threshold has eight true positives, five true negatives, one false positive and one false negative. A true positive is a predicted one whose authored target is one; a false positive is a predicted one whose target is zero. A false negative predicts zero when the target is one. These meanings depend on the declared positive class, so state that class instead of treating positive as a universally good or bad outcome.
| Final selected-model outcome | Count | Meaning in the authored inspection task |
|---|---|---|
| True positive | 8 | Correctly flags an authored positive case |
| True negative | 5 | Correctly leaves an authored negative case unflagged |
| False positive | 1 | Flags a case whose authored target is zero |
| False negative | 1 | Misses a case whose authored target is one |
Accuracy is the proportion of all evaluated cases with the correct target prediction:
Precision asks what proportion of predicted positives are truly positive, while recall asks what proportion of actual positives were found:
The accuracy is about 86.67 percent and precision and recall about 88.89 percent for these fifteen authored test cases. Keep exact fractions and the small denominator visible. The majority baseline's validation precision is undefined because it predicts no positives; the program records null, not a fabricated zero or one. Its validation recall is zero because it misses all nine validation positives. Undefined and zero express different situations.
Inspect concrete held-out cases. Record R018 has reading eighteen and authored target one because it is an explicit fixture exception; the selected threshold predicts zero, a false negative. R078 has reading 78 and target zero; it predicts one, a false positive. These were computed after the validation choice. Their errors are not permission to change the threshold and present the same test as untouched final evidence.
Consequences matter. In the fictional inspection task, a false negative could represent a missed review and a false positive an unnecessary review. A classroom cost model might assign five units to a false negative and one to a false positive, making these two errors cost six units. Those costs are invented assumptions, not measured harm or a decision policy. A changed use, such as estimating a numeric number of items to restock, changes the target type and may require regression and a numerical error measure instead of this binary classifier and accuracy.
Prepare a report with purpose, feature definition, label source, split method, candidate family, baseline, selection rule, expected and actual results, error examples and limitations. The source fixture is visible and designed, the split is deterministic, there are only fifteen test cases and no measured real-world distribution. We performed a reproducible teaching computation, not an independent blind trial, a fairness audit or a certified safety study. More rows of the same artificial pattern would not automatically fix those limitations.
For transfer to a permissioned greenhouse task, define a new inspection flag and explain why the input would be available before a decision. Keep repeated observations of the same plant or batch in appropriate groups, and use chronological separation if the intended claim concerns future conditions. Revisit the baseline, error consequences, missingness, season and sensor coverage. Agree on review or abstention outside the checked scope. The transfer is a proposed investigation; it was not run by this program.
Independent check: reconstruct the four test counts, identify which denominators each metric uses, and explain what would happen to the selection if only the test labels changed. Hint: the existing workflow must still fit the same models and make the same validation choice. Partial check: the final report could change, but that change must not flow backward into fitting or selection.
Interactive Tasks
Quiz: Test Your Knowledge
Which stage fits model parameters in the demonstrated workflow? (Training) (!Final test reporting) (!Publishing a score) (!Renaming source groups)
Which split matches the source hundred-record example? (70 training 15 validation 15 test) (!100 training and the same 100 test) (!15 training 70 test and no validation) (!70 training and 70 validation with all records repeated)
What does the majority baseline learn from training labels? (The most common target with a declared tie rule) (!The test target of every future item) (!A guarantee that positives do not exist) (!The final report after model deployment)
What was actually fitted for the threshold candidate? (A cut chosen from training mistakes across 101 integer candidates) (!A language model from pupil messages) (!A rule selected using the final test score) (!A real industrial inspection certificate)
Which rows choose between the fitted candidates? (Validation rows) (!Final test rows relabelled as training) (!All rows including hidden outcomes) (!The source identifiers as model features)
Why is a related group crossing training and test a problem? (The test may no longer provide appropriately separate evidence) (!The numeric ratio becomes impossible to write) (!Every group name must contain a real person) (!It proves the model has no parameters)
What does the selected model's final thirteen correct out of fifteen describe? (Accuracy on this authored test fixture) (!Guaranteed accuracy on every future task) (!A demographic fairness certification) (!A completed real-camera trial)
What is recall for eight true positives and one false negative? (Eight ninths) (!Eight thirteenths) (!Thirteen fifteenths) (!One eighth)
How is precision handled when there are no predicted positives? (It is reported as undefined) (!It is always reported as perfect) (!The missing denominator is silently changed to one) (!Every negative is counted as a positive)
What should follow redevelopment based on test results? (Document that use and obtain appropriate new final evaluation evidence) (!Call the same repeatedly tuned test untouched) (!Delete the old results and claim a blind trial) (!Assume extra rows guarantee fairness)
Memory Game
| Training | Data used to fit parameters |
| Validation | Data used for the declared candidate choice |
| Test | Data reserved for the final chosen-candidate report |
| Baseline | A simple comparison method |
| Leakage | Inappropriate information entering development or evaluation |
| Recall | The fraction of actual positives found |
Drag and Drop
| Seventy source records | The training partition in the original example |
| Candidate choice | Use validation under the declared rule |
| Related-source overlap | Review grouping and split before fitting |
| No predicted positives | Precision is undefined |
| Changed target type | Reconsider model family and suitable error measure |
Crossword Puzzle
| TRAINING | Stage that fits parameters |
| VALIDATION | Partition for candidate choice |
| BASELINE | Simple comparator for a candidate |
| LEAKAGE | Inappropriate information flow into development |
| PRECISION | Fraction of predicted positives that are truly positive |
| RECALL | Fraction of actual positives found |
LearningApps
Optional extra practice. Choose an appropriate activity with your teacher. If the external collection is unavailable, use the native activities and independent tasks in this course.
Cloze Text
Open Tasks
Easy
- Label the six workflow stages: Draw target dictionary split fitting validation choice and final reporting. State the data permitted to influence each.
- Account for all hundred rows: Explain the seventy fifteen fifteen allocation and why correct counts alone cannot establish group independence.
- Define the synthetic target: Describe x y key group and role, and explain why neither a real safety decision nor a pupil attribute is being predicted.
- Read one false negative: Use R018 to state feature target prediction and error category without treating the authored exception as a discovered labelling mistake.
Standard
- Compare training and validation reports: Use both candidate reports to explain why threshold fifty is selected and why training accuracy is not final independent evidence.
- Repair a grouped split plan: A related source group appears in train and test while counts remain seventy fifteen fifteen. Explain the information risk and a correction that preserves source evidence.
- Reject test driven selection: Explain why passing test rows into selection is blocked and why manually renaming their role would not solve the methodological problem.
- Calculate three metrics: Reconstruct test accuracy precision and recall from eight true positives five true negatives one false positive and one false negative. Explain undefined precision for the baseline.
Advanced
- Trace changed test labels safely: Describe a controlled code check that changes test labels only. State which fitted parameters and validation choice must remain unchanged and which final evidence can change.
- Design a future conditions split: Propose grouping and time separation for permissioned greenhouse observations. Define label availability candidate choice baseline and final evaluation limitations.
- Change target and consequences: Compare an inspection flag with a numeric restocking target. Explain renewed model metric error-cost data-rights and review decisions without inventing experimental results.
- Write an evidence honest report: Combine provenance split checks actual program results rejected leakage attempts concrete errors small-sample limits and a human fallback. Distinguish a visible synthetic exercise from a blinded real-world trial.
Learning Check
- Check the source split: Design the three-way allocation and identify what each partition does.
- Check the fitted choice: Give the actual models and validation comparison.
- Correct two leakage plans: Identify the blocked cross-group and test-selection mistakes and suitable corrections.
- Check the final error counts and metrics: Compute the reported test measures and interpret the concrete errors.
- Diagnose repeated test tuning: Explain how repeated changes after inspecting the test alter the claim.
- Transfer with changed assumptions: Propose a new task and a defensible evidence plan.
Worked checks and assessment guidance
- The original hundred records produce seventy training fifteen validation and fifteen test rows. Fit parameters on training rows, choose between the predeclared fitted candidates on validation rows, then report the fixed choice on test rows. The deterministic residue split is a reproducible teaching design, not a universal ratio random sample or proof of real-world independence.
- Training has thirty-eight zeros and thirty-two ones, so the fitted baseline always predicts zero. Enumerating cuts zero through one hundred on training mistakes returns threshold fifty with two training mistakes. Validation correct counts are six of fifteen for the baseline and thirteen of fifteen for threshold fifty, selecting that threshold under the declared equal-error rule. Test labels do not make this choice.
- A test row sharing a training source group is rejected; investigate the relationship and keep related observations together under a justified split, rather than changing the marker to conceal it. Passing test rows to selection is rejected because selection requires validation. Choose using validation, freeze the candidate and use test only for the final report. Counts or role strings alone cannot authenticate independence or prevent dishonest relabelling.
- Eight true positives plus five true negatives give thirteen of fifteen correct. Precision and recall are both eight ninths, using predicted-positive and actual-positive denominators respectively. R018 is a false negative and R078 a false positive. The validation baseline predicts no positives, so precision is undefined and recall zero. An invented cost of five per false negative plus one per false positive gives six units here, not measured real harm.
- The test has then influenced development and is no longer untouched final evidence. Preserve and document the history, make development choices using suitable development data and obtain an appropriate new final evaluation after fixing the choice. Re-running the same visible synthetic fixture does not create a new blinded sample or establish real deployment performance.
- For greenhouse inspection, check the flag definition input availability permissions grouped or temporal separation baseline consequences coverage and human review. A numeric restocking target could instead need regression and a numerical error measure. Record these as proposed changes requiring execution and new evidence. The current computation fits a simple threshold on authored data and certifies neither task safety nor fairness.
Learning evidence
Retain the purpose and five-field dictionary, the complete split with group checks, the fitted baseline and threshold, separate training and validation reports and fixed candidate choice. Add the final confusion counts exact metric fractions concrete error cases and both rejected leakage demonstrations. Record the deterministic visible synthetic scope, any test-informed redevelopment and new-evaluation requirements. Keep transfer plans separate from executed evidence and specify a proportionate human review or abstention route.
OER and Wikipedia
DfE GCSE Computer Science subject content, programming algorithms and evaluation anchor inspected 15 November 2026. The supervised-learning Year 10 sequence and seventy fifteen fifteen example are editorial enrichment, not an additional statutory AI syllabus or an exam-board specification.
scikit-learn common pitfalls, data-leakage and training-only preprocessing discussion inspected 15 November 2026. scikit-learn evaluation design documentation, the training validation and final-test distinction and grouping caveat inspected 15 November 2026. Our original standard-library implementation does not import scikit-learn, execute its examples or claim every recommended evaluation procedure was performed.
ETL jobb.jpg, Hansth, public domain, unchanged generic data-pipeline illustration with non-English labels. It is not this experiment's schema or an image of model training. Exact source metadata retained preview bytes and native image embedding checked; preview visually inspected.
https://www.youtube.com/watch?v=OeU5m6vRyCk — CodeAI: AI: What is Machine Learning? https://www.youtube.com/watch?v=x2mRoFNm22g — CodeAI: AI: Training Data & Bias. Actual retained metadata embedding availability and preview bytes checked and previews visually inspected. Unchanged external embeds imply no transcript copying endorsement or relicensing. All required methods are explained in the written course.
The complete record fixture targets exceptions groups model fitting code selection rules tests and transfer cases are original authored constructions. Actual execution fits one simple numeric threshold and a majority baseline, with test rows excluded from fitting and selection. It uses no local generative LLM paid AI API or personal learner data. The published visible fixture is not a blind independent real-world study. Author checks are not independent expert certification.
Linked learning areas
aiMOOC Projects
Schulfach+


aiMOOCs


aiMOOC Projekte


MOOCwiki · Deutsch
Nach dem Lernen ist vor dem Lernen
Entdecke direkt den nächsten Lernkurs. Weitere Inhalte erscheinen, wenn Du weiter nach unten scrollst.
Zur MOOCwiki-HauptseiteMediathek
Mediathek wird aus dem Wiki geladen ...
Keine passenden Inhalte gefunden. Bitte ändere Suche oder Filter.
NEWSLernweltNOAH fragen