English:Neural network training, loss and generalisation - AI Year 8
Introduction

Calculate a unit's weighted sum, change a weight and inspect what training actually does. An executed four-update experiment will show why lower training loss is not a guarantee of better unused-case performance. You will select a saved checkpoint with validation evidence, then evaluate it on separate fictional test cases. No real learner or sensor data is used.
From weighted input to a declared output
A mathematical unit combines input values using weights and a bias. A weight scales the contribution of an input; the bias adds a constant. An activation determines how the combined value becomes the unit's output. Multiple connected units can form a larger network. The biological analogy is limited: these calculations do not establish consciousness, human understanding or a model of how a particular human learns.
For two inputs, the weighted sum is:
Use the supplied weights w1=0.4, w2=0.3 and bias b=−0.2. For inputs two and one, multiply 0.4 by two and 0.3 by one, then add −0.2. The result is 0.8+0.3−0.2=0.9. Our declared threshold demonstration outputs one when z is at least 0.5 and zero otherwise. Therefore this case outputs one. The number 0.9 is a weighted sum, not automatically a 90 percent probability or accuracy score.
Try the second sum first: swap the inputs to one and two without swapping the weights. Hint: each weight remains attached to its own input position. Partial step: the two contributions are 0.4 and 0.6. After attempting, check: 0.4+0.6−0.2=0.8, so the threshold output is again one. Equal binary outputs do not imply equal internal sums.
Now change only w1 from 0.4 to 0.1, leaving w2 and the bias fixed. The first input pair gives 0.2+0.3−0.2=0.3 and the second gives 0.1+0.6−0.2=0.5. The first output changes to zero; the second remains one because equality is included at the threshold. A controlled change helps explain a contribution, but manually changing a weight is not itself evidence that a model was trained.
Investigate the boundary with x2 fixed at one and the original weights. The sum is 0.4 times x1 plus 0.1. At x1=0.99 it is 0.496 and the threshold output is zero. At x1=1 it is exactly 0.5 and outputs one. At x1=1.01 it is 0.504 and outputs one. The exact arithmetic matters: do not replace equality with an unspecified rounding decision.
3Blue1Brown introduces a larger neural network visually. Its many-unit example is not the one-unit program below and does not validate our fictional fixture. Use it to recognise inputs, connections and layers; the independent calculations here supply the required evidence. The training experiment in the next chapter uses an identity activation, which returns z unchanged. It deliberately does not differentiate or train through this hard threshold. Keep the two demonstrations separate.
A real parameter update in a small exact model
Training adjusts parameters according to a defined objective and data. Our original executable experiment uses one linear unit with two weights and a bias, identity activation and squared error. It is a deliberately minimal illustration of a gradient update, also interpretable as linear regression. There are no hidden layers, deep-network backpropagation or language-model calls in the retained program. A supplied mathematical update is enough to inspect the core idea without pretending that a large neural network has been trained.
For one training row, loss is:
t is the supplied numerical reference target. Squaring makes the loss nonnegative and emphasises larger errors; it removes the sign, so record the prediction and target as well. Mean squared error averages these squared differences when there is more than one row. This numerical training loss is not the binary classification error of the earlier threshold exercise.
Our single fictional training row has x1=1, x2=0 and target one. Start all three parameters at zero and set the learning rate to one eighth. The supplied slope factor and simultaneous update are:
You may use these formulas as given without studying calculus yet. Delta describes how the squared loss changes with the unit's sum; multiplying by an input gives the corresponding weight's slope. Eta is the learning rate. Compute every new parameter from the same old values. Do not update one weight and then accidentally use it to calculate a different parameter's old error.
At the start, z=0 and z−t=−1, so delta=−2. The first weight changes from zero to one quarter, the second stays zero because x2=0, and the bias becomes one quarter. The new training prediction is one half and its squared error is one quarter, down from one. The second update begins with that one-half prediction; delta is −1. It adds one eighth to w1 and the bias, producing w1=b=three eighths. The prediction is three quarters and training loss one sixteenth.
Predict before checking: why does w2 remain zero through every update? Hint: inspect its input multiplier, not the final target. Partial step: that training feature is always zero. It receives no gradient contribution in this fixture. This does not prove that the second feature is irrelevant to every possible task. The tiny training set cannot establish how it should behave outside the shown evidence.
The complete standard-library Fraction program below was actually executed. It runs exactly four updates, retains five checkpoints including the initial state and uses validation only to select among those saved states. It does not stop the training loop at the selected checkpoint; selection after the fixed run illustrates the idea behind monitoring and early stopping without claiming an unimplemented stopping controller. Its final fixture is evaluated only after checkpoint selection.
"""Original exact one-unit identity-activation training and checkpoint selection."""
from fractions import Fraction as F
import json
def number(value):
if type(value) not in [int, F]:
raise ValueError("Use exact integers or fractions")
return F(value)
def predict(parameters, features):
if type(parameters) not in [list, tuple] or len(parameters) != 3:
raise ValueError("Supply two weights and one bias")
if type(features) not in [list, tuple] or len(features) != 2:
raise ValueError("Supply two feature values")
w1, w2, bias = map(number, parameters)
x1, x2 = map(number, features)
return w1 * x1 + w2 * x2 + bias
def threshold(value):
return int(number(value) >= F(1, 2))
def mean_loss(parameters, rows):
if type(rows) not in [list, tuple] or not rows:
raise ValueError("Supply nonempty feature feature target rows")
total = F(0)
for row in rows:
if type(row) not in [list, tuple] or len(row) != 3:
raise ValueError("Each row needs two features and a target")
x1, x2, target = row
total += pow(predict(parameters, (x1, x2)) - number(target), 2)
return total / len(rows)
def update(parameters, training_row, learning_rate):
rate = number(learning_rate)
if rate <= 0:
raise ValueError("Use a positive learning rate")
if type(training_row) not in [list, tuple] or len(training_row) != 3:
raise ValueError("Supply one feature feature target training row")
x1, x2, target = map(number, training_row)
error = predict(parameters, (x1, x2)) - target
gradients = (2 * error * x1, 2 * error * x2, 2 * error)
return tuple(number(p) - rate * g for p, g in zip(parameters, gradients))
def demonstrate():
supplied = (F(2, 5), F(3, 10), F(-1, 5))
changed = (F(1, 10), F(3, 10), F(-1, 5))
forward = [dict(features=x, weighted_sum=predict(supplied, x),
threshold_output=threshold(predict(supplied, x)))
for x in [(2, 1), (1, 2)]]
boundaries = [dict(x1=x, weighted_sum=predict(supplied, (x, 1)),
threshold_output=threshold(predict(supplied, (x, 1))))
for x in [F(99, 100), F(1), F(101, 100)]]
training = [(1, 0, 1)]
validation = [(F(1, 2), 0, F(1, 2))]
final = [(F(1, 4), 0, F(1, 4)), (F(3, 4), 0, F(3, 4))]
parameters = (F(0), F(0), F(0))
checkpoints = []
for step in range(5):
checkpoints.append(dict(step=step, parameters=parameters,
training_loss=mean_loss(parameters, training),
validation_loss=mean_loss(parameters, validation)))
if step != 4:
parameters = update(parameters, training[0], F(1, 8))
selected = min(checkpoints, key=lambda r: (r["validation_loss"], r["step"]))
return dict(supplied_forward=forward,
changed_weight_forward=[predict(changed, x) for x in [(2, 1), (1, 2)]],
threshold_boundaries=boundaries, training=training, validation=validation,
checkpoints=checkpoints, selected_step=selected["step"],
selected_parameters=selected["parameters"],
final_after_selection=dict(rows=final,
predictions=[predict(selected["parameters"], r[:2]) for r in final],
mse=mean_loss(selected["parameters"], final),
zero_baseline_mse=mean_loss((0, 0, 0), final)),
training_activation="identity", threshold_used_for_training=False,
hidden_layers_trained=False, real_sensor_data=False,
local_generative_model_called=False)
if __name__ == "__main__":
print(json.dumps(demonstrate(), indent=2, default=str))Use a school-approved local Python environment if you execute it yourself, and retain actual output. The contract accepts exact integers and fractions, rejects missing or malformed rows and requires a positive learning rate. A positive rate alone does not guarantee improvement: from the zero initial state, rate one gives w1=b=two, training prediction four and loss nine instead of one. The gentle rate one eighth is a declared choice for this bounded example, not a universal optimum.
Read loss curves and keep a final test separate
The validation row is separate from training: x1=one half, x2=zero, target one half. It does not supply a gradient or replace the training target. The four-update experiment produces the following exact record. Epoch here means one pass over our one-row training set; checkpoint zero is the initial state before any pass.
| Checkpoint | w1 and bias | Training loss | Validation loss |
|---|---|---|---|
| 0 | 0 | 1 | 1/4 |
| 1 | 1/4 | 1/4 | 1/64 |
| 2 | 3/8 | 1/16 | 1/256 |
| 3 | 7/16 | 1/64 | 25/1024 |
| 4 | 15/32 | 1/256 | 169/4096 |
Training loss decreases at every shown update. Validation loss initially improves, reaches its smallest shown value at checkpoint two and then worsens. Expressing checkpoint-two validation loss as 4/1024 makes its comparison with checkpoint three's 25/1024 direct. Merely choosing the last state because it has the smallest training loss would miss this distinction. A widening training-validation gap can indicate overfitting; also examine the data, objective and evaluation design rather than assuming a single universal diagnosis.
Our declared selection rule chooses the lowest validation loss, preferring the earlier checkpoint if values tie. It selects checkpoint two with parameters three eighths, zero, three eighths. Validation is now model-selection evidence, so do not call its score an independent final assessment. The one validation row was invented for the exercise and is too small to establish reliable generalisation. The falling loss does not demonstrate fairness, causal understanding or suitability for a real decision.
Keep the selected state fixed for the final fixture. Test A has x1=one quarter, x2=zero, target one quarter. Its prediction is fifteen thirty-seconds; subtracting the target gives error seven thirty-seconds. Test B has x1=three quarters, x2=zero, target three quarters. Its prediction is twenty-one thirty-seconds and error minus three thirty-seconds. The squared errors are 49/1024 and 9/1024; their mean is 29/1024. A constant-zero baseline has squared errors one sixteenth and nine sixteenths, with mean five sixteenths. The selected state has lower loss on these two cases. Two authored cases do not prove a population-level advantage.
Attempt a monitoring report: state which checkpoint you would retain and what remains untested. Hint: name the split used for every claim. Partial step: checkpoint two is selected by validation, not by the final labels. Check after attempting that your report distinguishes lower training loss, a selection result and final measured arithmetic. If you change the model after inspecting final errors, those cases become development evidence; obtain new independent final cases for a new performance claim.
CodeAI discusses training data and bias. Connect it to the extreme limits of a one-row training fixture: many relationships remain unidentified, especially behaviour of the second feature. The generic image at the start is not this loss table, and the video's examples are not our measurements. None of these representations is a biological brain.
For a new fictional craft-kit task, define two permitted measurements, their units, a numerical target and a loss that matches the purpose. Keep training, checkpoint selection and final evaluation apart. Decide how missing inputs, changed units and different error costs will be handled before trying a model. A linear unit may be too limited; adding hidden layers is a new design requiring fresh tests, not a guaranteed upgrade. A classroom trace or program run is evidence of a calculation, not authorisation to automate decisions about people.
Interactive Tasks
Quiz: Test Your Knowledge
What is the supplied weighted sum for inputs two and one? (0.9) (!0.5) (!1.1) (!0.3)
What is the supplied weighted sum after swapping the inputs to one and two? (0.8) (!0.9) (!0.4) (!0.5)
What does the threshold demonstration output at exactly 0.5? (One) (!Zero) (!A missing value) (!A training gradient)
Which activation is used in the executed training experiment? (Identity returning the weighted sum) (!The earlier hard threshold) (!A biological neuron) (!A hidden language model)
After the first update what are w1 and the bias? (Both one quarter) (!Both zero) (!Both one) (!Both minus one quarter)
Why does w2 remain zero in the experiment? (Its training input is always zero) (!All second features are universally irrelevant) (!The validation target removes it) (!It receives the largest gradient)
Which checkpoint has the lowest shown validation loss? (Checkpoint two) (!Checkpoint zero) (!Checkpoint three) (!Checkpoint four)
Why is lower training loss insufficient for a final quality claim? (Unused-case loss can worsen while training loss falls) (!Training loss measures every possible future case) (!A lower loss proves consciousness) (!Validation and training are identical by definition)
What happens with learning rate one from the zero initial state? (Training loss rises to nine) (!Training loss is guaranteed to vanish) (!The parameters cannot change) (!The example becomes a deep network)
When should the retained final fixture be evaluated? (After selecting and freezing the checkpoint) (!As a gradient source at every update) (!Before choosing every future feature) (!Only after deleting its reference targets)
Memory Game
| Weight | A parameter scaling an input contribution |
| Bias | A parameter adding a constant to the weighted sum |
| Activation | The rule converting the combined value to an output |
| Loss | A declared measure of prediction error |
| Gradient | A direction and rate of change of the objective |
| Checkpoint | A saved parameter state that can be evaluated later |
Drag and Drop
| Calculate the weighted sum from unchanged parameters | Forward calculation |
| Compare prediction with the declared numerical target | Loss calculation |
| Adjust parameters using the supplied slopes and learning rate | Training update |
| Select among saved states using separate evidence | Validation selection |
| Evaluate the frozen choice on new cases and a baseline | Final evaluation |
Crossword Puzzle
| WEIGHT | What parameter scales an input contribution? |
| BIAS | What parameter adds a constant? |
| LOSS | What measures the declared error objective? |
| GRADIENT | What describes the objectives local rate and direction of change? |
| CHECKPOINT | What is a saved parameter state? |
| VALIDATION | What split selects among saved states here? |
LearningApps
Optional extra practice. Choose an appropriate activity with your teacher. If the external collection is unavailable, use the native activities and independent tasks in this course.
Cloze Text
Open Tasks
Easy
- Trace two weighted sums: Calculate both supplied input pairs before checking the chapter. Hint: keep the weight attached to its input position. Partial step: the first pair contributes 0.8 and 0.3 before bias. Record the sum separately from the threshold output.
- Change a single weight: Change only w1 to 0.1 and calculate both sums again. Hint: keep the other weight and bias fixed. Explain why equality makes the second threshold output different from a strict greater-than policy.
- Inspect the exact boundary: Predict outputs at x1 values 0.99 one and 1.01 with x2 one. Hint: calculate the weighted sums first. Partial step: at x1 one the sum is exactly one half. Retain equality explicitly.
- Label model components: Draw the two-input unit and distinguish weights bias activation target and loss. Hint: the training unit uses identity while the earlier output demonstration uses a threshold. Do not present either as a human brain.
Standard
- Reconstruct the first update: Start from zero parameters and use the supplied slopes at learning rate one eighth. Hint: the first error is minus one. Partial step: delta is minus two. Calculate all new parameters from the same old values then compare the new loss.
- Read the checkpoint record: Plot or describe training and validation losses for checkpoints zero through four. Hint: use separate labels and a sensible scale. Explain why the last training minimum is not the validation minimum.
- Document a selected model: Apply the declared checkpoint rule and record the chosen parameters without reading final labels into selection. Hint: compare exact fractions. Explain which evidence is now consumed by model selection.
- Check a large learning rate: Predict the first update with learning rate one before optionally running the program. Hint: both the first weight and bias change. Compare the new squared loss with the initial one and state the limit of this counterexample.
Advanced
- Audit final-case arithmetic: Recalculate both final predictions their signed errors squared errors and mean then compare constant zero. Hint: convert targets to thirty-seconds first. Retain all denominators and explain why two cases cannot prove general reliability.
- Investigate an unidentified feature: Explain what the always-zero second training input leaves unknown. Hint: its gradient contribution vanishes. Propose new fictional training and validation evidence without assuming a real causal relation or collecting personal data.
- Transfer to a craft-kit target: Define two permitted features a numerical target and an appropriate loss for fictional cards. Hint: declare units and error consequences first. Separate fitting selection final evaluation and a baseline while identifying model limitations.
- Critique an overconfident learning claim: Respond to a claim that lower training loss proves fairness and human-like understanding. Hint: use the checkpoint counterexample and the limited biological analogy. Produce an evidence-based report distinguishing arithmetic execution from practical competence.
Learning Check
- Calculate and qualify two sums: Give both original weighted sums and threshold outputs.
- Trace the controlled weight change: What happens when only w1 becomes 0.1?
- Explain two actual parameter updates: Give the first two training states and losses.
- Select from validation honestly: Which checkpoint is retained and what does the curve refute?
- Compute a final comparison: Give the two frozen-model predictions final MSE and baseline MSE.
- Explain a transfer and rate limit: What must be checked for another task and why is a positive learning rate insufficient?
Worked checks and assessment guidance
- For inputs two and one the sum is 0.8 plus 0.3 minus 0.2 giving 0.9. For one and two it is 0.4 plus 0.6 minus 0.2 giving 0.8. Both meet the declared at-least-half threshold and output one. These sums are not automatically probabilities.
- The sums become 0.3 and 0.5. Their threshold outputs are zero and one because equality is included. Changing one parameter while holding the rest fixed makes the contribution inspectable but is not itself a fitted training process.
- The one training row is one zero target one. From zero delta is minus two so learning rate one eighth produces first weight and bias one quarter with second weight zero. Prediction one half gives loss one quarter. Delta then minus one produces first weight and bias three eighths prediction three quarters and training loss one sixteenth. The training activation is identity and the hard threshold is not differentiated.
- Checkpoint two has validation loss one over256 lower than the values at zero one three and four. Its parameters are three eighths zero three eighths. Training loss continues falling after this while validation loss rises so lower training loss does not guarantee better unused-case results. The five saved states were all computed; no automatic early-stop controller is claimed.
- The selected model predicts15over32 and21over32. Prediction-minus-target errors are7over32 and minus3over32. Squared errors49over1024 and9over1024 average29over1024. Constant zero gives mean5over16. This is a result on two fictional cases with the state already selected not a reliability or fairness guarantee.
- Define features units target loss and evidence splits then compare a baseline and inspect missing or boundary inputs. Our first update at rate one gives weights and bias two prediction four and loss nine so positivity does not ensure improvement. The zero second training feature leaves its role unidentified. A different task or larger network needs new evidence and no biological or deployment competence is inferred.
Learning evidence
Retain two weighted-sum calculations, the controlled weight change, three exact threshold boundaries, both first updates and all five loss checkpoints. Identify the identity activation, fixed learning rate, untouched final selection order and the feature whose role remains unidentified. Include the two-case final calculation, zero baseline and a justified transfer plan with limitations. Retain actual Python outputs if executed; a paper trace or course completion is not a practical deployment certificate.
OER and Wikipedia
Google: Gradient descent supplies primary context for updates of weights and bias using loss. Google: Interpreting loss curves explains why training and unused-case behaviour must be distinguished. Relevant source sections inspected 15 November 2026. The original exact one-unit experiment here is a teaching construction, not Google's dataset or a deep network benchmark. DfE Computing, Key Stage 3 supplies abstraction context; this AI route is editorial enrichment.
Overfitting.svg, Chabacano, CC BY-SA 4.0. Original classification-boundary illustration unchanged; it is not the executed experiment's loss graph or evidence of its numerical results.
https://www.youtube.com/watch?v=aircAruvnKk — 3Blue1Brown: But what is a neural network? https://www.youtube.com/watch?v=x2mRoFNm22g — CodeAI: AI: Training Data & Bias. Actual metadata, playability, embedding and preview bytes verified; preview images visually inspected. Original recordings embedded unchanged without transcript copying, endorsement, redistribution or relicensing implied. The complete text and original calculations work without playback.
All training, validation and final cases are fictional dimensionless teaching values. The complete standard-library Fraction program was actually executed. It trains one identity-activation unit through four finite updates, saves five states and selects a checkpoint before its separate final calculation. It does not train the threshold, hidden layers or a language model, use private learner data, invoke a local generative LLM or call a paid API.
Linked learning areas
aiMOOC Projects
Schulfach+


aiMOOCs


aiMOOC Projekte


MOOCwiki · Deutsch
Nach dem Lernen ist vor dem Lernen
Entdecke direkt den nächsten Lernkurs. Weitere Inhalte erscheinen, wenn Du weiter nach unten scrollst.
Zur MOOCwiki-HauptseiteMediathek
Mediathek wird aus dem Wiki geladen ...
Keine passenden Inhalte gefunden. Bitte ändere Suche oder Filter.
NEWSLernweltNOAH fragen