Zum Inhalt springen

English:Neural network training and gradient descent - AI Year 9

Aus MOOCsWiki Staging
aiMOOC-Siegel aiMOOC

Neural network training and gradient descent - AI Year 9

QR-Code



Introduction

A generic diagram comparing a smooth boundary with a boundary that closely follows individual points
A flexible fit and useful generalisation are different questions. This generic illustration does not show our numerical training experiment.

Follow inputs through two fixed ReLU features into a trainable output layer. Calculate loss and parameter updates, monitor a separate validation case and compare with a simple baseline on new cases. The exact program was actually executed on original fictional numbers. It trains three output parameters, not a general deep network or a model of a human brain.


Distinguish a prediction loss and a decision

A neural-network diagram describes mathematical operations and adjustable parameters. It does not demonstrate consciousness or the way a biological brain learns. In our small regression example, input x enters two fixed hidden features: h1 is the larger of zero and x, and h2 the larger of zero and minus x. This rectified linear operation, ReLU, separates positive and negative input contributions. The hidden weights and biases remain fixed throughout this exercise.

h1=max⁡(0,x),h2=max⁡(0,−x)

The output is w1 times h1 plus w2 times h2 plus bias b. Only w1, w2 and b are trained. For x=1 the hidden features are one and zero; for x=-1 they are zero and one; for x=0 both are zero so the output is b. The final output has an identity activation and is a numerical prediction, not a probability constrained between zero and one.

ŷ=w1h1+w2h2+b

The two fitting examples are x=1 with target one and x=-1 with target zero. The separate validation example is x=one half with target one half. Two final inspection cases, kept outside fitting and checkpoint selection, are x=one quarter with target one quarter and x=minus one half with target zero. All values are authored teaching fixtures, not observations of pupils or a real service. Their target construction supports arithmetic practice rather than a population claim.

Mean squared error, MSE, averages the squared numerical prediction errors over the chosen examples. With all parameters zero, both training predictions are zero. The two training errors are minus one and zero, so training MSE is one half. Validation prediction is also zero, giving validation squared error one quarter. Report which dataset each loss describes; a training loss without its role can be misleading.

L=1n∑i=1n(ŷi−yi)2

Loss is not accuracy. As a separate simple threshold demonstration, suppose target class is one and predictions are 0.6 and 0.9, with a decision threshold of 0.5. Both produce the same correct class label, but squared errors are 0.16 and 0.01. Numerical closeness changes while that one-case decision accuracy does not. The main course model remains a regression example; we are not secretly turning its MSE into classification accuracy.

The supplied preparation brief includes an illustrative loss log 12, 8, 9. Its first change is minus four and the next plus one, so the number improves then worsens if smaller loss is preferred. This log was supplied as a teaching example, not produced by our program's training run. Without knowing which dataset it describes and the update conditions, that three-number sequence alone cannot diagnose overfitting or prove that an optimiser failed.

Try first: draw input, two fixed hidden features, two trainable output weights, bias and output. Hint: mark which parameters are frozen. Then change only w1 by one quarter: at x=1 the output rises by one quarter, while at x=-1 it stays unchanged because h1 is zero. Check the calculation rather than describing the network as if it had developed a new human understanding.


Calculate a gradient step and monitor its size

Gradient descent changes parameters in the direction opposite the loss gradient. A derivative here describes how the chosen training loss changes when a particular output parameter changes. In the program, we differentiate only the three output parameters; no derivative through the fixed hidden weights is required. This is not a general backpropagation implementation for every layer of a deep network.

Let e be prediction minus target. Each example contributes two times e times h1 to the w1 derivative, two times e times h2 to the w2 derivative, and two times e to the bias derivative. Average these contributions across the training examples. This follows the MSE definition with its division by n; changing the loss convention would also change the gradient scale.

∇L=2n∑iei(h1i,h2i,1)

At the zero-parameter start, the positive-input training error is minus one and the negative-input error zero. The averaged gradient is therefore minus one, zero, minus one in the order w1, w2, b. Choose learning rate one quarter and subtract rate times gradient: the new parameters are one quarter, zero, one quarter. The learning rate is a chosen hyperparameter; the weights and bias are the trained model parameters.

θt+1=θt−η∇L(θt),η=14

After this first update, training predictions are one half for x=1 and one quarter for x=-1. Squared errors are one quarter and one sixteenth, so MSE is five thirty-seconds. Validation prediction is three eighths, which differs from one half by minus one eighth; validation loss is one sixty-fourth. Both losses improve at this first step, but that observation is not a theorem that every step under every setting improves every dataset.

3Blue1Brown explains gradient descent with visual intuition. Use the recording to connect a slope, an update direction and a step size. The animation is not a measurement of our Fraction experiment, and our fixed-feature output-layer example is narrower than training a fully adjustable deep network. All required calculations and limits are given in text.

Try a deliberately large learning rate of one on the same zero start. The first updated parameters become one, zero, one. Training predictions are then two and one, each one unit away from its target. Training MSE becomes one, worse than the initial one half. A positive learning rate can therefore overshoot. The fixed-feature squared-loss problem here is convex in its output parameters, but an unsuitable step size can still fail to decrease the loss. General fully adjustable neural-network objectives also need not have this simple geometry.

The complete program below actually calculates gradients and applies four updates at rate one quarter. It retains all five checkpoints, including step zero. A separate one-step run checks the large rate. Inputs and targets use exact Fraction values so the displayed small fractions can be checked without floating-point rounding. The finite requested loop is not an unlimited optimiser, and invalid rates or update counts are rejected.

"""Exact output-layer training with two fixed ReLU features, not general backprop."""
from fractions import Fraction as F
import json

TRAIN = [(F(1), F(1)), (F(-1), F(0))]
VALIDATION = [(F(1, 2), F(1, 2))]
TEST = [(F(1, 4), F(1, 4)), (F(-1, 2), F(0))]


def exact(value):
    if type(value) not in (int, F):
        raise ValueError("Expected an integer or Fraction")
    return F(value)


def checked_parameters(parameters):
    if not isinstance(parameters, tuple) or len(parameters) != 3:
        raise ValueError("Expected output weights w1 w2 and bias")
    return tuple(exact(v) for v in parameters)


def hidden(x):
    x = exact(x)
    return max(F(0), x), max(F(0), -x)


def predict(parameters, x):
    w1, w2, bias = checked_parameters(parameters)
    h1, h2 = hidden(x)
    return w1 * h1 + w2 * h2 + bias


def checked_examples(examples):
    if not isinstance(examples, list) or not examples:
        raise ValueError("Expected nonempty pairs of exact input and target")
    for pair in examples:
        if not isinstance(pair, tuple) or len(pair) != 2:
            raise ValueError("Expected an input-target tuple")
        for value in pair:
            exact(value)


def loss(parameters, examples):
    checked_parameters(parameters)
    checked_examples(examples)
    errors = [predict(parameters, x) - target for x, target in examples]
    return sum(e * e for e in errors) / len(errors)


def gradient(parameters, examples):
    checked_parameters(parameters)
    checked_examples(examples)
    totals = [F(0), F(0), F(0)]
    for x, target in examples:
        h1, h2 = hidden(x)
        error = predict(parameters, x) - target
        for i, feature in enumerate((h1, h2, F(1))):
            totals[i] += 2 * error * feature
    return tuple(v / len(examples) for v in totals)


def train(steps=4, rate=F(1, 4)):
    rate = exact(rate)
    if type(steps) is not int or steps not in range(1, 21) or not rate > 0:
        raise ValueError("Expected one to twenty updates and a positive exact rate")
    parameters = (F(0), F(0), F(0))
    records = []
    for step in range(steps + 1):
        records.append(dict(step=step, parameters=parameters,
                            train_loss=loss(parameters, TRAIN),
                            validation_loss=loss(parameters, VALIDATION)))
        if step != steps:
            derivatives = gradient(parameters, TRAIN)
            parameters = tuple(p - rate * g for p, g in zip(parameters, derivatives))
    return records


def demonstrate():
    records = train()
    selected = min(records, key=lambda r: r["validation_loss"])
    parameters = selected["parameters"]
    baseline = (F(0), F(0), F(0))
    return dict(records=records, selected_checkpoint=selected["step"],
                selected_test_predictions=[predict(parameters, x) for x, t in TEST],
                selected_test_loss=loss(parameters, TEST),
                zero_baseline_test_loss=loss(baseline, TEST),
                too_large_rate_records=train(1, F(1)),
                supplied_illustrative_log=[12, 8, 9],
                threshold_demo=dict(predictions=[F(3, 5), F(9, 10)],
                    squared_errors=[F(4, 25), F(1, 100)],
                    labels_at_half=[1, 1], target=1),
                fixed_hidden_features=True, output_parameters_trained=3,
                general_backpropagation_implemented=False,
                online_early_stopping_implemented=False)


if __name__ == "__main__":
    print(json.dumps(demonstrate(), default=str, indent=2))

Attempt before checking: recompute the first bias update and both training errors. Hint: average contributions from both cases instead of forgetting the division by two. Partial step: the initial bias gradient is minus one. Check after attempting that the new bias is one quarter and the new training loss five thirty-seconds. For a new input x=-one half, h1 is zero and h2 one half; explain which output weight can affect it while keeping every other value fixed.


Select a checkpoint and challenge generalisation

The executed run makes four fixed updates. Training loss decreases at every recorded step in this particular small-rate run, but validation loss reaches its minimum at step two and then rises. That pattern is a reason to investigate generalisation and a plausible overfitting warning. One authored validation point cannot prove a general causal diagnosis or guarantee useful performance in a population.

Checkpoint Training MSE Validation squared error
0 1/2 1/4
1 5/32 1/64
2 41/512 0
3 365/8192 49/16384
4 3281/131072 121/16384

Our declared selection rule picks the checkpoint with the smallest validation loss after the four-update run, with an earlier checkpoint winning an exact tie. It selects step two, where w1=three eighths, w2=minus one sixteenth and b=five sixteenths. At validation x=one half, the output is three sixteenths plus five sixteenths, exactly one half. This is checkpoint selection after a completed finite run, not an implemented online early-stopping controller. Validation is used for selection and is therefore not an untouched final test.

Now inspect the two reserved test cases once, without choosing new settings from their results. At x=one quarter, the selected network predicts 13/32 against target one quarter, an error of 5/32. At x=minus one half it predicts 9/32 against target zero, an error of 9/32. Their MSE is 53/1024. A constant-zero baseline has test MSE one thirty-second, or 32/1024, which is lower on these two cases. Perfect fit on one validation point has not made this trained network better than the baseline on the supplied test fixture.

Ltest=(5/32)2+(9/32)22=531024>132

Report this unfavourable comparison honestly. Do not replace the test cases, tune on them and still describe a later result as an independent final test. If the evidence is used to redesign the method, obtain appropriately new evaluation cases. This tiny constructed example is enough to refute an automatic claim of success; it is not enough to estimate a reliable future error rate or identify every reason for poor generalisation.

CodeAI discusses training data and bias. Connect representation and case selection to what a loss curve can hide. More updates or a lower training objective do not establish that relevant users, contexts or error consequences were represented. A useful review examines data origin, split roles, model restrictions, learning settings, baseline, validation use and the remaining evidence needed before any real service.

Transfer workshop: use the same arithmetic with another set of small fictional input-target pairs. Mark which parameters and feature functions remain fixed, define train, validation and test roles before calculation, and predict the effect of changing one weight. Keep expected and actual outputs plus invalid-input and rate checks. State what the numbers represent in the new context and which consequences would require a human reviewer. Do not convert a toy score into a judgement about an actual pupil or a medical or employment decision.

This is an editorial Year 9 application of a broad AS/A-level computing-related mathematics source objective. It is not a statutory Year-specific deep-learning requirement or an A-level qualification. If earlier pipeline material is not yet available, the essential bridge is stated here: fit parameters on training data, use separate validation data for settings or checkpoint choices, and reserve test cases for later evaluation. The exercises introduce the required arithmetic rather than assuming that every later AI topic is already mastered.


Interactive Tasks


Quiz: Test Your Knowledge

Which parameters are trained in this example? (Two output weights and the output bias) (!Every hidden weight in a deep network) (!The real brains of two people) (!The final test targets)



What are the two hidden features for input minus one? (Zero and one) (!One and zero) (!Minus one and minus one) (!One half and one half)



What is initial training MSE at zero parameters? (One half) (!One quarter) (!Zero) (!One hundred percent)



Why is loss not the same as accuracy? (Different numerical errors can produce the same thresholded label) (!Every smaller loss is a larger percentage) (!Regression predictions always have a true class label) (!Accuracy is the gradient of every model)



What is the initial gradient in output-weight and bias order? (Minus one zero minus one) (!One one one) (!Zero zero zero) (!Minus four zero minus four)



What happens in the first update at learning rate one quarter? (Parameters become one quarter zero one quarter) (!Every test label changes) (!All hidden weights are trained) (!Training loss becomes a guarantee)



What does learning rate one demonstrate in the separate first step? (Training loss can rise despite a positive rate) (!Every positive rate always lowers loss) (!The model has human consciousness) (!Validation data may be used as training features)



Which checkpoint is selected by the declared validation rule? (Step two) (!Step four because it is latest) (!Step zero because it is first) (!A checkpoint chosen from the test losses)



Which model has lower loss on the two reserved test cases? (The constant-zero baseline) (!The selected network by definition) (!Both models must tie) (!There is no baseline calculation)



What does the executed checkpoint procedure actually implement? (Selection after four fixed updates) (!An online early-stopping controller) (!General backpropagation through every layer) (!A certified real-world deployment)





Memory Game

Parameter An adjustable value in the model
Learning rate A chosen hyperparameter controlling update size
Gradient Derivatives of the loss with respect to parameters
Loss A declared numerical objective on specified examples
Validation Separate examples used for choices such as checkpoints
Baseline A simpler method used for comparison





Drag and Drop

Compute the two fixed rectified features Hidden transformation
Combine features with adjustable weights and bias Output prediction
Average squared numerical errors MSE calculation
Subtract learning rate times gradient Parameter update
Compare selected model with zero on reserved cases Final fixture inspection





Crossword Puzzle

WEIGHT A multiplier in the output model
BIAS An additive parameter
GRADIENT Loss derivatives that guide updates
LOSS The objective being reduced
RELU A rectified linear activation
BASELINE A simpler comparison method





LearningApps

Optional extra practice. Choose an appropriate activity with your teacher. If the external collection is unavailable, use the native activities and independent tasks in this course.


Cloze Text

Complete the explanation.
The hidden ReLU features stay

. Only the output weights and

are trained. Mean squared error is a

. The learning rate is a

. The update moves opposite the

. A very large step can

. Checkpoint two minimises the supplied

loss. The reserved test comparison favours the zero

. Four fixed updates do not implement online early

. An artificial network is not proof of human-like

.



Open Tasks


Easy

  1. Draw the numerical path: Label x both fixed hidden features two output weights bias and identity output. Hint: distinguish the frozen and trainable parts. Work through x one minus one and zero without a biological claim.
  2. Calculate initial losses: Compute both training errors and the separate validation loss at zero parameters. Hint: the two-case mean includes division by two. Keep dataset roles distinct.
  3. Separate a label from a loss: Use predictions zero point six and zero point nine with target one and threshold one half. Hint: both labels match but squared errors differ. Explain why the main regression MSE is not a percent accuracy.
  4. Read the supplied three-number log: Describe the changes twelve to eight to nine and what remains unknown. Hint: the source does not identify it as the actual program run. State why that sequence alone cannot diagnose overfitting.


Standard

  1. Reproduce the first update: Calculate the three initial derivatives and parameters after rate one quarter. Hint: average two times error times each feature including the bias feature one. Check the five-thirty-seconds training loss.
  2. Change one weight or input: At a recorded checkpoint change only w1 then compare positive and negative inputs. Hint: h1 is zero for negative x. Explain what stays fixed rather than claiming a full-network retraining.
  3. Compare step sizes: Calculate the separate first step with rate one and compare its loss with the small-rate result. Hint: a positive step can be too large. Distinguish this convex output-layer case from fully adjustable general networks.
  4. Inspect validation selection: Use the five-checkpoint table to select a model and describe the warning pattern after step two. Hint: training improves while validation worsens. Keep the one-point validation limitation visible.


Advanced

  1. Audit the test comparison: Recompute both selected predictions errors MSE and zero-baseline MSE. Hint: the lower baseline loss must be reported honestly. Explain why a perfect validation value did not establish useful generalisation.
  2. Distinguish a controller from a record: Explain why selecting from a fixed completed run is not online early stopping. Hint: the loop always performs four updates. Propose what a real stopping procedure would need without claiming it exists here.
  3. Design boundary and reproducibility checks: Record source versions exact parameters gradients rate roles expected outputs and invalid cases. Hint: test zero or malformed rates and compare an independent numerical derivative. Do not weaken the permitted input contract to hide a failure.
  4. Transfer the model cautiously: Choose new fictional pairs and revisit units feature meaning target provenance splits baseline and error consequences. Hint: keep a final evaluation separate from redesign decisions and explain which human review would be required for real use.



Learning Check

  1. Explain the model and objective: What is fitted and what does its loss mean?
  2. Verify the first training step: Give the initial gradient update and losses.
  3. Challenge guaranteed improvement: What do the supplied log and large-rate check show?
  4. Interpret the selected checkpoint: Why is step two selected and what warning follows?
  5. Compare with the baseline: Calculate the final fixture result without claiming success.
  6. Write a responsible new-context record: What would make another exercise reproducible and appropriately limited?


Worked checks and assessment guidance

  1. Two fixed ReLU features are max zero x and max zero minus x. Two output weights and one bias form an identity numerical output. MSE averages squared errors on a named dataset. It is not classification accuracy a calibrated probability or a demonstration of biological understanding.
  2. The zero start has training loss one half and validation loss one quarter. The initial gradient is minus one zero minus one. Rate one quarter gives parameters one quarter zero one quarter with training loss five thirty-seconds and validation loss one sixty-fourth. Hidden features remain fixed.
  3. The supplied illustrative twelve eight nine sequence improves then worsens but is not the actual executed run or enough to diagnose overfitting. In the actual separate rate-one update parameters become one zero one and training loss rises from one half to one. A positive learning rate does not guarantee improvement or global optimality after a finite run.
  4. Validation loss is smallest at step two with parameters three eighths minus one sixteenth five sixteenths and validation output one half. Later training loss falls while the single validation loss rises which merits generalisation review. Selection occurs after four fixed updates not via an online stopping controller and that validation point is not an untouched test.
  5. Selected predictions are thirteen thirty-seconds and nine thirty-seconds with errors five thirty-seconds and nine thirty-seconds. MSE is fifty three over one thousand twenty four. Zero baseline MSE is one thirty-second or thirty two over one thousand twenty four and is lower. The tiny authored test does not estimate a reliable future rate and should not be retuned while still called independent.
  6. Declare inputs targets units provenance fixed versus trained parameters loss convention gradients learning rate update bound split roles checkpoint rule baseline expected and actual tests and remaining limits. Use invented examples and seek appropriate review for any real consequential task. The broad AS/A-level anchor is an editorial Year9 application not a statutory deep-learning or qualification claim.


Learning evidence

Retain the labelled fixed-feature network diagram, initial forward calculations, exact gradient and first update, five-checkpoint table and independent rate-one check. Add selected versus baseline test calculations, the source-log distinction, invalid-input tests and a new-context plan. Separate output-layer training from general backpropagation, validation selection from an untouched test and an authored numerical example from real-world competence.


OER and Wikipedia

Google Machine Learning Crash Course: Gradient descent supports the loss-gradient-update procedure. Google: Hyperparameters explains the learning-rate and parameter distinction and overshooting risk. Google: Activation functions explains ReLU and nonlinear hidden transformations. Relevant primary sections inspected 15 November 2026. Our fixed-feature three-parameter derivation finite run and exact examples are original teaching constructions, not those sources' datasets or a complete deep-network implementation.

DfE: GCE AS and A level computer science subject content, section 6, supplies the broad computing-related mathematics anchor. The Year 9 neural-training decomposition is editorial, not a statutory Year-specific requirement or A-level qualification.

Overfitting.svg, Chabacano, CC BY-SA 4.0. Generic illustration embedded unchanged; it is not the loss trace of our original numerical experiment.

https://www.youtube.com/watch?v=IHZwWFHWa-w — 3Blue1Brown: Gradient descent, how neural networks learn, Deep Learning Chapter 2. https://www.youtube.com/watch?v=x2mRoFNm22g — CodeAI: AI: Training Data & Bias. Actual metadata playability embedding and preview bytes verified; previews visually inspected. Videos embedded unchanged without transcript copying endorsement redistribution or relicensing implied. The text contains all required methods and calculations independently of playback.

All input-target pairs rate comparisons baseline and exact checkpoint results are original fictional constructions. The supplied twelve eight nine log is separately labelled. The complete bounded standard-library program was actually executed and trains only the output layer over fixed hidden features. It does not implement general backpropagation online early stopping a live AI service local generative LLM or paid API. Author checks are not independent expert certification.


Linked learning areas


aiMOOC Projects



Schulfach+




aiMOOCs



aiMOOC Projekte









MOOCwiki · Deutsch

Nach dem Lernen ist vor dem Lernen

Entdecke direkt den nächsten Lernkurs. Weitere Inhalte erscheinen, wenn Du weiter nach unten scrollst.

Zur MOOCwiki-Hauptseite

Mediathek

Suche verfeinern

Dein Lernplan bestimmt, was oben steht. Alle anderen Inhalte bleiben erreichbar. Seitenaufrufe beeinflussen die Auswahl nicht.

Inhalte werden geladen ...

Mediathek wird aus dem Wiki geladen ...