Zum Inhalt springen

English:Embeddings, attention and transformer architectures - AI Year 9

Aus MOOCsWiki Staging
aiMOOC-Siegel aiMOOC

Embeddings, attention and transformer architectures - AI Year 9

QR-Code



Introduction

A two-dimensional coordinate grid
A vector can be represented by coordinates. These generic axes are not measured language-model features or a map of word meanings.

Words become numbers inside a language model, but numbers are not dictionary definitions or evidence that an answer is true. Follow tokens into supplied vectors, calculate a small attention example and challenge what its weights can explain. You will use the ambiguous sentence about a dog and a ball, then audit a fictional prepared answer. The bounded program is a teaching calculation, not a trained transformer, real model trace or online text generator.


Distinguish tokens identifiers and representations

Begin with the source sentence: The dog chased the ball because it moved. The pronoun it leaves more than one plausible referent without further context. A reader may favour the ball, but the sentence alone does not formally settle the intended reference. Keep that uncertainty visible rather than treating a coloured arrow in a network sketch as proof.

A token is a unit chosen by a tokenizer. Depending on the real system it may be a word, part of a word or another text unit. Our limited teaching tokenizer lowercases an English sentence and keeps runs of letters from a to z. It produces the eight tokens the, dog, chased, the, ball, because, it, moved. This is not the tokenizer of an actual deployed model, a complete multilingual text processor or a guarantee that every word corresponds to one model token.

The program assigns identifiers in order of first appearance: the is zero, dog one, chased two, ball three, because four, it five and moved six. The resulting sequence is zero, one, two, zero, three, four, five, six. The repeated word the reuses identifier zero. These integer labels are lookup addresses, not ranks of importance, physical measurements or definitions. Swapping a consistently used identifier scheme need not change the content represented.

An embedding maps an identifier to a numerical vector. A real model may learn a high-dimensional embedding table. Later layers can produce representations that depend on surrounding context. For our exercise, two-dimensional values are supplied by the author so that you can check the arithmetic. Assign dog the vector one, zero and ball zero, one. This assignment is not learned from language, and the first coordinate is not proven to be an animal concept. You cannot infer a dictionary meaning from an arbitrary coordinate name.

edog=(1,0),eball=(0,1)

The optional word-embedding illustration by Fschwarzentruber uses a simplified relation between countries and capitals. Its verified diagram is supplementary, not an empirical measurement from our program or proof that all semantic relationships are linear. The introductory coordinate image supports only the idea of specifying a numerical position; it is not a language-model visualisation.

Compare two new contexts for another ambiguous word: the bank beside the river and the bank approved a loan. The spelling bank is the same, while the intended senses can differ. An initial lookup may reuse a vector for the same token, and contextual processing can alter the representation. Our code does not infer either sense: this comparison is an interpretation exercise. Ask what information the surrounding text supplies and which uncertainty remains.

To read a schematic transformer, distinguish four layers of description. An instruction asks for behaviour. Context supplies material the model can process. Generated text is a candidate continuation. External evidence is material you independently inspect to support a particular factual claim. A source-looking citation printed in generated text is still part of the output until you open and verify it. Our prepared examples do not ask a live AI to generate anything.

Attempt first: list the sentence's tokens and locate both appearances of the. Hint: remove the final punctuation under this deliberately limited rule. Partial check: there are eight token positions but seven distinct identifiers. Then write two plausible interpretations of it and one extra sentence that would disambiguate each. Label your added context as a construction rather than evidence about a real speaker's intention.


Calculate attention without inventing a model trace

Attention combines a query with keys and values. A query-key compatibility score determines weights; the output mixes value vectors using those weights. For the common scaled dot-product operation, divide each dot product by the square root of the key dimension, apply softmax and take a weighted sum. This small calculation illustrates one operation. It does not implement learned projections, a complete self-attention layer, positional encoding, a feed-forward block or a transformer.

Use two supplied keys, dog as one, zero and ball as zero, one, with matching value vectors for this first case. Choose the supplied query two, zero and key dimension two. Its dot products are two and zero. Scaling gives square root of two and zero. Softmax exponentiates and normalises these scores. All displayed decimal results are rounded from the actual floating-point run, not exact rational weights.

si=q⋅kidk,ai=exp⁡(si)∑jexp⁡(sj)

The dog weight is approximately 0.8044296825 and the ball weight 0.1955703175. Both are nonnegative and sum to one within floating-point precision. With our first values, the output has those same two coordinates. A weight of about 0.8044 is not an 80.44 percent probability that it means dog or that an answer is factually correct. It is a normalised mixing coefficient inside this declared operation.

o=∑iaivi

Now change only the query to zero, two. Scores, weights and the two output coordinates swap. A zero, zero query gives equal scores, weights one half and one half, and output one half, one half. We supplied these queries to compare mechanisms; the program did not derive them from the dog's sentence or automatically resolve the pronoun. More context could affect a trained representation, but this sketch is not a literal record of such a model.

Supplied query or change Weights on dog then ball Output
Query 2, 0 About 0.8044 then 0.1956 About 0.8044, 0.1956
Query 0, 2 About 0.1956 then 0.8044 About 0.1956, 0.8044
Query 0, 0 0.5 then 0.5 0.5, 0.5
Allow only the first key 1 then 0 1, 0
Both values changed to 1, 1 Weights depend on the query 1, 1 under both queries

The equal-values comparison is a useful counterexample. Keep both keys, use either query and set both value vectors to one, one. The two attention distributions differ but their weighted sums are the same vector, one, one. If you inspected only the weights you would miss the role of values. This construction does not prove that every attention map is useless; it demonstrates why weights alone are not a complete explanation of an output or a causal account of a real system.

For a different controlled change, double only the first coordinate of the dog value from one to two, keeping the original query and keys. The weights stay the same, but the first output coordinate becomes about 1.6088593650. Weight patterns alone therefore do not determine the numerical output. A separate mask can forbid a key: when only the first key is allowed, its normalised weight is one even with the second query. Our caller supplies a Boolean mask; the function does not automatically construct a causal sequence mask.

3Blue1Brown visually explains attention. Connect its queries, keys and values to the worked calculation while keeping its larger illustrated system separate from our supplied numbers. The text contains the method independently of playback. The complete standard-library program below was actually executed. Bounds and input checks keep it a finite calculation without model downloads or service calls.

"""A bounded, supplied-vector attention calculation, not a language model."""
import json
import math
import re


def vector(value):
    if not isinstance(value, (list, tuple)) or len(value) not in range(1, 9):
        raise ValueError("Expected one to eight vector components")
    if any(type(x) not in (int, float) or not math.isfinite(x) or abs(x) > 100
           for x in value):
        raise ValueError("Expected finite numeric components bounded by one hundred")
    return tuple(float(x) for x in value)


def attention(query, keys, values, allowed=None):
    q = vector(query)
    if (not isinstance(keys, list) or not isinstance(values, list)
            or len(keys) not in range(1, 17) or len(keys) != len(values)):
        raise ValueError("Expected one to sixteen key-value pairs")
    k = [vector(row) for row in keys]
    v = [vector(row) for row in values]
    if any(len(row) != len(q) for row in k) or len(set(map(len, v))) != 1:
        raise ValueError("Key and query dimensions must match; values must share a size")
    if allowed is None:
        allowed = [True] * len(k)
    if (not isinstance(allowed, list) or len(allowed) != len(k)
            or any(type(x) is not bool for x in allowed) or not any(allowed)):
        raise ValueError("Expected a Boolean mask allowing at least one key")
    scores = [math.fsum(a * z for a, z in zip(q, row)) / math.sqrt(len(q))
              for row in k]
    peak = max(s for s, use in zip(scores, allowed) if use)
    terms = [math.exp(s - peak) if use else 0.0 for s, use in zip(scores, allowed)]
    total = math.fsum(terms)
    weights = [x / total for x in terms]
    output = [math.fsum(w * row[j] for w, row in zip(weights, v))
              for j in range(len(v[0]))]
    return dict(scores=scores, weights=weights, output=output)


def tokens_and_ids(text):
    if not isinstance(text, str) or not text.strip() or len(text) > 4096:
        raise ValueError("Expected a nonempty short teaching sentence")
    tokens = re.findall(r"[a-z]+", text.lower())
    if not tokens:
        raise ValueError("Expected alphabetic teaching tokens")
    vocabulary = dict()
    for token in tokens:
        if token not in vocabulary:
            vocabulary[token] = len(vocabulary)
    return dict(tokens=tokens, ids=[vocabulary[t] for t in tokens], vocabulary=vocabulary)


def demonstrate():
    keys = [[1, 0], [0, 1]]
    values = [[1, 0], [0, 1]]
    equal_values = [[1, 1], [1, 1]]
    return dict(
        tokenisation=tokens_and_ids("The dog chased the ball because it moved."),
        supplied_keys=keys, supplied_values=values,
        dog_query=attention([2, 0], keys, values),
        ball_query=attention([0, 2], keys, values),
        tied_query=attention([0, 0], keys, values),
        first_key_only=attention([0, 2], keys, values, [True, False]),
        equal_values_dog_query=attention([2, 0], keys, equal_values),
        equal_values_ball_query=attention([0, 2], keys, equal_values),
        changed_value=attention([2, 0], keys, [[2, 0], [0, 1]]),
        large_scores=attention([100, 100], [[100, 100], [-100, -100]], values),
        learned_embeddings=False, real_model_trace=False,
        transformer_implemented=False, text_generated=False)


if __name__ == "__main__":
    print(json.dumps(demonstrate(), indent=2))

The stable softmax implementation subtracts the largest allowed score before exponentiation. This does not change the ideal normalised ratios, but avoids exponentiating a huge positive number. In the extreme supplied case it returns weights one and zero in finite precision; that zero can reflect numerical underflow, not a universal theorem that the key has exactly zero ideal weight. The function rejects all-masked rows rather than dividing by zero or inventing an output. Other rejected inputs include nonfinite components, mismatched dimensions and a mask of the wrong length.

Try first: predict what happens if the two values are equal but the query changes. Hint: distribute the weights over each coordinate. Since the two weights sum to one, each output coordinate remains one. Keep this analytical check beside the executed result and identify which components the counterexample deliberately holds fixed.


Place the mechanism in an architecture and verify claims

A transformer is a neural-network architecture built from attention-based processing blocks and additional transformations. In the original encoder-decoder design, tokens become embeddings with positional information; blocks combine attention, feed-forward transformations, residual connections and normalisation. Multiple heads can form different mixtures in parallel. These are components, not separate human thinkers or guaranteed interpretable concepts. Encoder-only and decoder-only variants also exist; not every transformer-based language model contains both full stacks.

In encoder self-attention, the queries, keys and values come from representations of the same input sequence. A decoder used for next-token generation must prevent access to later target tokens at each prediction position. Encoder-decoder attention instead obtains keys and values from encoder output. Position information distinguishes order, because the same collection of token embeddings alone does not specify every sequence relationship. Our supplied-vector function covers only query-key scoring, a caller-supplied mask and value mixing; it neither constructs these architectures nor trains them.

Draw a schematic using these labels: input text, tokenizer, token identifiers, embedding lookup plus position information, attention-based blocks, output scores, output token distribution. For a generative decoder, add the next selected token to the context and repeat as appropriate to the actual model. Use a separate arrow for external evidence retrieval if a real application includes it. Generation does not automatically perform retrieval, and a diagram of either route is not proof that an individual factual statement has been checked.

The research paper Attention is not Explanation reports limitations of treating attention weights as explanatory importance. Our equal-values construction is a smaller independent demonstration of a related interpretive issue, not a reproduction of its experiments or a claim that later explanation research has one simple final answer. To investigate a particular system, consider the values and other computations, controlled interventions, task and evaluation evidence rather than equating one heat map with complete reasoning.

CodeAI discusses training data and bias. Connect that context to what an architecture sketch does not show: dataset provenance, representation gaps, evaluation conditions and the consequences of errors. No pupil needs a live generative account or personal text sample for this course. Use invented sentences and teacher-prepared outputs. Do not upload school records, names, photographs or private learner work.

Here is an original fictional verification exercise. The supplied archive card is the whole evidence available in the exercise: Archive A, card blue, label Water, location shelf 2. A prepared candidate answer contains three claims. It says, The blue card is labelled Water. Every blue object is safe to touch. It moved. This is authored teaching text, not output obtained from a language model. Check the three claims separately rather than trusting the answer's fluency.

Candidate claim Evidence decision Appropriate response
The blue card is labelled Water Supported by the exact label in the fictional archive card Keep it with the narrow source reference
Every blue object is safe to touch Unsupported: the card states no safety rule about objects Remove the claim or seek relevant evidence; do not use it for a real safety decision
It moved Ambiguous referent and no movement evidence in the supplied card Ask what it refers to and request evidence before asserting movement

A narrow supported fact does not justify a larger claim. The word Water on a card says nothing about whether an actual substance is drinkable, safe to touch or even present. A confident-sounding answer or a high attention weight cannot supply that missing evidence. Rewrite the candidate as: The supplied fictional archive record labels card blue as Water and places it on shelf 2. It gives no safety or movement information. This preserves the supported content while clearly limiting its scope.

Transfer the routine to a new fictional school-garden catalogue. Declare a record's author, date, permission and what each field means. Check every generated or prepared claim against the particular record and distinguish supported, unsupported and ambiguous statements. If you change language, tokenization, model, task or context, recheck which assumptions and sources remain appropriate. Neither the same prompt nor the same vector diagram guarantees factual correctness.

The route is editorial Year 9 enrichment attached to a broad AS/A-level computing data-representation source anchor, not a statutory Year-specific transformer requirement or an A-level qualification. A useful test log records the supplied inputs, vector dimensions, masking choices, expected and actual values, numeric tolerance, rejected inputs and what was not implemented. Your evidence is a reproducible operation and a careful claim audit, not a trained language model or a record of its private reasoning.


Interactive Tasks


Quiz: Test Your Knowledge

How many teaching-token positions and distinct identifiers does the source sentence produce? (Eight positions and seven distinct identifiers) (!Seven positions and eight distinct identifiers) (!Eight positions and eight different meanings) (!One position for the whole sentence)



What is a token identifier in the supplied program? (A lookup label assigned by first appearance) (!A dictionary definition) (!An attention probability) (!A ranking of human importance)



What is true of the supplied dog and ball vectors? (Their two coordinates are authored for arithmetic) (!They were learned from a large language corpus) (!They reveal two verified universal semantic axes) (!They are literal traces from a deployed model)



What are the scaled scores for query two zero with the supplied keys? (Square root of two and zero) (!Two and two) (!One half and one half) (!Zero and square root of two)



What does the approximately 0.8044 attention weight represent? (A mixing coefficient in the declared operation) (!The probability that the pronoun means dog) (!An independently verified truth score) (!The accuracy of a trained text generator)



What happens with query zero zero? (The two allowed keys receive equal weights) (!Every value is rejected as unsafe) (!Only dog is allowed automatically) (!The model resolves the pronoun from external evidence)



What does the equal-values counterexample demonstrate? (Different attention weights can give the same output) (!Attention always provides complete causal explanations) (!Every possible attention map must be useless) (!Values never affect the output)



What does the supplied mask function do with every key forbidden? (It raises an explicit error) (!It invents a verified token) (!It divides by zero and calls it a result) (!It silently trains a new embedding)



Which statement about transformer architecture is accurate? (Encoder-only and decoder-only variants also exist) (!Every architecture needs two complete stacks) (!Every attention head is a human thinker) (!Position information guarantees factual truth)



Which prepared claim is supported by the fictional archive record? (The blue card is labelled Water) (!Every blue object is safe to touch) (!The card moved to another room) (!An actual substance is safe to drink)





Memory Game

Token A unit selected by a tokenizer
Identifier A lookup label for a token
Embedding A numerical vector representation
Query A vector compared with keys
Attention A weighting and value-mixing operation
Transformer An architecture built from attention-based blocks and other transformations





Drag and Drop

Split the teaching sentence into permitted units Tokenize
Map an identifier to a numerical representation Look up an embedding
Compare a query with supplied keys Calculate compatibility scores
Normalise scores over allowed positions Calculate attention weights
Open the exact supporting record for each claim Verify evidence





Crossword Puzzle

TOKEN A chosen text-processing unit
VECTOR An ordered numerical representation
QUERY A vector compared with keys
SOFTMAX An exponential normalisation operation
MASK A restriction on allowed attention positions
CONTEXT Surrounding material that can affect interpretation





LearningApps

Optional extra practice. Choose an appropriate activity with your teacher. If the external collection is unavailable, use the native activities and independent tasks in this course.


Cloze Text

Complete the explanation.
The teaching sentence contains eight token

. The repeated word the reuses an

. An embedding is a numerical

. Our supplied coordinates were not

. A query is compared with

. Softmax supplies normalised

. The output mixes the

. Forbidding a position uses a

. Equal values can hide changes in

. An architecture sketch is not a real model

. A printed citation still needs source

. Unclear reference makes a claim

.



Open Tasks


Easy

  1. Inspect tokenization: Write all eight teaching-token positions and their identifiers for the dog sentence. Hint: both instances of the use the same identifier. Explain why a real system might split text differently.
  2. Label a representation: Draw supplied dog and ball vectors on two generic coordinate axes. Label the values as authored rather than learned and explain why the axes are not verified dictionary meanings.
  3. Preserve ambiguity: Give two possible readings of it in the source sentence and add different invented context for each. Hint: distinguish your new information from what the original sentence actually says.
  4. Trace the operation: Make four labelled cards for query, keys, weights and values. Explain how the weighted output is constructed without claiming that a weight is a truth probability.


Standard

  1. Reconstruct the numbers: Calculate both dot products for query two zero, divide by square root of two, and use a teacher-approved calculator for softmax. Record unrounded intermediate values and compare the output to the stated rounded decimals.
  2. Change one component: Double only the first dog-value coordinate and predict which output changes while the weights stay fixed. Run only the supplied bounded program if a teacher-approved environment is available, then retain expected and actual results.
  3. Challenge the heat map: Compare the two query cases when both values are one one. Hint: use the fact that weights sum to one. Explain the counterexample and why it does not prove every attention visualisation has no possible use.
  4. Audit the prepared answer: Classify the three fictional archive claims separately and rewrite the answer using only supported details. Preserve the distinction between unsupported information, ambiguity and an actual contradiction.


Advanced

  1. Design boundary checks: Plan tests for zero queries, one allowed key, all keys forbidden, unequal vector dimensions and nonfinite inputs. State expected outputs or rejection, tolerance and limits; do not accept fabricated results for invalid cases.
  2. Compare architecture routes: Draw an encoder-only, decoder-only and encoder-decoder schematic. Label where keys and values come from and what a causal restriction prevents. Identify the components absent from the toy function.
  3. Transfer the claim audit: Create three invented school-garden records and a candidate paragraph with a supported, unsupported and ambiguous claim. Prepare a source-by-claim table and explain why changing subject or model requires renewed checks.
  4. Create an accessible mini-exhibit: Build an original poster or narrated diagram with a text transcript, the dog-sentence ambiguity, one calculated attention row and the equal-values counterexample. Attribute media and distinguish a schematic from a trained model trace or practical qualification.



Learning Check

  1. Separate units and meaning: Explain token, identifier and embedding using the source sentence and supplied vectors.
  2. Calculate the first attention row: Show scaled scores, weights and output for the original query, and interpret the number honestly.
  3. Explain the counterexample: Why can different attention patterns yield the same result, and what does changing a value show?
  4. Check numerical and mask boundaries: State the zero-query, single-allowed-key and all-masked results and the stable-exponentiation limit.
  5. Place the calculation accurately: Describe the wider architecture and distinguish the toy routine from a transformer.
  6. Verify and transfer: Audit all three archive claims and propose the next step for a new catalogue.


Worked checks and assessment guidance

  1. The limited lowercase-letter tokenizer returns eight positions with identifiers zero one two zero three four five six. Seven distinct identifiers occur because the repeats. An identifier is a lookup label; an embedding is a vector, here authored dog one zero and ball zero one. Actual tokenizers and learned representations can differ, and neither an identifier nor these coordinates is a dictionary definition.
  2. Dot products are two and zero. With key dimension two, scores are square root of two and zero. Softmax gives about 0.8044296825 and 0.1955703175. Matching basis-vector values give the same output coordinates. These are mixing weights, not probabilities of pronoun reference or factual truth. The query was supplied rather than inferred from the sentence.
  3. When both values are one one, any two nonnegative weights summing to one produce one one. The two supplied queries change the weights but preserve that output. With the original basis values, doubling only the first dog-value coordinate leaves weights fixed but raises the first output coordinate to about 1.6088593650. Values and other processing matter, so weights alone do not completely explain an output.
  4. A zero query gives equal weights one half each. If only the first key is allowed, the weights are one and zero even with the ball-oriented query. An all-false mask is rejected. Subtracting the largest allowed score before exponentiation preserves ideal normalised ratios and avoids positive overflow. Extreme floating-point zero weights can reflect underflow rather than exactly zero ideal mathematical weight.
  5. Tokenization and embedding plus positional information feed attention-based blocks with other transformations such as feed-forward layers residual connections and normalisation. Encoder-only decoder-only and encoder-decoder variants exist. Queries keys and values have distinct roles; causal decoder restrictions exclude future target positions. The program implements only a supplied-vector attention operation and caller-supplied mask, not learned projections positional encoding full blocks training or generation.
  6. The supplied record supports only the blue card label Water and shelf two. It supplies no universal safety rule; the safety claim is unsupported. It moved has an unclear referent and lacks movement evidence. Keep the narrow supported facts and request clarification or relevant evidence for the rest. In another fictional catalogue, recheck record meaning provenance permissions date and exact claim support rather than relying on fluency a citation shape or an attention score.


Learning evidence

Retain the eight-token sequence and seven-identifier lookup, explicitly supplied embeddings, architecture schematic, scaled dot products, softmax weights and weighted outputs. Add the equal-values and changed-value comparisons, mask and invalid-input tests, declared floating-point tolerance, the fictional claim-by-source audit and a new-context plan. Keep actual executed arithmetic separate from learned embeddings, a real model trace, causal explanation, generated text and independently verified facts.


OER and Wikipedia

Vaswani and colleagues: Attention Is All You Need, sections 3.1 to 3.5, is the primary source for scaled dot-product attention and the original architecture components. Google Machine Learning Crash Course: Large language models and Transformers supplies the introductory token and architecture-variant context. Relevant sections inspected on 15 November 2026. Our supplied-vector examples, limited tokenizer and computations are original rather than a reconstruction of the paper's trained models.

Jain and Wallace: Attention is not Explanation, 2019 research abstract inspected, supports caution about treating weights as explanatory importance. The equal-values construction here is an independent tiny arithmetic illustration, not a replication or a final claim about every later explanation method. 3Blue1Brown: Attention lesson supplies the corresponding visual explanation.

DfE GCE AS and A level computer science subject content, section 4 data representation, is the broad source anchor. The Year 9 decomposition is editorial enrichment and not an A-level qualification or statutory Year-specific transformer requirement.

2D Cartesian Coordinates.svg, H Padleckas, public domain, embedded unchanged as generic numerical axes. Word embedding illustration.svg, Fschwarzentruber, CC BY-SA 4.0, is an optional external country-capital illustration. Its original metadata and preview were verified; the native parser could not embed it, so native display is not claimed.

https://www.youtube.com/watch?v=eMlx5fFNoYc — 3Blue1Brown: Attention in transformers, step-by-step, Deep Learning Chapter 6. https://www.youtube.com/watch?v=x2mRoFNm22g — CodeAI: AI Training Data and Bias. Actual metadata embedding availability and preview bytes were checked and previews visually inspected. These unchanged external embeds do not imply transcript copying endorsement or relicensing. The complete method remains in the written material.

All supplied vectors, sentence calculations, restricted code and fictional archive and garden exercises are original. The standard-library program was actually executed but does not learn embeddings, train a transformer, resolve a real pronoun, generate text or access an AI service. Author checks are not independent expert certification. No personal learner data, local generative model or paid API is used.


Linked learning areas


aiMOOC Projects



Schulfach+




aiMOOCs



aiMOOC Projekte









MOOCwiki · Deutsch

Nach dem Lernen ist vor dem Lernen

Entdecke direkt den nächsten Lernkurs. Weitere Inhalte erscheinen, wenn Du weiter nach unten scrollst.

Zur MOOCwiki-Hauptseite

Mediathek

Suche verfeinern

Dein Lernplan bestimmt, was oben steht. Alle anderen Inhalte bleiben erreichbar. Seitenaufrufe beeinflussen die Auswahl nicht.

Inhalte werden geladen ...

Mediathek wird aus dem Wiki geladen ...