Zum Inhalt springen

English:Programming data pipelines with modular code - AI Year 10

Aus MOOCsWiki Staging
aiMOOC-Siegel aiMOOC

Programming data pipelines with modular code - AI Year 10

QR-Code



Introduction

A generic data pipeline with an input filter intermediate copies and several destinations
A pipeline connects stages. The original diagram has non-English labels and does not depict our invented records or a trained model.

Break a data task into functions, declare what each accepts and returns, and test both the stages and their connections. Begin with counting positive values in two minus one zero four, then clean an invented stock-change dataset containing a missing value, a duplicate and an inconsistent label. The complete bounded program was actually executed. It does not train a model, authenticate a real source or silently replace missing data.


Give each function a clear contract and trace

A function groups a reusable operation with inputs and an output. Naming it makes a program easier to describe, but does not make its logic correct. A contract states which inputs it accepts, what its result means and what happens outside the allowed range. Test the contract with normal, boundary and invalid cases instead of treating a neat function name as evidence of success.

Our first contract is simple: count the values strictly greater than zero in a list of at most two hundred integers between minus ten thousand and ten thousand. Booleans, missing values, strings and floating-point numbers are outside this particular teaching contract. That restriction is a design choice, not a claim that every data pipeline must reject all non-integer measurements. A different domain may need a different validated representation.

For the source list two, minus one, zero, four, start a counter at zero. Two is positive, so increment to one. Minus one is not positive, so keep one. Zero is not positive, so keep one. Four is positive, so finish at two. The empty list returns zero, and the all-negative list minus three, minus one returns zero. Returning zero from a valid empty list is different from pretending that missing or invalid values were observed as zero.

Next value Is it greater than zero? Counter after the step
2 Yes 1
-1 No 1
0 No 1
4 Yes 2

An intentionally defective comparison function counts values greater than or equal to zero. On the source list it returns three, because it includes zero. The correct function returns two. Document the correction precisely: the requirement is positive rather than nonnegative, so change the predicate from greater-than-or-equal-to-zero to greater-than-zero. Both functions still pass empty and all-negative tests. A test containing zero is necessary to expose this particular error; adding only more negative examples would miss it.

The executed program retains the defective function under the explicit name buggy_count_includes_zero solely as a counterexample. The main pipeline calls the correct count_positive function. This is an authored before-and-after teaching comparison, not an undisclosed failure of the published algorithm or a claim that every defect can be found by one test.

Corey Schafer introduces Python functions, parameters and return values. Connect those ideas to a contract and trace before examining the full program. His tutorial is an external supporting video, not the source of our original fixture or proof that a function is correct. The text provides all required methods independently of playback.

Think about interfaces between stages. A record-cleaning function returns retained records and review information. A transformation extracts validated numeric values from those records. The counting function receives a numeric list, not a raw mixture of records, strings and missing fields. If a caller bypasses cleaning, the next stage must not quietly accept an incompatible shape. Decomposition helps locate a defect only when the inputs, outputs and assumptions are also explicit.

Try first: trace the counter on zero alone, then on one, zero, minus one. Hint: positive excludes zero. Partial check: the zero-only correct result is zero, while the deliberately defective comparison gives one. Write a normal, boundary and invalid-input test and explain which requirement each checks rather than recording only passed or failed.


Validate records then clean transform and report

The original record dataset describes signed changes in a fictional stock count, measured in whole items. A negative value is a decrease, zero is no change and a positive value is an increase. The positive field is a Boolean annotation defined here as whether value is strictly greater than zero. It is checked for consistency with that declared rule; it is not an independently observed machine-learning target or a prediction to be fed back into a model.

Every record has record_id, value, positive, unit and source. Unit must be items_delta, and source must match the authored marker synthetic-stock-v1. Record identifiers are fictional row identities, not people. Checking the marker verifies this fixture's expected field value; it does not authenticate a real publisher or establish that a dataset is accurate, representative or permissioned.

Raw case Value and annotation Declared handling
R01 2 and true Retain
R02 -1 and false Retain once
R03 0 and false Retain
R04 4 and true Retain
Second identical R02 -1 and false Remove one repeated copy under the declared policy
R05 Missing value, false annotation Quarantine; do not invent a measurement
R06 3 and false annotation Quarantine because the annotation contradicts the defined rule
R07 Boolean true used as a value Quarantine because the value must be an exact integer

Validation checks each record's exact field set, identity format, unit, source marker, value type and range, and annotation consistency. Invalid rows retain an original input index and reason in a quarantine list. Quarantine means separated for appropriate review, not deleted evidence or an automatic correction of the source. We do not turn a missing value into zero or change the inconsistent annotation without evidence.

After validation, group the valid rows by identity. Identical copies retain one record and count the extra copies removed. Conflicting valid versions of one identity are all quarantined, rather than accepting whichever appeared first. The program compares only rows that passed initial validation; an organisation with a different risk policy may also need to hold an entire identity when an invalid copy appears. State the policy instead of claiming that one duplicate rule is universal.

Add another valid R01 with value nine and annotation true to create the separate conflict fixture. Both R01 versions now conflict and are held for review. The retained identifiers become R02, R03 and R04; the positive count is one from the value four. Five raw rows are quarantined and one identical repeat is removed. A different valid measurement under the same identifier is not an identical duplicate.

The complete bounded program below was executed. Its stages are checked_values and count_positive, row_problem, clean, a projection to numeric values, and reporting. A future approved model stage would need its own data, purpose, interface and evaluation; this program uses a transparent counting rule and no model. The deliberately buggy comparison is excluded from the main pipeline.

"""Bounded synthetic integer pipeline with explicit rejected records and duplicate policy."""
import json
import re


def checked_values(values):
    if not isinstance(values, list) or len(values) > 200:
        raise ValueError("Expected at most two hundred integer values")
    if any(type(v) is not int or not -10000 <= v <= 10000 for v in values):
        raise ValueError("Expected exact bounded integers, not Boolean or missing values")


def count_positive(values):
    checked_values(values)
    count = 0
    for value in values:
        if value > 0:
            count += 1
    return count


def buggy_count_includes_zero(values):
    checked_values(values)
    return sum(value >= 0 for value in values)


def fixtures():
    base = dict(unit="items_delta", source="synthetic-stock-v1")
    rows = [dict(base, record_id="R01", value=2, positive=True),
            dict(base, record_id="R02", value=-1, positive=False),
            dict(base, record_id="R03", value=0, positive=False),
            dict(base, record_id="R04", value=4, positive=True)]
    return rows + [dict(rows[1]),
                   dict(base, record_id="R05", value=None, positive=False),
                   dict(base, record_id="R06", value=3, positive=False),
                   dict(base, record_id="R07", value=True, positive=True)]


def row_problem(row):
    if not isinstance(row, dict) or set(row) != {"record_id", "value", "positive", "unit", "source"}:
        return "schema"
    if not isinstance(row["record_id"], str) or not re.fullmatch(r"R[0-9]{2}", row["record_id"]):
        return "identity"
    if row["unit"] != "items_delta" or row["source"] != "synthetic-stock-v1":
        return "unit_or_source"
    if row["value"] is None:
        return "missing_value"
    if type(row["value"]) is not int or not -10000 <= row["value"] <= 10000:
        return "value_type_or_range"
    if type(row["positive"]) is not bool or row["positive"] != (row["value"] > 0):
        return "inconsistent_label"
    return None


def clean(rows):
    if not isinstance(rows, list) or len(rows) > 200:
        raise ValueError("Expected at most two hundred synthetic records")
    groups, quarantine = dict(), []
    for index, row in enumerate(rows):
        problem = row_problem(row)
        if problem:
            quarantine.append(dict(index=index, reason=problem))
        else:
            groups.setdefault(row["record_id"], []).append((index, row))
    retained = []
    duplicates = 0
    for identity in sorted(groups):
        entries = groups[identity]
        first = entries[0][1]
        if any(row != first for index, row in entries):
            quarantine.extend(dict(index=index, reason="conflicting_identity") for index, row in entries)
        else:
            retained.append(dict(first))
            duplicates += len(entries) - 1
    return dict(retained=retained, duplicate_rows_removed=duplicates,
                quarantine=sorted(quarantine, key=lambda row: row["index"]))


def pipeline(rows):
    cleaned = clean(rows)
    values = [row["value"] for row in cleaned["retained"]]
    return dict(raw_rows=len(rows), retained_ids=[row["record_id"] for row in cleaned["retained"]],
                values=values, positive_count=count_positive(values),
                duplicate_rows_removed=cleaned["duplicate_rows_removed"],
                quarantine=cleaned["quarantine"], model_used=False)


def demonstrate():
    data = fixtures()
    conflict = dict(data[0], value=9, positive=True)
    return dict(source_values=[2,-1,0,4], source_count=count_positive([2,-1,0,4]),
                deliberately_buggy_count=buggy_count_includes_zero([2,-1,0,4]),
                empty_count=count_positive([]), all_negative_count=count_positive([-3,-1]),
                main=pipeline(data), empty_pipeline=pipeline([]), conflict=pipeline(data+[conflict]),
                data_authored=True, source_identity_authenticated=False,
                missing_values_imputed=False, model_trained=False)


if __name__ == "__main__":
    print(json.dumps(demonstrate(), indent=2))

The main eight-row fixture retains R01 through R04 with values two, minus one, zero and four. It reports positive_count two, one duplicate row removed and three quarantined rows. Check the accounting: four retained plus one removed duplicate plus three quarantined equals eight raw rows. In the nine-row conflict case, three plus one plus five equals nine. An empty valid pipeline returns empty lists and a zero count, while an invalid top-level container raises an error.

The verified CodeAI recording provides data-and-machine-learning context. Use it to ask why definitions and data quality matter before any possible model stage. The stock fixture is original and was not a training experiment. A larger raw collection can contain more duplicates, missingness or contradictions, so row count alone does not establish usefulness.

Attempt before checking: identify the reason for each quarantined main-fixture row and explain why the three reasons are different. Hint: a Boolean true can behave numerically in some Python operations, but our contract requires type int exactly. Partial check: R07 is a value-type problem, whereas R06 has an integer value but a contradictory annotation. Keep those failure categories separate in the test log.


Test modules and integration then transfer carefully

Testing a modular pipeline needs both component and integration evidence. A unit-style check can verify that count_positive returns two for the source list, zero for empty input and zero for all-negative input. A separate row check can identify a missing value. An integration check must show that only retained validated records reach the counting stage, that duplicate handling preserves the declared policy and that the final report accounts for every input row.

Compare expected and actual results in a structured log. For the deliberately defective function, record the source input, expected two, actual three, error type zero incorrectly included, correction to a strict comparison and the corrected result two. Retest the empty, all-negative and zero-containing cases. This documents one specific correction and its evidence; it does not imply that the name function or a passed sample proves the entire program correct.

Test type Original test Expected outcome or rejection
Normal count 2, -1, 0, 4 2 positives
Empty boundary Empty numeric list 0
All-negative boundary -3, -1 0
Zero boundary 0 alone Correct count 0; defective comparison 1
Invalid numeric input Missing value, Boolean, string or non-integer measurement Explicit rejection under this contract
Main integration Eight authored records 4 retained, 1 repeat removed, 3 held, 2 positives
Conflicting valid identity Add R01 with value 9 Both R01 versions held; 3 retained and 1 positive

Preserve data meaning at every boundary. Extracting values is a transformation, but it does not convert units or fill missingness. If a caller sends full dictionaries to count_positive, the contract rejects them rather than guessing which field to use. If a record arrives with another unit or source marker, this fixture holds it for review. That is a conservative choice for the exercise, not evidence that every differently formatted real record is useless.

The source marker and expected annotation rule do not establish source authenticity, sample representativeness or permission for a new purpose. Keep provenance, collection method, field definitions, units, missing-value policy and intended task in a data dictionary. A later model could introduce training-validation-test separation, leakage and outcome evaluation requirements that this deterministic count does not implement. Do not describe this pipeline as model training or classify its data-consistency checks as predictive accuracy.

Transfer the design to an invented museum light-reading dataset. You may still separate schema validation, quality checks, identity handling, transformation and reporting. However, lux has a different meaning from signed stock change, the allowed range may be nonnegative, and a label such as above a declared threshold must have a new definition. Copying the items_delta unit and positive annotation unchanged would be a semantic error even if the code ran without a syntax exception.

Use fictional records or appropriately permissioned material and teacher-approved execution. No pupil names, private stock records, credentials, sensor connections or model service are required. A paper trace can demonstrate understanding; a documented run demonstrates particular execution results. Neither proves practical suitability for a real warehouse, museum or safety decision. Keep a human reviewer for unresolved source conflicts rather than silently changing data to make the pipeline pass.

For your final portfolio, provide a data dictionary, module diagram, normal and boundary traces, invalid-input decisions, main-fixture accounting and the documented comparison-predicate correction. Add a new-domain test plan and a clear limitation or next improvement. This route applies the GCSE algorithm and programming source anchor as editorial Year 10 AI preparation; it is not a separate qualification or a claim that a single tested function completes a production data system.


Interactive Tasks


Quiz: Test Your Knowledge

What does the correct source-list count return? (Two) (!Three) (!Four) (!Zero)



Why does the deliberately defective comparison return three? (It counts zero as well as positive values) (!It rejects every negative value as a missing row) (!It learns a new model parameter) (!It authenticates the source marker)



What should a valid empty numeric list return here? (Zero) (!A fabricated positive observation) (!A copied previous result) (!A qualification award)



What does the positive annotation mean in this fixture? (Whether the integer stock change is greater than zero) (!An independently measured future model outcome) (!Whether a real pupil is successful) (!Whether the data publisher has been authenticated)



How is the missing R05 measurement handled? (Quarantine without inventing a value) (!Replace it with zero silently) (!Copy a neighbouring value automatically) (!Include it as a positive count)



What is done with an identical second R02 under the declared policy? (Retain one copy and count one repeated row removed) (!Treat it as a new independent identity) (!Quarantine every unrelated record) (!Increase the measurement value)



What happens when a conflicting valid R01 is added? (Both valid R01 versions are held for review) (!The first version wins by arrival order) (!The larger number is automatically true) (!The unit is silently changed)



Why is Boolean true rejected as a numeric value? (The teaching contract requires exact integers rather than Booleans) (!Every Boolean is invalid in every Python program) (!The source has proved that the value is malicious) (!The code must train a model first)



What does a function name establish about correctness? (Nothing without a contract and suitable tests) (!Automatic correctness of every input) (!That integration checks are unnecessary) (!That missing data is always zero)



What must change when transferring to museum light readings? (Units ranges label meaning and relevant tests must be rechecked) (!Only the title while keeping every assumption) (!The past results become universal accuracy estimates) (!Private learner names must be added)





Memory Game

Function A reusable operation with inputs and an output
Parameter A named input in a function definition
Return The value a function supplies to its caller
Validation Checking the declared form and allowed values
Quarantine Separating unresolved records for review
Interface The contract connecting stages





Drag and Drop

Declare each field unit and source assumption Define the data
Check schema types ranges and annotation consistency Validate records
Handle identical and conflicting valid identities Clean under a declared policy
Extract retained numeric values without inventing measurements Transform the representation
Count positives and account for every raw row Report results





Crossword Puzzle

FUNCTION A named reusable operation
PARAMETER A named input in a definition
RETURN Provide a result to the caller
VALIDATION Check form types and allowed values
QUARANTINE Hold unresolved records for review
INTERFACE A contract connecting stages





LearningApps

Optional extra practice. Choose an appropriate activity with your teacher. If the external collection is unavailable, use the native activities and independent tasks in this course.


Cloze Text

Complete the explanation.
A reusable operation is a

. A named definition input is a

. The supplied result is a

value. A positive count excludes

. The defective comparison includes

. Input form and bounds need

. Identical repeated rows use a declared duplicate

. Conflicting identities are held in

. Missing values are not silently

. Extracting retained values is a

. Connected stages share an

. A new domain requires renewed

.



Open Tasks


Easy

  1. Trace the source count: Write the counter value after two minus one zero and four. Hint: zero is not positive. Give the final result and state the exact numeric input contract.
  2. Compare two predicates: Predict the zero-only result for the correct and deliberately defective functions. Explain why empty and all-negative tests alone fail to distinguish them.
  3. Make a data dictionary: Describe record_id value positive unit and source for the invented stock changes. Distinguish a signed change from an absolute stock level and a source marker from authentication.
  4. Sort the three problems: For R05 R06 and R07 identify missingness inconsistent annotation and value-type error. Explain why quarantine preserves a review question rather than supplying a guessed correction.


Standard

  1. Document one correction: Record input expected result actual defective result cause strict-comparison correction and corrected result. Retest normal empty all-negative and zero-containing lists.
  2. Check the main accounting: Trace validation valid-identity grouping value extraction and counting. Show why four retained plus one identical repeat plus three quarantined equals eight raw rows.
  3. Test the conflict fixture: Add the supplied valid R01 with value nine in a copy of the fictional data. Predict before using approved execution, then record both quarantined versions the retained values and the positive count.
  4. Test an interface failure: Propose a call that sends raw records rather than numeric values into count_positive. State expected rejection and explain how stage contracts locate the error without guessing a field.


Advanced

  1. Compare duplicate policies: Explain the implemented valid-row grouping policy and a more conservative policy that holds an identity after any suspicious copy. Discuss evidence review workload and context without declaring either universally correct.
  2. Transfer to a new domain: Design invented museum lux records with a declared range threshold label missing-value rule and provenance. Specify which pipeline stages transfer and which stock assumptions must change.
  3. Plan a later model stage: Draw a proposed model interface after validated transformation and identify additional split leakage label and outcome checks. Keep the proposal separate from the actually executed deterministic count.
  4. Create a reproducible evidence pack: Produce an original annotated module diagram or narrated walkthrough with a text transcript, complete fixture trace boundary and invalid tests correction record provenance limits and a justified next improvement. Use no personal records or live sensor connections.



Learning Check

  1. Trace and correct the count: Explain the source case and the documented predicate correction.
  2. Interpret fields and validation: State the record meanings and distinct main-fixture failures.
  3. Follow the main pipeline: Give retained records duplicate handling quarantine and report.
  4. Handle a conflicting identity: What changes after adding the valid R01 with value nine?
  5. Test modules and connections: Give normal boundary invalid and integration checks and explain their limits.
  6. Transfer responsibly: What must change for a museum light-reading pipeline?


Worked checks and assessment guidance

  1. The correct counter moves from zero to one after two, remains one after minus one and zero, then becomes two after four. The intentionally defective nonnegative comparison counts zero and returns three. Changing the condition to strictly greater than zero corrects this requirement. Empty and all-negative cases both return zero under either predicate, so a zero-containing test is necessary to expose this defect.
  2. Values are whole-item signed stock changes, with positive defined as value greater than zero. The unit is items_delta and source is the authored synthetic-stock-v1 marker. R05 lacks a measurement, R06 has value three with a false annotation, and R07 uses Boolean true where an exact integer is required. These cases are quarantined for different reasons. Matching a source marker does not authenticate the publisher or prove representativeness.
  3. The eight-row fixture retains R01 through R04 with values two minus one zero four. An identical second R02 contributes one removed repeat. The three invalid rows remain in the quarantine record. Transformation extracts validated values and the correct count returns two. Four retained plus one removed repeat plus three held rows equals eight. No missing value is changed to zero and no model is used.
  4. Both valid R01 versions disagree and are quarantined rather than selecting the first or largest. R02 R03 and R04 remain with minus one zero four, giving one positive. There is still one identical repeated R02 removed, with five quarantined rows overall. Three retained plus one removed plus five held equals nine. Only initially valid rows enter this identity comparison under the stated policy.
  5. Check the mixed source list, empty list, all-negative and zero-only lists; reject missing non-integer Boolean or wrong-shape numeric inputs under the declared contract. Separately test row schema identity unit source type range and annotation. Integration checks must connect retained records to numeric extraction and ensure row accounting. A passed function sample does not establish every interface provenance permission or future model requirement.
  6. Retain the staged approach but redefine lux meaning allowed range threshold label missingness source permissions and conflict review. Signed items_delta assumptions cannot be copied unchanged. Document expected and actual or not-run tests and human handling of unresolved records. Any future model needs separate purpose data split leakage and outcome evaluation; the published example only executes deterministic counting on invented records.


Learning evidence

Retain module contracts, the full mixed-list trace, empty negative and zero boundary tests, the explicit defective-to-correct predicate record and rejected-input checks. Add the five-field dictionary, eight-row and conflict-fixture accounting, quarantine reasons, unchanged source evidence and a new-domain plan. Distinguish a function name from tested correctness, a missing measurement from zero, a marker from authenticated provenance and deterministic reporting from model training.


OER and Wikipedia

Python tutorial: Defining functions and Python tutorial: Data structures, relevant function list and dictionary material inspected 15 November 2026. The bounded code fixture and tests are original, not copied tutorial examples or a general data-ingestion framework.

DfE GCSE Computer Science subject content, algorithms programming and modular-interface skills, supplies the source anchor. The Year 10 pipeline decomposition is editorial AI preparation, not a statutory Year-specific AI syllabus or qualification.

ETL jobb.jpg, Hansth, public domain, embedded unchanged as a generic pipeline illustration with non-English labels. It is not our stock data, module architecture or evidence of model training. Original metadata and preview bytes are retained and verified.

https://www.youtube.com/watch?v=9Os0o3wzS_I — Corey Schafer: Python Tutorial for Beginners 8, Functions. https://www.youtube.com/watch?v=OeU5m6vRyCk — CodeAI: AI: What is Machine Learning?. Actual retained video metadata embedding availability and preview bytes checked; previews visually inspected. Unchanged external embeds imply no transcript copying endorsement or relicensing. Complete methods remain in the written course.

All values identifiers source markers units annotations duplicate and conflict cases code and test logs are original fictional teaching constructions. The deliberately buggy comparison is labelled and excluded from the main pipeline; its output and the strict-predicate correction were actually demonstrated. The standard-library modules and complete pipeline were executed without model training source authentication imputation external services paid APIs or personal learner records. Author checks are not independent expert certification.


Linked learning areas


aiMOOC Projects



Schulfach+




aiMOOCs



aiMOOC Projekte









MOOCwiki · Deutsch

Nach dem Lernen ist vor dem Lernen

Entdecke direkt den nächsten Lernkurs. Weitere Inhalte erscheinen, wenn Du weiter nach unten scrollst.

Zur MOOCwiki-Hauptseite

Mediathek

Suche verfeinern

Dein Lernplan bestimmt, was oben steht. Alle anderen Inhalte bleiben erreichbar. Seitenaufrufe beeinflussen die Auswahl nicht.

Inhalte werden geladen ...

Mediathek wird aus dem Wiki geladen ...