English:Natural Language Processing
Natural Language Processing
Introduction
Natural language processing (NLP) is the field that develops computational methods for analyzing, representing, understanding, and generating human language. It connects Computer science, Artificial intelligence, Machine learning, Linguistics, Computational linguistics, and Information retrieval. NLP systems work with text, speech transcripts, documents, conversations, and other language data.
In this university-level aiMOOC, you will learn how language becomes data, how linguistic structure can be modeled, how statistical and neural methods represent meaning, why transformers changed modern NLP, and how to evaluate systems critically. You will also study failure modes, bias, privacy, and responsible deployment. The aim is not only to know terminology but to make sound design decisions and justify them with evidence.
The diagram above gives a compact historical view of major NLP modeling families. The field has moved through overlapping symbolic, statistical, recurrent-neural, attention-based, and transformer-based approaches. Older methods remain useful when they fit the problem, especially when interpretability, limited data, or low computational cost matters.
The Stanford CS224N lecture above introduces modern NLP and word vectors. While watching, identify one problem that is primarily linguistic, one that is primarily computational, and one that requires both perspectives.
Learning Goals
By the end of this aiMOOC, you should be able to explain the main levels of language analysis, prepare text for modeling, compare sparse and dense representations, distinguish major NLP task types, explain self-attention and transformer architectures, choose suitable evaluation measures, perform systematic error analysis, and assess ethical risks in real applications.
You should also be able to design a small NLP experiment from problem definition to reporting. University-level work requires you to separate training data from evaluation data, document assumptions, compare against a meaningful baseline, and discuss limitations rather than reporting a single score without context.
Why Language Is Difficult for Computers
Human language is productive, context-sensitive, ambiguous, socially situated, and constantly changing. A system that handles words as isolated symbols can miss relationships among morphology, syntax, semantics, discourse, and real-world context.
Consider the sentence I saw the student with the telescope. It can mean that you used a telescope to see the student, or that the student had the telescope. This is a case of syntactic attachment ambiguity. The word bank can refer to a financial institution or the side of a river, which illustrates lexical ambiguity. Pronouns can be ambiguous too: in Alex told Jordan that they had won, the referent of they depends on context.
| Level | Central question | Typical NLP concern |
|---|---|---|
| Morphology | How are words built from smaller meaningful parts? | Inflection, derivation, stemming, and lemmatization |
| Syntax | How are words organized into phrases and sentences? | Part-of-speech tagging and parsing |
| Semantics | What do words and sentences mean? | Word sense, semantic similarity, and meaning representation |
| Pragmatics | What does an utterance mean in context? | Implicature, speaker intention, and context dependence |
| Discourse | How does meaning develop across sentences? | Coreference, coherence, and document-level relations |
A parse tree makes one syntactic analysis explicit. Real NLP systems may use constituency trees, dependency graphs, or learned representations that encode structural information without producing a visible tree.
From Raw Text to Tokens
Before a model can learn from language, text normally has to be represented as units that a computational system can process. Tokenization segments text into tokens such as words, punctuation marks, characters, or subword pieces. Modern neural language models often use subword tokenization because it can represent rare words and productive word forms without requiring one vocabulary entry for every possible word.
Preprocessing decisions should follow the task. Lowercasing may reduce sparsity but can remove information useful for named-entity recognition. Removing punctuation may simplify some bag-of-words tasks but can damage sentiment, dialogue, or authorship signals. Stemming reduces words to approximate stems, while lemmatization aims to map inflected forms to dictionary lemmas. Unicode normalization, sentence segmentation, language identification, and handling of URLs, emojis, or code-switching can also matter.
For multilingual NLP, tokenization is not a neutral technical detail. Languages differ in writing systems, word-boundary conventions, morphology, and available training data. A tokenizer that is efficient for one language may split another language into many more pieces, increasing sequence length and potentially affecting model quality or cost.
N-grams and Local Context
An n-gram is a contiguous sequence of n items, often words or characters. A bigram contains two adjacent items and a trigram contains three. Classical n-gram language models estimate the probability of a token from a limited context. They are simple and interpretable, but they struggle with long-range dependencies and unseen sequences unless smoothing is used.
A skip-gram is related but allows gaps between selected items. The following Wikimedia Commons diagram illustrates a skip-gram pattern.
Representing Words and Documents
Computers need numerical representations. A bag-of-words representation records which terms occur and how often, while TF-IDF downweights terms that are frequent across many documents. These sparse representations can be strong baselines for classification, search, and interpretability.
Word embeddings represent words as dense vectors. Distributional methods learn from patterns of co-occurrence: words that appear in similar contexts tend to receive similar vectors. Such representations can capture useful regularities, although simple word embeddings assign one vector to a word type even when that word has several meanings.
The image shows a simplified geometric intuition for word embeddings. Real embedding spaces are high-dimensional, and apparent vector relations should be interpreted cautiously rather than treated as exact linguistic laws.
Contextual embeddings instead compute token representations using surrounding context. The representation of bank in river bank can therefore differ from the representation of bank in bank account. Contextual representations are central to transformer-based NLP.
Core NLP Tasks
NLP tasks differ in the type of input, output, supervision, and evaluation they require. A practical system may combine several tasks.
| Task | Typical input | Typical output | Example |
|---|---|---|---|
| Text classification | Document or sentence | Category | Spam detection or topic labeling |
| Part-of-speech tagging | Token sequence | Label for each token | Noun, verb, adjective, and other grammatical classes |
| Named-entity recognition | Token sequence | Entity spans and types | Person, organization, place, or date |
| Parsing | Sentence | Syntactic structure | Dependency graph or constituency tree |
| Machine translation | Text in one language | Text in another language | English to French translation |
| Question answering | Question plus context or knowledge source | Answer | Answering a factual question from documents |
| Automatic summarization | One or more documents | Shorter text | Condensing a report |
| Natural language generation | Prompt, data, or hidden state | Text sequence | Dialogue response or report generation |
Named-entity recognition is usually evaluated at the span and type level. A model must identify both where an entity begins and ends and what kind of entity it is.
Machine Translation as an Example of System Design
Machine translation shows how NLP design has changed. Earlier systems could rely heavily on linguistic rules or statistical phrase correspondences. Neural sequence-to-sequence systems learned continuous representations, and transformer models made attention the central mechanism for relating source and target tokens.
The Vauquois triangle is a historical conceptual model for machine translation. Moving upward in the triangle represents increasingly abstract intermediate representations between source and target languages. Modern neural systems do not map neatly onto every historical layer, but the diagram remains useful for comparing direct, transfer-based, and interlingual ideas.
Learning Paradigms in NLP
Supervised learning uses labeled examples such as sentences paired with sentiment labels or token spans paired with entity types. Unsupervised learning seeks structure without task-specific labels. Self-supervised learning creates training signals from the data itself, for example by predicting missing or next tokens. Transfer learning applies knowledge learned in one training stage or task to another.
A strong workflow often starts with a simple baseline before moving to a large neural model. For a small labeled text-classification dataset, a TF-IDF representation plus logistic regression may be competitive, fast to train, and easy to inspect. A larger pretrained transformer may offer better contextual modeling but usually requires more computation and more careful analysis.
Important distinctions include training model parameters, validation or development-set selection of hyperparameters and design choices, and final testing on data that has not influenced model development. Mixing these roles creates data leakage and can produce misleading performance estimates.
Transformers and Large Language Models
A Transformer processes token representations with repeated layers containing attention and feed-forward components. In self-attention, each token forms a query that is compared with keys from tokens in the sequence; the resulting weights determine how value vectors are combined. Multi-head attention allows several learned attention patterns to operate in parallel.
Because self-attention by itself does not encode token order, transformer systems add positional information. Residual connections and normalization support optimization in deep networks. The original transformer architecture used an encoder for the source sequence and a decoder for the target sequence, a design well suited to sequence-to-sequence tasks such as translation.
The Stanford lecture above develops self-attention and transformer design in more detail. As you watch, focus on the trade-off between the ability to connect distant tokens and the computational cost of attention over long sequences.
Encoder, Decoder, and Encoder-Decoder Models
Encoder-only transformers are commonly used to build contextual representations for understanding tasks. Decoder-only transformers generate text autoregressively, predicting subsequent tokens from prior context. Encoder-decoder transformers map an input sequence to an output sequence and are natural choices for tasks such as translation and some forms of summarization.
The GPT diagram above illustrates a decoder-only generative transformer. GPT-style systems are pretrained on next-token prediction and can then be adapted through prompting, fine-tuning, instruction tuning, preference optimization, tool use, retrieval, or other methods. A large language model can perform many tasks through a common text interface, but task versatility does not guarantee factual accuracy, fairness, robustness, or suitability for a particular domain.
Evaluation and Error Analysis
Evaluation should match the real decision or communication problem. For classification, useful measures include accuracy, precision, recall, F1 score, macro-averaged scores for imbalanced classes, and the confusion matrix. For sequence labeling such as named-entity recognition, span-level precision, recall, and F1 are common. For retrieval, measures such as precision at k, recall at k, mean reciprocal rank, or normalized discounted cumulative gain may be appropriate.
Generated language is harder to evaluate. Metrics such as BLEU and ROUGE compare outputs with references, but surface overlap does not fully capture meaning, factuality, usefulness, or style. Learned metrics and model-based evaluators can help, but they can also introduce their own biases and failure modes. Human evaluation remains important when quality is multidimensional or open-ended.
A confusion matrix reveals which labels are being confused rather than hiding all errors inside one aggregate score. Always inspect examples behind the numbers. Error analysis can expose annotation problems, domain mismatch, rare phenomena, systematic bias, and unexpected shortcuts.
The Stanford benchmarking lecture above discusses why evaluation goals matter and why both automatic and human evaluation can be necessary for modern language models.
Experimental Validity
A credible NLP result depends on more than a metric. You should define the population and use case, document data collection, prevent train-test contamination, compare against meaningful baselines, report uncertainty where practical, and test robustness on relevant subgroups or domains.
Data leakage occurs when information from the evaluation set influences training or model selection. Leakage can happen through duplicate documents, preprocessing fitted on the full dataset, labels embedded in metadata, or repeated benchmarking that indirectly tunes decisions to the test set. A high score under leakage is not evidence of real generalization.
Responsible and Ethical NLP
Language technologies can affect people through search ranking, hiring tools, moderation, translation, education, healthcare documentation, public services, and everyday communication. Responsible NLP therefore requires technical evaluation together with social and institutional analysis.
Bias and representation matter because training corpora reflect uneven participation, stereotypes, historical inequalities, and domain-specific assumptions. Evaluate performance across relevant languages, dialects, demographic groups, and use contexts when such categories are legitimate and ethically collected. Avoid treating one benchmark as universal evidence of fairness.
Privacy matters because text may contain names, locations, health information, confidential business information, or other personal data. Data minimization, access control, careful logging, redaction where appropriate, and clear retention policies can be as important as model architecture.
Factuality and hallucination matter in generated text. Fluent output can be unsupported or wrong. Retrieval, citation mechanisms, constrained generation, verification, and human review can reduce risk, but no technique should be assumed to eliminate errors completely.
Transparency and provenance matter because users need to know what a system can and cannot do. Record model versions, data sources when legally and ethically possible, prompt or configuration choices, evaluation conditions, and known limitations. For high-impact applications, define escalation paths and the role of qualified human decision-makers.
A Practical NLP Workflow
A reproducible project can be organized as a sequence of decisions rather than a race to choose the largest model.
| Stage | Questions you should answer |
|---|---|
| Problem definition | What decision, prediction, retrieval, or generation task is being solved, and for whom? |
| Data design | What population does the corpus represent, how was it collected, and what permissions or licenses apply? |
| Annotation | What is the label definition, how are disagreements handled, and is inter-annotator agreement informative? |
| Split strategy | What belongs in training, validation, and test sets, and could duplicates or temporal leakage cross the boundaries? |
| Baseline | What simple method establishes a meaningful reference point? |
| Modeling | Which representation and architecture fit the task, data volume, latency, and resource constraints? |
| Evaluation | Which automatic and human measures reflect success and failure? |
| Error analysis | Which linguistic phenomena, groups, domains, or edge cases explain the remaining errors? |
| Deployment | How will drift, misuse, privacy, cost, and changing user needs be monitored? |
Keep a record of every transformation from raw data to final result. Reproducibility improves when code, environment details, random seeds, dataset versions, hyperparameters, prompts, and evaluation scripts are documented.
Mini Case Study: Classifying University Support Messages
Imagine that a university wants to route incoming support messages into categories such as admissions, finance, IT support, and student services. The objective is not simply to maximize accuracy; it is to route messages reliably while protecting student privacy and allowing staff to correct mistakes.
A sensible baseline could use TF-IDF features with logistic regression. A stronger contextual model could fine-tune a pretrained transformer. If the classes are imbalanced, macro-F1 can complement accuracy because it gives each class equal importance before averaging. The confusion matrix can reveal whether, for example, finance messages are systematically routed to student services.
Before deployment, you would inspect ambiguous examples, duplicate messages, sensitive information, out-of-domain requests, and differences across language varieties. You would also define what happens when model confidence is low and how staff feedback is incorporated without silently contaminating the evaluation set.
Further Study and Open Research Resources
For deeper study, use reliable resources that expose methods, assumptions, and research evidence. The Speech and Language Processing third-edition draft by Dan Jurafsky and James H. Martin provides a broad university-level treatment. The Stanford CS224N course site provides lecture materials on neural NLP and language models. The ACL Anthology provides open access to research papers in computational linguistics, speech, and NLP.
A useful reading habit is to ask four questions of every paper: What problem is defined? What data is used? What comparison establishes improvement? What limitations remain? This habit helps you distinguish a strong empirical claim from an impressive-looking number without adequate context.
The Stanford lecture above focuses on dependency parsing and shows how linguistic structure can be connected to computational modeling.
Interactive Tasks
Quiz: Test Your Knowledge
What is the main purpose of tokenization in NLP? (Segment text into units a model can process) (!Translate every sentence into another language) (!Remove every punctuation mark from text) (!Guarantee that every word has one meaning)
What distinguishes contextual embeddings from simple static word embeddings? (They can represent the same word differently in different contexts) (!They assign every sentence exactly one category) (!They require all words to be written in lowercase) (!They eliminate the need for evaluation data)
What does self-attention primarily allow a transformer to do? (Combine information from relevant tokens in a sequence) (!Store all training documents as exact copies) (!Convert every token directly into a grammar rule) (!Guarantee factual truth in generated text)
Which task identifies spans such as people organizations and places? (Named entity recognition) (!Language modeling) (!Machine translation) (!Sentence segmentation)
What is data leakage? (Use of evaluation information during training or model selection) (!Loss of spaces during tokenization) (!Compression of an embedding vector) (!Removal of rare words from a vocabulary)
Why can macro F1 be useful for imbalanced classification? (It gives each class equal weight before averaging) (!It ignores all minority classes) (!It measures only training speed) (!It replaces the need for a test set)
Which transformer family is naturally suited to mapping one sequence to another? (Encoder decoder) (!Decoder only) (!Bag of words) (!Rule dictionary)
What kind of ambiguity appears in the sentence I saw the student with the telescope? (Syntactic attachment ambiguity) (!Character encoding ambiguity) (!Dataset split ambiguity) (!Numerical rounding ambiguity)
Why are subword tokenizers useful for rare or newly formed words? (They can compose words from smaller learned pieces) (!They always preserve complete words as single tokens) (!They remove the need for a vocabulary) (!They guarantee equal token counts across languages)
Why is human evaluation often important for open ended generation? (Quality can include dimensions that automatic overlap scores miss) (!Human judges always agree perfectly) (!Automatic metrics cannot process text) (!Generated text has no measurable properties)
Memory Game
| Tokenization | Splitting text into units that a model can process |
| Embedding | Dense numerical representation of linguistic items |
| Lemmatization | Mapping inflected forms to dictionary lemmas |
| Attention | Mechanism that weights relationships among sequence elements |
| Entity | Named span such as a person place or organization |
| Benchmark | Dataset and protocol used to compare system performance |
Drag and Drop
| Match the correct terms. | Topic |
|---|---|
| Data preparation | Clean document and split the corpus without leakage |
| Baseline model | Establish a simple reference level of performance |
| Training | Fit model parameters using the training data |
| Evaluation | Measure performance on data not used for fitting |
| Error analysis | Inspect failures to discover systematic weaknesses |
...
Crossword Puzzle
| Tokenization | What process segments text into computational units? |
| Embedding | What dense numerical representation can encode linguistic similarity? |
| Attention | What mechanism weights relationships among tokens in a transformer? |
| Transformer | What neural architecture is built around attention and feed-forward layers? |
| Semantics | What field studies meaning in words and sentences? |
| Benchmark | What standardized dataset and protocol supports system comparison? |
LearningApps
Cloze Text
Open-Ended Tasks
Easy
- Tokenization Lab: Choose five English sentences containing punctuation, contractions, numbers, and unusual names; compare two tokenization strategies, record the resulting tokens, and explain which differences could matter to a model.
- Ambiguity Annotation: Create ten short examples of lexical, syntactic, referential, or pragmatic ambiguity and write two plausible interpretations for each example.
- Entity Recognition Poster: Annotate a public university announcement for people, organizations, places, dates, and other useful entity types, then create a one-page visual explaining difficult boundary decisions.
- Corpus Observation: Collect a small legally reusable text sample, calculate frequent words and collocations, and write a short reflection on what frequency can and cannot tell you about meaning.
Standard
- Sentiment or Topic Baseline: Build a small TF-IDF plus logistic-regression classifier on an open dataset, report a train-validation-test split, and analyze at least ten errors rather than reporting only accuracy.
- Practitioner Interview: Interview a researcher, librarian, translator, software developer, or digital-humanities practitioner about one real language-processing workflow and summarize how linguistic and technical decisions interact.
- Translation Error Study: Compare two machine-translation systems on at least twenty source sentences, design an error taxonomy, annotate the outputs, and discuss where automatic comparison would miss important quality differences.
- Explain Attention Video: Produce a three-minute teaching video that explains queries, keys, values, and self-attention with your own example and a simple visual rather than copying an existing diagram.
Advanced
- Fairness Audit: Select an open NLP dataset or model, define a defensible subgroup or language-variety comparison, measure performance differences, inspect examples, and distinguish observed disparities from claims about their causes.
- Reproducible Transformer Experiment: Fine-tune or probe a pretrained transformer for a clearly defined task, compare it with a simpler baseline, document hyperparameters and random seeds, and report uncertainty or repeated-run variation where feasible.
- Retrieval Project: Build a small semantic-search system over openly licensed documents, evaluate retrieval quality with a hand-labeled query set, and analyze the consequences of poor retrieval for a downstream question-answering system.
- Field Investigation: Visit a university NLP, AI, language-technology, or digital-humanities lab, or attend a live online research seminar when a visit is not possible; interview a participant and produce a report connecting current practice to evaluation, ethics, and reproducibility.
Learning Assessment
- From Need to NLP Task: Given a real communication problem, define the NLP input, output, users, constraints, and failure costs, then justify whether automation is appropriate.
- Baseline Versus Transformer: Compare a sparse linear baseline with a pretrained transformer on the same dataset and explain performance, computation, interpretability, and data requirements rather than selecting a winner from one score.
- Failure Diagnosis: Use a confusion matrix and a sample of incorrect predictions to propose a taxonomy of errors, connect each category to a possible cause, and design one follow-up experiment for each major cause.
- Validity Review: Inspect a hypothetical training pipeline for duplicate documents, preprocessing leakage, repeated test-set use, and hidden label cues, then redesign the evaluation procedure.
- Risk Assessment: Analyze an NLP application for privacy, bias, factuality, misuse, accessibility, and accountability risks, then propose technical and organizational safeguards.
- Metric Critique: Compare automatic translation scores with human judgments on a small sample and explain why agreement or disagreement occurs.
- Replication Plan: Choose one open NLP paper and write a replication plan covering data, preprocessing, baseline, model configuration, evaluation, uncertainty, computational resources, and expected limitations.
Evidence of Learning
Evidence of learning should demonstrate more than recall. It should show that you can connect linguistic phenomena with computational representations, design a valid experiment, interpret quantitative evidence, and communicate limitations.
| Evidence type | What strong evidence looks like |
|---|---|
| Knowledge | Accurate explanation of tokenization, linguistic structure, representations, learning paradigms, transformers, evaluation, and responsible NLP |
| Skills | Reproducible preprocessing, baseline construction, model comparison, metric selection, error analysis, visualization, and critical reading of research |
| Products | Annotated corpus sample, experiment notebook, model card or system report, evaluation table, error taxonomy, poster, video, interview report, or research presentation |
| Transfer achievement | Ability to select and justify an NLP approach for a new domain while considering users, data quality, generalization, privacy, fairness, resource limits, and monitoring |
A high-quality portfolio makes the reasoning traceable: another student should be able to understand what you did, why you did it, what evidence supports your conclusion, and what remains uncertain.
OERs on the Topic
The English Wikipedia article below offers an openly accessible overview that you can use as a starting point for further exploration. Compare its organization and references with the course material and with research literature from the ACL Anthology.
Linked Learning Areas
Natural language processing sits at the intersection of language, computation, statistics, and human-centered system design. The links below connect the core ideas in this aiMOOC to neighboring learning areas.
