Learning outcomes
By the end of this project you should be able to: write a complete program that reads a text file, counts how often each word appears, and reports the most common word and its count; explain how each part maps onto a lesson you have already completed; and trace the whole program so you can predict and verify its output.
Prerequisites
Everything. This is the capstone. You need reading a file lesson 9, the counting idiom lesson 8, the loop lesson 5, and the decomposition mind-set lesson 10. If any of these is shaky, this is the place to go back and shore it up.
The big idea
This program is the one PY4E opens its whole course with — counting the most common word in a piece of text — and by now you can build it yourself from the ideas you already have. It is not a new concept. It is the moment all the separate lessons snap together into one real, useful program. Notice that once you can count words, you can count almost anything: letters, tags, student scores, error types. The pattern is the same, and the pattern is the point.
Why this matters before the syntax
Do not try to write the whole program in one go. Build it one piece at a time, running each piece and checking the output, exactly as you practised in lesson 11. Here is the road map in four steps: first, read the file and print each line to confirm you are reading it at all. Second, split each line into words and print them, so you can see the text breaking apart. Third, feed each word into the counting dictionary and print the whole dictionary. Fourth, add the loop that finds the most common word. Each stage is testable and each builds on the last, so if something is wrong you know exactly which stage broke.
The data — a small text file
For this project you will work on a short file so you can verify the answer by hand. Create words.txt with these three lines.
| words.txt (create this file) |
|---|
| the cat sat on the mat |
| the dog chased the cat |
| the cat ran home |
Before you run anything, count by hand: 'the' appears five times, 'cat' three times, and 'sat', 'on', 'mat', 'dog', 'chased', 'ran', 'home' once each. That manual count is your prediction — the reference the program's output must match. This is tracing and testing in action: you have an expected result before you run the program.
Worked example 1: stage 1 — read the file
First, prove you are reading the file. Open it, loop over it, and print each line stripped.
for line in open("words.txt", "r"):
print(line.strip())the cat sat on the mat
the dog chased the cat
the cat ran homeWorked example 2: stage 2 — split into words
Next, break each line into individual words so you can see the raw material you are counting.
for line in open("words.txt", "r"):
print(line.split())['the', 'cat', 'sat', 'on', 'the', 'mat']
['the', 'dog', 'chased', 'the', 'cat']
['the', 'cat', 'ran', 'home']Worked example 3: stage 3 — count the words
The heart of the program: run the counting idiom over every word in every line. Print the dictionary to check your hand count.
counts = {}
for line in open("words.txt", "r"):
for w in line.split():
counts[w] = counts.get(w, 0) + 1
print(counts){'the': 5, 'cat': 3, 'sat': 1, 'on': 1, 'mat': 1, 'dog': 1, 'chased': 1, 'ran': 1, 'home': 1}This is the exact idiom from lesson 8, wrapped in two nested loops: the outer one over the file's lines, the inner one over the words in each line. The dictionary counts holds every distinct word as a key and its frequency as a value. Compare this against your manual prediction: 'the' has 5, 'cat' has 3, and every other word has 1. If the program does not match, you have found a bug while it is still small.
Worked example 4: stage 4 — find the most common word
Now loop over the dictionary to find the word with the largest count. You track the best answer as you go.
counts = {}
for line in open("words.txt", "r"):
for w in line.split():
counts[w] = counts.get(w, 0) + 1
top_word = None
top_count = None
for w, c in counts.items():
if top_count is None or c > top_count:
top_word = w
top_count = c
print("Total distinct words:", len(counts))
print("Most common word:", top_word, "appears", top_count, "times")Total distinct words: 9
Most common word: the appears 5 timesThe loop sets up two trackers, top_word and top_count, and starts with neither known. For each word and its count, it compares against the current best; if the new count is higher — or if nothing has been set yet — it remembers this word as the new best. This is the maximum-finding pattern, the same shape you used to sum a total, just deciding on a best rather than accumulating. The output confirms your hand count: nine distinct words and 'the' appearing five times.
Common mistakes
Watch outTrying to write the whole program at once. Build it in the four stages above, running and checking after each. A bug in a ten-line version is trivial to find; a bug in a forty-line version is not.
Watch outNot stripping or splitting correctly. If words come back with trailing newlines, strip() each line first; if you want word tokens, use split() with no argument.
Watch outForgetting to handle the empty dictionary. If the file has no lines, counts is empty, top_count stays None, and the program should report that gracefully rather than crash.
Watch outComparing counts with < instead of > in the maximum-finding loop. The trackers start as None; ensure you use a comparison that keeps the larger count, and guard the first iteration.
Check your understanding
Why build the program in four stages instead of one pass?
In the maximum-finding loop, why do the trackers start as None rather than a number like 0?
What happens to the program if words.txt is empty?
Multiple choice
What is the core data structure used to count word frequencies?
What does split() with no argument return?
How does the counting idiom add a new word to the tally?
What is the purpose of the second loop that scans counts.items()?
Why is it important to verify the program against a manual count?
Which loop structure does the program use to read the whole file?
If the program should handle an empty file safely, what must it check?
How does this capstone program relate to the earlier lessons?
Summary and next
You built a complete word-count program from the ideas of every previous lesson: read a file, split it into words, count them in a dictionary, and find the most common. You verified each stage against a manual prediction. That is not just a program — it is a demonstration that the separate pieces of the course compose into something genuinely useful.
Where the course goes from here: tuples let you sort the counted words (PY4E chapter 10), and from there the same counting idea extends to real data — scraping, parsing, and analysing text at scale. You have the foundation. Everything after this is this same pattern, applied to bigger and messier data.