Warm-up · Activity 1 of 7
// A4.3 · ~30 min · Advanced
The cost of data-structure choices
After this lesson you pick a set or dict for lookups and a deque for work at the left end, and show why: count the work, then time the ratio.
You will be able to
- Use a set or dict instead of a list for membership tests and lookups, and explain why
- Use a deque for adding and removing at the left end instead of list.pop(0) and insert(0, x)
- Show a difference by counting operations and by timing ratios as the input grows
Predict · Activity 2 of 7
Predict before you read on: Key counts how often == is called on it. How many comparisons does one lookup of the last key take in the list, and in the set?
class Key: comparisons = 0 def __init__(self, n): self.n = n def __eq__(self, other): Key.comparisons += 1 return self.n == other.n def __hash__(self): return hash(self.n) keys = [Key(i) for i in range(1000)] key_set = set(keys) Key.comparisons = 0 Key(999) in keys in_list = Key.comparisons Key.comparisons = 0 Key(999) in key_set print(in_list, Key.comparisons)Practice · Activity 3 of 7
Fill in the method that takes the first item off the queue in constant time.
first = queue.____()first = queue.()Practice · Activity 4 of 7
items is a list with a million elements. Which operation takes time that grows with its length?
Practice · Activity 5 of 7
Match each operation to its cost.
Brain teaser · Activity 6 of 7
Brain teaser. Someone "optimised" a lookup by turning the vocabulary into a set. Key now counts calls to __hash__. What does this print?
class Key: hashes = 0 def __init__(self, n): self.n = n def __eq__(self, other): return self.n == other.n def __hash__(self): Key.hashes += 1 return hash(self.n) words = [Key(i) for i in range(10)] vocabulary = [Key(i) for i in range(100)] found = [w for w in words if w in set(vocabulary)] print(Key.hashes)Apply · Activity 7 of 7
Mini-task. For n = 1,000, 10,000 and 100,000, look up a missing value in a list and in a set of range(n). First count the == calls with a Key class like the one above, then time both lookups with timeit.repeat, the data built outside the timed lambda. Print one line per n. What happens to each count and each time when n grows tenfold?
Check your work against this list
Build it yourself
Read the worked example, then write the exercises. Your code runs in your browser or on your computer and is never uploaded.
Worked example
Counting and timing two choices
Part 1 counts the == calls for one lookup of the last key in a list and in a set, for three sizes: exact numbers, the same on every machine. Part 2 times list against set membership and list.pop(0) against deque.popleft() for 100,000 items. The seconds change on every run, so the program prints only whether each ratio clears a wide threshold. Print t_list and t_set as well to see the raw figures.
main.py
import timeit
from collections import deque
class Key:
"""An int wrapper that counts how often Python compares it."""
comparisons = 0
def __init__(self, n: int) -> None:
self.n = n
def __eq__(self, other: object) -> bool:
Key.comparisons += 1
return isinstance(other, Key) and self.n == other.n
def __hash__(self) -> int:
return hash(self.n)
# 1. Count the work: == calls for one lookup of the last key.
for n in (1000, 2000, 4000):
keys = [Key(i) for i in range(n)]
key_set = set(keys)
Key.comparisons = 0
Key(n - 1) in keys
in_list = Key.comparisons
Key.comparisons = 0
Key(n - 1) in key_set
print(f"n={n}: list compares {in_list}, set compares {Key.comparisons}")
# 2. Time it. The seconds vary; ratios this large do not.
n = 100_000
data = list(range(n))
data_set = set(data)
t_list = min(timeit.repeat(lambda: n - 1 in data, number=20, repeat=5))
t_set = min(timeit.repeat(lambda: n - 1 in data_set, number=20, repeat=5))
print("list 'in' at least 100x slower than set 'in':", t_list / t_set >= 100)
items = list(range(n))
queue = deque(items)
def list_left() -> None:
items.pop(0) # shifts every other item one place
items.append(0)
def deque_left() -> None:
queue.popleft() # constant time at either end
queue.append(0)
t_pop = min(timeit.repeat(list_left, number=200, repeat=5))
t_popleft = min(timeit.repeat(deque_left, number=200, repeat=5))
print("list.pop(0) at least 10x slower than deque.popleft():", t_pop / t_popleft >= 10)
Run it with
python main.pyOutput
n=1000: list compares 1000, set compares 1
n=2000: list compares 2000, set compares 1
n=4000: list compares 4000, set compares 1
list 'in' at least 100x slower than set 'in': True
list.pop(0) at least 10x slower than deque.popleft(): True- Doubling n doubles the list’s comparisons; the set needs one, whatever the size.
- The set’s single comparison confirms that the key in the slot really is equal: equal hashes are not enough.
- The measured ratios are in the thousands and hundreds, far above the thresholds printed.
- Each timed function keeps the list the same length, so every run measures the same work.
Change it and run it
Tab indents and Shift+Tab outdents. To leave the editor with the keyboard, press Esc, then Tab.
The first run downloads Python for your browser (up to 6.5 MB) and keeps it cached. Your code stays on your device.
Exercises
Exercise 1 of 2
Duplicates without scanning
find_duplicates returns the items that occur more than once, each listed once, in the order in which they first repeat. It works, but it remembers what it has seen in lists, so every check scans them. Keep the result exactly the same and make both checks hash lookups. A test counts the == comparisons for 401 keys.
Tab indents and Shift+Tab outdents. To leave the editor with the keyboard, press Esc, then Tab.
The first run downloads Python for your browser (up to 6.5 MB) and keeps it cached. Your code stays on your device.
Hints
Hint 1
seen and reported are only ever asked "is this item in you?": that is what a set is for.
Hint 2
Start them as set() and use .add() instead of .append().
Hint 3
duplicates itself stays a list, because its order is part of the result.
Show a solution
One way to solve it. Yours can look different and still pass the checks.
from collections.abc import Hashable
def find_duplicates(items: list[Hashable]) -> list[Hashable]:
"""Items that occur more than once, each listed once, in order of their second occurrence."""
seen: set[Hashable] = set()
reported: set[Hashable] = set()
duplicates = []
for item in items:
if item in seen:
if item not in reported:
reported.add(item)
duplicates.append(item)
else:
seen.add(item)
return duplicates
Run it on your computer
Install Python 3.14 or newer. Save these files in one folder, open a terminal in that folder, and run the commands below.
main.py
from collections.abc import Hashable
def find_duplicates(items: list[Hashable]) -> list[Hashable]:
"""Items that occur more than once, each listed once, in order of their second occurrence."""
seen = []
reported = []
duplicates = []
for item in items:
if item in seen:
if item not in reported:
reported.append(item)
duplicates.append(item)
else:
seen.append(item)
return duplicates
test_main.py
from main import find_duplicates
class Key:
"""An int wrapper that counts how often Python compares it."""
comparisons = 0
def __init__(self, n):
self.n = n
def __eq__(self, other):
Key.comparisons += 1
return isinstance(other, Key) and self.n == other.n
def __hash__(self):
return hash(self.n)
def __repr__(self):
return f"Key({self.n})"
def test_duplicates():
"""3 and 1 repeat: [3, 1], each once"""
got = find_duplicates([3, 1, 3, 2, 1, 3])
assert got == [3, 1], f"got {got!r}, expected [3, 1]"
def test_no_duplicates():
"""Distinct items give an empty list"""
got = find_duplicates(["a", "b", "c"])
assert got == [], f"got {got!r}, expected []"
def test_comparisons():
"""400 distinct keys need almost no == comparisons"""
items = [Key(i) for i in range(400)] + [Key(7)]
Key.comparisons = 0
got = find_duplicates(items)
assert [k.n for k in got] == [7], f"got {got!r}, expected [Key(7)]"
assert Key.comparisons < 50, f"find_duplicates made {Key.comparisons} == comparisons; a list is scanned item by item, a set is not"
On macOS and Linux, type python3 wherever these commands say python, as in the first lesson.
Run the program:
python main.pyRun the checks (needs learnrun.py in the same folder):
python learnrun.py testDownload learnrun.pyExercise 2 of 2
A bounded history
History keeps the last size events, oldest first. The starter removes the oldest event with list.pop(0), which shifts all the others. Store the events in a collections.deque with maxlen=size instead, so that a full deque drops its oldest event by itself in constant time. Keep the attribute name events, and keep oldest() and recent() working; recent() must return a new list.
Tab indents and Shift+Tab outdents. To leave the editor with the keyboard, press Esc, then Tab.
The first run downloads Python for your browser (up to 6.5 MB) and keeps it cached. Your code stays on your device.
Hints
Hint 1
from collections import deque, then self.events: deque[str] = deque(maxlen=size).
Hint 2
With maxlen set, append on a full deque discards an item from the other end: add needs no if.
Hint 3
events[0] is still the oldest event, and it is fast: deque indexing is quick at both ends.
Show a solution
One way to solve it. Yours can look different and still pass the checks.
from collections import deque
class History:
"""The last `size` events, oldest first."""
def __init__(self, size: int) -> None:
self.size = size
self.events: deque[str] = deque(maxlen=size)
def add(self, event: str) -> None:
self.events.append(event) # a full deque drops its oldest item itself
def oldest(self) -> str:
return self.events[0]
def recent(self) -> list[str]:
return list(self.events)
Run it on your computer
Install Python 3.14 or newer. Save these files in one folder, open a terminal in that folder, and run the commands below.
main.py
class History:
"""The last `size` events, oldest first."""
def __init__(self, size: int) -> None:
self.size = size
self.events: list[str] = []
def add(self, event: str) -> None:
self.events.append(event)
if len(self.events) > self.size:
self.events.pop(0) # shifts every other event
def oldest(self) -> str:
return self.events[0]
def recent(self) -> list[str]:
return list(self.events)
test_main.py
from collections import deque
from main import History
def test_keeps_last_three():
"""With size 3, adding a to e keeps c, d and e"""
h = History(3)
for event in "abcde":
h.add(event)
assert h.recent() == ["c", "d", "e"], f"recent() returned {h.recent()!r}, expected ['c', 'd', 'e']"
def test_oldest():
"""oldest() is the first event still kept"""
h = History(2)
for event in ["login", "view", "logout"]:
h.add(event)
assert h.oldest() == "view", f"oldest() returned {h.oldest()!r}, expected 'view'"
def test_uses_bounded_deque():
"""events is a deque with maxlen equal to size"""
h = History(4)
ok = isinstance(h.events, deque) and h.events.maxlen == 4
assert ok, f"events is {h.events!r}; expected deque(maxlen=4), which drops the oldest item in constant time"
def test_recent_is_a_copy():
"""recent() returns a new list, not the internal store"""
h = History(2)
h.add("a")
h.recent().append("x")
assert h.recent() == ["a"], f"changing the list from recent() changed the history: {h.recent()!r}"
On macOS and Linux, type python3 wherever these commands say python, as in the first lesson.
Run the program:
python main.pyRun the checks (needs learnrun.py in the same folder):
python learnrun.py testDownload learnrun.pyCommon mistakes
Putting lists in a set
rows = [[1, 2], [3, 4], [1, 2]]
unique = set(rows)
print(len(unique))
What Python prints
TypeError: cannot use 'list' as a set element (unhashable type: 'list')Why, and the fix
Set elements and dict keys must be hashable, and a list is not, because it can change. Convert each row to a tuple first: set(tuple(row) for row in rows) gives 2 unique rows.
Writing {} for an empty set
seen = {}
seen.add("ada")
What Python prints
AttributeError: 'dict' object has no attribute 'add'Why, and the fix
{} is an empty dict, not an empty set, because dicts had the braces first. Write seen = set() for an empty set; {"ada"} with at least one element is a set.
Giving deque.pop an index
from collections import deque
queue = deque(["a", "b", "c"])
first = queue.pop(0)
What Python prints
TypeError: deque.pop() takes no arguments (1 given)Why, and the fix
deque.pop() always removes from the right and takes no index. To take the first item, call queue.popleft(), which is what replaces list.pop(0).
Python in the browser: Pyodide 314.0.7, MPL-2.0. Licence and source
Exit ticket
5 questions, no hints. Score 80% or more to complete the lesson.
Finish every activity above to unlock the exit ticket.