Polling-unit records were spread across many pages and collected by hand. The team needed one analysis-ready dataset — every State, LGA, Ward and Polling Unit — for civic auditing and stakeholder reports.
How I built it
01
Navigate
A recursive Selenium scraper walks each state, handling dynamic pagination and state filters.
02
Fix the labels
A semantic mapping layer corrects the source’s labelling errors during extraction.
03
Structure
A processing step turns flat scraped lists into a State → LGA → Ward → Polling Unit hierarchy.
04
Export
Multi-sheet Excel workbooks per state, plus CSV, for auditing.
05
Test
Offline tests for configuration, processing and HTML parsing run in GitHub Actions.
// how the data moves
polling_pipeline — live
How it worksdata flows ↓
01
State portalsrecords spread across many pages
02
Selenium scraperwalks every state and page
03
Label mappingfixes mislabelled columns
04
Excel workbooksone sheet per level
[ DECISION LOG ]
What went wrong, and what I decided.
// 4 decisions
D1
The source data was mislabelled.
what happened
Some “LGA” columns actually contained State names.
decision
Added a semantic mapping layer that corrects these during extraction.
result
A consistent hierarchy instead of hand-fixing every sheet.
D2
Pages loaded dynamically and timed out.
what happened
A simple page-by-page scrape missed records.
decision
Prototyped page-load and timeout handling first, then built recursive, state-by-state navigation.
result
The full chain across all 37 regions.
D3
Testing against a live government site is fragile.
what happened
Tests that hit the real site fail for reasons that aren’t bugs.
decision
Wrote offline tests against saved HTML and config, run in CI.
result
Changes can be checked without touching the live site.
D4
The people using the data work in spreadsheets.
what happened
A database would have been harder for them to audit.
decision
Exported multi-sheet Excel workbooks, one sheet per level.
result
Reports stakeholders could open and check directly.