ES ← Back to Portfolio

Contacto Legislativo

A civic technology platform that transformed publicly available legislative information into citizen-friendly insights and performance indicators for the Mexican Chamber of Deputies.

Role Lead Developer
Status Archived
Type Civic Technology
Web Scraping Data Engineering ETL Backend Development Civic Technology Data Visualization

The Problem

Public legislative information in Mexico was technically available but practically inaccessible. Citizens who wanted to understand their elected representatives faced a fragmented landscape of government websites, inconsistent document formats, and no consolidated view of performance.

Basic questions had no easy answer:

  • Who represents my district?
  • How active is my representative?
  • What initiatives have they proposed?
  • Are they complying with transparency requirements?

Answering any of these required navigating multiple government websites and manually reviewing documents — a barrier that effectively kept citizens disconnected from their own legislative process. The goal was to consolidate and simplify access to this information.

Data Pipeline Architecture

The platform was built around an automated data pipeline that moved raw government data through several processing stages before reaching citizens.

Official Chamber of Deputies Sources
Web Scrapers
Data Extraction
Data Normalization
Unified Data Model
Metrics Generation
Citizen Dashboards & Reports

Data Engineering

The most technically significant component of the platform was the automated data pipeline. Custom scrapers collected legislative information directly from official public sources — without any API or structured data feed being available.

The extracted information then moved through a multi-stage ETL process before being stored in a unified data model used throughout the application:

Extraction

Custom scrapers navigated government websites to collect raw legislative data, handling pagination, session state, and inconsistent markup across different periods.

Cleaning & Validation

Raw data was validated for completeness and consistency. Missing fields, duplicate records, and format anomalies were detected and resolved before transformation.

Normalization

Data from different legislative periods and source formats was mapped to canonical domain models, enabling unified queries across the full historical dataset.

Enrichment & Metrics

Derived indicators were computed from the normalized data — activity scores, compliance rates, and district mappings that didn't exist in any source system.

Technical Challenges

Government Websites Not Designed for Automation

Public websites were built for human consumption. Pages relied on session state, JavaScript rendering, inconsistent HTML structure across legislative periods, and had no stable identifiers for records — making machine extraction unreliable.

Solution

Developed custom extraction logic per source, capable of navigating inconsistent structures, maintaining session context, and recovering gracefully from partial failures without corrupting the dataset.

Inconsistent Data Formats Across Legislative Periods

Information appeared in different formats depending on the legislative period and which section of the site it originated from. The same deputy could appear under different name spellings, district codes, or party affiliations across sources.

Solution

Created a normalization layer with explicit mapping rules and a canonical domain model that reconciled identity across sources, enabling reliable cross-period queries.

Derived Metrics That Didn't Exist in Source Systems

The citizen-facing indicators — activity scores, transparency compliance rates, initiative counts by category — were not available anywhere. They had to be computed from the raw collected data through domain-specific business logic.

Solution

Designed a metrics generation layer that computed all derived indicators from the unified data model, separating business logic from storage and making metric definitions easy to evolve.

Metrics & Analytics

All citizen-facing indicators were generated by the platform from processed legislative activity data — none of them existed as ready-made data points in source systems.

📊
Percentage of deputies without registered initiatives
Transparency declaration compliance rate
📋
Legislative activity indicators per deputy
📁
Initiative counts by category and period
🏛️
Commission participation and attendance
📍
District representation lookup

Impact

The platform transformed fragmented public information into accessible civic insights. For the first time, citizens could answer basic questions about their representatives without navigating government bureaucracy.

Citizens could quickly identify their representative, review legislative activity, check public accountability indicators, and verify transparency compliance — all in one place.

The end result was a meaningful reduction in the effort required to understand representative performance, directly improving access to public information for everyday citizens.

Although the project is now archived, it demonstrated that civic technology can bridge the gap between publicly available government data and the citizens it represents — a model applicable to any jurisdiction with open legislative records.

Contacto Legislativo — district deputy lookup screen
Contacto Legislativo — party funding indicator and civic initiative screen
Contacto Legislativo — original promotional infographic showing legislative activity statistics

Historical promotional material used during the active phase of Contacto Legislativo.

Technologies & Architecture

Data Collection
Custom Scrapers HTTP Automation HTML Parsing
Processing
ETL Pipelines Data Normalization Domain Modeling Metrics Computation
Backend
Java Spring REST APIs
Persistence
SQL Database Relational Modeling
Presentation
Web Application Dashboards Data Visualization Reports

Lessons Learned

1
Resilient scrapers require defensive design from day one. Government websites change without notice. Building extraction logic that degrades gracefully and fails loudly — rather than silently producing corrupt data — is essential for any long-running scraping operation.
2
Imperfect public datasets require explicit data contracts. Working with real-world government data means accepting inconsistency as the baseline. Defining explicit normalization rules and canonical models early prevents those inconsistencies from leaking into application logic.
3
ETL pipelines need to be independently testable. Each stage of the pipeline — extraction, normalization, enrichment — should be verifiable in isolation. Coupling stages makes it nearly impossible to diagnose where data quality issues originate.
4
Translating complex data into simple metrics is the hardest product problem. The engineering challenge is not the pipeline — it is deciding which indicators matter to citizens and how to present them without distorting the underlying data. Domain expertise and user perspective are as important as technical skill.
5
Technology can meaningfully improve civic engagement. Making public information genuinely accessible — not just technically available — changes how citizens interact with their government. The same pattern applies to any domain where raw open data exists but no usable interface does.