🇦🇪 UAE Public Data Engine

A high-performance Python FastAPI engine for standardizing UAE Open Data (Real Estate & Infrastructure) into consistent, developer-friendly APIs.

---

🎯 Vision & Impact

The UAE is rapidly becoming a global leader in smart city innovation and data-driven governance. However, accessing and integrating public data from various government sources remains fragmented and technically challenging for developers.

This project bridges that gap by providing a standardized, high-performance API layer that transforms raw UAE open data into consistent, well-documented JSON schemas.

Strategic Alignment

• UAE National Strategy for AI 2031: Supports the national vision of becoming a world leader in AI by making government data more accessible for AI/ML applications
• Dubai Open Data Strategy: Aligns with Dubai's initiative to make government data available to businesses and researchers
• Market Transparency: Enables real estate professionals, investors, and researchers to access standardized property transaction data
• Developer Ecosystem: Lowers the barrier for PropTech startups and developers building UAE-focused applications

Who Benefits

• Real Estate Professionals: Access standardized transaction data for market analysis
• PropTech Startups: Build applications on consistent, well-documented APIs
• Researchers & Analysts: Access normalized datasets for urban planning and economic research
• AI/ML Engineers: Obtain clean, structured data for training models on UAE real estate patterns

---

⚡ Features

• 9 DLD Datasets: Transactions, Rents, Projects, Valuations, Land, Buildings, Units, Brokers, Developers
• Standardized Schemas: Consistent JSON responses across all data sources
• Aggregation Endpoints: Group by area, property type, date ranges
• Summary Endpoints: Market overview, price trends, developer performance
• Matching Endpoints: Cross-dataset joins for enriched views
• FastAPI: Modern, high-performance async web framework
• Dockerized: Production-ready containerization with multi-stage builds
• Poetry: Modern dependency management and virtual environments
• Linting & Formatting: Pre-configured Ruff for code quality
• CI/CD: Automated testing and linting via GitHub Actions
• OpenAPI Documentation: Auto-generated interactive API docs at /docs

---

🛠 Tech Stack

Component	Technology
Engine	Python 3.11 + FastAPI
Data Models	Pydantic v2
Data Loading	CSV to in-memory
Architecture	Microservice (Dockerized)
Data Sources	Dubai Land Department (DLD)
CI/CD	GitHub Actions

Data Sources

Source	Data Type	Status
Dubai Land Department (DLD)	Real Estate Transactions, Rents, Projects	✅ Integrated
Dubai Pulse	Infrastructure, Transportation, Environment	🚧 Planned
Abu Dhabi DMT	Transport & Mobility	📋 Planned

---

🚀 Getting Started

Prerequisites

• Python 3.11+
• Poetry
• Docker & Docker Compose (optional, for containerized run)

Installation

1. Clone the repository:

   git clone https://github.com/Dynamic-Web-Lab/uae-public-data-engine.git
   cd uae-public-data-engine

2. Install dependencies:

   make install
   # OR
   poetry install

Development

Start the development server with hot-reload:

make run
# App will be running at http://localhost:8000
# Interactive docs at http://localhost:8000/docs
# ReDoc at http://localhost:8000/redoc

Run linter and tests:

make check

Docker

# Build and run with Docker Compose
docker-compose up --build

# Or build manually
docker build -t uae-public-data-engine .
docker run -p 8000:80 uae-public-data-engine

---

📊 API Endpoints

Data Endpoints

Method	Endpoint	Description
GET	/api/v1/transactions/	List transactions with filters
GET	/api/v1/transactions/{id}	Get transaction by ID
GET	/api/v1/rents/	List rental contracts
GET	/api/v1/projects/	List development projects
GET	/api/v1/valuations/	List property valuations
GET	/api/v1/land/	List land parcels
GET	/api/v1/buildings/	List buildings
GET	/api/v1/units/	List individual units
GET	/api/v1/brokers/	List licensed brokers
GET	/api/v1/developers/	List registered developers

Aggregation Endpoints

Method	Endpoint	Description
GET	/api/v1/aggregation/transactions/by-area	Transaction metrics by area
GET	/api/v1/aggregation/transactions/by-date	Transaction trends over time
GET	/api/v1/aggregation/rents/by-area	Rental metrics by area
GET	/api/v1/aggregation/projects/by-status	Projects by completion status
GET	/api/v1/aggregation/projects/by-developer	Projects per developer

Summary Endpoints

Method	Endpoint	Description
GET	/api/v1/summary/market-overview	High-level market snapshot
GET	/api/v1/summary/price-trends	Price per sqm trends

Matching Endpoints

Method	Endpoint	Description
GET	/api/v1/matching/transaction/{id}/details	Full transaction with context
GET	/api/v1/matching/area/{code}/profile	Complete area profile
GET	/api/v1/matching/broker/{id}/transactions	Broker transaction history
GET	/api/v1/matching/search	Unified search across datasets

Sample Response

{
  "status": "success",
  "source": "dubai_land_department",
  "metadata": {
    "total_items": 20,
    "filters_applied": {
      "area": "Dubai Marina"
    }
  },
  "data": [
    {
      "transaction_number": "TN-2024-001234",
      "transaction_date": "2024-01-15",
      "transaction_type": "Sale",
      "area": "Dubai Marina",
      "property_sub_type": "Apartment",
      "amount": 1850000.00,
      "property_size_sqm": 116.00,
      "rooms": 2
    }
  ],
  "pagination": {
    "page": 1,
    "page_size": 50,
    "total_pages": 1,
    "total_items": 20
  }
}

---

📂 Project Structure

.
├── app/
│   ├── api/
│   │   └── v1/
│   │       ├── router.py              # Main v1 router
│   │       └── endpoints/             # API endpoints
│   │           ├── transactions.py
│   │           ├── rents.py
│   │           ├── projects.py
│   │           ├── aggregation.py
│   │           ├── summary.py
│   │           └── matching.py
│   ├── schemas/
│   │   ├── common.py                  # Shared models
│   │   └── dld/                       # DLD Pydantic models
│   ├── services/
│   │   ├── data_loader.py             # CSV data loading
│   │   ├── aggregation.py             # Aggregation logic
│   │   └── matching.py                # Data joining logic
│   ├── core/
│   │   └── config.py
│   └── main.py
├── data/
│   ├── dld/                           # DLD CSV data files
│   │   ├── transactions/
│   │   ├── rents/
│   │   ├── projects/
│   │   └── ...
│   └── dubai-pulse/                   # Future data sources
├── tests/
├── .github/
├── pyproject.toml
├── Dockerfile
├── docker-compose.yml
└── Makefile

---

🗺 Roadmap

Phase 1: Foundation (Current)

• Project scaffolding with FastAPI
• Docker configuration
• CI/CD pipeline
• Basic health check endpoint
• Pydantic models for 9 DLD datasets
• Sample CSV data files
• Basic CRUD endpoints
• Aggregation endpoints
• Summary endpoints
• Matching/join endpoints

Phase 2: Data Integration

• Dubai Pulse API integration
• Pydantic models for infrastructure data
• Schema validation and transformation

Phase 3: Production Features

• Rate limiting
• Authentication
• Caching layer
• Monitoring and metrics

Phase 4: Advanced Analytics

• Time series analysis
• Predictive analytics
• Market forecasting

---

🤝 Contributing

We welcome contributions! This project especially needs help with:

• New Data Sources: Mapping Abu Dhabi DMT schemas, Sharjah data
• Data Models: Expanding Pydantic models for different property types
• Tests: Improving test coverage
• Documentation: API usage examples, tutorials

Please read the CONTRIBUTING.md and CODE_OF_CONDUCT.md for details.

Good First Issues

• Add schema for Abu Dhabi property transactions
• Create data model for commercial real estate
• Add unit tests for schema validation
• Improve error handling for API rate limits

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

Built with ❤️ for the UAE developer community

Report Bug · Request Feature