Expert Data Engineer – Contract-Driven Airflow Ingestion Pipeline

Job Title

Expert Data Engineer Needed – Contract-Driven Airflow Ingestion Pipeline (Python, CI/CD)

Project Scope

Small to Medium (initial dataset implementation, potential follow-up work)

Level of Expertise

Expert

Location

Global (must be comfortable working with a European-based team)

Contract Type

Fixed-price preferred (milestone-based per dataset)

Description

We are building a structured data platform and are looking for an experienced Data Engineer to implement a production-grade ingestion pipeline for a predefined dataset.

This is a standards-driven execution task. Architecture, repository structure, schema templates, and data contracts are already defined. Your role is implementation and integration — not architecture design.

For a given dataset, you will:

• Implement ingestion from the approved source (API or structured dataset)
• Land raw data into object storage (bronze layer)
• Transform data into a predefined canonical schema (contract-driven)
• Implement automated data quality checks
• Integrate into an existing CI/CD workflow
• Deliver a production-ready Apache Airflow DAG
• Ensure idempotent execution and safe reprocessing
• Emit structured logs and telemetry metrics

You will work independently but in coordination with our internal data and platform team. Code review is mandatory before merge.

Deliverables

• Working Airflow DAG (production-ready)
• Python ingestion module
• Contract-compliant transformation logic
• Data quality validation layer
• CI/CD integration
• Technical documentation (README + assumptions)

Required Experience

• Strong Python (production-grade code, not notebook scripts)
• Apache Airflow (real production DAGs required)
• Experience building ingestion pipelines from APIs and/or structured datasets
• Schema-driven or contract-driven data processing
• CI/CD integration

(GitHub Actions, GitLab CI, etc.)

• Cloud storage experience (S3 or equivalent)
• Understanding of idempotent pipeline design

Important

• Architecture and schema design are predefined and must not be modified.
• Clean, maintainable, and standards-compliant code is required.
• Please share examples of Airflow DAGs you have implemented.

This initial project covers one dataset. Strong performance may lead to additional domain implementations.

Contract duration of 1 to 3 months. with 30 hours per week.

Mandatory skills:

Python, Data Modeling, CI/CD, data contract, DQ, Parquet, schema validation, Apache Airflow, Data Engineering, Data Integration