Full-Stack Data Engineer: Java

We are looking for a versatile developer to help us bridge the gap between our core enterprise application and our data science environment.

The project involves building a synchronized pipeline that moves data from a transactional database into an analytics engine.

Responsibilities

Java Development:

Maintain and extend a Spring Boot microservice that handles user authentication and transactional logging. You will be responsible for creating REST endpoints that trigger data sync tasks.

Python Integration:

Develop a Python-based worker (using Pandas or Scikit-Learn) that pulls raw data from the database, applies a transformation/scoring model, and pushes the results back to a reporting table.

SQL Optimization:

Write and optimize complex PostgreSQL/MySQL queries.

You will need to design a schema for "Reporting Tables" that allows for efficient data retrieval without locking the main production tables.

API Orchestration:

Ensure the Java backend and Python services communicate reliably via a message broker (RabbitMQ/Kafka) or direct REST calls.

Technical Requirements

Java: 3+ years experience (Spring Boot, Hibernate/JPA, Maven/Gradle).

Python:

Proficiency in data manipulation (Pandas, NumPy) and creating lightweight APIs (FastAPI or Flask).

SQL:

Expert level in writing Joins, CTEs, and Window Functions. Experience with database indexing and query plan analysis.

Experience

Previous work with Docker and microservices architecture is a plus.

Deliverables

A Java-based API module for data ingestion.

A Python script for automated data cleaning and scoring.

A SQL migration script to set up the new analytics schema.

Brief documentation on how to run the integrated environment.

Contract duration of 1 to 3 months.

Mandatory skills:

Python, Java, SQL