Software Engineer- Data Engineer

You are a highly skilled Data Ingestion Engineer responsible for designing, building, and maintaining scalable, reliable, and secure data ingestion pipelines on AWS. Your deep experience in ingesting data from diverse sources into a centralized data platform ensures data quality, observability, and performance.

Key Responsibilities:

• Design and implement batch and real-time data ingestion pipelines using AWS-native services
• Ingest data from databases, APIs, files, event streams, and third-party SaaS systems
• Build fault-tolerant, scalable, and reusable ingestion frameworks
• Handle CDC (Change Data Capture) from transactional systems

AWS & Cloud Engineering:

• Leverage AWS services such as S3, Glue, Lambda, Step Functions, Kinesis, SNS/SQS, EMR / Spark
• Optimize data ingestion for cost, performance, and scalability
• Implement secure ingestion using IAM, KMS, VPC endpoints, Secrets Manager

Data Quality, Monitoring & Reliability:

• Implement data validation, schema enforcement, and error handling
• Build monitoring and alerting using CloudWatch, logs, and metrics
• Ensure idempotency, replayability, and recoverability of ingestion pipelines
• Manage late-arriving data and schema evolution

DevOps & Automation:

• Build CI/CD pipelines for ingestion jobs using Git, Terraform / CloudFormation
• Containerize ingestion services using Docker (ECS/EKS is a plus)
• Follow Infrastructure as Code (IaC) best practices

Collaboration & Documentation:

• Work closely with analytics, platform, and downstream data consumers
• Create clear documentation for ingestion patterns, SLAs, and runbooks
• Participate in design reviews and mentor junior engineers

Required Skills & Qualifications:

Core Skills:

• 5+ years in Data Engineering with a strong focus on data ingestion and cloud-native pipelines
• Strong expertise in AWS data services
• Proficiency in Python (mandatory); SQL required
• Experience with Spark / PySpark for large-scale ingestion
• Solid understanding of data formats (JSON, Avro, Parquet, CSV)

Ingestion Patterns:

• Batch ingestion (file-based, database extracts)
• Streaming ingestion (event-driven architectures)
• API-based ingestion (REST, OAuth, rate limits)
• CDC tools or patterns (DMS, Debezium, custom CDC)

Data Architecture:

• Experience working with data lakes (S3) and lakehouse architectures
• Familiarity with Medallion architecture (Bronze / Silver / Gold)
• Understanding of partitioning, compaction, and metadata management

Nice-to-Have:

• Experience with Snowflake, Redshift, or Databricks
• Knowledge of dbt and downstream transformations
• Banking / FinTech domain experience
• Exposure to data governance, lineage, and catalog tools
• Experience with DBT Cloud, Airflow, or AWS Managed Workflows (MWAA)

Soft Skills:

• Strong problem-solving and debugging skills
• Ability to work independently in complex environments
• Excellent communication with both technical and non-technical teams
• Ownership mindset for production data pipelines

Example KPIs for Success:

• Reduction in data ingestion failures and latency
• Improved data freshness and SLA adherence
• Scalable ingestion frameworks adopted across teams
• Well-documented, observable, and maintainable pipelines

About Convera:

Convera is the largest non-bank B2B cross-border payments company globally, providing smarter money movements to customers. With a focus on diversity and inclusion, Convera offers a rewarding and innovative environment for growth-minded individuals. Competitive perks and benefits include a great salary, annual bonus opportunities, career growth, and a flexible work approach. Join Convera if you are ready to unleash your potential and transform Business to Business payments. You are a highly skilled Data Ingestion Engineer responsible for designing, building, and maintaining scalable, reliable, and secure data ingestion pipelines on AWS. Your deep experience in ingesting data from diverse sources into a centralized data platform ensures data quality, observability, and performance.

Key Responsibilities:

• Design and implement batch and real-time data ingestion pipelines using AWS-native services
• Ingest data from databases, APIs, files, event streams, and third-party SaaS systems
• Build fault-tolerant, scalable, and reusable ingestion frameworks
• Handle CDC (Change Data Capture) from transactional systems

AWS & Cloud Engineering:

• Leverage AWS services such as S3, Glue, Lambda, Step Functions, Kinesis, SNS/SQS, EMR / Spark
• Optimize data ingestion for cost, performance, and scalability
• Implement secure ingestion using IAM, KMS, VPC endpoints, Secrets Manager

Data Quality, Monitoring & Reliability:

• Implement data validation, schema enforcement, and error handling
• Build monitoring and alerting using CloudWatch, logs, and metrics
• Ensure idempotency, replayability, and recoverability of ingestion pipelines
• Manage late-arriving data and schema evolution

DevOps & Automation:

• Build CI/CD pipelines for ing