Senior Data Engineer - Onsite, Missouri

Must Have Skills:

" AWS Redshift

" AWS EC2/S3

" Python

Detailed Job Description:

Data Engineer Lead to join our team and lead the development of a robust data integration pipelines using AWS services. The ideal candidate will be responsible for designing, building, and optimizing scalable data workflows that meet our client's data processing, storage, and analytics needs.

Location: St. Louis, MO - USA

WFH / Remote may not be possible for this role.

Key Responsibilities:

Design & Develop Data Pipelines:

" Architect and implement end-to-end data pipelines using AWS Cloud Services including Redshift - S3, EMR, Glue, Step Functions.

" Manage data ingestion framework processes from AWS EC2/S3, ensuring secure and efficient data transfer.

" Implement initial data routing, validation, and transformations

Data Processing & Transformation:

" Integrate using Pl/SQL, Python, Spark to perform complex data transformations and analytics.

" Optimize PL/SQL and Spark jobs for processing large-scale datasets with a focus on performance and resource utilization.

" Handle both historical and incremental data loads, ensuring data consistency and integrity.

Data Storage & Management:

" Define and implement data storage strategies across S3, RDS, and Redshift, adhering to business requirements.

" Manage data catalog creation and schema management within AWS.

Automation & Orchestration:

" Develop and manage workflows using One Automation or Apache Airflow to automate data processing tasks.

" Implement monitoring, error handling, and retries within the orchestration framework.

Security & Compliance:

" Ensure data security with encryption (AES-256, TLS) and IAM role-based access controls.

Performance Monitoring & Optimization:

" Optimize Redshift Code Data Structures and Stored Procedures for efficient executions and handling. Implement advanced Redshift features like Data Sharing, WLM Query Execution Optimization and features like Auto MV, Concurrency Scaling, etc.

" Continuously optimize Spark job configurations and data flows for maximum throughput and minimal latency.

Required Qualifications:

" 7+ years of experience in Data Engineering using Pl/SQL and Python with a focus on development in Redshift, EC2, Glue, Step Functions, and Spark.

" Proven track record of building scalable data pipelines in cloud environments.

" Strong expertise in Spark, Hadoop, and distributed computing using AWS Services.

" In-depth knowledge of AWS services (S3, Glue, Redshift, RDS, Lambda, Step Functions).

" Experience with data formats (JSON, CSV, Parquet, Avro) and transformation techniques.

" Strong problem-solving skills and ability to troubleshoot complex data processing issues.

" Excellent communication skills with the ability to document and explain technical details clearly.

Minimum Years of Experience:

7+ years