AWS Data Engineer

Role : AWS Data Engineer – Kinesis (Band 3)

• Design, develop, and maintain real-time data streaming pipelines using AWS Kinesis for processing and analyzing high-velocity data streams.
• Collaborate with stakeholders to gather requirements and design efficient data streaming architectures for real-time analytics, monitoring, and alerting.
• Build and optimize Kinesis Data Streams, Kinesis Data Firehose, and Kinesis Data Analytics to process real-time data from various sources.
• Develop and implement ETL processes to transform, enrich, and aggregate streaming data for downstream consumption in data lakes or data warehouses such as AWS S3, Redshift, and Dynamo DB.
• Integrate Kinesis with other AWS services like Lambda, S3, and EMR for real-time data ingestion, processing, and storage.
• Optimize streaming applications for performance and scalability, ensuring efficient resource usage and minimal latency.
• Develop and maintain robust error handling, data validation, and retry mechanisms within Kinesis pipelines to ensure data consistency and reliability.
• Implement security best practices for data streaming, including encryption in transit and at rest, access control using IAM roles and policies, and data retention policies.
• Monitor and troubleshoot AWS Kinesis streams using Cloud Watch metrics, alarms, and logs, ensuring high availability and minimal downtime.
• Collaborate with Dev Ops teams to automate the deployment and scaling of streaming pipelines using AWS Cloud Formation, Lambda, and CI/CD tools.
• Work closely with data analysts and data scientists to ensure real-time data is processed and delivered for immediate insights and reporting.
• Implement data partitioning, sharding, and key management strategies to handle high-throughput data streams and ensure scalability.
• Optimize the cost of real-time data processing by tuning the configurations of Kinesis and associated AWS services like Lambda and Firehose.
• Stay current with AWS updates and best practices for streaming data, recommending improvements to the architecture as new features and services become available.
• Provide guidance and mentorship to junior engineers in real-time data processing, ensuring best practices in streaming technologies and AWS cloud services.
• Create and maintain detailed technical documentation for data pipelines, including data flow diagrams, stream processing logic, and operational guidelines.

Experience:

• Kinesis Data Streams: 8 years (preferred)
• AWS: 8 years (preferred)

Work authorisation:

• United Kingdom (preferred)