Senior Data Engineer
Printed.com Cramlington
You will play a crucial role in designing, building, and maintaining our data platform, with a strong emphasis on streaming data, cloud infrastructure, and machine learning operations.
Key Responsibilities:
- Architect and Implement Data Pipelines:
- Design, develop, and maintain scalable and efficient data pipelines.
- Optimize ETL processes to ensure seamless data ingestion, processing, and integration across various systems.
- Streaming Data Platform Development:
- Lead the development and maintenance of a real-time data streaming platform using tools like Apache Kafka, Databricks, Kinesis.
- Ensure the integration of streaming data with batch processing systems for comprehensive data management.
- Cloud Infrastructure Management:
- Utilize AWS data engineering services (including S3, Redshift, Glue, Kinesis, Lambda, etc.) to build and manage our data infrastructure.
- Continuously optimize the platform for performance, scalability, and cost-effectiveness.
- Communications:
- Collaborate with cross-functional teams, including data scientists and BI developers, to understand data needs and deliver solutions.
- Leverage the project management team to coordinate project, requirements, timelines and deliverables, allowing you to concentrate on technical excellence.
- ML Ops and Advanced Data Engineering:
- Establish ML Ops practices within the data engineering framework, focusing on automation, monitoring, and optimization of machine learning pipelines.
- Data Quality and Governance:
- Implement and maintain data quality frameworks, ensuring the accuracy, consistency, and reliability of data across the platform.
- Drive data governance initiatives, including data cataloguing, lineage tracking, and adherence to security and compliance standards.
Requirements
Experience:
- 3+ years of experience in data engineering, with a proven track record in building and maintaining data platforms, preferably on AWS.
- Strong proficiency in Python, experience in SQL and PostgreSQL. PySpark, Scala or Java is a plus.
- Familiarity with Databricks and the Delta Lakehouse concept.
- Experience mentoring or leading junior engineers is highly desirable.
Skills:
- Deep understanding of cloud-based data architectures and best practices.
- Proficiency in designing, implementing, and optimizing ETL/ELT workflows.
- Strong database and data lake management skills.
- Familiarity with ML Ops practices and tools, with a desire to expand skills in this area.
- Excellent problem-solving abilities and a collaborative mindset.
Nice to Have:
- Familiarity with containerization and orchestration tools (e.g., Docker, Kubernetes).
- Knowledge of machine learning pipelines and their integration with data platforms.
- 22 days holiday Plus 8 days bank holidays
- Staff discounts & Friends and Family discounts
- Cycle to work scheme and Tech Scheme
- Breakfast and drinks provided
- Charity day per annum supported
- Summer and Christmas Parties
- Street food days
- Perkbox membership
- Enhanced Maternity & Paternity leave
- Employee referral scheme
Newcastle upon Tyne, 5 mi from Cramlington
Senior Data Engineer (Azure Databricks Python Spark SQL) *Newcastle Onsite* to £130k
Are you a technologist Data Engineer with a strong record of academic achievement?
You could be progressing your career in an impactful role at a start-up...
DWP DigitalNewcastle upon Tyne, 5 mi from Cramlington
and is radically transforming how we detect and prevent fraud and error in the welfare system.
You'll play a vital role in delivering innovative and insightful data products, and work across the data engineering lifecycle.
Integrating a variety of source data...
DWPNewcastle upon Tyne, 5 mi from Cramlington
a vital role in delivering innovative and insightful data products, and work across the data engineering lifecycle.
p>
Integrating a variety of source data, you'll use your judgement to figure out the best approach, so that appropriate data quality...