Recruitment news
Senior Data Engineer
Remuneration: | cost-to-company |
Location: | Cape Town |
Job level: | Senior |
Type: | Permanent |
Company: | THE SKILLS MINE (PTY) LTD |
Requirements
- Five-plus years of experience as a data engineer or relevant experience
- Proven experience in designing and implementing data pipelines
- Experience with end-to-end testing of analytics pipelines
- Expertise in managing and optimising Elasticsearch clusters, including performance tuning and scalability
- Strong proficiency with data extraction, transformation, and loading (ETL) processes
- Familiarity with data modelling and schema design for efficient data storage and retrieval
- Good programming and scripting skills using languages like Python, Scala, or Java
- Knowledge of DevOps and automation practices related to data engineering
- Experience with data engineering in an agile / scrum environment
- Familiarity with ksqlDB / Kafka or other stream processing frameworks
- Familiarity with data lakes and querying them
- Experience with integrating machine learning models into data pipelines
- Kafka / ksqlDB
- Python
- Redis
- Elasticsearch, cluster management and optimisation
- AWS S3
- PostgreSQL
- AWS
- Designing, implementing, and maintaining robust data pipelines, ensuring the efficient and reliable flow of data across systems
- Developing and maintaining Elasticsearch clusters, fine-tuning them for high performance and scalability
- Collaborating with cross-functional teams to extract, transform, and load (ETL) data into Elasticsearch for advanced analytics and search capabilities
- Troubleshooting data pipeline and Elasticsearch issues, ensuring the integrity and availability of data for analytics and reporting
- Participating in the design and development of data models and schemas to support business requirements
- Continuously monitoring and optimising data pipeline and Elasticsearch performance to meet growing data demands
- Collaborating with data scientists and analysts to enable efficient data access and query performance
- Contributing to the evaluation and implementation of new technologies and tools that enhance data engineering capabilities
Posted on 18 Sep 12:43, Closing date 18 Oct
Recent jobs by The Skills Mine | City | Date posted | |
Senior Data Engineer | Cape Town | 18 Sep 12:43 | |
Coordinator - Marketing and Events | Johannesburg | 16 Sep 09:25 | |
Senior Back-end Engineer | Cape Town | 13 Sep 15:50 | |
Regional Sales Manager | Durban | 10 Sep 16:21 | |
Customer Success Manager | Durban | 10 Sep 13:22 | |
Web Developer and SQL Admin | Johannesburg | 10 Sep 13:20 | |
Full Stack Developer - Ruby on Rails | Johannesburg | 06 Sep 15:29 | |
Regional Product Manager - (6 Months FTC) | Johannesburg | 04 Sep 13:56 | |
HR and Talent Acquisition Business Partner | Johannesburg | 04 Sep 09:22 | |
Portfolio Manager - Credit Risk | Johannesburg | 03 Sep 16:28 | |
More jobs... |
Senior Data Engineer
Requirements Five-plus years of experience as a data engineer or relevant experienceProven experience in designing and implementing data pipelinesExperience with...
Senior Python JavaScript AWS Software Engineer
Are you a Senior Python JavaScript Software Engineer looking for a new opportunity in an environment that empowers salesforce to reach every customer, through an...
Senior Back-end Engineer
Requirements Over five years of experience in back-end web development.In-depth expertise in PostgreSQL and RDBMS in general.Experience working with message brokers and...
Senior DevOps Engineer
New work in at an on-demand delivery and digital tech shop in SA, owned by one of our major retail groups. The current need is for a DevOps engineer with key...
Java System Integration Engineer
Become a member of a team enthusiastic about the challenges posed by integrating complex systems comprising diverse components and subsystems currently in search for...
Senior DevOps Engineer
New work in at an on-demand delivery and Digi tech shop in SA, owed by one of our major retail groups.
Data Engineer - SQL DBA
A Fintech department, delivering financial information to the company that is timely, accurate, reliable, and relevant data originating from a multitude of external...
Software Developer /Data Engineer
A dynamic client who leverages the power of AI to analyse and interpret data with unprecedented accuracy, speed and gives actionable improvement suggestions are looking...
Mechanical Engineer
Our seeking a highly skilled and experienced Mechanical Engineer to join our team on a world-changing mission.
Data Engineer
A fintech department, delivering financial information to the company that is timely, accurate, reliable, and relevant data originating from a multitude of external...
Senior Data Engineer
A fintech department delivering timely, accurate, reliable, and relevant financial information from various external sources is seeking a senior data engineer.
Office Administrator & Co-Ordinator
Office Administrator amp Co-Ordinator - George Western Cape nbsp
Point of Sales Cashier & Administrator
Point of Sales Cashier amp Administrator - George Western Cape nbsp
Mid-Level NodeJS JavaScript Developer
Looking for a mid-weight full-stack developer that will be helping with building a platform that will work globally to be used for deliveries to the world’s...
Mid-Level NodeJS JavaScript Developer
Looking for a mid-weight full-stack developer that will be helping with building a platform that will work globally to be used for deliveries to the world’s...
Full-Stack PHP JavaScript Developer
Looking for a mid-weight full-stack developer that will be helping with building a platform that will work globally to be used for deliveries to the world’s...
JavaScript Node.js Cloud Developer
In search for a Full Stack JavaScript Node.js Developer that will focus on both front-end and back-end development, currently offering this remote work opportunity, you...
Senior Data Engineer
Requirements Five-plus years of experience as a data engineer or relevant experienceProven experience in designing and implementing data pipelinesExperience with...
Senior Back-end Engineer
Requirements Over five years of experience in back-end web development.In-depth expertise in PostgreSQL and RDBMS in general.Experience working with message brokers and...
Junior to Mid-level Software Developers
Our client is a dynamic force transforming the insurance industry through innovative, customer-centric solutions.
Full Stack Java Developer
Are you looking to expand your skills in the Global Market? We are looking for a Full Stack Java developer, to become a part of an international renowed asset management...
Java System Integration Engineer
Become a member of a team enthusiastic about the challenges posed by integrating complex systems comprising diverse components and subsystems currently in search for...
Java Angular Fullstack Developer
Opportunity to flex your Dev muscle on high-performance tools and platforms! You will be based in a progressive performance driven environment that offers collaboration...