Senior Data Engineer - Databricks

This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Senior Data Engineer - Databricks based in Brazil. We are looking for an experienced Senior Data Engineer to design, develop, and maintain scalable data pipelines using Databricks and modern cloud technologies. In this role, you will integrate and transform data from diverse sources, ensuring it is reliable, secure, well-governed, and ready to support business decisions. You will work extensively with Python, SQL, PySpark, and Delta Lake while applying the Medallion architecture to build robust data solutions. Collaborating with business stakeholders, analysts, data scientists, and platform engineers, you will translate complex requirements into high-quality data products. You will also promote modern DataOps practices and help improve engineering standards across the team. This opportunity combines hands-on technical work, cross-functional collaboration, and the chance to make a meaningful impact through trusted, data-driven solutions.

Design, develop, maintain, and optimize scalable data pipelines using Databricks, PySpark, Delta Lake, and SQL. Integrate data from diverse sources, including APIs, relational databases, files, and other enterprise systems. Build and maintain robust data ingestion and transformation processes aligned with the Medallion architecture and modern data engineering practices. Collaborate with business and analytics teams to understand data requirements and deliver solutions that support strategic objectives. Ensure the quality, reliability, security, and governance of ingested and processed data throughout the data lifecycle. Apply modern DataOps practices, including code versioning, automated data testing, CI/CD pipelines, and infrastructure as code. Troubleshoot and resolve performance, scalability, and reliability issues across data pipelines and distributed processing environments. Document data engineering processes, technical decisions, standards, and best practices to support maintainability and knowledge sharing. Provide technical guidance and mentorship to colleagues while collaborating with data scientists, analysts, and platform engineers. Contribute to continuous improvement in data architecture, pipeline efficiency, and engineering quality. Requirements: Proven experience designing, developing, and managing large-scale data pipelines using Databricks, PySpark, Delta Lake, and SQL. Strong programming skills in Python and SQL for data processing, transformation, and pipeline development. In-depth understanding of ETL/ELT frameworks, data warehousing concepts, and distributed data processing. Hands-on experience with modern DataOps practices, including Git version control, CI/CD pipelines, automated testing, and infrastructure as code. Familiarity with cloud platforms such as AWS, Azure, or Google Cloud Platform and their associated data services. Strong problem-solving and analytical skills, with the ability to troubleshoot and optimize data solutions for performance, scalability, and reliability. Proficiency with Git and collaborative software development workflows. Advanced English proficiency for effective communication, technical collaboration, and documentation. Nice to have: Experience with data contracts, schema evolution, and maintaining compatibility between services and data consumers. Familiarity with data quality frameworks such as Great Expectations, Soda, dbt tests, or custom-built validation solutions. Experience integrating data solutions with Power BI. Benefits: Health and dental insurance. Meal and food allowances. Childcare assistance. Extended paternity leave. Access to gyms and health and wellness professionals through Wellhub (Gympass) and TotalPass partnerships. Profit Sharing and Results Participation (PLR). Life insurance. Access to a continuous learning platform and professional development resources. Discount club with partner offers. Free online resources dedicated to physical, mental, and overall well-being. Pregnancy and responsible parenting courses. Partnerships with online learning platforms. Access to a language-learning platform. Additional benefits and well-being initiatives.