Data & AI Engineer

This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Data & AI Engineer based in Brazil. This role offers the opportunity to design and deliver modern data platforms and production-ready AI solutions for enterprise clients. You will build scalable Lakehouse architectures, optimize data pipelines, and transform complex datasets into reliable foundations for analytics and machine learning. Working with technologies such as Databricks, Apache Spark, Delta Lake, and MLflow, you will help bring AI initiatives from concept to real-world deployment. You will collaborate closely with data architects, data scientists, engineering teams, and client leaders in a fast-paced, delivery-focused environment. The position combines hands-on engineering, cloud optimization, MLOps, and data governance to create secure, efficient, and scalable solutions. It is ideal for an engineer who values technical excellence, ownership, continuous improvement, and measurable business impact.

Lakehouse Architecture and Data Pipelines: Design, develop, and optimize scalable ETL/ELT pipelines using Databricks, Apache Spark, and Delta Lake across cloud environments, including AWS, Azure, and Google Cloud Platform. Data Ingestion and Transformation: Build robust workflows for processing structured and unstructured data, automating ingestion, storage, and feature engineering to support analytics and real-time AI workloads. Performance and Cost Optimization: Tune Spark jobs and data processing workflows to improve latency, throughput, scalability, and cloud cost efficiency across client environments. MLOps and AI Deployment: Collaborate with machine learning engineers and data scientists to operationalize AI models within Databricks, using MLflow for experiment tracking, model versioning, and deployment. CI/CD and Infrastructure Automation: Implement automated deployment pipelines and Infrastructure-as-Code practices using tools such as Terraform and Databricks CLI to ensure reliable, repeatable production releases. Data Governance and Security: Establish and maintain data access controls, security policies, and governance standards using Unity Catalog and cloud-native capabilities, supporting requirements such as GDPR, HIPAA, and SOC 2. Client Collaboration: Work directly with client technology leaders, product managers, architects, and engineering teams to translate complex data and AI requirements into practical technical solutions and measurable business outcomes. Documentation and Knowledge Transfer: Produce clear technical documentation and coordinate smooth solution handoffs to client engineering teams, ensuring maintainability and long-term operational success. Requirements: Professional Experience: At least three years of experience in cloud-native data engineering, including a minimum of two years of hands-on experience with Databricks, Apache Spark, Delta Lake, and MLflow. Programming and Data Engineering: Strong proficiency in Python, SQL, and Spark-based frameworks, with experience designing and maintaining scalable data processing solutions. Cloud and Lakehouse Technologies: Practical experience building data architectures across cloud environments and implementing modern Lakehouse patterns for enterprise data workloads. MLOps and Model Lifecycle Management: Familiarity with MLflow and the processes required to track, version, deploy, and operationalize machine learning models. DevOps and Automation: Experience implementing CI/CD pipelines and Infrastructure-as-Code solutions using Terraform, Databricks CLI, or comparable tools. Databricks Certification: An active Databricks Certified Data Engineer Associate or Professional certification, or Databricks Machine Learning Professional certification, is strongly preferred. Consulting and Agile Delivery: Proven experience working in fast-paced Agile or DevOps consulting environments, delivering cloud-agnostic solutions and adapting to evolving client requirements. Communication and Collaboration: Strong communication skills, with the ability to explain technical concepts clearly, work effectively with multidisciplinary teams, and maintain productive client relationships. Ownership and Continuous Improvement: A proactive, accountable approach to problem-solving, with a commitment to engineering quality, continuous learning, and delivering measurable outcomes. Benefits: Remote-First Work: The flexibility to work remotely while collaborating with a globally connected team. Collaborative Environment: The opportunity to work alongside skilled professionals in a supportive, innovative, and fast-paced setting. Technical Growth: Exposure to modern Lakehouse architectures, cloud platforms, data engineering, MLOps, and production AI solutions. Meaningful Client Impact: The opportunity to build scalable, production-ready solutions that address real business challenges. Inclusive Workplace: A culture that values diversity, equal opportunity, collaboration, curiosity, and individual initiative.