Senior Data Engineer

Our client is looking for a Senior Data Engineer to design, scale, and maintain high-performance ingestion, Change Data Capture (CDC), and delivery pipelines. In this role, you will architect enterprise-grade data platforms to process large-scale metadata, algorithmic feature extraction, and licensable asset datasets. This team sits at the foundation of the data processing and feature enrichment pipelines. Driving consistency and performance across an ever-growing data lake, we provide guardrails and sound system design patterns to inform the creation and fine-tuning of the industry's cutting-edge foundational large language models. You will leverage Google Cloud Platform (GCP) as the primary compute foundation of a multi-cloud data processing ecosystem, alongside hybrid data warehouses and query engines including Snowflake and ClickHouse to provide low-latency data exposure, incremental delta tracking, and seamless integration endpoints for downstream AI/ML consumption. Primary Tech Stack: GCP, ClickHouse, Snowflake, dbt Core, Apache Airflow, Python, SQL. Responsibilities

1. Ingestion Architecture & GCP Foundation:

• Architect, build, and optimize scalable job creation, batching, and high-throughput ingestion pipelines hosted on GCP.
• Implement serverless and containerized data pipelines using GCP infrastructure (BigQuery, Dataflow, Pub/Sub, Cloud Storage, GKE).

2. Enterprise ETL, CDC & Incremental Processing:

• Design robust Change Data Capture (CDC) and incremental update workflows to enable fast delta synchronization across platforms.
• Build dbt transformation models and Airflow orchestrations following Data Vault / dimensional modeling patterns.
• Implement delta tracking methods (e.g., hash keys, hash diffs) to track row-level asset modifications and metadata events.

3. Multi-Engine Analytics & Query Optimization:

• Maintain and optimize Snowflake integration layers, handling deduplication logic, canonical content filtering, and policy restrictions.
• Deploy and manage ClickHouse as a high-concurrency analytical engine to deliver sub-second metadata retrieval and indexing.
• Construct dedicated integration views and API query layers allowing internal and external clients to consume real-time delta updates.

4. Data Quality, Governance & Scalability:

• Enforce metadata schema standardization, canonical asset mapping, and automated validation layers across diverse asset types (images, video, 3D models, audio).
• Collaborate closely with cross-functional teams to integrate metadata pipelines with feature extraction tools, search indexes, and licensing platforms. Requirements
• Google Cloud Platform (GCP): 4+ years of production experience with core GCP data technologies (BigQuery, Dataflow, Cloud Pub/Sub, GKE, Google Cloud Storage).
• Database & Query Engines: Deep technical proficiency with both Snowflake (warehousing, semi-structured queries, integration layers) and ClickHouse (real-time aggregation, low-latency columnar storage, indexing).
• Enterprise ETL / CDC: Proven experience architecting CDC pipelines and incremental delta loading for multi-terabyte dataset scale.
• Data Transformation & Orchestration: Hands-on mastery of dbt Core/Cloud and Apache Airflow in an enterprise CI/CD setting (e.g., GitHub Actions).
• Programming Skills: Advanced skills in Python and SQL; familiarity with Go, Java, or Scala is a strong plus. Nice-to-Have Qualifications:
• Experience with Data Vault 2.0 architecture, hash-key generation, and raw/staging/integration layer design.
• Background handling AI/ML metadata, computer vision feature vectors, C2PA provenance headers, or semantic search configurations.
• Familiarity with media metadata schemas (images, 3D model stems, video derivatives).
• Experience with other major cloud providers such as AWS, Azure, or Oracle Cloud. Key Success Metrics (First 90 Days)
• GCP Pipeline Optimization: Standardize ingestion batching and pipeline compute infrastructure on GCP.
• ClickHouse & Snowflake Integration: Establish streamlined integration views exposing low-latency metadata delta changes.
• Automated CDC Workflows: Finalize automated dbt delta transformations with hash tracking across primary warehouse layers. We offer*: Flexible working format - remote, office-based or flexible A competitive salary and good compensation package Personalized career growth Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more) Active tech communities with regular knowledge sharing Education reimbursement Memorable anniversary presents Corporate events and team buildings Other location-specific benefits *not applicable for freelancers