Senior Software Development Engineer - Big Data

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Software Development Engineer – Big Data based in the United States . We are seeking a highly experienced Senior Software Development Engineer to design and build scalable data platforms that power analytics, machine learning, and AI-driven products. In this role, you will develop reliable, secure, and high-performance data pipelines and services that transform complex datasets into valuable business insights. You will also contribute to the development of a shared AI context layer and agent platform, enabling teams to build intelligent applications and deliver enhanced customer experiences. Working at the intersection of software engineering, distributed systems, and artificial intelligence, you will collaborate with cross-functional teams to turn business and technical requirements into production-ready solutions. You will help establish engineering best practices, improve platform reliability, and mentor other engineers. This fully remote opportunity offers the chance to solve complex data challenges at scale while shaping the future of AI-powered technology in a collaborative and innovative environment.

Design and develop scalable data infrastructure: Architect, implement, and maintain secure, observable, and reliable data pipelines and services that collect, process, and store large, diverse datasets for analytics, machine learning, and product applications. Optimize data processing and storage: Develop efficient data models, storage strategies, and processing frameworks to support high-volume, real-time, and batch workloads while balancing performance, reliability, and infrastructure costs. Lead technical excellence: Drive technical design reviews, code reviews, and engineering best practices, providing guidance and mentorship to engineers on data quality, system resilience, and operational excellence. Collaborate across teams: Partner with product management, analytics, data science, and platform engineering teams to translate business objectives and modeling requirements into robust, production-grade data solutions and APIs. Build AI and agentic platform capabilities: Develop and maintain shared context-layer services, agent tools, authentication mechanisms, and APIs that enable teams across the organization to build and deploy AI-powered applications at scale. Ensure reliability and data quality: Implement automated testing, monitoring, alerting, and incident response processes to improve service availability, data integrity, and overall platform performance. Drive technology innovation: Evaluate and introduce modern big data technologies, development tools, and architectural patterns to improve developer productivity, scalability, and end-to-end data lifecycle management. Establish engineering standards: Document data flows, system interfaces, and operational runbooks while contributing to shared frameworks, reusable components, and best practices for big data engineering. Support continuous improvement: Identify opportunities to strengthen platform architecture, simplify operations, and improve the efficiency and maintainability of distributed data systems. Requirements Education: Bachelor's degree in Computer Science, Computer Engineering, or a related technical field, or an equivalent combination of education and professional experience. Professional experience: At least 8 years of software engineering experience, with a strong focus on building, deploying, and operating data-intensive applications or distributed systems. Big data engineering expertise: Demonstrated experience designing, implementing, and supporting production-grade data pipelines and platforms using modern big data, streaming, and/or cloud-native technologies. Data architecture and systems design: Strong understanding of scalable data processing, data modeling, storage strategies, distributed computing, system reliability, and performance optimization. Production engineering: Experience developing and maintaining reliable data services, including automated testing, monitoring, observability, alerting, and operational troubleshooting. AI and agentic technologies: Hands-on experience building agentic solutions or AI-powered capabilities using major AI tools and shared internal platforms. Experience with Databricks is considered an advantage. API and platform development: Ability to build robust services, integrations, authentication mechanisms, and APIs that support scalable data and AI applications. Cross-functional collaboration: Proven ability to work effectively with product, analytics, data science, and engineering stakeholders to deliver solutions aligned with business objectives. Technical leadership: Strong problem-solving skills and the ability to lead design discussions, conduct code reviews, mentor engineers, and promote consistent engineering standards. Communication and documentation: Excellent written and verbal communication skills, with the ability to explain complex technical concepts, document architectures, and collaborate effectively across distributed teams. Adaptability and continuous learning: Ability to navigate evolving technologies, evaluate new tools, manage competing priorities, and continuously improve engineering practices in a fast-paced environment. Benefits Competitive base salary: Compensation varies by work location: $164,400–$262,600 annually in California, Connecticut, Maryland, Massachusetts, New Jersey, New York, Washington State, and Washington, D.C. $156,200–$249,400 annually in Colorado, Hawaii, Illinois, Maine, Minnesota, Nevada, Ohio, Rhode Island, Vermont, and Virginia. Compensation for other eligible US locations may vary. Equity awards: Eligibility for equity compensation based on factors including experience, performance, and geographic location. Remote work flexibility: Work remotely from an eligible location within the United States, with flexibility supported by a distributed-work model. Career development: Opportunities to work on large-scale data platforms, emerging AI technologies, and innovative engineering initiatives. Technical leadership opportunities: Influence architectural decisions, establish engineering standards, and mentor colleagues across technical teams. Collaborative environment: Work alongside professionals in software engineering, data science, analytics, product development, and cloud infrastructure. Inclusive workplace: Join an environment committed to equal employment opportunities, collaboration, innovation, and professional growth.