Senior Data Platform Engineer
About the role
Senior data platform role focused on architecting and building Crustdata's core data infrastructure, including data warehouse/lake, ETL/ELT pipelines, real-time streaming, and support for data science and ML workloads.
Responsibilities
- Architect & Build: Design, build, and maintain our core data infrastructure, including our data warehouse and data lake, using modern cloud technologies (AWS, GCP, or Azure).
- Pipeline Development: Develop and scale robust, fault-tolerant data pipelines (ETL/ELT) to ingest and process massive volumes of structured and unstructured data from diverse sources.
- Enable Data Science & ML: Create the foundational platform to support our data scientists and ML engineers. This includes building systems for feature engineering, model training, and deploying ML models into production.
- Orchestration at Scale: Implement and manage workflow orchestration for hundreds of daily data jobs, ensuring reliability, monitorability, and efficiency using tools like Airflow, Dagster, or Prefect.
- Real-time Infrastructure: Build and manage real-time data streaming pipelines using technologies like Kafka or Flink to power live dashboards and time-sensitive product features.
- Data Quality & Governance: Champion data quality and reliability. Implement frameworks for data validation, testing, and monitoring to ensure our data is accurate and trustworthy.
Qualifications
- 3+ years of professional software engineering experience, with a significant focus on data engineering or building backend systems at scale.
- Strong programming skills in Python or another modern language (e.g., Java, Go).
- Hands-on experience with modern big data technologies such as Spark, Flink, or Dask.
- Practical experience with workflow management tools like Temporal, Airflow, Dagster, or Prefect.
- Pragmatic problem-solver who can navigate ambiguity, manage complexity, and take ownership of projects from inception to completion.
- Startup mentality: excited to work in a fast-paced, collaborative environment and wear multiple hats.
Skills
About Crustdata
Real-time B2B data via simple APIs
Crustdata provides live company and people data via APIs and full dataset delivery. We make hard to get data easy to use at scale. We have developed technology that allows us to pipe in live data from over a dozen different data sources and deliver this data instantly to our customers. We serve use cases like: automatic pipeline building, pipeline prioritization, champion watching, company and people triggers for sales and marketing automation, investment deal sourcing Our goal is index all the important data on the web and deliver it to customers in an easy-to-use way.
More open roles at Crustdata
- ML Engineer Intern | Summer 20267mo agoHybrid (San Francisco, CA) · Engineering
- Founding ML Engineer (India)7mo agoHybrid (Bengaluru, KA, IN / Bengaluru, Karnataka, IN) · Engineering
- Founding ML Engineer7mo agoHybrid (San Francisco, CA) · Engineering / Machine Learning
- Software Engineering Intern (Forward Deployed) | Summer 20268mo agoSan Francisco, CA · Engineering
- Founding Forward Deployed Engineer11mo agoSan Francisco, CA · Engineering