← All roles

Senior Software Engineer, Data Engineering

Distributed Spectrum· Engineering
Apply on company site
New York City$160k - $200kFull-timePosted 2w ago

About the role

Senior Data Engineer role at Distributed Spectrum to design and operate a data lakehouse and vector database infrastructure for AI/ML workloads, from ingestion through storage, cataloging, retrieval, and governance.

Responsibilities

  • Architect and build our data lakehouse: ingestion pipelines, open table formats, partitioning and compaction strategies, cataloging, and governance.
  • Design and operate vector database infrastructure for embedding storage, similarity search, and retrieval at scale — choosing, tuning, and evolving the right systems for our workloads.
  • Build large-scale data management systems: lifecycle and retention, lineage, versioning of datasets for reproducible training, quality monitoring, and cost management across petabyte-class storage.
  • Design data platform capabilities that directly support ML use cases — feature and embedding pipelines, training-set assembly, evaluation datasets, and low-latency retrieval for agents.
  • Establish schema and data-contract standards across teams, and build tooling that lets researchers and engineers self-serve.
  • Own reliability and performance of data pipelines in production, including observability and failure handling.
  • Mentor engineers and lead technical design across the data domain.

Qualifications

  • 4–6+ years of software / data engineering experience, including several years designing and operating large-scale data platforms in production.
  • Deep experience with lakehouse architectures and technologies — e.g., Apache Iceberg, Delta Lake, or Hudi; Parquet; Spark, Flink, Trino, DuckDB, or similar engines.
  • Hands-on experience with vector databases and embedding retrieval (e.g., pgvector, Milvus, Qdrant, Weaviate, Pinecone, LanceDB, or FAISS-based systems), including indexing trade-offs and scaling.
  • Strong experience with AWS or another major cloud provider — object storage, managed data services, compute, and cost optimization at scale.
  • Fluency in Python and SQL; experience with orchestration tools (Airflow, Dagster, Prefect, Step Functions, etc.).
  • Demonstrated experience designing data architectures specifically for ML / AI workloads: training data pipelines, feature stores, dataset versioning, or retrieval systems.
  • Strong grasp of data modeling, consistency, and the trade-offs between batch and streaming.

Skills

Required:Data lakehouse architectureVector databases and embedding retrievalCloud platforms (AWS)PythonSQLOrchestration toolsML data architectures

Benefits

  • Above-market salary, equity, and benefits package.
  • Early Series A Equity
  • Excellent health, dental, and vision coverage
  • 401(k) match - up to 4% of your salary
  • Flexible PTO
  • Daily office lunches in NYC
Last updated Oct 7, 2026. Interested? Apply on the company site.

About Distributed Spectrum

The World’s First AI Signals Expert

Gain actionable insights from the radio spectrum in real-time with small, cheap and efficient sensors from Distributed Spectrum

radio spectrumsensorsAI signalsthreat detection
Company profileWebsite

More open roles at Distributed Spectrum