← All roles

Data Engineer - Foundational

Harmattan AI· R&D
Apply on company site
ParisFull-timePosted 7mo ago

About the role

Data Engineer on the Foundational team responsible for building high-performance data infrastructure to power foundational models, managing terabytes to petabytes of raw, unstructured video data (EO and IR) and enabling efficient ML workloads.

Responsibilities

  • Multi-Modal Ingestion Pipeline: Build ETL/ELT pipelines to extract, decode, and store raw Electro-Optical (EO) and Infrared (IR) video from field logs into highly optimised formats like WebDataset, TFRecords, or Parquet.
  • Sensor Synchronisation & Alignment: Develop algorithms to programmatically synchronise EO and IR frames temporally and spatially to provide paired inputs for model training.
  • High-Throughput Data Loading: Architect storage-to-GPU pipelines to ensure multi-node training clusters maintain >90% GPU utilisation without I/O bottlenecks.
  • Distributed Processing: Write and optimise distributed data processing jobs using tools like Apache Spark, Ray, or Apache Beam to process thousands of hours of tactical video logs.
  • Data Quality & Versioning: Implement automated quality checks to filter corrupted or blank frames and maintain 100% reproducible training runs through robust versioning and lineage tracking.
  • Infrastructure Evaluation: Assess and implement advanced storage solutions (e.g., MinIO, S3 tiering) to manage growing datasets while optimising for cost and latency.

Qualifications

  • BS or MS in Computer Science, Software Engineering, or Distributed Systems preferred; deep knowledge of operating systems, networking, and parallel computing.
  • 5-6+ years of experience building and maintaining terabyte-scale pipelines for unstructured data (video, images, or point clouds).
  • Proven track record of maximising multi-node GPU utilisation and optimising data loaders for frameworks like PyTorch or JAX.
  • Strong command of distributed computing tools (Spark, Ray, Beam) and ML data versioning tools (DVC, Apache Iceberg, or Pachyderm).
  • A systems-thinker who thrives in a fast-paced startup environment and treats messy data as an engineering problem solvable via automation.
  • 100% dedication to Harmattan AI’s mission of providing a defensive edge to allied nations through ethical, high-impact technology.

Skills

Required:ETL/ELT pipeline designMulti-modal data synchronizationHigh-throughput data loadingDistributed processingData quality & versioning
Nice to have:Storage solutions
Last updated Jul 15, 2026. Interested? Apply on the company site.
Harmattan AI logo

About Harmattan AI

Building the Future of Autonomous Warfare.

Building the Future of Autonomous Warfare.

Autonomous WarfareUASPayloadCounter-UAS
Company profileWebsite

More open roles at Harmattan AI