Member of Technical Staff, LLM & Production
San Jose$180k - $450kfull-timePosted 1w ago
About the role
Engineer at the intersection of model research and production systems building evaluation harnesses, data pipelines, and RL training environments to measure and productionize multimodal LLM capabilities.
Responsibilities
- Build and own end-to-end evaluation suites for LLM capabilities: task specs, grading logic, and the dashboards researchers check every day.
- Run the eval harness as a production service — sandboxed execution, tool calling, multi-turn and agentic tasks, deterministic replay, and caching — so a full evaluation pass is cheap and gives the same answer twice.
- Own the data pipeline from source to training: collection, filtering, dedup, labeling workflows, synthetic data generation, and the quality gates that catch contamination and distribution drift before anything reaches a run.
- Build RL training environments — reward functions, simulators, verifiers, task generators — and keep them stable as they fan out across many workers.
- Work side by side with researchers: turn a hypothesis into a runnable eval or environment within days, then help make sense of what comes back.
- Pull capabilities out of the product layer and into the model. Build the evals and environments that teach the model to do those things natively, and retire the workarounds once it can.
- Measure everything: eval regressions, environment health, throughput, cost — and make it obvious when a metric moved for the wrong reason.
- Raise the bar for research code across the team: tests, versioned datasets and evals, and clean boundaries between training, inference, and evaluation.
Qualifications
- 3+ years in software engineering, including 1-2 years hands-on with LLM training, evaluation, or data infrastructure.
- Deep Python fluency and real distributed-systems experience: you’ve chased failures across a fleet of machines, dealt with unreliable workers, and know where the compute budget actually goes.
- You've built evaluation infrastructure, not just run benchmarks — designing tasks, graders, and the plumbing around them.
- Working knowledge of SFT and RL for language models.
- Production instincts: you write tests, insist on reproducibility, and treat research tooling as something others depend on — because they do.
- Comfortable in ambiguity next to researchers: handed a half-formed idea, you can make it measurable.
Skills
Required:Python
Benefits
- Note: Benefits information is not included in the provided job description.
Last updated Oct 15, 2026. Interested? Apply on the company site.
About Hark
building the most advanced personal intelligence in the world.
At Hark, we are building the most advanced personal intelligence in the world.
personal intelligenceartificial intelligence
More open roles at Hark
- Warehouse Associate4d agoSan Jose, CA · Supply Chain
- Industrial Designer1w agoHybrid (San Jose, CA) · Design
- Member of Technical Staff, Hark Desktop2w agoHybrid (San Jose) · Product Engineering
- Application Security Engineer2w agoSan Jose, CA · Privacy & Security
- Manager, Enterprise Sales2w agoSan Jose · Commercial Operations