← All roles

Applied Researcher: On-Device Multimodal Reasoning

Apple· Machine Learning and AI
Apply on company site
Hybrid (Sunnyvale, CA)$150.4k - $277.6kfull-timePosted 2mo ago

About the role

Applied research role focused on on-device multimodal reasoning with small model regimes (sub-10B parameters, down to sub-1B). Design, train, and optimize compact vision-language models that reason over images, video, and 3D scene content within strict compute/memory/latency budgets, enabling real-time, privacy-preserving multimodal understanding on Apple devices.

Responsibilities

  • Design, train, and post-train compact vision-language models (under ~10B, including sub-1B) that perform multi-step visual reasoning, grounded visual understanding, and language generation within on-device resource budgets
  • Research and implement efficient reasoning techniques — compressed and latent chain-of-thought, reasoning-trace distillation, early-exit and budget-aware reasoning, adaptive compute allocation, and test-time scaling that maximizes reasoning quality per FLOP
  • Own the decoding stack for on-device inference: speculative and self-speculative decoding, draft models and multi-token prediction, structured/constrained generation, KV-cache compression and quantization, prefill/decode scheduling, and streaming latency (TTFT, tokens/sec)
  • Build reasoning over real-time perception experts: enable a small VLM to invoke and reason over on-device 3D scene reconstruction and geometry, human body/hand pose and mesh recovery, object detection, localization, and tracking — designing representations, interfaces, and training signals for limited context
  • Develop structured-output fusion and expert scheduling: compact tokenizations for geometry/body parameters/detections; policies for which perception models run at which resolution and frame rate; mechanisms for the reasoner to resolve conflicts between experts and visual features
  • Apply reasoning-focused post-training: supervised distillation from frontier teachers, preference and RL methods (GRPO, RLVR, STaR, rejection sampling), verifier- and reward-guided decoding, and process supervision for multimodal and spatially grounded chains
  • Drive visual token efficiency and feature compression: token pruning, merging, resampling; adaptive resolution and frame-rate policies; learned connectors that preserve reasoning accuracy at reduced visual token budgets
  • Develop distillation, pruning, and quantization strategies that preserve multimodal reasoning fidelity at reduced model sizes — including QAT and mixed-precision inference across language decoder, vision encoder, and perception experts
  • Explore hybrid architectures balancing long-context visual reasoning with inference efficiency and bounded memory (stateful components like SSM/Mamba)
  • Optimize reasoning and encoder architectures for Apple silicon (Neural Engine, GPU, ANE-aware patterns); profile latency, memory footprint, thermal behavior, and power across devices
  • Collaborate with hardware, compiler, and platform teams to co-design model architectures that exploit device-specific acceleration capabilities
  • Build evaluation frameworks measuring reasoning fidelity, grounding, hallucination, robustness under budgets; generate accuracy-vs-latency and accuracy-vs-power curves

Qualifications

  • MS in Computer Science, Machine Learning, AI, Computer Vision, or related field (or equivalent practical experience)
  • Strong foundation in deep learning with experience in LLM or VLM training, post-training, or inference optimization
  • Experience with multimodal models in resource-constrained environments, including reasoning quality, decoding, or model compression
  • Proficiency in Python and PyTorch; familiarity with inference and optimization toolchains (quantization, distillation, pruning) — e.g., vLLM/SGLang, llama.cpp, MLX, CoreML

Skills

Required:Multimodal model developmentOptimization and model compressionDecoding strategiesOn-device ML frameworksProgramming

Benefits

  • Comprehensive medical and dental coverage
  • retirement benefits
  • discounted products and free services
  • educational expense reimbursement (tuition)
  • discretionary bonuses or commissions possible
  • relocation assistance
  • Apple Employee Stock Purchase Plan
  • discretionary employee stock units for eligible employees
Last updated Aug 7, 2026. Interested? Apply on the company site.

About Apple

Discover the innovative world of Apple and shop everything iPhone, iPad, Apple Watch, Mac, and Apple TV, plus explore accessories, entertainment, and expert device support.

Discover the innovative world of Apple and shop everything iPhone, iPad, Apple Watch, Mac, and Apple TV, plus explore accessories, entertainment, and expert device support.

AirPods Max 2Apple ArcadeApple CardApple Fitness+Apple MusicApple TVApple TV+Apple Watch Series 11
Company profileWebsite

More open roles at Apple