← All roles

Principal AI Software Engineer

Microsoft· Software Engineering
Apply on company site
Redmond, WA, Mountain View, CA, Hillsboro, OR$142.8k - $274.8kFull-timePosted 4w ago

About the role

Lead systems AI software prototyping and optimization within Azure hardware systems, focusing on memory hierarchy, AI inference workloads, and hardware/software co-design to reduce total cost of ownership for Azure deployments.

Responsibilities

  • Lead full system software prototyping to develop proof-of-concepts for memory TCO reduction (memory tiering/pooling, overcommit) for Azure usages and deployment scenarios.
  • Develop insights via workload characterization to identify systems optimization opportunities.
  • Collaborate with workload experts and partner ISVs to engineer TCO-optimized solutions for Azure compute fleets.
  • Influence hardware architecture and industry alignment with data-driven analysis and recommendations (3-6 year horizon).
  • Lead characterization/optimization of Large Language Model (LLM) inference workloads focusing on KV Cache capacity, placement, migration, and utilization across GPU HBM, host DRAM, CXL memory expansion/pooling, SSD, and emerging memory tiers.
  • Develop proof-of-concepts and evaluation frameworks for memory-tiering architectures for AI inference (CXL pooled memory, memory expansion, context-memory platforms, SSD-backed cache tiers).
  • Design and execute workload characterization studies for agentic, multi-turn, coding, reasoning, and long-context AI workloads to quantify memory consumption, latency, throughput, token efficiency, and utilization.
  • Analyze end-to-end data movement across GPU, CPU, storage, and networking; optimize via GPU Direct Storage (GDS), GPU Direct RDMA (GDR), peer-to-peer transfers, and distributed inference pipelines.
  • Develop software prototypes, framework extensions, and instrumentation to evaluate KV Cache offload, prefetching, migration, compression, deduplication, and memory-overcommit techniques.
  • Build performance models and simulation frameworks to predict memory hierarchy innovations’ impact on large-scale inference deployments.

Qualifications

  • Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in C, C++, C#, Java, JavaScript, or Python OR equivalent experience.

Skills

Required:Memory hierarchy and system performance analysis
Nice to have:KV Cache and AI inference optimization

Benefits

  • Find additional benefits and pay information here: https://careers.microsoft.com/us/en/us-corporate-pay
Last updated Sep 24, 2026. Interested? Apply on the company site.

About Microsoft

Cloud ComputingConsumer ElectronicsSoftwareTechnologyliquid-ai-recruiting-target
Company profileWebsite

More open roles at Microsoft