Key Responsibilities
Build, operate, and continuously improve self-hosted AI inference services for internal applications, customer-facing products, and future Inference-as-a-service offerings.
Define and implement standard model-onboarding workflows covering model intake, compatibility validation, packaging, runtime selection, optimization, deployment, endpoint registration, testing, release, and lifecycle management.
Provision and manage secure, scalable inference endpoints for common AI application patterns, including interactive generation, RAG, embeddings, reranking, batch processing, multimodal use cases, tool calling, and agentic workflows.
Develop reusable deployment templates, APIs, SDKs, configuration standards, and self-service workflows for users to request, configure, access, monitor, update, and retire model endpoints.
Work with leading inference frameworks and toolkits, such as TensorRT-LLM, TensorRT, SGLang, vLLM, Triton Inference Server, NVIDIA Dynamo, NVIDIA NIM, CUDA, cuDNN, NCCL, and related serving, profiling, and observability tools.
Optimize model-serving performance using appropriate techniques, including quantization, compilation, batching, continuous batching, request routing, KV-cache management, prefix caching, speculative decoding, load balancing, model routing, memory optimization, and distributed parallelism.
Build and validate reusable inference recipes that specify compatible model versions, framework and runtime versions, precision formats, GPU configurations, topology requirements, scaling approaches, scheduler profiles, benchmark results, and expected performance envelopes.
Use quantization and optimization approaches such as NVFP4, FP8, INT8, TensorRT compilation, kernel optimization, efficient attention mechanisms, and memory-management techniques while maintaining agreed model-quality targets.
Design distributed inference configurations for large models, including tensor, pipeline, expert, context, and data parallelism where appropriate.
Work with the Kubernetes and proprietary scheduler team to define endpoint resource profiles, placement requirements, topology preferences, priority classes, quota models, autoscaling rules, capacity reservations, and workload-management policies.
Contribute inference workload characteristics, benchmarks, and performance profiles to the Model-to-Grid product so that endpoint placement, scheduling, capacity planning, and AI-factory operations can make more informed decisions.
Build benchmarking and qualification workflows using controlled experiments, reproducible baselines, load tests, latency tests, throughput tests, concurrency tests, scaling tests, performance profiling, regression testing, and internal or industry-standard benchmark methodologies where relevant.
Measure and improve key inference indicators, including time-to-first-token, inter-token latency, tokens per second, requests per second, end-to-end latency, concurrency, GPU utilization, memory efficiency, cache hit rate, scaling efficiency, power efficiency, and cost efficiency.
Establish automated performance-regression testing and release qualification for model versions, runtime and toolkit upgrades, CUDA and driver changes, Kubernetes releases, scheduler changes, networking and storage changes, and new GPU platforms.
Build operational observability for inference services, including endpoint availability, request volume, latency, queueing, errors, GPU utilization, GPU memory use, cache behavior, capacity, cost, power, and service-level objectives.
Partner with the agentic applications team to provide fit-for-purpose self-hosted endpoints for agent planning, retrieval, tool use, summarization, diagnosis, recommendation, optimization, and AI-factory operations.
Expose governed inference, benchmark, recipe, performance, and capacity information to agentic systems, allowing them to recommend suitable models, identify degradation, diagnose bottlenecks, plan optimization experiments, and validate results.
Work with Product, UX, DevOps, Platform, Infrastructure, Security, and Global Operations teams to ensure that inference provisioning, model selection, endpoint configuration, performance visibility, quota management, and troubleshooting are clear, secure, and operationally supportable.
Required Skills & Abilities
5+ years of software engineering experience, including 3+ years in AI inference, model serving, ML systems, high-performance computing, distributed systems, or comparable performance-critical environments.
Demonstrated experience building, operating, or materially improving production model-serving platforms, inference APIs, GPU-backed services, AI developer platforms, or multi-tenant AI systems.
Hands-on experience with one or more modern inference frameworks, such as TensorRT-LLM, TensorRT, SGLang, vLLM, Triton Inference Server, NVIDIA Dynamo, NVIDIA NIM, Hugging Face Text Generation Inference, or equivalent technologies.
Strong understanding of the NVIDIA AI software stack, including CUDA, cuDNN, NCCL, TensorRT, GPU profiling, distributed communication, and GPU performance analysis.
Practical understanding of LLM and generative-AI serving behavior, including prompt processing, token generation, batching, context length, concurrency, KV-cache management, prefill and decode performance, request scheduling, model routing, and latency-throughput trade-offs.
Experience with model optimization methods, including quantization, compilation, calibration, mixed precision, kernel fusion, memory optimization, caching, speculative decoding, parallelism, and accuracy-performance validation.
Strong Python skills and working proficiency in C++ or Go for inference services, APIs, automation, benchmarking, profiling, runtime integrations, and performance-critical development.
Experience with distributed inference or training patterns, including tensor, pipeline, expert, context, and data parallelism; collective communication; fault handling; and multi-node scaling.
Familiarity with Kubernetes, containers, CI/CD, GitOps, service APIs, autoscaling, workload scheduling, observability, and production multi-tenant platform operations.
Understanding of high-performance GPU infrastructure, including GPU topology, NVLink, NVSwitch, PCIe, NUMA, NIC affinity, RDMA, RoCEv2, network fabrics, storage throughput, and their impact on inference performance.
Experience with inference benchmarking, performance profiling, reproducibility, load testing, regression testing, and analysis of throughput, latency, utilization, scaling, power, and cost metrics.
Familiarity with model-serving use cases such as RAG, embeddings, reranking, multimodal inference, agentic applications, model routing, and tool-calling workflows.
Understanding of security and governance for inference services, including identity, authentication, authorization, tenant isolation, quotas, rate limiting, secrets handling, audit logging, abuse prevention, and data protection.
Self-hosted inference platform engineering and production ownership.
Model serving, endpoint provisioning, lifecycle management, and developer self-service experience.
Inference-as-a-service foundations, including multi-tenancy, scalable endpoint operations, usage visibility, quotas, service profiles, and operational supportability.
High-performance LLM, generative-AI, embedding, reranking, and multimodal inference optimization.
Practical use of modern inference frameworks, model-serving toolkits, GPU profiling tools, and benchmarking methods.
Development of validated, repeatable, versioned inference recipes and deployment configurations.
Quantitative performance engineering across latency, throughput, GPU utilization, memory, scaling, power, energy, cost, and reliability.
Model-to-Grid thinking: connecting endpoint workload characteristics to benchmarking, scheduler policies, topology-aware placement, capacity, power, thermal state, and AI-factory operations.
Ability to provide reliable, governed, and cost-efficient inference services for agentic applications and autonomous operational workflows.
Cross-functional collaboration with AI applications, Model-to-Grid, scheduling, DevOps, Platform, SDI, Security, UX, product, and global operations teams.
Reliable, secure, and scalable self-hosted inference endpoints are available for priority internal applications, external products, Model-to-Grid capabilities, and agentic systems.
Reduction in the time required to onboard, validate, optimize, deploy, provision, update, and retire a supported model endpoint.
Adoption of standardized inference deployment workflows supported model catalogues, endpoint templates, APIs, SDKs, recipes, and self-service capabilities.
Demonstrated foundations for future Inference-as-a-service offerings, including defined service profiles, endpoint lifecycle controls, tenant isolation, quota management, usage measurement, observability, support processes, and release governance.
Improvement in inference performance for priority workloads, measured by latency, time-to-first-token, inter-token latency, tokens per second, requests per second, concurrency, GPU utilization, memory efficiency, and scaling efficiency.
Reduction in cost per request, cost per token, energy per inference task, and avoidable resource over-provisioning, while maintaining agreed quality, availability, and reliability standards.
Number and adoption of benchmark-validated, documented, versioned inference recipes across supported models, frameworks, toolkits, precision formats, endpoint types, GPU configurations, and deployment topologies.
Effective use of benchmarking and performance-validation practices to prevent regressions across model updates, runtime upgrades, CUDA or driver changes, infrastructure changes, scheduler releases, and new GPU platforms.
Percentage of production endpoints with appropriate authentication, authorization, tenant isolation, quota controls, rate limits, metering, observability, audit logging, documentation, support runbooks, and rollback procedures.
Contribution of inference workload profiles and performance data to Model-to-Grid scheduling, capacity planning, placement quality, operational visibility, power efficiency, and user time-to-results.
Reliability and performance of endpoints supporting agentic applications, measured through agent task latency, response quality, tool-call completion, workflow success rate, safe automation outcomes, and reduced reliance on unmanaged external model services.
