tensorrt-llm - TensorRT-LLM LLM Inference Optimization
Optimizes LLM inference with NVIDIA TensorRT for high-throughput, low-latency production deployment on NVIDIA GPUs.
Tags
Updated: 2026-03-19Capabilities
Typical Inputs
Typical Outputs
What this skill does
- optimize LLM inference
- compile models with TensorRT
- serve models via HTTP API
- generate text from prompts
- configure sampling parameters
- deploy across multiple GPUs
- apply quantization to models
- manage batch requests
- enable in-flight batching
- use tensor parallelism
- use pipeline parallelism
Inputs
- LLM model
- input prompts
- sampling parameters
- GPU devices
- model weights
Outputs
- generated text
- inference server
- API responses
- compiled model
Requirements
- NVIDIA GPU (A100/H100/GB200)
- CUDA 13.0.0+
- TensorRT 10.13.2+
- Python 3.10-3.12
- tensorrt-llm package