sglang - LLM serving framework with RadixAttention prefix caching
Provides structured generation and high-performance serving for LLMs and VLMs using RadixAttention prefix caching.
Tags
Updated: 2026-09-24Capabilities
Typical Inputs
Typical Outputs
What this skill does
- Serve LLM and VLM models
- Cache prefixes automatically using RadixAttention
- Generate structured JSON outputs
- Constrain generation with regex
- Constrain generation with grammar
- Execute function calling agent workflows
- Process multi-modal image inputs
- Run speculative decoding
- Execute batch inference
Inputs
- Model paths
- User prompts and queries
- JSON schemas
- Regex patterns
- EBNF grammar definitions
- Tool definitions
- Image file paths
Outputs
- Generated text responses
- Structured JSON data
- Regex constrained text
- Tool invocation requests
- OpenAI API server responses
Requirements
- Python environment
- PyTorch library
- Transformers library
- CUDA compatible GPU