llm-inference-batching-scheduler
Guidance for optimizing LLM inference request batching and scheduling problems. This skill applies when designing batch schedulers that minimize cost while meeting latency and padding constraints, involving trade-offs between batch count, shape selection, and padding ratios. Use when the task involves grouping requests by sequence lengths, managing shape compilation costs, or optimizing multi-objective scheduling with hard constraints.
>_
Quick Installnpxskills add benchflow-ai/skillsbench--skill llm-inference-batching-schedulerInstructions
No instructions available
