torch-tensor-parallelism by benchflow-ai

torch-tensor-parallelism

benchflow-ai

9382423 months ago

Guidance for implementing tensor parallelism in PyTorch, including ColumnParallelLinear and RowParallelLinear layers. This skill should be used when implementing distributed tensor parallel operations, sharding linear layers across multiple GPUs, or simulating collective operations like all-gather and all-reduce for parallel computation.

>_
Quick Install
npxskills add benchflow-ai/skillsbench--skill torch-tensor-parallelism
Instructions

No instructions available