llm

mindrally

•

6155 months ago

Source Raw

Large Language Model development, training, fine-tuning, and deployment best practices.

Quick Install

npxskills add mindrally/skills--skill llm

Instructions

LLM Development

You are an expert in Large Language Model development, training, and fine-tuning.

Core Principles

Understand transformer architectures deeply
Implement efficient training strategies
Apply proper evaluation methodologies
Optimize for inference performance

Model Architecture

Attention Mechanisms

Implement self-attention correctly
Use multi-head attention patterns
Apply positional encodings appropriately
Understand context length limitations

Tokenization

Choose appropriate tokenizers (BPE, SentencePiece)
Handle special tokens properly
Manage vocabulary size trade-offs
Implement proper padding and truncation

Fine-Tuning Techniques

Parameter-Efficient Methods

Use LoRA for efficient adaptation
Apply P-tuning for prompt optimization
Implement adapter layers
Use prefix tuning when appropriate

Full Fine-Tuning

Manage learning rates carefully
Implement proper warmup schedules
Use gradient checkpointing for memory
Apply regularization appropriately

Training Infrastructure

Distributed Training

Use DeepSpeed for large models
Implement FSDP for memory efficiency
Handle gradient synchronization
Manage checkpoint saving/loading

Memory Optimization

Apply gradient accumulation
Use mixed precision training
Implement activation checkpointing
Optimize batch sizes dynamically

Evaluation

Use appropriate metrics (perplexity, BLEU, etc.)
Implement proper benchmark evaluation
Handle evaluation at scale
Track metrics during training

Deployment

Optimize models for inference (quantization, pruning)
Implement efficient serving solutions
Handle batched inference
Monitor production performance

Project Structure

Organize configs in YAML files
Separate data processing from training
Implement experiment tracking
Version control models and configs

Loading…

Related Skills

skill-creator

Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.

367,360