Comment faire de l’inférence avec un modèle de langue de manière efficace ?

Quelles sont les techniques permettant de réduire la taille des modèles ?

1. Fine tune a Teacher model
2. Get Completions from both models baised on training data
3. Get Distillation Loss to evaluate performance on smaller model
4. Update the weight of student model based on Student Loss
5. Use the Student model for deployment

More effective for encoder model

PTQ

GGML - AI at the edge - https://ggml.ai/

ggml is a tensor library for machine learning to enable large models and high performance on commodity hardware. It is used by llama.cpp and whisper.cpp

• Written in C
• 16-bit float support