LLM Fine-Tuning: When PEFT Makes Sense and When It Doesn’t

Fine-tuning has a reputation problem. Some see it as a magic wand, sure that feeding a model some company data will make it understand the domain. Others treat any change to the weights as a relic of the pre-prompting era.