FSDP QDoRA (more advancement in finetuning)

Answer.AI announced QDoRA, a quantized, FSDP-compatible training method that it says matches full weight training in accuracy for continued pre-training while using far less memory. The company says preliminary experiments on Llama 2 and initial Llama 3 runs were promising, and that QDoRA is likely the best way for most people to train language models. On Llama-2-7b fine-tuned on the Orca-Math dataset with 10k samples, QDoRA scored an exact match of 0.176, compared with 0.182 for full fine-tuning and 0.098 for QLoRA. The article notes that full fine-tuning uses far more memory than these alternatives.