ICDPO: Effectively Borrowing Alignment Capability of Other LLMs via In-Context Direct Preference Optimization

ICDPO (In-Context Direct Preference Optimization) uses the states of a large language model before and after in-context learning to build an instant scorer, enabling it to borrow the human preference alignment capability of a stronger LLM through ICL and generate more aligned responses. Experiments show that it outperforms two fine-tuning-free baselines and is competitive with SFT + LoRA; a two-level retriever and an upgraded scorer can further improve the results.