Exploring Methods to Detect Recommender System Data Memorization in Large Language Models

Researchers explore methods to detect memorization of recommender system data by large language models, comparing three approaches on MovieLens-1M: jailbreak prompt engineering, unsupervised latent knowledge discovery, and automatic prompt engineering (APE). Previous extraction of memorized data from MovieLens-1M relied solely on manual prompt engineering. Experiments with LLaMA models show that jailbreak prompts do not improve retrieval of memorized entries and are unstable; unsupervised methods probing internal activations via CCS and Cluster-Norm reliably distinguish real from fictional movie titles but fail on numerical data for users and ratings; APE retrieves entry-level information with moderate success but struggles to recover numerical interactions. The authors consider automatically optimized prompts the most promising strategy for extracting memorized samples.