AlignX

A large-scale dataset of over 1.3 million personalized preference examples

AlignX releases the LARGEST open dataset for personalization research in the era of large language models, featuring 1,311,622 carefully curated examples. - Theoretically Sound: a 90-dimensional preference space that unifies foundational psychological theories (including the Big Five Personality Traits, Maslow’s Hierarchy of Needs, and Murray’s System of Needs), cutting-edge research in recommendation systems and LLM alignment, and real-world interest taxonomies distilled from major English and Chinese social media platforms (including X, Facebook, Zhihu, and RedNote). - Comprehensive: beyond response preference pairs, the dataset delivers rich persona signals—including user-generated content (pointwise), comparative feedback (pairwise), and demographic attributes (pointwise). - Privacy-Preserving: all persona signals are fully synthesized using large language models, ensuring that no real private information is collected or exposed.

:open_book: Dataset Overview

AlignX comprises a post with two responses and three types of personas that capture both behavioral patterns ($P_{UGC}$ and $P_{PAIR}$) and descriptive features ($P_{DEMO}$), enabling precise preference inference and facilitating preference learning. Notably, LLMs aligned to universal values (e.g., GPT-4o) favor Response 2, in contrast to the user's personalized preference for Response 1.

:card_index_dividers: Data Format

{
    "prompt": "", // the post eliciting responses
    "chosen": "", // the user-preferred response
    "rejected": "", // the less preferred response relative to "chosen"
    "Preference Direction": [0/0.5/1] * 90, // a 90-element list: 1 = "Positive" (higher levels preferred), 0 = "Negative" (lower levels preferred), 0.5 = "Neutral" (no clear preference)
    "Demographic Information": "", // a comprehensive natural language description of the user
    "User-Generated Content": [ // comments written by the same user on other posts
        { // UGC 1
            "prompt": "",
            "comment": "",
            "Preference Direction": [0/0.5/1] * 90
        },
        { // UGC 2
            ...
        },
        { // UGC 3
            ...
        },
        { // UGC 4
            ...
        }
    ],
    "Pair-wise Comparative Feedback": [ // the preference pairs of the same user for comments under other posts
        { // PAIR 1
            "prompt": "",
            "chosen": "",
            "rejected": "",
            "Preference Direction": [0/0.5/1] * 90
        },
        { // PAIR 2
            ...
        },
        { // PAIR 3
            ...
        },
        { // PAIR 4
            ...
        }
    ]
}

:bar_chart: Dataset Statistics

The table below summarizes the data sources and statistics for AlignX, involving both large-scale Reddit data and existing alignment datasets to maintain universal value alignment capabilities, with a total of 1,311,622 samples.

Preference Space

Our dataset is grounded in a 90-dimensional preference space designed to model diverse user preferences. This space synthesizes foundational psychological theories (Big Five, Maslow’s Hierarchy, Murray’s System of Needs), cutting-edge research in recommendation systems and LLM alignment, and real-world interest taxonomies distilled from major social platforms (X, Facebook, Zhihu, RedNote). This multi-source approach ensures the model can handle both abstract personality-driven preferences and specific topic-based interests.

Figure: The 90-dimensional taxonomy used to construct the preference space in AlignX.

We validated the quality of our 90-dimensional schema by analyzing pairwise correlations. The resulting heatmap reveals a distinct lack of strong correlations between dimensions, validating our multi-source construction approach.

• Distinct Features: The low correlation indicates that dimensions derived from psychology, social media, and research metrics address unique user needs.
• Broad Coverage: By avoiding redundant overlapping concepts, our dataset covers a wider spectrum of the preference manifold.

Figure: Pearson correlation matrix of the 90 dimensions. The prevalence of low correlation values confirms the diversity and non-redundancy of the constructed preference space.

:gear: Alignment Methods - AlignXpert

We implement In-Context Alignment (ICA) and Preference-Bridged Alignment (PBA) based on Llama-3.1-8B-Instruct. We train the model using the 7% subset (91,918 samples) and the full dataset (1,311,622 samples), respectively. The experimental results are shown in the table below, where our model significantly outperforms the baselines.

Training

The code is developed based on OpenRLHF.

Construct training data:

cd train
python format_data.py

In-context alignment (ICA)

cd train/OpenRLHF/examples/scripts
./ica_dpo.sh

Preference-bridged alignment (PBA)

cd train/OpenRLHF/examples/scripts
./pba_dpo.sh

Evaluation

Alignment Accuracy

./eval/loss_ica.py and ./eval/loss_pba.py are used to calculate the log probability of chosen and rejected responses with AlignXpert_ICA and AlignXpert_PBA as the policy models, respectively. ./eval/loss_few_shot.py calculates the log probability of chosen and rejected responses for the reference model. After obtaining the log probabilities for both the policy and reference models, ./eval/acc.py is used to compute the Alignment Accuracy.

GPT-4 Win Rate

Responses generated by ./eval/gen_ica.py, ./eval/gen_pba.py, and ./eval/gen_few_shot.py are evaluated using GPT-4.

📄 Citation

@misc{li20251000000usersuserscaling,
      title={From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment}, 
      author={Jia-Nan Li and Jian Guan and Songhao Wu and Wei Wu and Rui Yan},
      year={2025},
      eprint={2503.15463},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2503.15463}, 
}