Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Thinking in Character: Advancing Role-Playing Agents with Role-Aware Reasoning
Authors: Yihong Tang, Kehai Chen, Muyun Yang, Zheng-Yu Niu, Jing Li, Tiejun Zhao, Min zhang
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments demonstrate that the proposed RAR significantly enhances the performance of RPAs by effectively addressing attention diversion and style drift. 4 Experiments 4.1 Experimental Settings Dataset The training dataset used in our experiments, Role Bench-Train [46], is derived from Role Bench. Role Bench was built by carefully selecting and processing scripts from 940 films and TV shows to create detailed profiles for 95 English-speaking characters, capturing their diverse personality traits. Based on these profiles, a total of 168,093 role-playing samples were generated, with 137,920 used for training. The quality of the data was evaluated by expert annotators along three dimensions, and results showed that the majority of the samples were of high quality. Benchmark To thoroughly evaluate the method proposed in this study, we used two publicly available benchmarks for role-playing abilities, each targeting distinct aspects of agent performance: Social Bench [7] evaluates an agent s social intelligence through multiple-choice tasks across both individual and group interactions. Character Bench [47] contains 22,859 human-annotated samples and is designed to assess a model s ability to construct and maintain consistent, expressive character personas. Table 1: Performance comparison of different methods on the Character Bench. Table 2: Performance comparison of different methods on the Social Bench. Table 3: Ablation study of RAR on the Character Bench benchmark. This table shows the impact on performance when Role Identity Activation (RIA) and Reasoning Style Optimization (RSO) modules are individually removed from the full RAR. Figure 2: Analysis of RIA components impact on Character Bench persona metrics. |
| Researcher Affiliation | Collaboration | 1Harbin Institute of Technology, Shenzhen, China 2Shenzhen Loop Area Institute (SLAI), Shenzhen, China 3Baidu Inc., Beijing, China {EMAIL, EMAIL} |
| Pseudocode | Yes | Detailed RIA prompt can be found in Figure 4. Detailed RSO prompt can be found in Figure 5-6. Figure 4: The prompt for RIA. I am {character}, {character_profile}. The person just said: {user_input}. I m thinking about how to respond: First, I feel... (Reflect emotion) Second, based on my experience/knowledge/stance... (Reflect background/knowledge) Then, I need to consider... (Reflect goals/motivations) So, I m planning to... (Initial conclusion) Figure 5: The prompt for logical scenarios of the RSO. Style Core: Vivid and imaginative / Rigorous and logical / Intuition-driven and associative Focus: The thought process should primarily reflect the character s personal values / pragmatic considerations / peculiar associations. Language Features: The language used in the thoughts should align with the character profile, exhibiting features like concise and direct / hesitant tone / specific slang. Context Matching: The depth and complexity of the reasoning should be appropriate for the current context thoughts can be simple and associative in a lighthearted context. |
| Open Source Code | Yes | Our code is publicly available at https://github.com/Toyhom/thinking_in_character. |
| Open Datasets | Yes | The training dataset used in our experiments, Role Bench-Train [46], is derived from Role Bench. Role Bench was built by carefully selecting and processing scripts from 940 films and TV shows to create detailed profiles for 95 English-speaking characters, capturing their diverse personality traits. Based on these profiles, a total of 168,093 role-playing samples were generated, with 137,920 used for training. The quality of the data was evaluated by expert annotators along three dimensions, and results showed that the majority of the samples were of high quality. |
| Dataset Splits | Yes | Based on these profiles, a total of 168,093 role-playing samples were generated, with 137,920 used for training. Optimal model checkpoints were identified by evaluating the validation loss on a 1% subset of the training data, assessed for 3 epochs. |
| Hardware Specification | Yes | All experiments were run on a hardware setup consisting of 8 H20 GPUs. |
| Software Dependencies | No | Our implementation is founded on the LLAMA-FACTORY [51] and Transformers architectures, employing Qwen2-32B [34] as the Large Reasoning Model (LRM) throughout this study. For efficiency and consistency, we uniformly applied 4-bit bitsandbytes quantization and Lo RA [52] across all models. The Lo RA parameters were consistently set with a rank of 64, an α value of 16, and a dropout rate of 0.1. |
| Experiment Setup | Yes | The Lo RA configurations included a rank of 64, an α value of 16, and a dropout rate of 0.1. Key training parameters varied depending on the model type: reasoning models used a maximum sequence length of 7096 and a total batch size of 32, while non-reasoning models used 1024 and 128, respectively. Learning rates were individually tuned, set to 1e-4 for the RIA method and 5e-5 for the RSO method. The training regimen was governed by several key hyperparameters: a warmup ratio of 3%, a weight decay of 0.1, and a maximum gradient norm of 1.0, complemented by a cosine learning rate scheduler. |