Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Projection-based Lyapunov method for fully heterogeneous weakly-coupled MDPs
Authors: Xiangcheng Zhang, Yige Hong, Weina Wang
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this section, we perform two sets of experiments to illustrate the numerical performance of the proposed ID policy for fully heterogeneous WCMDPs. We increase the number of arms as N {100, 200, 400, 800, 1600, 3200}. Each arm s MDP has 10 states and 4 actions, with parameters generated uniformly at random in a proper sense. We simulate the policy for 2 104 time steps over 4 replications for each N. To illustrate the performance more clearly, we measure the optimality ratio, defined as the ratio between the long-run average reward achieved by a policy and the LP relaxation upper bound Rrel N. Confidence intervals are calculated using the batch means method with a batch size of 4000, but they are typically too small to be visible on figures. Figure 1 shows that the optimality ratio of the ID policy becomes increasingly close to 1 as N increases. |
| Researcher Affiliation | Academia | Xiangcheng Zhang1 Yige Hong2 Weina Wang2 1Weiyang College, Tsinghua University 2 Computer Science Department, Carnegie Mellon University EMAIL EMAIL |
| Pseudocode | Yes | Algorithm 1 ID reassignment 1: Input: optimal state-action frequencies (y i (s, a))i [N],s S,a A, budgets (εk)k [K] 2: Output: new arm ID, recorded in new ID(i), for each arm with old ID i [N] 3: Compute (C k,i)i [N],k [K] and the set of active constraints A using (8) 4: if A = then 5: new ID(i) i for all i [N] ϑ No need for ID reassignment 6: else 7: Initialize F ϑ Set of arms that have been assigned new IDs 8: Initialize Dk {i [N]: C k,i ϖ} for all k A 9: ϖ εmin/4 mink [K] εk/4; d 10: for ϱ = 0, 1, . . . , N/d 1 do 11: I(ϱ) [ϱd + 1 : (ϱ + 1)d]; j ϱd + 1 12: for k A do 13: if $ {new ID(i) I(ϱ)} < ϖ then 14: Pick any i from Dk and set new ID(i) j; remove i from Dk for all k ; add i to F 15: j j + 1 16: For all i [N]"F, assign values to their new ID(i) s randomly from [N]\{new ID(i ): i F} |
| Open Source Code | Yes | The complete code for these experiments is available on Git Hub [53] |
| Open Datasets | No | Each arm s MDP has 10 states and 4 actions, with parameters generated uniformly at random in a proper sense. The N-armed problem has 4 budget constraints, with cost functions also generated randomly. More details are provided in Appendix B.1. |
| Dataset Splits | No | The paper does not mention specific dataset splits like training, validation, or test sets. It describes generating problem instances on the fly and simulating them. |
| Hardware Specification | Yes | The complete code for these experiments is available on Git Hub [53], and all results can be reproduced within 24 hours on a standard PC (e.g., 6-Core Intel Core i7). |
| Software Dependencies | No | The paper does not explicitly list any specific software dependencies with version numbers (e.g., Python 3.x, PyTorch 1.x, or specific solver versions) used for the implementation or experiments. |
| Experiment Setup | Yes | In the first set of experiments, we demonstrate the asymptotic optimality of the ID policy. We increase the number of arms as N {100, 200, 400, 800, 1600, 3200}. Each arm s MDP has 10 states and 4 actions, with parameters generated uniformly at random in a proper sense. The N-armed problem has 4 budget constraints, with cost functions also generated randomly. More details are provided in Appendix B.1. We simulate the policy for 2 104 time steps over 4 replications for each N. |