Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees
Authors: Yifei Zhou, Ayush Sekhari, Yuda Song, Wen Sun
ICLR 2024 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimentally, in challenging rich-observation environments, we show that our approach outperforms a state-of-the-art hybrid RL baseline which only relies on off-policy policy optimization, demonstrating the empirical benefit of combining on-policy and off-policy learning. |
| Researcher Affiliation | Academia | Yifei Zhou University of California, Berkeley {EMAIL Ayush Sekhari MIT EMAIL Yuda Song Carnegie Mellon University EMAIL Wen Sun Cornell University EMAIL |
| Pseudocode | Yes | Algorithm 1 Hybrid Actor-Critic (HAC) ... Algorithm 2 Hybrid Fitted Policy Evaluation (HPE) ... Algorithm 3 Hybrid NPG with Parameterized Policies (HNPG) ... Algorithm 4 Practical Finite-Horizon HNPG ... Algorithm 5 Finite-Horizon Hybrid Fitted Policy Evaluation (FHPE) |
| Open Source Code | Yes | Our code is publicly available at https://github.com/Yifei Zhou02/HNPG. |
| Open Datasets | Yes | In the Cifar100 augmented combination lock setting, the RL agent can only access images from the training set during training and will be tested using images from the test set. (Krizhevsky, 2009). |
| Dataset Splits | No | The paper mentions using "training set" and "test set" for the Cifar100 dataset, but does not explicitly provide details about a validation split (percentages, sample counts, or methodology). |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., exact GPU/CPU models, processor types, or memory amounts) used for running its experiments. |
| Software Dependencies | No | The paper mentions using 'multi-layer perceptron', 'Generalized Advantage Estimation (GAE)', and 'conjugate gradient algorithm', but does not provide specific software names with version numbers (e.g., Python, PyTorch, TensorFlow versions) required to replicate the experiment. |
| Experiment Setup | Yes | The pseudocode and hyperparameters are provided in Appendix E. We provide the hyperparameters of HNPG for both continuous Comblock and image-based continuous Comblock in Table 1. In addition, we provide the hyperparameters we tried for RLPD baseline for both Comblock settings in Table 2. |