Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees
Authors: Yifei Zhou, Ayush Sekhari, Yuda Song, Wen Sun
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimentally, in challenging rich-observation environments, we show that our approach outperforms a state-of-the-art hybrid RL baseline which only relies on off-policy policy optimization, demonstrating the empirical benefit of combining on-policy and off-policy learning. |
| Researcher Affiliation | Academia | Yifei Zhou University of California, Berkeley {yifei_zhou@berkeley.edu Ayush Sekhari MIT sekhari@mit.edu Yuda Song Carnegie Mellon University yudas@cs.cmu.edu Wen Sun Cornell University ws455@cornell.edu |
| Pseudocode | Yes | Algorithm 1 Hybrid Actor-Critic (HAC) ... Algorithm 2 Hybrid Fitted Policy Evaluation (HPE) ... Algorithm 3 Hybrid NPG with Parameterized Policies (HNPG) ... Algorithm 4 Practical Finite-Horizon HNPG ... Algorithm 5 Finite-Horizon Hybrid Fitted Policy Evaluation (FHPE) |
| Open Source Code | Yes | Our code is publicly available at https://github.com/Yifei Zhou02/HNPG. |
| Open Datasets | Yes | In the Cifar100 augmented combination lock setting, the RL agent can only access images from the training set during training and will be tested using images from the test set. (Krizhevsky, 2009). |
| Dataset Splits | No | The paper mentions using "training set" and "test set" for the Cifar100 dataset, but does not explicitly provide details about a validation split (percentages, sample counts, or methodology). |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., exact GPU/CPU models, processor types, or memory amounts) used for running its experiments. |
| Software Dependencies | No | The paper mentions using 'multi-layer perceptron', 'Generalized Advantage Estimation (GAE)', and 'conjugate gradient algorithm', but does not provide specific software names with version numbers (e.g., Python, PyTorch, TensorFlow versions) required to replicate the experiment. |
| Experiment Setup | Yes | The pseudocode and hyperparameters are provided in Appendix E. We provide the hyperparameters of HNPG for both continuous Comblock and image-based continuous Comblock in Table 1. In addition, we provide the hyperparameters we tried for RLPD baseline for both Comblock settings in Table 2. |