Fast and Fine-grained Autoscaler for Streaming Jobs with Reinforcement Learning
Authors: Mingzhe Xing, Hangyu Mao, Zhen Xiao
IJCAI 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments conducted on real-world datasets demonstrate the superiority of our method over six competitive baselines. |
| Researcher Affiliation | Collaboration | 1School of Computer Science, Peking University 2Huawei Noah s Ark Lab |
| Pseudocode | No | The paper does not contain any pseudocode or algorithm blocks. |
| Open Source Code | Yes | The code and Appendix are available at https://github.com/xmzzyo/sure. |
| Open Datasets | Yes | Dataset. We use Clarknet Trace 1 as workloads, which describes the number of HTTP requests to the servers recorded in 20,000 minutes. The workloads are highly varying in time, and show periodicity characteristics. As for computing jobs, since we focus on long-running streaming jobs, we only keep the jobs in Alibaba Cluster Dataset 2 that were running for more than 2,000 minutes following Mondal et al.. The details of datasets can be found in Appendix D.1. 1ftp://ita.ee.lbl.gov/html/contrib/Clark Net-HTTP.html 2https://github.com/alibaba/clusterdata |
| Dataset Splits | No | The paper states: 'For each job, we first train an autoscaler and then use it to perform testing on this job for ten times and take the average reward as the final result.' It does not explicitly define training, validation, and test dataset splits with percentages, counts, or references to predefined splits. |
| Hardware Specification | No | The paper discusses experiments on 'computing clusters' and a 'simulated computing system' but does not specify any particular hardware details such as GPU models, CPU types, or memory sizes used for the experiments. |
| Software Dependencies | No | The paper mentions using 'REINFORCE [Williams, 1992]' for training but does not provide specific version numbers for any software dependencies, libraries, or programming languages used. |
| Experiment Setup | Yes | The details of parameter settings and modifications of all compared methods are in Appendix D.2. |