ReACTR: Realtime Algorithm Configuration through Tournament Rankings
Authors: Tadhg Fitzgerald, Yuri Malitsky, Barry O'Sullivan
IJCAI 2015 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate the Re ACTR methodology on three datasets. The first two are variations of combinatorial auction problems which were used in the evaluation of the original Re ACT configurator... For comparison we evaluate both versions of Re ACTR against a state-of-the-art static configurator, SMAC. ... Figure 5(a) shows the rolling average (total time to date/instances processed) on the circuit fuzz dataset... Finally, Table 1 shows the amount of time each configuration technique requires to go through the entire process. |
| Researcher Affiliation | Collaboration | Tadhg Fitzgerald Insight Centre for Data Analytics University College Cork, Ireland tadhg.fitzgerald@insight-centre.org Yuri Malitsky IBM T.J. Watson Research Centre, New York, USA ymalits@us.ibm.com Barry O Sullivan Insight Centre for Data Analytics University College Cork, Ireland barry.osullivan@insight-centre.org |
| Pseudocode | Yes | Algorithm 1 Components of the Re ACT algorithm |
| Open Source Code | No | The paper states: 'We use a highly-rated open source Python implementation of True Skill for our experiments [Zongker, 2014] using all of the default settings (µ = 25, σ=8.3 ).' and provides a link to that external library. However, it does not state that the code for Re ACTR, the methodology described in this paper, is open source or publicly available. |
| Open Datasets | Yes | We evaluate the Re ACTR methodology on three datasets. The first two are variations of combinatorial auction problems... generated using the Combinatorial Auction Test Suite (CATS) [Leyton-Brown et al., 2000]... The third dataset comes from the Configurable SAT Solver Competition (CSSC) 2013 [UBC, 2013]. This dataset was independently generated using Fuzz SAT [Brummayer et al., 2010]. |
| Dataset Splits | Yes | After this filtering the regions dataset contains 2,000 instances (split into a training set of 200 and a test set of 1800) while the arbitrary dataset has 1,422 instances (200 training and 1,222 test)... The resulting dataset contained 884 instances which was split into 299 training and 585 test. |
| Hardware Specification | Yes | All experiments were run on a system with 2 X Intel Xeon E5430 processors(2.66Ghz) and 12 GB RAM. |
| Software Dependencies | Yes | solved using the state-of-the-art commercial optimizer IBM CPLEX [IBM, 2014]. [IBM, 2014] refers to 'IBM ILOG CPLEX Optimization Studio 12.6.1'. |
| Experiment Setup | Yes | We use a highly-rated open source Python implementation of True Skill for our experiments [Zongker, 2014] using all of the default settings (µ = 25, σ=8.3 ). The solver time-out for both datasets is set to 500 seconds. We use a timeout of 300 seconds for this dataset. Furthermore, because Re ACTR uses six cores, six versions of SMAC are trained using the shared model mode option, which allows multiple SMAC runs to share information. |