Active Ranking of Experts Based on their Performances in Many Tasks
Authors: El Mehdi Saad, Nicolas Verzelen, Alexandra Carpentier
ICML 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this section, we perform some numerical simulations on synthetic data to compare our Algorithm with a benchmark procedure from the literature. |
| Researcher Affiliation | Academia | 1INRAE, Mistea, Institut Agro, Univ Montpellier, Montpellier, France. 2Institut f ur Mathematik, Universit at Potsdam, Germany. |
| Pseudocode | Yes | Algorithm 1 Compare(δ, ϵ)”,”Algorithm 2 Try-compare(δ, s, h)”,”Algorithm 3 Active-ranking(δ)”,”Algorithm 4 Best-expert(δ)”,”Algorithm 5 Binary-search(δ, i, r, a, b)”,”Algorithm 6 Max-search(δ, S, ϵ) |
| Open Source Code | No | The paper does not contain an explicit statement about releasing source code for the described methodology, nor does it provide a link to a code repository. |
| Open Datasets | No | We perform some numerical simulations on synthetic data... The means of performances of the suboptimal expert Mπ(2) are drawn from [0, 1/2] following the uniform law. We build a gaps vector s that is s-sparse, the non-zero tasks are drawn uniformly at random from Jd K, and the value of the kth non-zero gap is set to k 3s 2. Then we consider Mπ(1) = Mπ(2) + s. |
| Dataset Splits | No | The paper describes numerical simulations on synthetic data but does not mention specific training, validation, or test dataset splits, cross-validation, or any other data partitioning methodology. |
| Hardware Specification | No | The paper does not specify any hardware details (e.g., CPU, GPU models, memory, or cloud resources) used for conducting the numerical simulations. |
| Software Dependencies | No | The paper mentions comparing its algorithm with the AR algorithm from (Heckel et al., 2019) but does not provide specific software dependencies or their version numbers used in the numerical simulations. |
| Experiment Setup | Yes | We focus on the specific problem of identifying the best out of two experts (n = 2) and d = 10 tasks. For each s Jd K, we consider the following scenario: the performances of both experts in each task follow a normal distribution with unit variance. The means of performances of the suboptimal expert Mπ(2) are drawn from [0, 1/2] following the uniform law. We build a gaps vector s that is s-sparse, the non-zero tasks are drawn uniformly at random from Jd K, and the value of the kth non-zero gap is set to k 3s 2. Then we consider Mπ(1) = Mπ(2) + s. Figure7 presents the sample complexity of Algorithm 1 with parameters (δ, 0) and AR from (Heckel et al., 2019), as a function of the sparsity ratio s/d for s Jd K. The results are averaged over 20 simulations for each scenario, in all simulations both algorithms made the correct output. [...] We varied the number of tasks d [4, 8, 16, 32, 64] while keeping the sparsity rate constant 1/3. |