Reparameterizable Subset Sampling via Continuous Relaxations
Authors: Sang Michael Xie, Stefano Ermon
IJCAI 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | 4 Experiments 4.1 Synthetic Experiments We check that generating samples from Algorithm 2 results in samples approximately from p(S|w). We define a subset distribution using weights w = [0.1, 0.2, 0.3, 0.4] and take subset size k = 2. Using Algorithm 2 and the top-k relaxation from [Pl otz and Roth, 2018], we sample 10000 relaxed k-hot samples for each temperature in {0.1, 1, 10} and take the the top-k values in the relaxed k-hot vector as the chosen subset. We plot the empirical histogram of subset occurrences and compare with the true probabilities from the subset distribution, with 95% confidence intervals. The relaxation produces subset samples with empirical distribution within 0.016 in total variation distance of p(S|w) for all t. This agrees with Theorem 1, which states that even for higher temperatures, taking the top-k values in the relaxed k-hot vector should produce true samples from (2). |
| Researcher Affiliation | Academia | Sang Michael Xie and Stefano Ermon Stanford University {xie, ermon}@cs.stanford.edu |
| Pseudocode | Yes | Algorithm 1 Weighted Reservoir Sampling (non-streaming) Input: Items x1, . . . , xn, weights w = [w1, . . . , wn], reservoir size k Output: Swrs = [ei1, . . . , eik] a sample from p(Swrs|w) 1: r [ ] 2: for i 1 to n do 3: ui Uniform(0, 1) 4: ri u1/wi i # Sample random keys 5: r.append(ri) 6: end for 7: [ei1, . . . , eik] Top K(r, k) 8: return [ei1, . . . , eik] |
| Open Source Code | Yes | 1Code available at https://github.com/ermongroup/subsets. |
| Open Datasets | Yes | We test our results on the Large Movie Review Dataset (IMDB) for sentiment classification [Maas et al., 2011]... We compare with parametric t-SNE [van der Maaten, 2009] on the MNIST [Le Cun and Cortes, 2010] and a small version of the 20 Newsgroups dataset [Roweis, 2009]2. |
| Dataset Splits | Yes | We use cross validation to choose temperatures t {0.1, 0.5, 1, 2, 5} according to the validation loss. ...and search over temperatures t = {0.1, 1, 5, 16, 64} using the validation set... |
| Hardware Specification | Yes | Results were obtained from a Titan Xp GPU. |
| Software Dependencies | No | The paper does not provide specific software dependency versions (e.g., Python, PyTorch, TensorFlow versions). |
| Experiment Setup | Yes | Following L2X, we use k = 10 for IMDB-word and k = 1 sentences for IMDB-sent. ...We fix k = 9 nearest neighbors to choose from m candidates and search over temperatures t = {0.1, 1, 5, 16, 64} using the validation set... For all experiments, we set t = 0.1 and train for 200 epochs with a batch size of 1000... |