Multi-Group Proportional Representation in Retrieval
Authors: Alex Oesterling, Claudio Mayrink Verdun, Alexander Glynn, Carol Long, Lucas Monteiro Paes, Sajani Vithana, Martina Cardone, Flavio Calmon
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate that existing methods optimizing for equal and proportional representation metrics may fail to promote MPR. Crucially, our work shows that optimizing MPR yields more proportional representation across multiple intersectional groups specified by a rich function class, often with minimal compromise in retrieval accuracy. Code is provided at https://github.com/alex-oesterling/multigroup-proportional-representation. and 5 Numerical Experiments In this section, we show that MOPR is effective in promoting more proportional representation across intersectional groups while preserving similarity between retrieved items and a given query. |
| Researcher Affiliation | Academia | Alex Oesterling Harvard University Claudio Mayrink Verdun Harvard University Carol Xuan Long Harvard University Alexander Glynn Harvard University Lucas Monteiro Paes Harvard University Sajani Vithana Harvard University Martina Cardone University of Minnesota Flavio P. Calmon Harvard University |
| Pseudocode | Yes | Algorithm 1 MOPR (Multi-group Optimized Proportional Retrieval) |
| Open Source Code | Yes | Code is provided at https://github.com/alex-oesterling/multigroup-proportional-representation. |
| Open Datasets | Yes | We conduct retrieval over three image datasets of faces: Celeb A [38], which includes labels for gender, age, and various other attributes, UTKFace [40], which contains gender, age, and race attributes, and Occupations [31], which contains gender attributes. We compute MPR for equal representation by constructing a synthetic dataset DC balanced over all attributes and for proportional representation using Fair Face [39] as the curated dataset DC |
| Dataset Splits | Yes | When training linear probes, we conduct a cross-validated hyperparameter search from 0.01 to 100 over ℓ2 regularized logistic regression models from scikit-learn [69] to find the optimal probe. |
| Hardware Specification | Yes | These experiments were conducted mainly on CPU after initially using a single A100 GPU to compute CLIP and Debias CLIP embeddings and run FAISS to compute the top 10k candidates for each. |
| Software Dependencies | No | To run MOPR, we use Gurobi [98] with an Academic License to solve Eqn. (9) each iteration of our cutting plane method and use CVXPY [99, 100] to solve our closed-form, quadratic program. To train our oracle estimators, we call scikit-learn s implementations of Linear Regression, Decision Trees with depth 3, and MLPs with one hidden layer of 64. The paper cites Gurobi's 2024 manual, implying a recent version, but does not provide specific version numbers for CVXPY or scikit-learn, which are key components. |
| Experiment Setup | Yes | Experimental Setup. We consider three classes of representation statistics C: linear regression, decision trees, and MLPs (the last two are presented in Appendix G). ... We retrieve k = 50 items for each of the above queries. For a given function class and query, we compute the baseline MPR and average cosine similarity given by the top 50 most similar items. Then, we conduct a parameter sweep over ρ starting from this max-MPR value, inputting each value to MOPR in Algorithm 1. and When training linear probes, we conduct a cross-validated hyperparameter search from 0.01 to 100 over ℓ2 regularized logistic regression models from scikit-learn [69] to find the optimal probe. ... To train our oracle estimators, we call scikit-learn s implementations of Linear Regression, Decision Trees with depth 3, and MLPs with one hidden layer of size 64. |