Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Disentangling Influence: Using disentangled representations to audit model predictions
Authors: Charles Marx, Richard Phillips, Sorelle Friedler, Carlos Scheidegger, Suresh Venkatasubramanian
NeurIPS 2019 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We show through both theory and experiments that disentangled influence audits can both detect proxy features and show, for each individual or in aggregate, which of these proxy features affects the classifier being audited the most. |
| Researcher Affiliation | Academia | Charles T. Marx Haverford College EMAIL Richard Lanas Phillips Cornell University EMAIL Sorelle A. Friedler Haverford College EMAIL Carlos Scheidegger University of Arizona EMAIL Suresh Venkatasubramanian University of Utah EMAIL |
| Pseudocode | Yes | DISENTANGLED-INFLUENCE-AUDIT(X, M) 1 for p in FEATURES(X) 2 (f , g, h) = DISENTANGLED-REPRESENTATION(X, p) // (h is not used) 3 M = g M 4 X = {f(x) for x in X} 5 SHAPp = DIRECT-INFLUENCE(X , p, M ) 6 return {SHAPp for p in FEATURES(X)} |
| Open Source Code | Yes | All data and code4 for the described method and below experiments is available in the Supplementary Materials. 4Code is available at: https://github.com/charliemarx/disentangling-influence |
| Open Datasets | Yes | It includes 5,000 instances of two variables x and y drawn independently from a uniform distribution over [0, 1]... The second synthetic dataset is the d Sprites dataset commonly used in the disentangled representations literature to disentangle independent factors that are sources of variation [23]... Finally, we will consider a real-world dataset containing Adult Income data that is commonly used as a test case in the fairness-aware machine learning community... [27]. |
| Dataset Splits | No | The paper does not provide specific details on validation dataset splits, such as percentages or sample counts. |
| Hardware Specification | Yes | The Titan Xp used for this research was donated by the NVIDIA Corporation. |
| Software Dependencies | No | The paper mentions software like 'shap' and 'Black Box Auditing' (available via pip install) but does not provide specific version numbers for these or any other software dependencies. |
| Experiment Setup | No | Preprocessing, model, and disentangled representation training information are included in the Supplementary Material. |