Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..

Journal of Artificial Intelligence Research (JAIR) - 2020

Documentation Rate of Empirical Papers by Reproducibility Variable

Distribution of Empirical Papers by Number of Documented Variables

Website:

Venue Year Papers
Reproducibility Score Reproducibility Score based on Gundersen et al. (2025). See Methods for details.
Documentation Score Documentation Score is the average score over the seven reproducibility variables for empirical research papers. See Methods for details.
% Empirical Percentage of papers that are empirical research vs theoretical research.
% Industry Percentage of empirical research papers with at least one author from Industry.
Website
JAIR 2020 84 0.48 3.83 71.43% 16.67%
Pseudocode
Open Source Code
Open Datasets
Dataset Splits
Hardware Specification
Software Dependencies
Experiment Setup
A Differential Privacy Mechanism that Accounts for Network Effects for Crowdsourcing Systems βœ… ❌ βœ… ❌ ❌ ❌ βœ… 3
A General Approach to Multimodal Document Quality Assessment ❌ βœ… βœ… βœ… ❌ ❌ βœ… 4
A Global Constraint for the Exact Cover Problem: Application to Conceptual Clustering βœ… ❌ βœ… ❌ βœ… βœ… βœ… 5
A Set of Recommendations for Assessing Human–Machine Parity in Language Translation ❌ ❌ βœ… βœ… ❌ ❌ βœ… 3
AMP Chain Graphs: Minimal Separators and Structure Learning Algorithms βœ… βœ… βœ… ❌ ❌ ❌ βœ… 4
ASNets: Deep Learning for Generalised Planning βœ… βœ… βœ… βœ… βœ… ❌ βœ… 6
Adapting Behavior via Intrinsic Reward: A Survey and Empirical Study βœ… βœ… βœ… ❌ ❌ ❌ βœ… 4
Adaptive Stress Testing: Finding Likely Failure Events with Reinforcement Learning βœ… βœ… βœ… ❌ βœ… ❌ βœ… 5
Adversarial Attacks on Crowdsourcing Quality Control ❌ βœ… βœ… ❌ ❌ ❌ βœ… 3
Agreement on Target-Bidirectional Recurrent Neural Networks for Sequence-to-Sequence Learning βœ… βœ… βœ… βœ… βœ… ❌ βœ… 6
An Evaluation of Communication Protocol Languages for Engineering Multiagent Systems ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
Annotator Rationales for Labeling Tasks in Crowdsourcing βœ… ❌ βœ… βœ… ❌ ❌ βœ… 4
Automated Conjecturing II: Chomp and Reasoned Game Play ❌ βœ… ❌ ❌ βœ… ❌ ❌ 2
Belief change and 3-valued logics: Characterization of 19,683 belief change operators ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
Best-First Enumeration Based on Bounding Conflicts, and its Application to Large-scale Hybrid Estimation βœ… ❌ βœ… ❌ βœ… ❌ βœ… 4
Blind Spot Detection for Safe Sim-to-Real Transfer βœ… βœ… ❌ ❌ ❌ ❌ βœ… 3
Bounds on the Size of PC and URC Formulas βœ… βœ… ❌ ❌ ❌ ❌ ❌ 2
Bridging the Gap Between Probabilistic Model Checking and Probabilistic Planning: Survey, Compilations, and Empirical Comparison ❌ βœ… βœ… ❌ βœ… βœ… βœ… 5
Catching Cheats: Detecting Strategic Manipulation in Distributed Optimisation of Electric Vehicle Aggregators βœ… ❌ βœ… ❌ ❌ ❌ βœ… 3
Compositionality Decomposed: How do Neural Networks Generalise? ❌ βœ… βœ… βœ… ❌ ❌ βœ… 4
Computing Bayes-Nash Equilibria in Combinatorial Auctions with Verification βœ… βœ… ❌ ❌ βœ… ❌ βœ… 4
Conservative Extensions in Horn Description Logics with Inverse Roles ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
Constraint and Satisfiability Reasoning for Graph Coloring βœ… βœ… βœ… ❌ βœ… βœ… βœ… 6
Contiguous Cake Cutting: Hardness Results and Approximation Algorithms βœ… ❌ ❌ ❌ ❌ ❌ ❌ 1
Contrasting the Spread of Misinformation in Online Social Networks βœ… ❌ βœ… βœ… βœ… βœ… βœ… 6
Credibility-limited Base Revision: New Classes and Their Characterizations ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
Deep Reinforcement Learning: A State-of-the-Art Walkthrough ❌ ❌ βœ… ❌ ❌ ❌ ❌ 1
Diagnosis of Deep Discrete-Event Systems ❌ ❌ ❌ ❌ βœ… ❌ ❌ 1
Epistemic Argumentation Framework: Theory and Computation βœ… ❌ ❌ ❌ ❌ ❌ ❌ 1
Fair Allocation with Diminishing Differences βœ… βœ… ❌ ❌ ❌ ❌ βœ… 3
Gradient-based Learning Methods Extended to Smooth Manifolds Applied to Automated Clustering βœ… ❌ βœ… βœ… βœ… βœ… βœ… 6
Graph Width Measures for CNF-Encodings with Auxiliary Variables ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
HTN Planning as Heuristic Progression Search βœ… βœ… βœ… ❌ βœ… ❌ βœ… 5
Hedonic Games with Ordinal Preferences and Thresholds βœ… ❌ ❌ ❌ ❌ ❌ ❌ 1
How to Do Things with Words: A Bayesian Approach ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
Image Captioning using Facial Expression and Attention ❌ ❌ βœ… βœ… ❌ ❌ βœ… 3
Improved High Dimensional Discrete Bayesian Network Inference using Triplet Region Construction βœ… βœ… βœ… ❌ βœ… βœ… βœ… 6
Improving Nash Social Welfare Approximations βœ… ❌ ❌ ❌ ❌ ❌ ❌ 1
Incompatibilities Between Iterated and Relevance-Sensitive Belief Revision ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
Incomplete Preferences in Single-Peaked Electorates βœ… βœ… βœ… ❌ βœ… ❌ ❌ 4
Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog ❌ βœ… ❌ βœ… ❌ ❌ βœ… 3
Learning the Language of Software Errors βœ… ❌ βœ… ❌ ❌ βœ… βœ… 4
Lifted Bayesian Filtering in Multiset Rewriting Systems βœ… ❌ βœ… ❌ ❌ ❌ βœ… 3
Mapping the landscape of Artificial Intelligence applications against COVID-19 ❌ ❌ βœ… ❌ ❌ ❌ ❌ 1
Maximin Share Allocations on Cycles βœ… ❌ ❌ ❌ ❌ ❌ ❌ 1
Modular Structures and Atomic Decomposition in Ontologies βœ… ❌ βœ… ❌ βœ… βœ… βœ… 5
Neural Machine Translation: A Review βœ… ❌ βœ… βœ… ❌ ❌ ❌ 3
On Sparse Discretization for Graphical Games ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
On the Complexity of Learning a Class Ratio from Unlabeled Data ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
Ontology Reasoning with Deep Neural Networks βœ… βœ… βœ… βœ… ❌ ❌ βœ… 5
Planning High-Level Paths in Hostile, Dynamic, and Uncertain Environments βœ… ❌ βœ… ❌ βœ… βœ… βœ… 5
Planning for Hybrid Systems via Satisfiability Modulo Theories βœ… βœ… βœ… ❌ ❌ βœ… βœ… 5
Point at the Triple: Generation of Text Summaries from Knowledge Base Triples ❌ βœ… βœ… βœ… βœ… ❌ βœ… 5
Predicting Strategic Behavior from Free Text ❌ βœ… βœ… βœ… ❌ ❌ βœ… 4
Preferences Single-Peaked on a Circle βœ… ❌ ❌ ❌ ❌ ❌ ❌ 1
Properties of Switch-List Representations of Boolean Functions ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
Qualitative Numeric Planning: Reductions and Complexity βœ… βœ… ❌ ❌ ❌ βœ… ❌ 3
Regret Bounds for Reinforcement Learning via Markov Chain Concentration βœ… ❌ ❌ ❌ ❌ ❌ ❌ 1
Representing Fitness Landscapes by Valued Constraints to Understand the Complexity of Local Search ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
Reviewing Autoencoders for Missing Data Imputation: Technical Trends, Applications and Outcomes ❌ ❌ βœ… ❌ ❌ ❌ ❌ 1
Robust Multi-Agent Path Finding and Executing ❌ βœ… βœ… ❌ βœ… ❌ βœ… 4
Saturated Cost Partitioning for Optimal Classical Planning βœ… βœ… βœ… ❌ ❌ ❌ βœ… 4
Scalable Planning with Deep Neural Network Learned Transition Models ❌ ❌ ❌ βœ… βœ… βœ… βœ… 4
Simulating Offender Mobility: Modeling Activity Nodes from Large-Scale Human Activity Data ❌ βœ… βœ… βœ… ❌ ❌ βœ… 4
Sliding-Window Thompson Sampling for Non-Stationary Settings βœ… ❌ ❌ ❌ ❌ ❌ βœ… 2
Solving Delete Free Planning with Relaxed Decision Diagram Based Heuristics βœ… βœ… βœ… ❌ ❌ βœ… βœ… 5
Structure from Randomness in Halfspace Learning with the Zero-One Loss ❌ ❌ βœ… βœ… ❌ ❌ βœ… 3
Subgoaling Techniques for Satisficing and Optimal Numeric Planning βœ… βœ… βœ… ❌ βœ… βœ… βœ… 6
TensorLog: A Probabilistic Database Implemented Using Deep-Learning Infrastructure βœ… ❌ βœ… βœ… βœ… ❌ βœ… 5
The 2^k Neighborhoods for Grid Path Planning βœ… ❌ βœ… ❌ βœ… ❌ βœ… 4
The Bottleneck Simulator: A Model-Based Deep Reinforcement Learning Approach ❌ ❌ βœ… βœ… βœ… ❌ βœ… 4
The Complexity Landscape of Outcome Determination in Judgment Aggregation ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
The Effects of Experience on Deception in Human-Agent Negotiation ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
The Force Awakens: Artificial Intelligence for Consumer Law ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
The Impact of Treewidth on Grounding and Solving of Answer Set Programs βœ… ❌ βœ… βœ… ❌ βœ… βœ… 5
The PETLON Algorithm to Plan Efficiently for Task-Level-Optimal Navigation βœ… ❌ ❌ ❌ βœ… ❌ βœ… 3
The Parameterized Complexity of Motion Planning for Snake-Like Robots βœ… ❌ ❌ ❌ ❌ ❌ ❌ 1
To Regulate or Not: A Social Dynamics Analysis of an Idealised AI Race ❌ ❌ ❌ ❌ ❌ ❌ βœ… 1
Towards Knowledgeable Supervised Lifelong Learning Systems βœ… βœ… βœ… βœ… ❌ ❌ βœ… 5
Towards Partial Order Reductions for Strategic Ability βœ… ❌ βœ… ❌ ❌ ❌ ❌ 2
Using Machine Learning for Decreasing State Uncertainty in Planning ❌ βœ… βœ… βœ… ❌ ❌ βœ… 4
Using Task Descriptions in Lifelong Machine Learning for Improved Performance and Zero-Shot Transfer βœ… ❌ ❌ ❌ βœ… ❌ βœ… 3
Variational Bayes In Private Settings (VIPS) βœ… βœ… βœ… βœ… ❌ ❌ βœ… 5
Vocabulary Alignment in Openly Specified Interactions ❌ ❌ ❌ ❌ ❌ ❌ βœ… 1