Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..

Journal of Artificial Intelligence Research (JAIR) - 2023

Documentation Rate of Empirical Papers by Reproducibility Variable

Distribution of Empirical Papers by Number of Documented Variables

Website:

Venue Year Papers
Reproducibility Score Reproducibility Score based on Gundersen et al. (2025). See Methods for details.
Documentation Score Documentation Score is the average score over the seven reproducibility variables for empirical research papers. See Methods for details.
% Empirical Percentage of papers that are empirical research vs theoretical research.
% Industry Percentage of empirical research papers with at least one author from Industry.
Website
JAIR 2023 100 0.51 4.11 61.0% 19.67%
Pseudocode
Open Source Code
Open Datasets
Dataset Splits
Hardware Specification
Software Dependencies
Experiment Setup
A Benchmark Study on Knowledge Graphs Enrichment and Pruning Methods in the Presence of Noisy Relationships 4
A Comprehensive Survey on Deep Graph Representation Learning Methods 0
A General Model for Aggregating Annotations Across Simple, Complex, and Multi-Object Annotation Tasks 5
A Logic of East and West 1
A Markov Framework for Learning and Reasoning About Strategies in Professional Soccer 4
A Model to Support Collective Reasoning: Formalization, Analysis and Computational Assessment 4
A Practical Approach to Discretised PDDL+ Problems by Translation to Numeric Planning 6
A Survey of Learning Criteria Going Beyond the Usual Risk 0
A Survey of Zero-shot Generalisation in Deep Reinforcement Learning 0
A Survey on Understanding and Representing Privacy Requirements in the Internet-of-Things 0
A Unified Linear Speedup Analysis of Federated Averaging and Nesterov FedAvg 3
Achieving Zero Constraint Violation for Concave Utility Constrained Reinforcement Learning via Primal-Dual Approach 2
Actor Prioritized Experience Replay 7
Amortized Variational Inference: A Systematic Review 1
An Overview of Environmental Features that Impact Deep Reinforcement Learning in Sparse-Reward Domains 2
Asymptotics of K-Fold Cross Validation 3
Automatically Finding the Right Probabilities in Bayesian Networks 4
Certified Dominance and Symmetry Breaking for Combinatorial Optimisation 6
Characterizing Tseitin-Formulas with Short Regular Resolution Refutations 0
Classes of Hard Formulas for QBF Resolution 0
Clustering what Matters: Optimal Approximation for Clustering with Outliers 0
Collective Belief Revision 0
Competitive Equilibria with a Constant Number of Chores 1
Complexity of Computing the Shapley Value in Partition Function Form Games 0
Computational Modelling of Quantifier Use: Corpus, Models, and Evaluation 5
Contract Scheduling with Predictions 2
Coopetition Against an Amazon 1
Decentralized Gradient-Quantization Based Matrix Factorization for Fast Privacy-Preserving Point-of-Interest Recommendation 4
Deciding FO-rewritability of Regular Languages and Ontology-Mediated Queries in Linear Temporal Logic 0
Diagnosing AI Explanation Methods with Folk Concepts of Behavior 0
Distributed Bayesian: A Continuous Distributed Constraint Optimization Problem Solver 4
Domain-Specific Heuristics in Answer Set Programming: A Declarative Non-Monotonic Approach 6
Dynamic Controllability of Temporal Plans in Uncertain and Partially Observable Environments 1
Efficient Multi-Goal Reinforcement Learning via Value Consistency Prioritization 5
Efficiently Explaining CSPs with Unsatisfiable Subset Optimization 6
Embedding Ontologies in the Description Logic ALC by Axis-Aligned Cones 0
Equivalence in Argumentation Frameworks with a Claim-centric View: Classical Results with Novel Ingredients 0
Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning 4
Exploiting Cultural Biases via Homoglyphs in Text-to-Image Synthesis 6
Exploiting Functional Constraints in Automatic Dominance Breaking for Constraint Optimization 5
FactGen: Faithful Text Generation by Factuality-aware Pre-training and Contrastive Ranking Fine-tuning 4
Fair Influence Maximization in Large-scale Social Networks Based on Attribute-aware Reverse Influence Sampling 4
Fair and Efficient Allocation of Scarce Resources Based on Predicted Outcomes: Implications for Homeless Service Delivery 3
Fairness in Forecasting of Observations of Linear Dynamical Systems 6
Favoring Eagerness for Remaining Items: Designing Efficient, Fair, and Strategyproof Mechanisms 1
Finite Materialisability of Datalog Programs with Metric Temporal Operators 1
First-Order Context-Specific Likelihood Weighting in Hybrid Probabilistic Logic Programs 4
FlexiBERT: Are Current Transformer Architectures too Homogeneous and Rigid? 6
FlexiBO: A Decoupled Cost-Aware Multi-Objective Optimization Approach for Deep Neural Networks 6
Generalizing Group Fairness in Machine Learning via Utilities 3
Generating Random SAT Instances: Multiple Solutions could be Predefined and Deeply Hidden 3
Graphmax for Text Generation 6
Hierarchical Decompositions and Termination Analysis for Generalized Planning 3
How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy 3
How to Tell Easy from Hard: Complexities of Conjunctive Query Entailment in Extensions of ALC 1
Improved Peel-and-Bound: Methods for Generating Dual Bounds with Multivalued Decision Diagrams 5
Improving Resource Allocations by Sharing in Pairs 1
Information Lattice Learning 4
Introduction to the Special Track on Artificial Intelligence and COVID-19 0
Liability Regimes in the Age of AI: a Use-Case Driven Analysis of the Burden of Proof 0
Lifted Reasoning for Combinatorial Counting 5
MDP Playground: An Analysis and Debug Testbed for Reinforcement Learning 6
Maintenance of Plan Libraries for Case-Based Planning: Offline and Online Policies 5
Maximisation of Admissible Multi-Objective Heuristics 3
Measuring Fairness Under Unawareness of Sensitive Attributes: A Quantification-Based Approach 5
Mimicking Behaviors in Separated Domains 0
Mining ℰℒ⊥ Bases with Adaptable Role Depth 2
Negative Human Rights as a Basis for Long-term AI Safety and Regulation 0
Non-Crossing Anonymous MAPF for Tethered Robots 4
Object-agnostic Affordance Categorization via Unsupervised Learning of Graph Embeddings 4
On Centralized Critics in Multi-Agent Reinforcement Learning 3
On Dynamics in Structured Argumentation Formalisms 0
On Expected Value Strong Controllability 5
On Fair Division under Heterogeneous Matroid Constraints 1
On the Complexity of Finding Set Repairs for Data-Graphs 1
On the Evaluation of (Meta-)solver Approaches 2
On the Parallel Parameterized Complexity of MaxSAT Variants 1
Online Bin Packing with Predictions 4
Optimal and Efficient Auctions for the Gradual Procurement of Strategic Service Provider Agents 2
Optimality Guarantees for Particle Belief Approximation of POMDPs 4
Prediction of Social Dynamic Agents and Long-Tailed Learning Challenges: A Survey 1
Program Synthesis with Best-First Bottom-Up Search 4
QNLP in Practice: Running Compositional Models of Meaning on a Quantum Computer 5
Qualitative Reasoning about 2D Cardinal Directions using Answer Set Programming 6
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text 1
Reviewer Assignment Problem: A Systematic Review of the Literature 0
Robust Control for Dynamical Systems with Non-Gaussian Noise via Formal Abstractions 4
SAlign: A Graph Neural Attention Framework for Aligning Structurally Heterogeneous Networks 5
Scalable Neural-Probabilistic Answer Set Programming 5
Select and Augment: Enhanced Dense Retrieval Knowledge Graph Augmentation 3
Semiring Reasoning Frameworks in AI and Their Computational Complexity 1
Sequence-Oriented Diagnosis of Discrete-Event Systems 4
Stackelberg Security Games with Contagious Attacks on a Network: Reallocation to the Rescue 3
The Complexity of Matching Games: A Survey 0
The Jiminy Advisor: Moral Agreements among Stakeholders Based on Norms and Argumentation 0
Towards Green Automated Machine Learning: Status Quo and Future Directions 1
Viewpoint: Artificial Intelligence Accidents Waiting to Happen? 0
Visualizing the Implicit Model Selection Tradeoff 4
Your College Dorm and Dormmates: Fair Resource Sharing with Externalities 1
Your Prompt is My Command: On Assessing the Human-Centred Generality of Multimodal Models 0