Can Neural Networks Learn Implicit Logic from Physical Reasoning?
Authors: Aaron Traylor, Roman Feiman, Ellie Pavlick
ICLR 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We design a set of experiments in which models are trained to track objects as they move throughout a visual scene, and then evaluated on a task from developmental psychology that requires reasoning about the location of a hidden object and is considered to be a face-valid test for the representation of (implicit) negation and disjunction. We find that, by most measures, object-tracking neural networks are unable to generalize zero-shot to the logical reasoning test, even when given training data which directly illustrates the requisite reasoning pattern. |
| Researcher Affiliation | Academia | Aaron Traylor,1 Roman Feiman,2 & Ellie Pavlick1 1 Department of Computer Science 2 Department of Cognitive, Linguistic & Psychological Sciences Brown University, Providence, Rhode Island, USA {aaron_traylor, roman_feiman, ellie_pavlick}@brown.edu |
| Pseudocode | No | The paper does not contain structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide concrete access to source code (specific repository link, explicit code release statement, or code in supplementary materials) for the methodology described in this paper. |
| Open Datasets | No | The paper describes generating its own training data (e.g., 'Physical Reasoning Schema', 'Two-Cup Ablations Training Schema', 'Invisible Displacement Training Schema') but does not provide concrete access information (link, DOI, repository, or formal citation with authors/year) for accessing these datasets or specify that they are publicly available. |
| Dataset Splits | No | The paper mentions training data size ('5,000 videos') and dev set size ('100 examples', '50 examples') but does not specify exact split percentages, sample counts for each split, or provide details on the splitting methodology needed to reproduce the data partitioning. |
| Hardware Specification | No | The paper does not provide specific hardware details (exact GPU/CPU models, processor types with speeds, memory amounts, or detailed computer specifications) used for running its experiments. |
| Software Dependencies | No | The paper mentions 'pygame and Box2D' but does not provide specific version numbers for these or any other key software components, libraries, or solvers needed to replicate the experiment. |
| Experiment Setup | No | The paper states 'Model training details are in the Appendix.' but these details, including concrete hyperparameter values or training configurations, are not present in the main text. |