Automatic Generation of Raven’s Progressive Matrices
Authors: Ke Wang, Zhendong Su
IJCAI 2015 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We have realized our approach and evaluated its efficiency and effectiveness. We show that our system can generate hundreds of valid problems per second with varying levels of difficulty. More importantly, we show, via a user study with 24 participants, that the generated problems are statistically indistinguishable from actual problems. |
| Researcher Affiliation | Academia | Ke Wang Zhendong Su Department of Computer Science University of California, Davis {kbwang,su}@ucdavis.edu |
| Pseudocode | Yes | Algorithm 1: Matrix generation |
| Open Source Code | No | The paper does not provide concrete access to the source code for the methodology. The URL provided (http://www.cs.ucdavis.edu/~su/rpm.html) is for a sample of synthesized problems, not the code itself. |
| Open Datasets | No | Due to copyright constraints, we do not include the APM problems that we purchased from Pearson at http://us.talentlens.com/pricing#ravens. |
| Dataset Splits | Yes | Each participant was given two test sets, each containing 30 problems randomly sampled from each of the two problem sets. Problems in the APM test are not uniformly distributed in terms of difficulty (there are 21 problems with one RPM constraint, 19 problems with two RPM constraints, and 5 with three constraints). So, we control the sampling process of the synthesized problems to maintain the correlation between the two sets. Specifically, we allow no more than 10 problems with three RPM constraints in the synthesized problem set. and we randomly partitioned the APM problems within each category that a participant has been tested on into two halves and summarized the participant s error rates for each half. |
| Hardware Specification | Yes | We conducted our experiments on a desktop with a 4th generation Intel Core i7-4770 processor and 16GB RAM, running Ubuntu 12.04 LTS. |
| Software Dependencies | Yes | We implement our RPM synthesis algorithm in Java SE 1.6 and interface with SVG technology [Ferraiolo, 2000] to render the figural elements in the browser. |
| Experiment Setup | Yes | Participants We invited twenty-four volunteers in total from our institution to participate in the study. ... Each participant was given two test sets, each containing 30 problems randomly sampled from each of the two problem sets. ... We control the sampling process of the synthesized problems to maintain the correlation between the two sets. Specifically, we allow no more than 10 problems with three RPM constraints in the synthesized problem set. ... The total time is 30 minutes for each of the test set. Half of the participants completed the APM test set before beginning the other test set, and the other half completed the test set with synthesized problems first. |