MultiScan: Scalable RGBD scanning for 3D environments with articulated objects
Authors: Yongsen Mao, Yiming Zhang, Hanxiao Jiang, Angel Chang, Manolis Savva
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We validate our dataset on instance segmentation and part mobility estimation tasks and benchmark methods for these tasks from prior work. Our experiments show that part segmentation and mobility estimation in real 3D scenes remain challenging despite recent progress in 3D object segmentation. To allow for reproducible benchmarks, we split the data into approximately 70/15/15% train/val/test sets partitioned by scene (see supplement for statistics). |
| Researcher Affiliation | Academia | Yongsen Mao, Yiming Zhang, Hanxiao Jiang, Angel X. Chang, Manolis Savva Simon Fraser University |
| Pseudocode | No | The paper does not contain any pseudocode or algorithm blocks. |
| Open Source Code | Yes | In summary, we contribute: [...] We open-source our code and our data under the MIT license. |
| Open Datasets | Yes | We introduce Multi Scan, a scalable RGBD dataset construction pipeline leveraging commodity mobile devices to scan indoor scenes with articulated objects and web-based semantic annotation interfaces to efficiently annotate object and part semantics and part mobility parameters. We use this pipeline to collect 273 scans of 117 indoor scenes containing 10957 objects and 5129 parts. The resulting Multi Scan dataset provides RGBD streams with per-frame camera poses, textured 3D surface meshes, richly annotated part-level and object-level semantic labels, and part mobility parameters. We open-source our code and our data under the MIT license. |
| Dataset Splits | Yes | To allow for reproducible benchmarks, we split the data into approximately 70/15/15% train/val/test sets partitioned by scene (see supplement for statistics). |
| Hardware Specification | Yes | The processing steps above are done automatically for each uploaded scan on a processing server (Intel i9-10900F CPU, 32GB RAM, Nvidia RTX 3090Ti GPU). |
| Software Dependencies | No | The paper mentions software like Open3D, Instant Meshes, Mesh Lab, and Minkowski Engine but does not provide specific version numbers for these dependencies. |
| Experiment Setup | Yes | For each method, we apply the same settings for both object and part segmentation experiments. In our experiments, we use point clouds with position (xyz) and color (rgb) information. For SSTNet [24], we train using the original implementations with a learning rate of 1e-3, and Adam W [27] as the optimizer. For Point Group [19] and HAIS [4], we train using a re-implementation2 with Minkowski Engine [5] as the backbone with a learning rate of 1.5e-3, and Adam [21] as the optimizer. We apply data augmentation during training for all three methods: random jitter and mirror in the horizontal plane. We train all methods on the Multi Scan train split for 512 epochs. We set batch size 4 and 64 for object and part segmentation, respectively. |