Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
OSKAR: Omnimodal Self-supervised Knowledge Abstraction and Representation
Authors: Mohamed Abdelfattah, Kaouther Messaoud, Alexandre Alahi
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experimental results show that OSKAR s unified pretrained encoder outperforms models with specialized architectures of similar size in action recognition (rgb, skeleton, frozen, low-shot) and localization, video-text retrieval, and video question answering. Project website: https://multimodal-oskar.github.io |
| Researcher Affiliation | Academia | Mohamed Abdelfattah Kaouther Messaoud Alexandre Alahi École Polytechnique Fédérale de Lausanne (EPFL), Switzerland {firstname.lastname}@epfl.ch |
| Pseudocode | No | The paper describes methods in narrative text and uses figures to illustrate architecture (e.g., Figure 2) but does not include any explicitly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | An anonymized version of our code is available as part of the supplementary materials. Codes will be opensourced to the community upon publication. |
| Open Datasets | Yes | We train OSKAR entirely with pseudo-labels on 10M videos from Open Human Vid3 [49] (13.2M videos, 16.7K hours). At the submission time of this paper, only 10M videos from Open Human Vid were publicly released. Downstream benchmarks include: Kinetics400 [45] and Something-Something V2 [37] (RGB action recognition), NTU60 [66] and NTU120 [54] (skeleton action recognition), AVA [39] (action localization), MSRVTT [84]/MSVD [16]/VATEX [79] (text-video retrieval), and MSRVTT-QA [84]/MSVD-QA [82]/TGIF-Frame QA [52] (video QA). |
| Dataset Splits | Yes | Downstream benchmarks include: Kinetics400 [45] and Something-Something V2 [37] (RGB action recognition), NTU60 [66] and NTU120 [54] (skeleton action recognition), AVA [39] (action localization), MSRVTT [84]/MSVD [16]/VATEX [79] (text-video retrieval), and MSRVTT-QA [84]/MSVD-QA [82]/TGIF-Frame QA [52] (video QA). on NTU60 and NTU120 XSub (Tab. 4) |
| Hardware Specification | Yes | Pretraining uses 256 GH200 GPUs. |
| Software Dependencies | No | The paper mentions specific tools and models used in data processing and pre-tokenization (e.g., YOLO11 [44], Mini CPM [87], Cog VLM [41], V-JEPA [12], Motion BERT [92], Word Piece [69]) and specific algorithms like Adam W [58], but it does not provide version numbers for any core software libraries or frameworks like PyTorch, TensorFlow, or Python. |
| Experiment Setup | Yes | We use standard Vi T-S, Vi T-B, and Vi T-L [24] backbones with learnable positional encodings. All target encoders use a shared EMA update parameter (λ = 0.998). Models are randomly initialized and trained on 500B tokens (10B warmup) using Adam W [58] (β1 = 0.9, β2 = 0.95), a base learning rate of 1e-4, cosine decay, batch size 8192, and weight decay 0.05. Transformers use Swi GLU [67] activations and bfloat16 [14] precision. Each model processes N s = N t = 128 tokens per step, with aggressive masking (<5%, 128 of 2640 tokens visible) via non-overlapping modality masks and Dirichlet-sampled allocation ratios (α = 0.5). |