AttNS: Attention-Inspired Numerical Solving For Limited Data Scenarios
Authors: Zhongzhan Huang, Mingfu Liang, Shanshan Zhong, Liang Lin
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our results on benchmarks, ranging from highdimensional problems to chaotic systems, showcase Att NS consistently enhancing various numerical solvers without any intricate model crafting. Next, in Section 5, we conduct experimental and theoretical analyses of Att NS, proving its strong generalization and robustness abilities akin to the attention mechanism in conventional deep learning tasks, while ensuring solver convergence. |
| Researcher Affiliation | Academia | 1Sun Yat-sen University, China 2Peng Cheng Laboratory, China 3Northwestern University, USA. |
| Pseudocode | Yes | Algorithm 1 The processing of Att NS and Att NS-m. |
| Open Source Code | No | https://github.com/dedekinds/NeurVec. (This link is for a baseline method, NeurVec, not the proposed Att NS.) |
| Open Datasets | Yes | Table 3: Summary of the datasets mentioned in this paper. Benchmark Type Dimension Data size Step size Generative Method Evaluation time T Spring-mass Train * 5k 1e-3 * 20 Spring-mass Validation * 0.1k 1e-5 RK4 20 Spring-mass Test * 5k 1e-5 RK4 20 2-link pendulum Train 4 20k 1e-3 RK4 10 2-link pendulum Validation 4 1k 1e-5 RK4 10 2-link pendulum Test 4 10k 1e-5 RK4 10 Elastic pendulum Train 4 20k 1e-3 RK4 50 Elastic pendulum Validation 4 1k 1e-5 RK4 50 Elastic pendulum Test 4 10k 1e-5 RK4 50 |
| Dataset Splits | Yes | Table 3: Summary of the datasets mentioned in this paper. ... Spring-mass Validation * 0.1k 1e-5 RK4 20 ... 2-link pendulum Validation 4 1k 1e-5 RK4 10 ... Elastic pendulum Validation 4 1k 1e-5 RK4 50 |
| Hardware Specification | No | The paper does not provide any specific hardware details such as GPU or CPU models, memory, or cloud instance types used for running the experiments. |
| Software Dependencies | No | The paper discusses various numerical methods and deep learning concepts (e.g., “Neural Networks”, “attention module”), but it does not specify any software libraries or dependencies with version numbers (e.g., “PyTorch 1.9”, “Python 3.8”, “CUDA 11.1”). |
| Experiment Setup | Yes | Input: A coarse step size tc; The number of step N which satisfied the evaluation time T = N tc; A given equation du/dt = f(u), u(0) = c0; A given numerical integration scheme S; The high-quality dataset D(traj(u)). The attention module Q[ |ϕ]; The learning rate η. We set d1 = 1024 and h = 2 by default. We adopt the coarse step size t = 2e 1 for the numerical solver and the fine step size 1e 3 for training the AHS. Table 4: The initialization of different benchmarks. |