Learning a Single Neuron with Bias Using Gradient Descent
Authors: Gal Vardi, Gilad Yehudai, Ohad Shamir
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | We theoretically study the fundamental problem of learning a single neuron with a bias term (x 7 σ( w, x + b)) in the realizable setting with the Re LU activation, using gradient descent. Perhaps surprisingly, we show that this is a significantly different and more challenging problem than the bias-less case (which was the focus of previous works on single neurons), both in terms of the optimization geometry as well as the ability of gradient methods to succeed in some scenarios. We provide a detailed study of this problem, characterizing the critical points of the objective, demonstrating failure cases, and providing positive convergence guarantees under different sets of assumptions. To prove our results, we develop some tools which may be of independent interest, and improve previous results on learning single neurons. [...] If you ran experiments... n/a |
| Researcher Affiliation | Academia | Gal Vardi Weizmann Institute of Science gal.vardi@weizmann.ac.il Gilad Yehudai Weizmann Institute of Science gilad.yehudai@weizmann.ac.il Ohad Shamir Weizmann Institute of Science ohad.shamir@weizmann.ac.il |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. It is a theoretical paper. |
| Open Source Code | No | The NeurIPS Checklist for this paper states: "If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... n/a". This indicates no open-source code is provided for the methodology. |
| Open Datasets | No | The NeurIPS Checklist for this paper states: "If you ran experiments... n/a". This indicates that no empirical studies with datasets were conducted, thus no publicly available dataset was used for training purposes in this paper's experiments. |
| Dataset Splits | No | The NeurIPS Checklist for this paper states: "If you ran experiments... n/a". This indicates no empirical studies were conducted, thus no training/validation/test splits are described. |
| Hardware Specification | No | The NeurIPS Checklist for this paper states: "If you ran experiments... n/a". Therefore, no hardware specifications for running experiments are provided. |
| Software Dependencies | No | The NeurIPS Checklist for this paper states: "If you ran experiments... n/a". Therefore, no specific software dependencies with version numbers are provided for experiment replication. The mention of 'Pytorch [Paszke et al., 2019]' refers to a general deep learning library, not a specific dependency for their work. |
| Experiment Setup | No | The NeurIPS Checklist for this paper states: "If you ran experiments... n/a". Therefore, no experimental setup details like hyperparameters or training settings are provided. |