Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks

Дата публикации: 17-08-2026 20:26:00

We study the implicit bias of the general family of steepest descent algorithms with infinitesimal learning rate in deep homogeneous neural networks. We show that: (a) an algorithm-dependent geometric margin starts increasing once the networks reach perfect training accuracy, and (b) any limit point of the training trajectory corresponds to a KKT point of the corresponding margin-maximization problem. We experimentally zoom into the trajectories of neural networks optimized with various steepest descent algorithms, highlighting connections to the implicit bias of popular adaptive methods (Adam and Shampoo).

Схожие новости

#	Наименование новости	Тональность	Информативность	Дата публикации
1	Optimization and Generalization of Gradient Descent for Shallow ReLU Networks with Minimal Width	0	3.84	17-08-2026
2	Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data	0	4.36	17-08-2026
3	A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks	0	10.97	17-08-2026
4	A Mean-Field Analysis of Neural Stochastic Gradient Descent-Ascent for Functional Minimax Optimization	0	9.82	17-08-2026
5	Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selection	0	6.96	17-08-2026
6	Reparameterized Complex-valued Neurons Can Efficiently Learn More than Real-valued Neurons via Gradient Descent	0	11.86	17-08-2026
7	Optimal Approximation and Generalization Errors for Deep Convolutional Neural Networks	0	7.31	17-08-2026
8	A Natural Primal-Dual Hybrid Gradient Method for Adversarial Neural Network Training on Solving Partial Differential Equation	0	7.94	17-08-2026
9	Atasi Cyberbullying, EU Social DigiThon 2021 Dirilis	0	0	27-09-2021
10	Monsanto против фермеров в бразильском суде выигрывает спор на 7,7 млрд долларов	0	0	14-11-2019

Классификация: . Схожих патентов: 0. Схожих новостей: 10. Тональность: 0. Информативность: 5. Источник: jmlr.org.