Gradient Span Algorithms Make Predictable Progress in High Dimension

Дата публикации: 17-08-2026 20:26:00

We prove that all 'gradient span algorithms' have asymptotically deterministic behavior on scaled Gaussian random functions as the dimension tends to infinity. This is a functional generalization of similar results for random quadratic functions and spin glasses. They explain the counterintuitive phenomenon that different training runs of many large machine learning models result in approximately equal cost curves despite random initialization on a complicated non-convex landscape. This 'predictable progress' phenomenon is exploited by the AutoML community: Since the optimization progress of a single run is already representative, multiple retries with the same hyperparameters are not necessary.

Схожие новости

#	Наименование новости	Тональность	Информативность	Дата публикации
1	Kernel-based Distributed Learning	0	7	17-08-2026
2	High-Dimensional Analysis of Gradient Flow for Extensive-Width Quadratic Neural Networks	0	8.7	17-08-2026
3	Graph-based Clustering Revisited: A Relaxation of Kernel k-Means Perspective	0	10.94	17-08-2026
4	Asymptotics of Stochastic Gradient Descent with Dropout Regularization in Linear Models	0	5.86	17-08-2026
5	Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selection	0	6.96	17-08-2026
6	Convergence of Decentralized Stochastic Subgradient-based Methods for Nonsmooth Nonconvex Optimization	0	8.78	17-08-2026
7	The Sample Complexity of Parameter-Free Stochastic Convex Optimization	0	5.7	17-08-2026
8	Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizes	0	7.52	17-08-2026
9	Near-optimal Delta-convex Estimation of Lipschitz Functions	0	9.71	17-08-2026
10	A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks	0	10.97	17-08-2026

Классификация: Пресс-релизы. Схожих патентов: 0. Схожих новостей: 10. Тональность: 0. Информативность: 6.38. Источник: jmlr.org.