Researchers have derived oracle inequalities for dense and sparse mixture-of-experts (MoE) models, according to a preprint published on arXiv October 14. The findings establish theoretical bounds on estimation error and address the statistical cost of adaptive routing in these models, which underpin many modern large-scale systems, the preprint states arXiv.
The study focuses on the estimation error of least-squares MoE estimators where both the experts and the gating network are Lipschitz in their parameters. The researchers represent each MoE as a gate-expert pair in a Gated Hilbert Space, using its empirical representation to derive an oracle inequality with a fast rate of order $\tilde{\mathcal O}\big((Kd_{\mathrm{pr},e}+d_{\mathrm{pr},g})/n\big)$ for dense MoEs with $K$ experts. For sparse Top-$k$ and Top-$p$ routing, the selection map is discontinuous in the gating parameters; the resulting bounds match the dense rate up to an additive routing cost governed by the Natarajan dimension of the selector arXiv.
The paper shows that such a routing-complexity condition cannot be dropped, as Lipschitz continuity in the parameters alone does not suffice. For linear softmax gating, the routing cost is of the same order as the gating term. For linear experts, the study establishes a matching minimax lower bound of order $Kd_{\mathrm{pr},e}/n$, indicating that the rates are optimal up to logarithmic factors arXiv.
The preprint, titled "Sparsity Saves Compute, Not Samples: Oracle Inequalities for Dense and Sparse Mixture of Experts," is not peer-reviewed and has not been independently validated. The study does not include empirical testing.