Figure 1. Parallel convolutional experts sharing an input (ResNet-50, CIFAR-100): pairwise kernel cosine similarity. The experts collapse onto nearly the same filter, so operator duplication yields little specialization.
|
|
|
|
|
|
Paper (in one week) |
Code |
|
| Mixture-of-Experts (MoE) scales language models by routing each input through a small set of independently parameterized experts. We show that copying this design into convolutional networks fails for a structural reason: parallel convolutional experts that read the same input channels learn nearly identical filters. We therefore move the expert axis from operator duplication to channel selection. We introduce Mixture of Channel Experts (MoCE), a structured sparse channel-mixing layer, inspired by MoE, that replaces pointwise (1×1) channel-reduction projections. In MoCE, an expert is a single output channel with a learned sparse support of k « C input channels. The selected channels are combined by a softmax whose temperature is predicted per input, so each expert can move between mean-like and max-like aggregation. A residual expert summarizes the unselected channels, and a load-balancing loss keeps channel coverage complete. MoCE replaces a dense projection whose cost is quadratic in C with a mechanism whose predicted savings hold in measured wall-clock time. Across ResNet backbones on ImageNet-1K, transfer learning, EfficientViT, and a strong modern training recipe, MoCE matches or exceeds dense baselines and prior channel-selection methods while reducing MACs by 16.7–21% and end-to-end latency. |
| A direct convolutional analogue of MoE duplicates an operator and routes each input to a subset of parallel experts. In a controlled ResNet-50 / CIFAR-100 diagnostic with eight convolutional experts sharing the same input representation and objective, the learned kernels are strongly aligned‐most pairwise cosine similarities lie between 0.85 and 0.97. Duplicating operators can therefore spend parameters without producing useful specialization, so we move specialization from duplicated operators to sparse channel supports. |
|
Figure 1. Parallel convolutional experts sharing an input (ResNet-50, CIFAR-100): pairwise kernel cosine similarity. The experts collapse onto nearly the same filter, so operator duplication yields little specialization.
|
| MoCE replaces a dense pointwise projection from C input channels to E ≤ C outputs. Each of the E−1 routed experts is a single output channel: learned routing logits fix a static top-k support of the input, and the selected channels are combined by a softmax whose temperature is predicted per input by a lightweight shared gate, sliding each expert between mean-like and max-like aggregation without changing which channels it reads. A residual expert aggregates channels selected by no routed expert (on average 7.7%), and a coverage regularizer equalizes channel usage while letting the hard supports reorder during training. Because supports are frozen at inference, every expert reduces to a static gather-and-multiply‐so the k/C + 2/E MAC reduction translates into measured speedups on off-the-shelf dense kernels rather than remaining a theoretical FLOP count. |
Figure 2. The MoCE layer. E−1 routed experts over learned static supports, each with a per-input temperature-gated softmax mixture, plus one residual aggregate; the outputs are concatenated to form the projected feature map.
|
Table 1. ImageNet-1K. Accuracies are mean ± std over three seeds; MoCE parameters are training / deployment counts of learned values. MoCE improves accuracy while reducing both MACs and deployed parameters.
|
Table 2. Matched conditional-channel methods under the ResNet-50 / ImageNet-1K protocol. MoCE reaches Squeeze-and-Excitation accuracy with 6.8 M fewer deployed parameters and 17% fewer MACs.
|
Table 3. CIFAR-100 from scratch and ImageNet→CIFAR transfer (three seeds). MoCE improves the accuracy–efficiency operating point throughout, most clearly on ResNet-50.
|
Table 4. MoCE in EfficientViT on CIFAR-100 (k=64, second feed-forward projection, s=2), three seeds. Sparse supports apply beyond ResNet-style backbones to token-wise linear projections, cutting MACs by 18–22% at preserved accuracy.
|
Table 5. FP32 batch time (ms), batch 512, ResNet-50 eval; medians after warm-up. The advantage grows in wide projections: the representative 2048→512 module runs at 0.331× the dense time, consistent with the k/C scaling as C grows from 256 to 2048.
|
Figure 3. Routing in the first MoCE layer. Top-left: top-k overlap between experts. Top-right: cosine similarity of routing-logit rows. Bottom: number of experts selecting each channel, against the uniform expectation. The learned supports diversify rather than collapse.
|
Figure 4. Learned temperatures across experts, spanning max-like (concentrated), neutral, and mean-like (near-uniform) regimes rather than collapsing to one mode‐evidence that the input-dependent gate is used.
|