MOE models can have experts unique to each layer or shared across layers, depending on design.

In many MOE models, each layer has its own set of experts. This means the experts are specialized for that layer's specific tasks and inputs. Other models share the same experts across multiple layers, which can save memory and reduce the number of parameters. Both approaches exist in research and practical applications. The choice affects how the model learns and performs. For example, Google's Switch Transformer uses unique experts per layer, allowing each layer to learn different features. Meanwhile, some experimental models explore sharing experts to reduce model size and speed up training, though this may limit the model’s flexibility.