Simon Willison discusses the innovative ’streaming experts‘ technique, which allows larger Mixture-of-Experts (MoE) models to run on hardware with insufficient RAM. This method involves dynamically streaming necessary expert weights from SSD as each token is processed, optimizing resource utilization for complex ML models.
Source: Simon Willison