2024年3月25日 · Here, we analyze encoder and decoder transformer models and show how memory bandwidth can become the dominant bottleneck for decoder models. We argue for a redesign in model architecture, training, and deployment strategies to …

2024年10月9日 · Our embodied LLM agent "WALL-E" is built upon model-predictive control (MPC). By optimizing look-ahead actions based on the precise world model, MPC significantly improves exploration and learning efficiency.

2024年12月5日 · We’re finally coming to terms with the idea that foundation LLMs have hit a wall. Thanks to decades of data creation and graphics innovation, we advanced incredibly quickly for a few years. But we’ve used up these accelerants and there’s none left to fuel another big leap.

2014年9月10日 · This is a repository with the data used for the AI and Memory Wall paper. We report the number of paramters, feature size, as well as the total FLOPs for inference/training for SOTA models in CV, Speech Learning, and NLP.

We argue for a redesign in model architecture, training, and deployment strategies to overcome this memory limitation. The amount of compute needed to train Large Language Models (LLMs) has recently been growing at a rate of 750×/2yrs.

2021年3月29日 · The memory wall problem involves both the limited capacity and the bandwidth of memory transfer. This entails different levels of memory data transfer.

2023年10月26日 · We show that contextual sparsity exists, that it can be accurately predicted, and that we can exploit it to speed up LLM inference in wall-clock time without compromising LLM's quality or in-context learning ability.

Overview of WALL-E (Left) and NeuroSymbolic Learning details (Right). The agent determines actions to take via MPC, where an LLM optimizes future steps’ actions by interacting with a neurosymbolic world model.

2024年12月23日 · Many were and probably still are wondering whether LLM Scaling Laws, which predict that increases in compute, data and model size lead to ever better models, have "hit a wall". Have we reached a limit in terms of how much we can scale the current paradigm: transformer-based LLMs?

2024年11月11日 · Orion 的情况可以检验人工智能领域的一个核心假设，即 scaling laws：只要有更多数据可供学习，并有更多的计算能力来促进训练过程，LLM 就能继续以相同的速度提升性能。