Go to technology
◉ Project Goal Ship collaboration features，推荐阅读新收录的资料获取更多信息
The first implementation of Mog used LLVM as the backend. LLVM can produce somewhat faster code due to its wide array of optimizations, but it had two major issues. First, compile times were not fast enough. The new compiler has compile times that are not quite as good as Go’s, but within an order of magnitude for programs under 1000 lines – fast enough that the start time for one-off scripts is not painful. Mog does not claim to provide zero-cost abstractions or arbitrary opportunities for low-level optimization. It compiles to native code, but an expert can still write faster C or C++.，详情可参考新收录的资料
Looking at the forward pass implementation of MoEGate we find:，详情可参考新收录的资料
We have one horrible disjuncture, between layers 6 → 2. I have one more hypothesis: A little bit of fine-tuning on those two layers is all we really need. Fine-tuned RYS models dominate the Leaderboard. I suspect this junction is exactly what the fine-tuning fixes. And there’s a great reason to do this: this method does not use extra VRAM! For all these experiments, I duplicated layers via pointers; the layers are repeated without using more GPU memory. Of course, we do need more compute and more KV cache, but that’s a small price to pay for a verifiably better model. We can just ‘fix’ an actual copies of layers 2 and 6, and repeat layers 3-4-5 as virtual copies. If we fine-tune all layer, we turn virtual copies into real copies, and use up more VRAM.