Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

想问一下作者大大是怎么做到模型大小这么小的,通过模型压缩或是参数共享吗,还是只是减少了layer、dim这些参数呢? #105

Open
ChinanBoys opened this issue Dec 27, 2024 · 1 comment

Comments

@ChinanBoys
Copy link

最近有在看MobileLLM那篇论文,不知道作者是不是用了论文里面的技术呢?如embedding share、GQA等呢

@jingyaogong
Copy link
Owner

更少的layer+hidden dim/dim,这些readme有写。

另外GQA和linear share大约一年前就普及了,minimind里同样标配。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants