Speedrunning ideas discussion

[KellerJordan]

Greetings GPT speedrunning enjoyers

I noticed some people were using the GitHub issues to suggest new ideas for the run.

I think it's more optimal to concentrate new ideas into a discussion here instead. There are a lot of interesting ideas.

But no promises that any runners actually try anything that's posted, of course.

[djbyrne]

Are you looking at any throughput improvements even if they are not algorithmic? i.e things like pre-fetching ?

[kklemon]

• Differential Transformer. According to the paper, it seems to improve the loss for a given model size, but due to the overhead of introducing an additional attention operation, it may not lead to improvements in wall clock time. However, using a lightweight attention layer, perhaps even just a single head for differential attention, might alleviate this issue. No empirical data to support this, but it might be worth a try.
• nGPT. Should be fairly easy to implement. The paper claims faster convergence per token seen, but I don't recall them showing or claiming faster wall clock convergence. However, given the not-so-large overhead of the approach, it should result in faster wall-clock convergence.
• Introduce lightweight RNN layers, as in SRU++. In the referenced paper, they (well, it's a single author) couple lightweight SRU RNN layers with attention and achieve impressive results in language modeling. To my knowledge, this approach has never really been scaled beyond relatively small benchmark datasets, but I don't see any reason why it shouldn't work on larger datasets. In this case, one could try it by simply applying the SRU operation within the feed-forward blocks, i.e. proj from d to 3d -> sru -> proj from d to d instead of proj from d to 4d -> proj from 4d to d, without changing the rest of the architecture too much.

I would be happy to hear other opinions on these ideas. I could also provide working implementations if someone who is not GPU-poor (@KellerJordan 👉👈 ) can evaluate them.

[pszemraj]

+1 for nGPT, I think somehow there are two discussion threads for this topic and I added nGPT unknowingly on the other one