Would GrokFast and other accelerate training techniques apply here?

[BradKML]

Thinking this might also be useful in making NanoGPT++ work even faster for fun https://github.com/ironjr/grokfast

[BradKML]

Some of the new acceleration techniques covered by Tunadorable:

• Patch level training https://github.com/shaochenze/PatchTrain
• Initializing with small model weights https://github.com/apple/ml-hypercloning