based

Any guess on the pre-training tokens/flops they consumed?

Technical report has a graph vs Kimi K2 with 1e21 FLOPs (but they don't claim that's the entirety of pretraining)

1e21 flops is hilariously wrong. for reference the llama 3 8B model (https://arxiv.org/pdf/2407.21783) used 10 times that many flops. This model is 350x bigger in total params and 12x bigger in active params and was trained on 3x the data.