By the way, the punch didn't actually break on the sight. While I was doing some light filing on the sight, the punch rolled off the bench and hit the concrete floor and broke.
I wasn't slamming the punch against the sight, as my post may have sounded upon a re-read.