Are you sure it's glued on? Seems like a silly thing to do, but possible I suppose.
You could always drill into the end of the shaft with a small hole cutter, which would cut through the handle, but go over the drive. Bound to work, what could possibly go wrong ?
Have you tried tapping it with a hammer, not that the spring clip has been removed, or turning it sharply clockwise and anti-clockwise to shake it loose?
Richard