...physical fitness can help a person be a better accountant or programmer or whatever. Those jobs aren't physical, but when people keep fit they tend to have better quality sleep, more energy, more altertness...
It's the same with any job. If you're a builder, you relax sitting in an armchair, if you're a programmer, you relax walking, swimming, or doing some exercise... You need health and a good sleep, always.
But when you're a drummer, I don't know. I wouldn't say it's not physical exercise, but it is, anyway, a sitting job. A lot depends on what you play and how easy you can make it. There is, for example, a hard song with a strong tiring beat with my band, which I don't like and the beat is very unfamiliar for my taste - I always tell them that when we play it, a guitar or whatever solo must follow it so as I could stand up and walk off stage for a few seconds and gather myself.
So fitness, both mentally and physically, is very important. If I was fitter, I would do that song laughing, but without it, I just suffer to play it right,