While I can't give you an informed medical opinion, I can relate to you my story which might help you in some way. My situation is somewhat opposite of yours.
I've been playing drums for about 33 years (with some time off here & there) At certain points I hit pretty hard but also worked to develop good technique. Over the years I never had any problems other than a few blisters.
But last May I moved into a house that was a fixer upper, I spent the summer and part of the fall rehabbing the house. Lots of demo work, lots of hammering, sanding, lots of painting, a couple days of using a jackhammer, and general wear and tear on my hands for about 12 to 14 hours per day.
By August I had developed some pretty bad tendonitis in both wrists and a constant numbness as well as occasional sharp pain in the two middle fingers of my right hand to the point where I couldn't sleep at night. I'm not a kid anymore, obviously I overdid it, but didn't have much choice in that matter. Some rest (and the completion of the rehab) fixed the problem though.
So I think the wear and tear of doing that kind of work is much more strenuous on the joints (especially over 28 years) than playing drums, even if you had the worst technique in the world.
That isn't to say that drumming couldn't give you some trouble, but depleting the cartilage in your thumbs in 6 years of playing less than 4 hours a week? I can't see how that can be blamed on drumming.