What sticking are you using? Simple RLRL?

T1-T2-T3-SN
RL-RL-RL-RL

Why not change the sticking, and/or bring in feet?

For example, you could switch LR at the floor tom, so you're not "waiting" for the left hand to play the last note on the floor tom in order to start the singles again.

T1-T1-T3-SN
RL-RL-LR-LR

Or end the fill with RR on the floor tom and use the space you create to get your LH back to the snare:

T1-T2-T3-SN
RL-RL-RR-LR

Or use your kick as another tom, and end with one or both feet:

T1-T2-T3-K-SN
RL-RL-RL-r-RL      

or

T1-T2-T3-KK-SN
RL-RL-RL-rl-RL

As mentioned ....

Using a different sticking, such as a double stroke just before you want to go back to the snare .....

or

Cross-over (or under) your left arm with your right.

Practice a variety of ways, starting slowly and gradually building up your speed. As you get comfortable with the motion, you'll sound better and better.

Practice practice practice.  Thats how i get it down, works every time.