Whenever I want to build drums around something I go right to drum loops. I have thousands and I just go through auditioning them in Ableton until I find one where the sounds work good with the sample. The rhythm is almost never right, but that doesn't matter. Then I immediately EQ that loop to fine tune it with what I have already. I actually do this a few times to find some loops that layer really well.
That will sound like shit, but then I pick out all the sounds and slice them to a drum rack. I end up with some nice layered kicks and snares, then shit like toms and hihats come from the individual loops and provide some more depth. Once a basic pattern is set up all I need to do is adjust the levels and panning. That's where you really get a sound you can't just get from relying on the loops as they are.