I feel like my best stuff has always always started with the drums first. The reason here being that when I start with drums, I've usually found an awesome break and either chopped it up and rearranged or just looped it in order to get ideas flowing about my melody sample (then go back and match the drums to whatever sequence I have going). In my opinion, drums are the most important part of a beat so when I'm starting out with a really nice break it usually inspires me to find a better sample and at the same time a sample that really matches well with the feel of the drums.
I mean I've sat on a drum pattern I've made from a break for weeks just trying sample after sample until I was finally satisfied, even when alot of the samples I passed up on were really nice. If I wasn't satisfied, I wasn't going to waste that break on something that wasn't 100% in my mind.