Good drumming/rhythm performance is all about subtlety and nuance. A couple of ideas...
1) When "programming" things like your hi-hat patterns or a shaker part, if you have the skills, play it in rather than step it in. If you are challenged by the tempo and don't want to quantize, _slow it down and then speed the pattern up later!_ You can quantize to a 16th-note rather than an 8th or a quarter, and when you slow the pattern down to play in the notes, you can double the click track so that it helps you audibly sub-divide (count) the beat better.
2) I like (and often have to) work quickly, but I want as much reality and human variation when programming/entering hi-hat parts, shakers, tambourine, etc. Let's assume you're like me-- what do you do? DON'T play all of the notes of a broken 16th-note hi-hat or shaker pattern in at once(!). For instance, if you're working with a two or four bar loop, on the first pass, play in only the quarter note downbeats ("1,2,3,4"), steady and even (pretty much at the same volume). On the _second_ pass, play the upbeat (in-between/"and") notes. If there's a third pass, you can add broken 16th-note variations or a hi-hat opening. When you play the hi-hat track back, the sum total of all of your in-time, but inconsistent (i.e. non-mechanical) note performances will give you a lot more realistic human variation that a real drummer would get in a live playing or recording situation on the first pass. Just don't overdo it, and try to resist the temptation to over-tweak (unless you just really played something so bad it's glaringly obvious). Do a number of shorter loops with the variations (same basic part, with same basic sounds, but slight alterations and adjustments), chain them together in a sequence, and you get some pretty realistic sounding drum parts. Good sound-selection, layering, and applying the technique to other rhythm instruments/sounds (such as snare drum) can fool a lot of people that you actually hired a drummer, rather than programmed a bunch of stuff in on a sampler, drum-machine, or DAW grid...
3) Have you ever tried putting in a basic pattern, using that as your "click," and layering your own "vocal beat" in on top? If I was a johnny beatbox, I'd be doing that all the time...
GJ