I liked the idea and I think that you should persevere with the swing sound, it makes for an interesting listen and I'd quite like to hear an MC over it.
Your initial drum selection for the actual sounds is really good, and I thought that with some of your over tracks that I had a quick listen to, but you need to work on fitting the rhythm in with the sample a bit more, I guess it depends on the genre but I feel as if the track would benefit from a more 'natural' feel with the drums, it's sounding a bit computerized to me.
You're heading in the right direction though, at least you are daring to be quite original with your sounds.