given that most grooves/swing templates are based on what happens off the beat, it is important for you to ensure that any musical event that is off the beat is treated with the same groove/swing factor (more or less), otherwise you may well end up with a mess of conflicting interpretations of what the beat is and how it is subdivided
However, there are times when using slightly different grooves/swing factors can add some interest to otherwise mundane rhythmic movement - experimentation is the only way to make these work, however... (wow started and ended with a however, very rare grammatically correct scansion)