You're not doing it right.
The 0 latency (well almost) is by auditioning direct out from the audio interface, that is how it's done. Obviously tightening the buffer in the pc will put too much strain on the cpu and not run a full project well, no mater how powerful the computer or pro a studio. When you're doing it right there should be no need to move audio files back and forth or ignore certain lagging sounds, with any direct monitoring interface.
Direct monitoring through the interface is the main reason many come with plugins built in. They enable you to add simple reverbs etc to the auditioning sound with out running through the DAW. You wouldn't record the effect on the audio, it's just to aid the singer. If you're wanting VST's enabled from the DAW on the auditioning audio track, this is when you might get issues.
I will add that Cubase has a feature know as ASIO guard. This enables you to have large buffer setting for your project but then sets tighter ones to any track you arm for recording. It takes care of the timing issues. Initially it will start the track with the latency, you play a long, intime, cubase places the audio in the right place (helps against the above mentioned issue).
The problem you are getting with FL must be because you arn't muting the track you're recording on. Are you hearing what you play then what you play again, quite like a loop? In Cubase if you monitor from the DAW you need to select the monitor button. If you monitor from the audio interface, you simply don not select that option and you won't hear the audio being recorded by the DAW (which will be later due to latency). You don't need to select it as you'll hear what you play through the interface.
You will need to learn how to use the software mixer for your audio interface. Enable the recording input channel to go direct to the outputs and have the DAW out puts running there to.
Edit: My friend uses FL but when ever he wishes to record he comes round to mine to use Cubase.