I think you're confusing latency with performance. I suspect you are thinking of this like a video game or something.
Latency, with respect to DAWs, means the number of samples a plugin needs to keep in memory to be able to perform it's task. Or how many samples out of sync the output is with the input.
For example... for a gain plugin all you have to do is multiply the current sample by a factor. So latency is nothing. Every output needs only the current input. Or output n corresponds to input n.
In Linear Phrase EQ overcomes the phase distortion created by filters but does that by working on a chunk of audio rather than one sample at a time. Some compressors have look-ahead modes. These too need to keep a buffer in memory for the next, say, 1 ms of samples. This means that the plugin can't give you an 'answer for a give sample until after it has collection more samples. So output n correspond to input n-5 (with 5 sample latency). The total number of samples it has to collect is it's latency. DAWs can take this latency into account and shunt all the samples back in sync with all the other pugins.
That is nothing to do with performance (except that all that shunting presumably uses buffering which takes up RAM and CPU). If the CPU is maxed out then your DAW can't produce sample data in reatime and you get drop-outs.
It does, however, impact input performance. More latency makes playing stuff in a pain.