Actually, it's a little bit of a myth that passive controllers don't color sound. They can color the sound FAR more than active controllers. Why? Because passive controllers are completely dependent upon the impedance of the connected devices. This is why one person will say a certain passive controller is absolutely transparent while the next will say it colored the sound almost like an EQ. And depending on how good the design is, they color the sound more or less depending on where the volume control is sitting. To be clear, I'm not saying passive controllers are bad. It's just that coloration, or lack thereof, is highly dependent upon what is going into it and what it's going out to. Both passive and active controllers process the sound, it's just which method they use to do it. I will say this though: you can make a quality passive controller for less money than a quality active one - but again, you've got the impedance issues to navigate and they can bite you in the butt.
This is why I suggested a switcher. Making a transparent controller is a lot harder than it initially seems. And the OP doesn't really have an issue with setting volume, or any of the other functions a controller can do (input switching, mono, L/R cut, phase, etc. etc. etc.). They just need to switch between two sets of speakers. Hell, you can pretty much go to radio shack and buy a toggle switch, four TRS jacks and a tiny bit of wire and build one that will be 100% transparent for $15 (maybe less!)
