This may be because your speakers are wired out-of-phase ("+" to "+" on one speaker, and "+" to "-" on another.  When this happens, the sound "cancels each other out" to some extent.

To test this, adjust the fader and balance to isolate a single speaker (say, the front-left speaker) and see if the apparent volume increases.