Well, it is true there is more to hearing than sound measurements.
The brain is a mighty complex beast, and it uses technologies we're only starting to do in smartphones. This is because human senses are poor instruments of detection and over millions of years, the brain has evolved ways to adapt.
It has been demonstrated that hearing is easily influenced by both sight - what you see affects what you hear, as well as memory - what you've heard and understood before also influences your hearing. The first you can see via one of the "pah pah dah dah" demo videos where the same sound is played, but the visual is changed. The latter has a demo where a heavily filtered sound is played, then it's explained what it really saying, and then every time the filtered sound is played, you understand it. You can even replay the test and you can decipher the sound.
And lets not forget how playing something just a bit louder makes it sound "better" automatically. After all, it's a common salesman trick - make the system you want to sell just a bit louder and people will flock to it.
This means you can't really just "hear" the improvement - you can listen to the exact same audio system, and depending on what you're looking at, it can sound completely different. Likewise, what you listened to before can influence what you hear. This also means it's trivially easy to manipulate people to say which system sounds better - there are plenty of people who say an MP3 file sounds better than the uncompressed version.
That's why we have both double blind tests and null tests. These attempt to eliminate the other human senses from influencing the opinion of the audio. After all, its not like manipulation hasn't happened before. And really, does anyone want to admit that someone managed to make them say a $5 MP3 player was better than a top of the line $1,000,000 system? I'm sure with enough preparation and trickery (including switching the visuals - play the MP3 player while making it appear the premium system is in use and vice versa) it would be possible. Or even just using the MP3 player the whole time while the premium system remains "off".