SoundGuys content and methods audit 2020

Spread the love
  • Yum

If you’re a regular reader of SoundGuys, you’ve probably noticed that we like to keep things as direct and unimpeachable as possible. However, we’re a collection of human beings—and we often screw up. Every so often I go through the way we do things to see if there’s any improvements that we can make, and if so: implement them. However, I thought it would be helpful for superfans and casual Googlers alike to see how we handle our work with things go awry.

Whether you’re a veteran review-junkie or someone just looking for the best product to buy, it can be very comforting to see a review with high ratings on Amazon or in print. Surely, you might say, that high score means this is the best product for me to buy! Unfortunately, that’s not always the case (and don’t call me Shirley).

Making buying decisions based on reviews is the smart way of shopping, but scores introduce some pitfalls.

The thing is: data can only tell you so much, and there’s often a piece of the puzzle missing when objective data is used to determine the “best” product in any category. You can’t just rely on a number to tell you everything you need to know about any product out there, simply because the only thing that matters is how you’ll like something. That’s why we spend so much time on our reviews to contextualize everything we’ve found with a set of headphones, earphones, or speakers.

Scoring audit: our scores diverge from user scores

After scraping all scores for all reviews on our site, we wanted to see how well we approximated value judgments for different aspects of products’ performance. I personally was looking for a variance of under 1.5 points in either direction, and we hit that—with one notable exception.

A plot showing the difference in user scores versus the SoundGuys staff scores

You can see that our users and staff agree on a lot… outside of isolation and attenuation.

Users rate isolation/attenuation much higher than we do

I’m not surprised by this result, as it’s one of those things that we fundamentally disagree with our readers on, and probably always will. The way we rate noise attenuation is built to reward both headphones and active noise cancelers on the same scale, so that way you can directly compare the effectiveness of all headphones to one another.

Because many ANC headsets offer better noise attenuation by a couple orders of magnitude, making a fair scoring scheme is difficult, and it doesn’t fit the mental models of many people who are only comparing headphones of a certain type. While many people will rate headphones that block out a bunch of noise highly, it might not be taking how well ANC headphones perform at the same task into account. Consequently, we couldn’t make an isolation/attenuation score for each product category because we then couldn’t directly compare products across categories.

Users rate sound quality lower than we do

A photo of the Beyerdynamic DT 990 PRO being worn by writer Adam Molina.

Humans aren’t robots, so it’s perhaps no surprise that our robot ears rate headphones differently than you do.

The other anomaly in our scoring is that our users are pretty much universally more negative on sound quality than we are. While that may be because the very nature of human ears means people disagree on a lot, I wasn’t expecting the near-universal dip in scores compared to ours.

While it’s not a huge difference, it’s still a difference—and that’s worth exploring. So I plotted all the counts of user scores for sound quality and:

A plot showing how users scores headphones

User scores for overall sound quality of headphones and earphones are most commonly rated 7.4 out of 10.

A normal distribution (like the one above) around 7.5 is something you’d expect with complete coverage of a single category of headphones, but it’s a little strange that it worked out this way for everything. I say this because the total breadth of coverage on SoundGuys tends to avoid both the upper and lower extremes of the headphones available, so our review selection is a little more curated. In that case, you wouldn’t necessarily expect scores to look like a nice, neat bell curve. If you were to try to review the top options of any industry, the scoring distribution would probably look a less pretty like the SoundGuys house scoring:

A plot showing the SoundGuys house scores for all headphones and earphones.

Our distribution of scores is a little more forgiving, but a lot less uniform in distribution than the user scores.

While that doesn’t tell us anything out of the gate, it’s possible that we should re-examine how we score sound quality, because even though most of the scores are well separated: they gravitate more toward the top of the scale than I’d like to see. Though it’s possible that most headphones nowadays are approaching the point where their sound quality is a settled issue—I don’t believe it. Not yet, anyway. We’re going to look to dial in these scores a bit more, but the good news is we have all of the test data on hand to do this. The big thing we have to do is make new scoring algorithms.

Proposed remedy: revisit scores for sound quality. Expand testing.

Be sure to check back as we explore new methods of scoring and analysis. While it’s tempting to acknowledge the difference and move on, we’re going to test a few things behind the scenes to hopefully bridge the gap between our readers and our own analysis. We feel like any excuse to improve our data collection is worth it.

An ideal gaming mic is not an ideal microphone

A man wearing the Corsair Void RGB Elite Wireless gaming headset with posters in the background.

A mic that close is susceptible to the proximity effect.

A while back we had a couple manufacturers cry foul when we rated some microphones lower than others because they attempted to reject low-end noise by being less sensitive to lower frequency sounds altogether. At the time we saw no reason to change our methodology, but after reviewing a large number of gaming headsets, we’re forced to conclude that we were wrong—so we’re changing the way we score microphones.

Think about it this way: with a gaming headset, the microphone is going to be pretty close to your mouth, right? Well, that means it’s susceptible to something called the “proximity effect.” In short, the proximity effect is what makes many radio DJs and podcasters sound like they have a boomy, bassy voice on the air. When you use a directional mic and get too close to it, you’ll notice that sounds under 200Hz can get up to a 30dB bump in emphasis. Great for recording kick drums, not so much for recording voices. By allowing that kind of emphasis, you can unintentionally make your voice tougher to understand over voice chat, which needs sounds in the highs to stand out for your brain to differentiate many consonant sounds.

A photo of the Shure SM7B dynamic microphone's' frequency response illustration on the back of the microphone.

There’s a good reason that the Shure SM7b allows you to de-emphasize low-frequency sounds right on the microphone.

You can probably see where this is going, but I’ll break it down anyway. In order to combat this, most audio professionals will de-emphasize low-frequency sounds to avoid the negative consequences of the proximity effect, and take all the work of equalizing gamers’ voices away so nobody has to think about it. In our view: this is the correct move, and we’ll be taking steps to address this in our scoring.

Normally we fight to keep all scores comparable to one another, but this is a situation where that isn’t the best idea. Namely, the results people get from normal use of gaming headset mics will be different than what they want them to be without low-end de-emphasis, so comparing a gaming headset mic against dedicated studio options is unfair.

Proposed remedy: create new scoring target for gaming headsets

In the near future, you’re going to see the scores of microphones change a bit, namely to account for gaming headsets and their varying low-end attenuation. It’s for the better that we move to meet the needs of consumers, so we’re going to do exactly that.

Wrapping up

Data usage on smartphone next to cash.

Our mission continues to be to give you the best bang for your buck in the audio world.

I do periodic audits of our methods and content, and this one was brewing for some time. Most of our slowness on some of these issues stems from needing a large corpus of data to analyze, and our readers have been very generous with their own opinions and assessments! We’re very appreciative of our readers’ help in grappling with a useful scoring system.

If you want to see changes, feel free to reach out on our contact page, or even just leave a question on our FAQ at the bottom of every article we have. While I can’t promise we’ll ever be perfect, I can promise that we’ll always try to do better than we did previously… and that’ll be true for as long as we’re around.

Happy listening.

Disclosure: We may receive affiliate compensation in connection with your purchase of products via links on this page. Even though we may receive compensation, we always give our honest opinions, findings, beliefs, or experiences on each product. See our ethics policy for more details.

Leave a Reply

Your email address will not be published. Required fields are marked *