I recently watched an interesting video Crinicle posted a couple of years ago about the “IGN Problem”: basically grade inflation in online reviews in the audio space. And because I’m me, it REALLY got me thinking. [And to be clear, it’s a thing that’s been churning in my brain for much longer than that, definitely encouraged by the thinking behind John Green’s fascinating “Anthropocene Reviewed”[1], AND conversations I had back in like … 2004 about Ivy League grade inflation.]
It boils down to this: it’s really hard to trust online reviews (basically, you SHOULDN’T trust online reviews) because so many of them are as inflated as a Harvard GPA. And it made me wonder: I’ve been doing reviews for more than two years now. I have scored 19 pieces of audio gear on a two-part, 10-point scale. What’s my average score? Turns out: pretty high!
| Pure | Cost-adjusted | |
| Average | 7.68 | 7.63 |
| Median | 8 | 8 |
| Mode | 9 | 9 |
| Minimum | 4 | 3 |
| Maximum | 10 | 11 |
An average of almost 7.7? A minimum of 4 or 3? I’m leaving so much scoring on the table! Am I IGN? Am I a Harvard professor, grading on a ridiculous curve? Have I lost the plot?!?!?!
The answer, obviously,[2] is a resounding no. It’s one thing to analyze the scoring of a professional organization that purports to review the field or some substantial subset thereof and make recommendations to the average person on; that’s certainly not what I’m doing here in my own little fun corner of the internet. BUT, I never made that explicit and I probably should. So here:
A GUIDE TO UNDERSTANDING PHIL’S SCORING:
- I’m a commercial transactions attorney, not an audio journalist. (Aka I do audio reviews in my free time because I think they’re fun, I love high-end audio, and it’s a creative outlet). I’m not a professional reviewer. So:
- I mostly only review things I like. I have a whole list of items that I’ve tested or used and decided not to review because I don’t have anything nice to say, or they’re too niche. Someday I might even publish it. This is because:
- I’m not really here to yuck anyone’s yums.[3] I started this whole thing in large part due to the fact that people often ask me about my impressions of device X, and I have found (and others have echoed) the fact that the human brain is genuinely bad at remembering what something sounded like fifteen minutes later, let alone five years later. If I don’t like something? Unless I have a really compelling reason to, I’m not going to take the time or energy to review it. Plus, like, let people live. Sometimes on Reddit I get someone telling me that I’m an idiot for a given opinion and my genuine response is … “Awesome. More power to you.”
- Legitimately, I have reviewed like … at best 10% of the gear that I use regularly, let alone things that I have tried or done testing on. I currently own like 100 pairs of headphones, have hi-fi systems in seven rooms in my house, and have a closet full of unused speakers and amps. Hell, there are eleven pairs of headphones in my laptop bag, maybe three of which I’ve reviewed. This is because:I spend HOURS on each review, even for stuff I’ve been using for years.[4] Probably ten or twenty hours of listening with notes along the way, plus at least three hours writing, actively A/Bing, retesting, editing, and publishing.[5] I have a real, grown-up, full-time job. I ain’t got the time to do that for things I don’t want to be able to point people to and say “hey, I think you’ll dig this thing. You should check it out.”[6] Ditto things that are out of production. Also because of the time:
- I have a pretty long backlog of things I want to review, and that’s even without considering the whiskey/wine side of my enthusiasm. I tend to focus on things that are still in production/are easily available, and I do hope to go back and update some things as new version become available.
- If I review something and give it a low score, it’s either because a) someone specifically asked me about it, b) I was super excited about it and then super disappointed, or c) it’s SO not for me, but might be for someone else with a different use case or preference.
So, here’s how you should read a particular score:[7]
- Above a 10: it’s fucking awesome. No notes. You will love it. Buy it now. [the MacBook Air 13” is a genuinely perfect product for my use case, and for what you get, it’s WEIRDLY cheap. Like Apple is hurting themselves selling it at this price point and I don’t get it.]
- 9-10: this is everything that I expect from a product. There are probably no major flaws, or if there are, they’re so counter-weighted by either overall quality or price that I’m totally fine overlooking those flaws. [The Meze 109 Pro are a damned near perfect headphone; they’re everything I’m looking for and I don’t think I would have any notes for the engineers as to what they might do to improve the product. Similarly, the Arya Stealth are almost perfect; my only (minor) criticism is that they can be a tiny bit treble heavy but a) it basically never bothered me during testing and b) it’s easily handled by EQ. If they were still their original MSRP the cost-sensitive would have been much lower, but at $599 they’re a pretty great deal.]
- 7-8: this is a good product that I like and will use regularly. [the 99 Noirs are a 7 pure, 8 cost-adjusted and they’re my favorite closed-backed headphones, and I carry them to Seattle every week and have for years).
- 5-6: this is a solid product, fit for a specific purpose. [the much-loathed Airpods are a 6 pure, 5 value, and I use them three times a day to listen to NPR while I walk Denali.]
- 3-4: I’m not going to use this, but it might be good for you (or for very specific purposes). [I loathe the Beyerdynamic DT 770 Pro, rated 4 & 4, BUT some people will really love their super-objective sound, and they’re probably what I would grab if I were mixing something.
- 1-2: I don’t think anyone should use this. I haven’t bothered to do a review of anything in this category and I can’t imagine I would, other than a one paragraph “don’t’ use this” review.
[1] I’m absolutely the kind of person who spends too much time thinking about what it means to judge everything on a five (or ten?) point scale and how that is or is not helpful for us as individuals, society, and as a species.
[2] At least to me.
[3] Mostly. But also sometimes, fuck you. Especially Airpods. Fuck Airpods.
[4] I have a 40- or 50-track (and ever-evolving) playlist of torture testing tracks for audio. It’s here: Spotify (tired), Apple Music (wired)): and Tidal (galaxy brain).
[5] I’ve owned the 6XX I reviewed last week since 2018, and I still spent probably fifteen hours listening to them before writing this review last week.
[6] The exception was the regular Airpods, which I HATED, just because I get asked about them frequently. And even on those, I’ve come around. They serve a purpose. So maybe don’t fuck Airpods, at least not as hard. (Except Bluetooth. Because definitely fuck Bluetooth.)
[7] Keeping in mind that I’m a single lawyer with no kids, no wife, and a golden retriever, and thus might have a super skewed sense of cost.
Phil's Superpower of Enthusiasm 