11 Oct 2011 @ 4:16 PM 

One of the most-hit columns I’ve ever written on Top Fermented was a “Beer Advocate vs. Rate Beer” column. It raises ire. Some people like the fact that I attempted to (poorly) apply statistics to compare the ratings on each site. Other people have bitched and moaned about how it’s a steaming pile of turd, which I won’t necessarily argue with – it has flaws. I’m pretty sure I even say that in the article itself.

Anyway, I see all of these links point here and I read them all and contemplate them and never really say anything because.. hey.. it’s like a two and a half year old column at this point, and it’s kind of meaningless now. Ratings systems are continually updated and people continue to use the site in new and different ways. However, the rating sites still over-appreciate dark, high alcohol, and hard-to-find ales and well-made low alcohol lagers consistently under-perform.

I’ve thought about it a lot and I am now of the opinion that wholesale beer rating is really to a point that it is no longer useful and, in fact, might even be detrimental to the market as a whole – and I don’t just mean Beer Advocate or Rate Beer, but beer review blogs, etc., and anon. The noise-to-signal ratio is just out of whack and the results are being given gravity that they don’t deserve.

Before the flames and trolls show up, let me state my case:

People rate beer by measures that are too subjective

Plain and simple. By and large, people rate beer based on whether or not they liked it, not whether or not it’s a good beer. Believe it or not those are two different things. I can’t stand Bud Light, but I won’t tell you that it’s a poorly made beer. It’s excellently made beer if you want a lite (yeah, I spelled it) lager. But it gets a 0 on Rate Beer (a 1 within the style) and a D- at Beer Advocate even though it is essentially the definition of the “light lager” style. Why? Because it lacks technical brewing skill and is rife with off-flavors? No. Because the bulk of the people who are rating it, like me, hate it. Rather than disconnecting themselves subjectively to actually answer whether or not it’s technically well-made and matches the style, they rate their own taste in graduated values of suck.

I do think that there is value in being able to have a list of ratings of beers that you have enjoyed for your own reference. It’s one of the reasons that I like Untappd – because it gives me a list of my own ratings for me to reference later. I don’t always remember a beer three months later. Have I tried this? Did I like it? 5 stars says, “Yes!” But just because I like it doesn’t mean that somebody else will. Taste is subjective. I like a really wide range of beers, but give me something with a load of Nugget hops and I will always, always hate it. 1 star-only and man did that suck. But! That doesn’t mean you won’t like it, so why should my personal rating mean anything to you?

If it’s a useless measure, we shouldn’t be using it to judge beers with.

There’s no way to tell that people are tasting good beer

And by good I mean “like the brewer intended”. Not old or oxidized or through infected taplines or in dirty, frosted glassware or drunk by a smoker or someone with an asshole for a mouth. Certainly, some people will note in the comments of their review about how it was served or what it looked like, etc., etc., but one look at the top comment under Bud Light really says it all:

…serving type was shot gunning at the football game.

Indeed, byteme94. I will now take your D+ more seriously because I know you put a lot of thought into it for those 4 seconds while it was passing through your esophagus. Was the fact that you didn’t immediately throw up what saved it from a D- or an F?

If someone is tasting a beer out of a dirty tapline and (and this is important) they don’t know what a dirty tapline tastes like, they think they just have a shitty beer and there’s no way for me, as a reader, to tell if this is in perfect serving conditions or if this is someone drinking beer out of their cat’s old food dishes before they give a beer a score (“drunk from a straight-sided shallow goblet”, indeed). I’m not going to look through 3,000 reviews. I’m going to look at the aggregate score. If the aggregate score is a composite of unreliable measures, then the aggregate score is unreliable.

There’s no way to tell if the people are good at tasting

Let’s take, for instance, Geary’s IPA in which the first review – which gives it a B (which is decent, if you consider C to be average) – mentions the word “buttery” twice. Once in the aroma and once in the flavor. He didn’t really care for the butteriness of the malt. Of course, he mentioned that he wouldn’t really expect bitterness or alcohol in an IPA, either. Now, I happen to know that Geary’s is brewed at Shipyard, and that Shipyard’s house yeast is Ringwood which has a VERY high flocculation rate. It tends to drop out of the solution really early and doesn’t really remove diacetyl (which tastes like butter) from the beer like it should unless you do some awesome tricks to keep that yeast in suspension – which Shipyard is generally pretty good at.

IPA shouldn’t be buttery. Malt does not taste buttery. This is an off-flavor. But the reviewer doesn’t know this (or that an IPA should be bitter, sadly). He just thinks (correctly) that it tastes like butter, and while he doesn’t really like it he also doesn’t know that it’s not supposed to be there at all so he doesn’t judge it as harshly as he could and maybe should. Or to look at it backwards, he is judging it as though the butteriness is supposed to be there, because he doesn’t know that it isn’t.

Is this a good, honest review of this beer? It certainly reflects whether or not the drinker likes it, but does it reflect the quality of the beer? ie – Why should this B count with the same weight as someone’s C who does know that their beer is diacetyl heavy? How do I know if the person who is reviewing the beer knows enough about the beer to give a good review? Just because you drink a lot doesn’t make you an expert. It just makes you drunk.

(I am positive that at this point in the article, at least one thread will start on a forum somewhere to discuss whether or not it matters if a beer is well-made if you like drinking it, anyway. Related: Who cares who makes your beer if you like drinking it? Answer: I do.)

The internet is untrustworthy in general

Sorry kids, but I just don’t have any reason to trust you. Just because a lot of people rate something doesn’t mean that there’s any sort of reasonable quality involved. You know that saying that’s something like, “50,000 people can’t all be wrong”? Well – actually, they can. It happens all the time.

A significant portion of this country believes that science and math are just these things that the educated elite make up to try to perpetuate grant funding because paying yourself off of grants is sooooo awesome. They believe things like vaccines are bad but polio is kinda okay. They believe that man and dinosaurs used to co-exist. Why on earth should I trust you, the internet, to know enough about beer to give me a decent recommendation if you can’t get broad “society has moved on” issues correct?

In Summary

Fact: You can’t measure something with an unreliable tool. If I’m allowed to make my own ruler that just has however many inches I want on it at whatever random intervals, I can use it to build the same thing every time. But as soon as I give you my plans you are up a creek without.. well.. a ruler. Have fun defining that cubit, bucko, because I measured it using MY forearm, not yours.

There’s no good way to cut through the noise of beer reviews to find out which ones are worth paying attention to and which ones aren’t. Since there’s no way to calibrate the tasters to make sure that they’re all tasting with the same objectivity, then there’s no way to say that any given set of ratings is even reasonably reliable and I won’t waste my time with them. Until we have some sort of Cicerone-weighted rating system or something like that, I’m calling shenanigans on beer rating, especially wholesale ratings sites like BA and RB. Their data is no longer worthy of consideration, by my estimation.

Make your own ratings and decide what you like for yourself. It’s far more valuable in the long run.

These ratings are being put forth as guides for consumers

Let me quote something to you from the comments of a blog that I ran across that I’m pretty sure sums up common sentiment. I know that I should quote who it’s from, but I don’t know them personally and I don’t want to get into any sort of pissing contest. This quote is in reference to a post recommending shelf tags from Rate Beer and Beer Advocate in retail establishments, much like you would see shelf tags from, say, Wine Spectator.

I do appreciate that the rankings are from a consortium of dedicated drinkers compared to wine, which historically was dominated by one individual or several publications.

Indeed. You know what I hate? Being able to make informed decisions based on reliable, consistent data. What I prefer is to make random guesses based on completely unreliable anonymous data. I mean – who needs Consumer Reports and a trained panel of experts when I can get a product rating from BoobLvr67?

That is the equivalent of trusting anonymous online ratings for beer (or anything, really, but let’s stick on topic).

What I’d Like To See…

…is some sort of rating system from people who are actually known trained tasters – Cicerones and/or BJCP judges – with ratings ranked in importance based on how skilled they’ve shown themselves to be. That would be better information. There’s still individual taster differences, but at least those tasters have been moderately calibrated. At least there’s a starting point beyond, “I signed up for the website.”

That’s a rating site I’ll trust, and those are shelf tags I want to see in retail establishments. Until we can get there, I’m dispensing with wholesale beer ratings in general.

Share
Tags Tags: , , , ,
Categories: appreciation, blog, industry, op-ed
Posted By: erik
Last Edit: 11 Oct 2011 @ 05 00 PM

EmailPermalinkComments (54)
 07 Oct 2011 @ 12:31 AM 

This week is a festival-y week. This past weekend was, of course, the Great American Beer Festival. This coming weekend gives us the World Beer Festival in Durham, NC. It’s my home beer fest, and this particular one will be the first one that my little startup brewery has a booth at, however unofficially.

It’s meant that I’ve spent the past week or so thinking a lot about the festivals themselves: What do I want out of them? What are people getting out of them? What are they all about?

This year was my first GABF, and for the most part it seemed like a nice beer festival. I quite enjoyed it. It was well organized, there were nice wide aisles, there was a good selection of beer and nice side events. (The Farm-to-Food Beer Pairing Pavilion was brilliant.) Indoor beer festivals are also my favorites because there’s no smoking and there are actual bathrooms and not port-a-johns. So, awesome on those counts. What I really noticed, overall, was something that Andy Crouch brought up in his recent GABF recap which is that there was a tremendous lack of brewers. There were, however, scads of volunteers (yay!) that didn’t know anything about the beer they were pouring (boo!). It’s something that made me reflect back upon local beer festivals and seeing brewers hanging out behind festival tents chatting with one another while volunteers were pouring their beer just feet away on the other side of a flap.

What’s going on here?

Before I answer that, let me ramble on a little more.

While I was in Denver, I had lunch with a friend who does not drink. Over the course of lunch, she asked, “So, what’s the point of this festival? Is it a good way to get exposure to a lot of people?” and I thought about it and had to answer: No. Not really. Not at all, actually. I drank beer from dozens of breweries, and I doubt that I can tell you more than a handful that I had and enjoyed.

And it’s a sort of woogy answer. Here in Durham, with an almost-open brewery, I am looking forward to getting a lot of exposure. But I am largely an exception. I am pre-new and many people haven’t heard of me (or have only heard of me have no idea what I’m all about) and this is an excellent way to get in front of my local crowd. But what about for everyone else?

Again, what I told my friend: I don’t think most brewers really care about getting in front of drinkers, anymore. They care about getting in front of the 1% of people that are in there that can actually help expand their market – the beer buyers, the bar owners, the restaurateurs, the distributors. Everybody else is just getting trashed.

And that, in a nutshell, is my problem with beer festivals.

To wit (and I’ve said this before): Beer festivals used to be about education. For years and years, they served as a way for a population that was eager to learn to get a wide array of beers easily, and to learn about a vast array of different styles in a way that just wasn’t available to them anywhere else. Now they can go do that at the package store. Total Wine has just as many beers as you’ll see at most beer festivals, maybe more, and it’s cheap to build your own six-pack. Sampling is just a lot easier in the marketplace than it used to be. That means when people go to beer festivals they’re not interested in learning. They already know what they’re looking for. They’re interested in drinking – which in and of itself is not a terrible thing – but as beer festival prices go up and up and up, people tend to try to get their money’s worth out of the price of their ticket and that generally means pounding as many 2 oz. samples as possible.

Is this true for every festival? Certainly not. But the bigger ones, the more well-known ones? Almost universally true.

From a brewer’s standpoint, if people aren’t there to learn from you there’s no incentive to try to engage with them. There’s nothing more disheartening than having somebody walk up to your booth and ask for “your lightest beer” or “whatever” and then just slug it back, regardless of what it was. We put a lot of work into making these products, and we’re proud of what they taste like. It’s frankly a little insulting to watch somebody pound a sample of your product without any thought. I’d rather you hated it and dump it out then to drink it without thinking about it.

If it’s not about educating consumers, it’s really about that small contingent of people that can effect a brewer’s bottom line in a real way, the people who will end up buying a large portion of product (eg – kegs or cases, not a bottle or a 6’er), and that makes hanging out at a booth and giving beer to people who don’t care fall under “not a good use of time” in most circumstances. After all, a brewer can hang out around the back of the booth and wait for that 1% to come around and focus on them while volunteers pour the beer.

So, if this is the overall trend, then what does the future of beer festivals look like? I struggle with this. I can’t help but think that brewers will get more and more jaded about spending large amounts of time and product going to beer festivals that help them less and less and that more and more people will stop showing up to something that is become more and more of a chugfest.

I offer these possible future solutions for beer festivals:

  • Stop pretending it’s about the beer and focus on something else. Get bands in and make it an all-day concert that happens to have a great beer selection (a la Brewgrass). Give brewers a chance to sell their beer instead of give it away and you’ll end up attracting a lot more breweries and take a lot of pressure off of the breweries.
  • Stop pretending it’s about the drinkers and make it a trade show. Make it industry-only. You want to see brewers show up and show off their products? Make sure that the only people there are buyers, retailers, and distributors. General public can have their own drunk-fests with all-volunteer staffs and maybe a few special guests.
  • Get the education back into beer festivals. Get rid of the damn band – no one cares – make it event-heavy. Put beer-and-food pairing sessions front and center, have talks by brewers about ingredients or techniques that are more than introductory bullshit schlock. Realize the fact that craft reaches a much larger portion of the marketplace than it used to and cater to that market. After all, those are the people that are going to beer festivals now.
  • Limit attendance. For the love of all that’s holy. Anybody will tell you: small beer festivals are more fun.

I’m sure there are dozens of other ways we could see festivals revitalized. It’s time for some innovation.

I enjoy beer festivals, but I am jaded by their expense to the brewer. You’ll see us at our locals, but not at many outside of our area. They need re-imagining to continue to be relevant to both brewers and drinkers and I kind of wish I was in a place to help move them along. Unfortunately (for festivals), I’ll be on the other side of the tent flap making beer.

Share
Tags Categories: beer festival, industry, marketing, media, op-ed Posted By: erik
Last Edit: 07 Oct 2011 @ 09 19 AM

EmailPermalinkComments (15)
\/ More Options ...
Change Theme...
  • Users » 130222
  • Posts/Pages » 204
  • Comments » 2,674
Change Theme...
  • HopsHops « Default
  • BarleyBarley

About



    No Child Pages.

Shirts



    No Child Pages.

Tour



    No Child Pages.