KCBS judge seating changes

[ClaytonSmoke]

BTW, I think beer comps have this down a little better as rookie judges are sat with master judges and the table can’t differ very much from the master judge. More consistency but at the cost of conformity.

Judging beer, you are judging to a defined style. Unfortunately this doesn't exist in BBQ. I do think the beer judge certification program really has it together, with a pretty challenging test and multiple levels of judges.

 

[Gary Tomato]

I was taught to judge each box on its merits, don't compare one with another, and there never was a starting point mentioned.

My scores have been consistent with the others at my table, so that's probably a good for competitors in Australia - KCBS comps have only been held here for four years.

I was fortunate enough to be trained in Melbourne, Australia, by Jim Johnston and the International Outreach team way back in 2015.

 

[D&D BBQ]

Judging beer, you are judging to a defined style. Unfortunately this doesn't exist in BBQ. I do think the beer judge certification program really has it together, with a pretty challenging test and multiple levels of judges.

Yeah, I was lucky enough to be asked to judge in a couple of homebrewing comps at Sierra Nevada brewery back in the 90s and was so impressed with how organized they were and helpful with printed guidelines for each style which we used to score the entries. Then we’d confer with the master judge at the table and adjust our scores accordingly to be near the master judge per the rules. Learned a lot about beer and became a fan of IPAs in short order because of that.
One of my favorite events I’ve been a part of is our local bbq club got together with the homebrewers association and held a joint contest wherein the bbq cooks judged their beers and the brewers judged our bbq. Just an informal thing with pork and beef ribs from us while they had a mix of styles going up against each other. Definitely need to do it again!

 

[enasnidx]

Just saw the board meeting quick notes:

Early 2017, it was determined that judges would be seated via the Shuffle. This was instituted nationwide. Mid-year, Mark Gibbs developed a method of seating of judges that was based on their scoring averages divided evenly among the number of tables to be utilized at a contest. This program was Beta tested across the country for several months and it was determined that it was very successful. It was rolled out to ALL reps in January of 2018 and mandated to be used for all Master Series contests.

I am in favor of this change but this is not the end of scoring issues. It may help a little at contests but there are still other issues. The fundamental issue of judge training differences still needs to be addressed.

I'm with you. I do think it's a good change, especially for those teams that are hit or miss to break into the top ten a lot (like me :redface. The TOA/TOD can make or break a contest for them. A little more consistency from table to table can make a big difference for those teams only doing 10-15 comps a year. Those are the teams that benefit the most from this change. The TOY contenders probably have less to gain and less to lose from this change.

However, my big concern is that the board looks at this and says "Good job everyone, problem solved." I agree with what most have shared, it's a band aid. There needs to be some major triage done.

 

[ModelMaker]

"Then we’d confer with the master judge at the table and adjust our scores accordingly to be near the master judge per the rules."

If that's what you guys want, maybe we can fill out the score cards with nothing but 8's and 9's mixed up a little bit and then you can wait for the electronic coin flip to see who the winner might be that day.
But then you'll complain the coin has 3 sides.
Ed

 

[rorkin]

Once again we are operating without facts in regard to the seating program.

Why has there not been an article in the bullsheet explaining the problem as perceived, the solution as implemented, the analysis of that change relative to the goal..

All we get is hearsay results and mandates out of Kansas City.

I have seen judges ranked by types of wood.. What does that even mean ?

 

[goodsmokebbq]

There are only two ways to beat the underlying statistics of this system.

One: lower the judge to judge variance through training or standards or whatever. But no amount of training will create the perfect judging pool AKA zero judge to judge variance. Some variance will remain and that will lead to the real problem, number two...

Two: you need to take more samples. Right now it is theoretically impossible for every table to have the same average with only 6 samples. Impossible. You can't drive the judge to judge variance low enough. Statistics wins... Every time...

We are running head long into a wall that will always remain.

 

[goodsmokebbq]

The new seating system, balancing the tables ahead of time based on a judges last few events is a noble effort, but I have a couple reservations.

A judge is being averaged on their last few events. Clearly they could have a low average because they just had bad food the last few times or maybe they are a low judge. We really don't have any way of knowing. Now envision this "low" judge hits a table of good food and scores high. He was supposed to be the low guy on the table. Now we see an artificially high table.

What average is being used? Overall? I think it is impossible to re-seating judges for each category, so they must be using the overall value. What if a judge scores high in chicken, ribs and pork, but is a low scorer for brisket. With three high categories and one low his average will be high. Thus he will be the "high" guy on the table for brisket. Where he will score low driving that brisket table down. Just one example of the hundreds of situations that could pop up.

I am also worried about the possibility of an unintended incentive for judges to raise their scores. Every judge will be looking around the table wondering if he / she is the low guy. We are reaching the ceiling quickly and I don't think we need any more incentive for a judge to raise their scores. Judges shouldn't be worried about their average when they are judging any single event. I can't help but think that this will be in the back of every judges mind.

And then there are new judges, with no or very little experience. How are they handled?

I really think we should let random be random, seat the judges randomly and deal with the underlying statistics of the system... or change the system...

 

[Pappy Q]

The new seating system, balancing the tables ahead of time based on a judges last few events is a noble effort, but I have a couple reservations.

A judge is being averaged on their last few events. Clearly they could have a low average because they just had bad food the last few times or maybe they are a low judge. We really don't have any way of knowing. Now envision this "low" judge hits a table of good food and scores high. He was supposed to be the low guy on the table. Now we see an artificially high table.

What average is being used? Overall? I think it is impossible to re-seating judges for each category, so they must be using the overall value. What if a judge scores high in chicken, ribs and pork, but is a low scorer for brisket. With three high categories and one low his average will be high. Thus he will be the "high" guy on the table for brisket. Where he will score low driving that brisket table down. Just one example of the hundreds of situations that could pop up.

I am also worried about the possibility of an unintended incentive for judges to raise their scores. Every judge will be looking around the table wondering if he / she is the low guy. We are reaching the ceiling quickly and I don't think we need any more incentive for a judge to raise their scores. Judges shouldn't be worried about their average when they are judging any single event. I can't help but think that this will be in the back of every judges mind.

And then there are new judges, with no or very little experience. How are they handled?

I really think we should let random be random, seat the judges randomly and deal with the underlying statistics of the system... or change the system...

Absolutely agree that it should just be random seating. Any attempt to seat per any criteria is a failed attempt at manipulating the outcome.

 

[gettinbasted]

Absolutely agree that it should just be random seating. Any attempt to seat per any criteria is a failed attempt at manipulating the outcome.

Bingo. We are using outcomes from past events to try to manipulate the outcomes of future events, period. This "solution" has zero to do with fixing judging (assuming it's that broken in the first place). All we are doing is "fixing" scores by dragging everyone to the middle, food quality be damned.

 

[jaestar]

When I took the CBJ class five years ago, he instructor said a nine was bbq you would drive 250 miles one way just to eat that bbq...no other stops or as part of a trip. Just drive to the bbq, eat and drive back home. That message was givem to me amd 25-30 other potential judges and I have never been told differently from KCBS.

 

[GrillBillie_D]

First, I agree that the board should do more explaining of exactly what the changes are and how their beta tests showed to be successful. What was their criteria and who evaluated it?

The reason I am in favor of this change is that in my area, we have a few judges who consistently score higher or lower than the "average" judge. I think that this change at least tries to minimize the impact of such judges to the teams....not eliminate. I agree that it is merely a bandaid on the real issues but I do think it may help in some small way.

... or change the system...

This is the elephant in the room that nobody really wants to address. Those who have the current system figured out don't want it changed and those who want it changed sound whiny (as some describe them....not me). My opinion is that competition BBQ has evolved to the point where we use exotic breeds of meat and have homogenized the flavors to the point where the existing system doesn't fit. I think it's time to expand the criteria just a bit.

Think about this for a minute...I'll use one example.....Sams Club final from last year....The winning score was 702.846 and 2nd was 702.8344. That is a difference of 0.0116. 24 entries from each team and the difference literally being decided by less than any single judge changing any single score by one point. And that tiny difference had a monetary change of $25,000 difference from 1st to 2nd. Look at how many positions in contests are separated by less than 2.2972 points. That's the equivalent of one judge out of the 20 taste scores that count changing by one single point.

 

[Pappy Q]

When I took the CBJ class five years ago, he instructor said a nine was bbq you would drive 250 miles one way just to eat that bbq...no other stops or as part of a trip. Just drive to the bbq, eat and drive back home. That message was givem to me amd 25-30 other potential judges and I have never been told differently from KCBS.

Must have been the same instructor I had as that what my class was taught. When I challenged the logic, she got real upset.

 

[enasnidx]

So what if they added fractional scoring to the point system? I'm not asking because I think it's a fix. I'm asking because I'm curious what you all think the impact would be.

Would judges be more willing to pass out a 6.5 or a 7.5 where they may have "rounded up" in the past (as I've heard some judges share they do - whether intentional or not). We've all heard the stories that some judges have been coached that "cooks spend a lot of money to compete", blah blah blah, so they are reluctant to pass out 7's (and of course 6's) knowing it could kill someone's chances of a call. Again, clearly not a fix, probably just another band aid, but does this benefit at all? Does it just add unnecessary time/complexity to the scoring? Does 7.5 just become the new 7? For better or worse, the rankings will certainly get tighter as more positions are decided by even smaller fractions of a point.

 

[Blanton]

So what if they added fractional scoring to the point system? I'm not asking because I think it's a fix. I'm asking because I'm curious what you all think the impact would be.

Would judges be more willing to pass out a 6.5 or a 7.5 where they may have "rounded up" in the past (as I've heard some judges share they do - whether intentional or not). We've all heard the stories that some judges have been coached that "cooks spend a lot of money to compete", blah blah blah, so they are reluctant to pass out 7's (and of course 6's) knowing it could kill someone's chances of a call. Again, clearly not a fix, probably just another band aid, but does this benefit at all? Does it just add unnecessary time/complexity to the scoring? Does 7.5 just become the new 7? For better or worse, the rankings will certainly get tighter as more positions are decided by even smaller fractions of a point.

I would see this as just one more variable to an already confusing issue. I agree issues still come down to:

• Lack of direction on what to score to. Much like the beer judging examples provided.
• And different starting points on the scale.

With that said, I do believe if you are turning in top notch product then it really doesn't matter. There is no way the top team can always miss the TOD at a contest. Us mortals just need to get better.

 

[Rusty Kettle]

Isn't seating based on past events just trying to inflate scores? What good does that do? It sure is not helping a new cook. Sure no one is probably going to complain about higher scores but is it really helping anyone? Me personally I would rather get an actual score that I earned than an inflated score to make me feel good about myself when I shouldn't.

If I deserve a 6 give me a 6. Why? How will I improve if I believe I am on the right track if I am not. Also, if its between a 6 or 7 or 7 or 8. Do me a real favor round down regardless. Why? So I know it wasn't the higher number. So if its not an 8 but it is almost an 8 give me a 7. Why? So I know I didn't get an 8. That way I can know I need to re-evaluate what I did and learn from it.
I didn't ask for a pat on the back. I asked for judges to be critical of my food and give honest feedback. If I wanted to get a pat on the back for being mediocre then I would call up my friends and family and ask them as its a heck of a lot cheaper.

I am asking judges to please give me the score I earned. If its not an 8 or whatever then give out the score earned. If its almost an 8 but you think its possibly a 7 then give a 7... Its still not an 8 so don't give an 8.
I spend alot of money for a real evaluation of my skills. Please don't pat me on the back if I don't deserve it. Make me earn my scores.

 

[GrillBillie_D]

Isn't seating based on past events just trying to inflate scores?

Rusty, the changes are not trying to inflate scores. It's really trying to average out the scores across the table. If you have two judges who historically score lower than the average judge, then they try not to seat them at the same table. Same goes for judges who score higher. They are mixed up, based on how they have scored in the past. There is no punishment to a judge who scores lower or higher. They are just seated differently.

I do agree with you that judges should give you a fair score. The problem is that different judges were trained differently as to what those numbers mean. Some were trained to start at 6 and go up. Some were told to start at 9 and go down. Some were told a 9 is something they would drive hours for. There is no consistency as to how different judges were trained or how they give you a score. That is really the core issue that still has not been addressed.

 

[rorkin]

This drive 250 miles approach illustrates the craziness of the system,.

Obviously you would drive 250 miles because you really like it.

But the rules are judge to the standard not what you like .

Sheesh !!

 

[ModelMaker]

So if 250 mi is a 9 then is 175 mi the illusive average (6)?
The problem I've seen is of the 6 different CBJ training Reps whose classes I have helped with or attended they each have this little bit of wisdom they each use to quantify a point in the power point presentation. Thus you have different thoughts taught throughout the country.

Now if you really want identical classroom instruction these little gems need to be avoided like the plague. Each class must be presented exactly the same everywhere.
No personal adlibs at all.
I also see no reason not to have judges recertify every (pick a number) years. If you want to judge put in the effort. If not then someone else would love to get some more contests.
Ed

 

[Jorge]

So if 250 mi is a 9 then is 175 mi the illusive average (6)?
The problem I've seen is of the 6 different CBJ training Reps whose classes I have helped with or attended they each have this little bit of wisdom they each use to quantify a point in the power point presentation. Thus you have different thoughts taught throughout the country.

Now if you really want identical classroom instruction these little gems need to be avoided like the plague. Each class must be presented exactly the same everywhere.
No personal adlibs at all.
I also see no reason not to have judges recertify every (pick a number) years. If you want to judge put in the effort. If not then someone else would love to get some more contests.
Ed

I've been told, by someone that was in a position to know, that at one time there was a proposal to do it via video having it produced by a college or university. The theory being that it would standardize instruction. It just needed board approval, which did not come.