Kant touch that!!!

[rob g]

I wouldn't have an issue with refresher classes but for the most part I think they will be a waste of time. The reps already do a pretty good job of highlighting the rules at every judges meeting and pointing out any new rulings from on high. We get to listen to the audio tape on judging before every comp to the point most people zone out. Until KCBS starts to define what constitutes a 6 or a 9, and why, we aren't likely to see any changes as the definition is at the discretion of the individual judge.

To be honest I tend to score lower than the table average when I check my scores after a comp. I tend to use the whole scoring range and give the sample a 9 when it deserves it or a 4 or 5 when it is lacking. At the same time I make a point of a comment card for 6 or under to justify why I gave the score I did. I usually have occasion at least once or twice a season to do double duty as a judge and table captain. I get the chance to review the other cards at my table. It blows me away when I score a 5 or 6 and another judge gives a 9 for the same meat. I realize meats from one box can vary in quality but I often see a 7 given as the low score from that judge for the day. The new seating system does average this out so most tables are balanced

 

[eggzlot]

Well it is officially the "offseason" when the judging threads start popping up. Is the system flawed? people think so - but then why do the same teams consistently win or finish in the top week after week? End of day, the cream usually rises to the top. If the judging was so flawed I think you'd see less consistency from teams placing in the top and getting calls week after week? Maybe someone could prove me statistically wrong but that is how I view the topic.

 

[Jason TQ]

Well it is officially the "offseason" when the judging threads start popping up. Is the system flawed? people think so - but then why do the same teams consistently win or finish in the top week after week? End of day, the cream usually rises to the top. If the judging was so flawed I think you'd see less consistency from teams placing in the top and getting calls week after week? Maybe someone could prove me statistically wrong but that is how I view the topic.

I doubt anyone in this conversation is saying the top teams don't still finish in the top regardless of whatever system is in place. That is abundantly clear. Maybe I missed that......Seems more of a general how do we improve the system.......I could be wrong.

 

[eggzlot]

I doubt anyone in this conversation is saying the top teams don't still finish in the top regardless of whatever system is in place. That is abundantly clear. Maybe I missed that......Seems more of a general how do we improve the system.......I could be wrong.

I am just playing devils advocate mostly - but if the same teams rise to the top, I am not sure if the system is that flawed. A few weekend warriors (like myself) may think so, but if the same teams week in and out consistently win, maybe the judging is fair across the board? I think if teams had very inconsistent scores week in and week out - especially the teams with 20+ contents a year - then maybe one could say judging is flawed and not accurate and we need to rip apart the system and start over.

This is the opinion of someone who did 5-7 contents for 3-4 years and stopped 2 years ago and also judged 3 contests as a CBJ within KCBS. Team got some calls, a RGC, etc. Take it for what its worth. As a casual observer these days I do see some flaws and areas improvement for KCBS but I think people bash judging too much sometimes. There is always something small to nitpick but as I’ll repeat, the cream does rise to the top time and time again.

 

[ModelMaker]

I believe KCBS needs to define "average" as those meats found on the judging table. This will narrow the wide variation in exposure to bbq to a smaller range. As an example my exposure to bbq was burgers, dogs and the occasional steak growing up as a kid here in Canada. Compare that to a kid growing up in the southern US that was eating low and slow bbq from the time they were old enough to lick the sauce off the bones. It will take new judges 3-4 competitions to get a handle on competition meat but their scores will soon come into line. If most of the meats are getting a 6 then the truly stellar sample actually has the ability, and room to move, to separate itself from the pack. In my opinion most of the samples I get tend to taste the same. It's usually how well the meat is cooked that makes it stand out for me. At the same time I think judges need to be a little more critical in their judging while using the complete scoring range. Giving only 7-9 scores makes judging easy but isn't particularly fair to the teams that actually have turned in a superior product.

If there's a nail sticking out of the beginning BBQ judges experience board, Mr G has stuck it right on the head!
Where the new judges BBQ experience starts makes all the difference in what their first few contests will play out score wise. But in a short time they will understand what a 6 is and what a 9 is.

I have helped with the KCBS CBJ class here in Iowa for probably 10 years. The presentation is smooth and consistent and each new judge starts off with the same ammunition so the variable is mostly life's BBQ experience.
Ed

 

[tduffy]

The new seating system does average this out so most tables are balanced

Not what I seen this year.

 

[rob g]

Not what I seen this year.

You can only do so much to balance the judges on a table by evening out the high scoring judges with low scoring judges. You have no control over the meat landing on that table.

 

[Serious Hog]

A 3 point scoring system means that on my worst day, my entries are going to score the same as the team that turned in excessively smoked, ash tray tasting garbage. That or they get a middle score that they didn't deserve. That's not how we fix judging.

I will be first to admit the 3 point system was less of a well thought out plan and more of a reaction to these judging threads that pop up week in and week out over the 5 years I've been on this forum (and probably long before that). I get that people don't think it's a good long term solution and respect the reasoning. But I'm not sure all the re-training and re-certification in the world is gonna change competitors belief that judging is not being done in an overall consistent way either. In my opinion, "unteaching" people's mindset, whether it be BBQ judges or anyone else, is just a really difficult thing to do.

Anyways on the scenario you mentioned, I was picturing a 7 (BAD) for the ash tray garbage. I'm guessing your worst entry is still some good tasting bbq, maybe just doesn't quite have the flavor, tenderness, or presentation you would get on a better day. That would get an 8 (GOOD).

 

[tduffy]

You can only do so much to balance the judges on a table by evening out the high scoring judges with low scoring judges. You have no control over the meat landing on that table.

The idea that balancing tables by scoring average only is a terrible idea in the first place. As I stated earlier I cooked a contest in July where a judge I knew had stated it created a table of all women and all old people. Stating that seating judges by scoring average is the best way to seat judges is an opinion. Tell me a table of judges of same gender or age is fair just because their scoring averages(before the contest) make even tables? Not buying it.

 

[rob g]

The idea that balancing tables by scoring average only is a terrible idea in the first place. As I stated earlier I cooked a contest in July where a judge I knew had stated it created a table of all women and all old people. Stating that seating judges by scoring average is the best way to seat judges is an opinion. Tell me a table of judges of same gender or age is fair just because their scoring averages(before the contest) make even tables? Not buying it.

Given the minimal data KCBS has to work with what do you suggest?
Experience level? I've seen master judges that only use 8 and 9. They use a 7 to knock a team out of contention but not embarrass them.
How is a table of seniors any different than a mixed age table or a table of ladies? Given equal experience levels you should see consistent scoring.
I think you will find most people, with at least a few comps under their belt, will score fairly consistently. Whatever they deem good or bad bbq will remain common from comp to comp.

While the KCBS seating program may not be perfect I think it is the best they are going to be able to do to eliminate the perceived variation between tables. Getting the judges to be more critical in their judging and using the whole scoring range would help open up the field for the top meats to stand out but wouldn't likely upset the table balance

 

[Serious Hog]

How is a table of seniors any different than a mixed age table or a table of ladies?

Taste buds begin to fade with age, so the way a table of all seniors perceives taste will likely vary from table of all younger judges. Women tend to like spicy food less than men. Neither of these are hard and fast absolutes, but they are tendencies that can change the balance of a table.

 

[tduffy]

Given the minimal data KCBS has to work with what do you suggest?
Experience level? I've seen master judges that only use 8 and 9. They use a 7 to knock a team out of contention but not embarrass them.
How is a table of seniors any different than a mixed age table or a table of ladies? Given equal experience levels you should see consistent scoring.
I think you will find most people, with at least a few comps under their belt, will score fairly consistently. Whatever they deem good or bad bbq will remain common from comp to comp.

While the KCBS seating program may not be perfect I think it is the best they are going to be able to do to eliminate the perceived variation between tables. Getting the judges to be more critical in their judging and using the whole scoring range would help open up the field for the top meats to stand out but wouldn't likely upset the table balance

I suggest a seating system that factors in many things and not just scoring average. Age, gender, contests judged, MCBJ etc.

A table of seniors will have a different pallate than a table of women.
Seating by scoring average did not eliminate any tables of death or angels. If anything I seen more of them this year than years past.

 

[Pappy Q]

I suggest a seating system that factors in many things and not just scoring average. Age, gender, contests judged, MCBJ etc.

A table of seniors will have a different pallate than a table of women.
Seating by scoring average did not eliminate any tables of death or angels. If anything I seen more of them this year than years past.

Using age and gender as a seating factor could potentially be viewed as discriminatory. There was a time in this area when some Reps used that criteria for seating. It’s my understanding there were some complaints to KCBS which lead to being not allowed.

 

[airedale]

Using age and gender as a seating factor could potentially be viewed as discriminatory. There was a time in this area when some Reps used that criteria for seating. It’s my understanding there were some complaints to KCBS which lead to being not allowed.

Even if you wanted to consider things like age, gender, and judging experience in addition to historical scores, how would you do it? Some kind of mathematical formula to produce a "judge coefficient" for each one? And would you grab weighting parameters out of the air?

It seems like considering as many judge characteristics as possible would be good, but I don't know how you could mathematically consider, in a validated fashion, anything but scoring history and number of comps. And even then how would you validate a theory that said more of comps produces a more objective judge? Does it?

 

[gettinbasted]

Seat judges completely at random.

There isn’t enough data, and what data their is completely omits the most important variable (quality of food judged). Cooks screamed and screamed for this seating system and they got it. It doesn’t work very well (thank God!). Seating by scoring average is trying to use past data to influence future outcomes. The outcome desired is all tables giving out the same scores. How does that make any sense?

I’d go further and say that the very act of tracking a judges performance has narrowed the scoring range judges will use even further. It’s human nature to not want to stand out.

KCBS judging is not broken. It’s the best system by far. I think most cooks get frustrated by the 2-3 low outlier judges at each contest and that’s an easy thing to blame on the ride home. Why only low? Because we don’t have enough room for a high outlier with the scoring range used.

Some common sense, well communicated directives could be used to get everyone on the same page, that, more than anything, will be beneficial.

The danger lies with no standards or communication. Where judges individually decide how to apply the scoring range. I think we have a little of that going on now.

As to the original post. Push, pull, kick, poke, whatever you want to a piece of meat, but ultimately mouth feel and the texture as you eat should determine a tenderness score.

 

[Slamdunkpro]

Seat judges completely at random.

^^This^^

You can't get meaningful data by averaging random events.

 

[GrillBillie_D]

Has anybody taken the time to talk to the people who created the judge seating program to see how the data they collect shows how scoring has changed? It's easy to speculate that the program is not working but what about the people who see the data and see how it has impacted overall scoring all across the country.

In my opinion, seating them by experience was not working. Seating completely randomly may be better. In the contests I was in this year, there were far less swings for high scoring tables vs. low scoring tables. It looked to me that things were better than before, at least in the contests I competed in.

 

[gettinbasted]

Has anybody taken the time to talk to the people who created the judge seating program to see how the data they collect shows how scoring has changed? It's easy to speculate that the program is not working but what about the people who see the data and see how it has impacted overall scoring all across the country.

In my opinion, seating them by experience was not working. Seating completely randomly may be better. In the contests I was in this year, there were far less swings for high scoring tables vs. low scoring tables. It looked to me that things were better than before, at least in the contests I competed in.

I’m so glad you brought this up! This isn’t personal to you, but allows me to make a broader point. What would the data need to show to prove that it’s working? What is the desired result? That scoring is now bunched up? That all tables now score exactly the same? This is the worst case scenario in my mind. This is a competition and we are actively trying to manipulate the results to be equal off of each table. The food is not equal.

This solution does not address the problem. It attempts to bury it by manipulating how each table scores. Keep in mind that it COMPLETELY IGNORES the quality of food that comes across each table.

Repeat after me. The problem is not TOA or TOD. That is a symptom, but our solution tries to eliminate them in a vacuum. Sometimes the top 3 entries or 5 of the bottom 10 entries should be off of the same table. Food quality matters.

I know I’m in the minority by a long shot in my thinking on this, but use some critical thinking skills. Write down the problem and then tell me how this solution does anything to solve it.

End of rant!

 

[airedale]

... Write down the problem and then tell me how this solution does anything to solve it. ...

Well, maybe your "solution" of making all tables scores similar is a straw man that does deserve to get knocked down.

I am not hampered by any actual experience or data here, but what I would do if I were "leveling" judges would be to look at how their historical scores compared to their table average. Then in assigning judges to a table I would try to match a judge that usually scored below the table average with one that usually scored above, then maybe add a couple of judges that typically scored near their table average, etc.

This would recognize that different tables had different food to judge but it would tend, I think, to reduce the effect of outlier judges at any table. And maybe over the long haul the outliers would move more towards table averages as they compared themselves to their tables.

But, that said, I am kind of agnostic on this new judge assignment thing. I am too new to really have the historical perspective that would let me have a valid opinion.

 

[Nordy]

Judging meat tenderness by pushing, tugging, draping, whatever with your hands is like determining softness of a blanket by licking and chewing it... Stupid.

Food is made to be eaten. Tenderness should be judged by "mouth feel" ie how it chews...

Happy New Year... is it BBQ season yet?

Nordy