KCBS Score - Judges' Scorecard is live

[rob g]

You're probably right, but I would like to see this posted as public information. There is no reason why we as judges should be provided with the ability to decide the results of a contest, and not be prepared to stand behind the scores we gave

As a judge I have no problem justifying the score I give. The only problem I have is remembering why I gave a particular piece of chicken a 6 by the time I finish judging brisket. I do comment cards for a 5 or less. If teams want a comment card for every sample we would need more time at the judging table to do this. Worst case we have 20 minutes to judge 6 samples, clean up and be ready for the next turn-in.

 

[Okie Sawbones]

yeah we wouldnt want judges to suffer the transparency that us cooks have to! hahahah

I think this is really useful for organizers and reps if they let them be more involved in the selection of judging They have no say but are responsible for seating them

Don't be snarky now. :cop: I for one welcome this wholeheartedly. It will let the organizers see 'that judge' who is 2 points below the table average, and then hopefully avoid their application. We don't need these people judging a cook. It hurts everyone.

 

[Big Poppa]

I was only kidding Seriously its amazing that its halfway through 2016....

I also do believe that it shouldnt be secret.

 

[ModelMaker]

This list will never indicate who the judge is except to the judge accessing it. Tablemates will not know who they judged with. Organizers will not be able to use for seating.
So what's the purpose or need?
I'm not a number cruncher, you can make numbers say whatever you want.
I don't care what the judges at my table scored, am I supposed to change how I score because of my neighbors?
What did I learn from this report on my last contest?
I gave out
16 - 9's
30 - 8's
15 - 7's
6 - 6's
2 - 5's
I scored higher than some, lower than some.
What did I learn, nothing that I needed to know.
Ed

 

[rob g]

Don't be snarky now. :cop: I for one welcome this wholeheartedly. It will let the organizers see 'that judge' who is 2 points below the table average, and then hopefully avoid their application. We don't need these people judging a cook. It hurts everyone.

Is the judge that is 2 points below the table out of line or are they just using the whole scoring range as it was originally intended? Is the rest of the table judges that only use 7,8,9? If you weed out the low judge will you end up with a panel of judges that only use 8 and 9? The scoring feedback is interesting but there are a lot of other factors coming into play i.e. equality of all samples in a box, initial training of judges etc.

 

[bignburlyman]

I was finally able to get logged in, and get to the info. Based on the info I am reading, my total score was never more than 1 point higher or lower than the table average, except for the 1 contest when I had 2 raw chicken entries :mmph: . I don't know yet what value this info provides, but it is interesting to look at. I could not tell you from this info if I was ever a "Table of Angels" or "Table of Death".

Edit to Add: I don't think all my contests are listed on the database.

 

[Right on Q]

Is the judge that is 2 points below the table out of line or are they just using the whole scoring range as it was originally intended? Is the rest of the table judges that only use 7,8,9? If you weed out the low judge will you end up with a panel of judges that only use 8 and 9? The scoring feedback is interesting but there are a lot of other factors coming into play i.e. equality of all samples in a box, initial training of judges etc.

The point is not whether judges use a 9 point scale or a 2 point scale. It's that all judges use the same scale and standards when they judge. This would help eliminate TOA and TOD at contests.

 

[DUBBAGA]

Crunched the numbers for 30 events that are listed, only taking the category averages (me vs table)

Chicken - 0.20 avg lower than table
Ribs - .019 avg higher than table
Pork - 0.45 avg lower than table
Brisket - 0.53 avg lower than table

My total average combined score is 127.6908367, divided by 4 categories = 31.92. Giving an entry 8 8 8 = 32 points. Therefore my average score is just slightly less than an 8 8 8. My table average combined score is 128.68465, or an average of 32.17 per category. Therefore per category, I am averaging 0.25 points lower than my table.

It doesn't seem like much, but put 2 or 3 judges like me who use a wider scoring range together, and it is irrefutable that we are skewing the final results.

More than ever before, I really want KCBS to use assigned seating based on some form of historical judging averages

 

[Jorge]

Crunched the numbers for 30 events that are listed, only taking the category averages (me vs table)

Chicken - 0.20 avg lower than table
Ribs - .019 avg higher than table
Pork - 0.45 avg lower than table
Brisket - 0.53 avg lower than table

My total average combined score is 127.6908367, divided by 4 categories = 31.92. Giving an entry 8 8 8 = 32 points. Therefore my average score is just slightly less than an 8 8 8. My table average combined score is 128.68465, or an average of 32.17 per category. Therefore per category, I am averaging 0.25 points lower than my table.

It doesn't seem like much, but put 2 or 3 judges like me who use a wider scoring range together, and it is irrefutable that we are skewing the final results.

More than ever before, I really want KCBS to use assigned seating based on some form of historical judging averages

How did you determine that you are part of the group that's skewing results? Greater standard deviation?

 

[DUBBAGA]

How did you determine that you are part of the group that's skewing results? Greater standard deviation?

I'm not smart enough to do that kind of fancy stuff. Just my feeling that if I am scoring XX.XXXX points lower than the table based on an average of 30 events, then it would be safe to reason that having multiple people with the same tendency at the same table would skew them

 

[rob g]

The point is not whether judges use a 9 point scale or a 2 point scale. It's that all judges use the same scale and standards when they judge. This would help eliminate TOA and TOD at contests.

I agree 100%. The problem as I see it there is definition of what constitutes a 9 or a 5. What is average? Should average only correspond to competition meat or to your complete bbq experience? Until some of these things are defined in the rules and training you will never get uniformity on the score cards. Every judge comes from a different backgtound.

 

[Jorge]

I'm not smart enough to do that kind of fancy stuff. Just my feeling that if I am scoring XX.XXXX points lower than the table based on an average of 30 events, then it would be safe to reason that having multiple people with the same tendency at the same table would skew them

Just using the work you've done, I'd contend that it indicates that there are too many judges not using the entire range of possible scores. I've got a problem believing that the number of cooks consistently turning in nothing but 8+ food is >= those turning in 8 or below.

Based solely off of the numbers you provided, I'm more inclined to think that the other judges should be scoring more like you are.

 

[CBQ]

It will let the organizers see 'that judge' who is 2 points below the table average, and then hopefully avoid their application

I have no issues with low scoring judges, except when they sit at the same damn table. (Birds of a feather, and all that.)

Organizers should not have access to this information when taking judge applications, however once judges arrive at a contest and sign in with their KCBS number, seating should be done via software that, as much as possible, creates tables that have an average close to the contest average. That is what would be most fair to the teams. Let everyone face the same mix of judges.

Educating judges is well and good (and KCBS should rightly try to make it uniform) but you can't fix human nature. People aren't the same. Fix the system instead.

 

[Big Poppa]

trust me he is the barber of so cal....;hahahaha Just kidding Scott

 

[Right on Q]

Using an example from this weekend. My ribs ranked #1 on my table and placed 19th out of 40 overall. A friend of mine got LAST on his table and placed 16th out of 40 in ribs.

Am I mad at each judge about how they scored? Not particularly. If they think my food deserved 888s that's fine. I am upset however that all of these low scoring judges sat at the same table.

However, looking from a judges perspective, if I were a low scoring judge and I sat down at a table how would I know the other judges are low scoring as well? I don't think judges are planning to create TOD and TOA when they pick their spots but KCBS has the data to easily correct this

 

[Vince RnQ]

I am upset however that all of these low scoring judges sat at the same table.

However, looking from a judges perspective, if I were a low scoring judge and I sat down at a table how would I know the other judges are low scoring as well? I don't think judges are planning to create TOD and TOA when they pick their spots but KCBS has the data to easily correct this

Exactly right! The vast majority of the judges are doing their job to the best of their ability. It is the system we are all forced to work with that is in need of change and KCBS does nothing toward that end. Put the blame where it belongs: on the KCBS Board of Directors.

Every team who lands on a TOA or TOD should send an email to the BoD telling them that they have tools at their disposal to prevent these things from happening. They'd be getting literally hundreds of emails each week. Maybe then some of the stubborn Old Guard would start to recognize that there is a significant problem.

 

[rickv14623]

I agree 100%. The problem as I see it there is definition of what constitutes a 9 or a 5. What is average? Should average only correspond to competition meat or to your complete bbq experience? Until some of these things are defined in the rules and training you will never get uniformity on the score cards. Every judge comes from a different background.

Been saying this as of late. I think some judges are judging that average = backyard + comp + local joint + chain joint.

You are judging comp Q against the "standards" of comp Q. It is like getting a white sauce entry or an ancillary that is not your favorite item. It is not do I like/dislike white sauce but is it an excellent example of what the cook intended.

 

[Candy Sue]

My opinion -- I am no longer on the board, but this is based on my board experience! This is an evolving process for KCBS! The judges score sheet is an important FIRST step. Second steps will be deeper analysis of judging and overall examination of trends, and setting up a method for reps and organizers to use the data to improve contests. I've always felt it was important for KCBS to give CBJs an objective picture on how he/she judged compared to the rest of the judges at that table, at that contest.

Personally, I believe I'd like to see decimals used. We've gotten to a mostly 7-8-9 scoring, which puts most judges in the 2 point range. If judging was (loosely) based on grade scoring (with a period in between) 6.0 to 9.9 would open up the scoring system. But that's just me. This would be a huge change.

 

[Right on Q]

Personally, I believe I'd like to see decimals used. We've gotten to a mostly 7-8-9 scoring, which puts most judges in the 2 point range. If judging was (loosely) based on grade scoring (with a period in between) 6.0 to 9.9 would open up the scoring system. But that's just me. This would be a huge change.

Adding decimals would helps spread out the scores for a while but eventually the scores would migrate to 9.8-9.9-10. Just look at the MiM, MBN scoring system and this is exactly what's happened over there.

I agree about the first steps and I'm glad to see the board doing something

 

[thirdeye]

I'm glad to see this information is finally available, I just need to figure out how to best utilize it, and which numbers are most important to me. Since my records only go back to 2013 I have 10 contests of data to sort through and the contests include a couple of first-year comps with only 15 teams and many first time cooks & first time judges... to comps with 45 or more teams with the majority of the judges having 12 or more contests under their belts, and usually at least one master judge at each table. The KCBS reps in my area all have similar methods for seating, based on years of experience and number of contests judged, so it seems there is a good mix at each table.

I'm generally right with the pack, but I occasionally see one or two lines where I have a 3 point swing from the table average, but then category average will tighten back up.