Men Have Earned Nearly Double What Women Have? The Study Was Flawed!

A recent study published at Maltese site, reported on by the AV Club, made the outrageous claim that “men have earned nearly double what women have”.

It is the belief of The Jeopardy! Fan that these claims are inherently biased by a poor use of a dataset and completely flawed to the point that it is irresponsible to report on them.

The methodology, while it acknowledges that the data is incomplete, appears to be working on the assumption that the incomplete data is representative.

As a Founding Archivist at J! Archive, I can categorically say that the data is far from representative, and not representative to the point that making any conclusions based on that data is improper and irresponsible.

One claim in the original study was that the 1991-92 season was “easiest”, because it had a ratio of “4.2 correct answers for every wrong answer”. That season, in J! Archive, has only 48 games out of 230 recorded. Of those 48 games, easily the featured set was Jerome Vered’s complete run of 5 games. It would therefore stand to reason that Jerome, who completely dominated and steamrollered his opponents in May of 1992, would boost that average significantly because his games take up so much of that season’s average. Its conclusion is thus flawed.

The other outrageous claim was comparing men with women. Again, the vast majority of the early games in J! Archive were put in there of players who later returned to the show in tournaments, either a Tournament of Champions, Million Dollar Masters, Super Jeopardy, the Ultimate Tournament of Champions, or the Battle of the Decades. Most of those “dominant” players were male. Thus, again, the dataset is inherently biased towards male players, and female players who might not have won enough (but still enough that it would have brought things back closer to its true average) are less included from earlier seasons (simply out of the fact that fewer of their games have appeared in private collections or video sharing sites over the years).

It is thus the opinion of The Jeopardy! Fan site that any story that reports on this study or dataset is inherently flawed and its conclusions are inherently invalid.

Looking to contact Andy Saunders for further information about this story? Check the About page!

4 Comments on "Men Have Earned Nearly Double What Women Have? The Study Was Flawed!"

  1. I respectfully disagree with your prima facie rejection of the conclusions simply because we don’t want it to be true.
    You criticize one non-provocative finding about the 1991 season because their data set was a mere 20% of the lot? And therefore, all other correlations and associations are null and void?

    It doesn’t sound like they were irresponsible or wreckless, nor did the article I read seem judgmental or authoritative.

    It is important to filter results. When Donald Trump says anything, you can start with it as false, and look for something factual. If any part is a lie, then anything from that source needs questioned.
    Thank you for describing a potential softness in the data review. Until you provide the contradictory and improved data set, I will accept the data at face value.
    We will not obtain equality and fairness by pretending we have more than we do.

    • Andy Saunders | February 5, 2017 at 12:18 am |

      Did you even look at the article? They listed clue values as 100, 200, 300, 400, 500, 600, 800, and 1000. This complete and utter lack of domain knowledge renders them, in my opinion, completely 100% unqualified to analyze this data.

      Look at this statement: “While round one questions worth $1,000 had the lowest likelihood of being answered correctly (almost 1-to-1), questions valued at $500 proved to be the second most difficult. Additionally, categories worth $600 and $800 had a ratio of over 2-to-1 in both rounds, which just goes to show – you never know which question is going to be the one that stumps you.”

      If you don’t know enough about the history of Jeopardy! to know that $500 and $1000 are the exact same difficulty level ($500 clues were at the bottom row of the Jeopardy! round until November 2001, before dollar values were doubled), you do not have the relevant expertise to make conclusions based upon the data.

      In terms of an improved data set? Until Seasons 1-13 reach the same level of completeness as Seasons 14-33 do on J! Archive, a competing data set does not exist in order to do a proper analysis. Which, frankly, was the entire point in my calling it flawed in the first place.

      I will continue to reject the conclusions made in the two articles wholesale. It was not a rejection out of not wanting it to be true; it was a rejection because, as a Founding Archivist of J! Archive, I am intimately aware of the limitations of the data set as it applies to early seasons and a knowledge of the inherent biases within those early seasons.

  2. Whoa, tiger

    They certainly did bungle the analysis of clue value vs. wrong answers. For that matter, the math on the answer ratio seems to fail my smell test, but I haven’t watched faithfully for 32 years. Maybe contestants have more recently learned to stay clam and don’t miss as many now. Most of the BA’s are well over 85%.

    Point is, they mined information and show a serious gender gap, and there is no exculpating information to indicate there isn’t one. Maybe we just need to throw out Ken Jennings and Brad Rutter (or whoever) to homogenize the data set.

    Again, I don’t see any overtones in the article study. It was a nice trivia tidbit, and yours was a nice counterpoint. I hope TPTB have their own data and make the clues progressive and choose the contestants fairly. We are interested in meeting new smart and interesting people on the show and admiring their skill and smarts.

    Thanks for providing an interesting website, or blog, or whatever this is.
    kdmeares/aka twelvefootboy

    • Andy Saunders | February 5, 2017 at 5:34 pm |

      Asserting that a proposition is true simply because it hasn’t been proven false (“they mined information and show a serious gender gap, and there is no exculpating information to indicate there isn’t one“) is a logical fallacy. Argumentum ad ignorantiam, I believe.

Comments are closed.

%d bloggers like this: