FiveThirtyEight Establishes Process for Pollsters to Review its Database of Their Polls

Yet another housekeeping item, but an important one, as we have established a process by which a pollster may audit polls as they appear in the database used in connection with our pollster ratings. That process is described below.

1. Pollsters may request, and FiveThirtyEight will provide, a list of all polls included under their names in our polling database. Requests should be sent to our polling “hotline” at 538poll@gmail.com. A contact phone number should be provided along with the request. Please be aware that it may occasionally take 72 hours or more to respond to requests, although response times will usually be much faster.

2. Pollsters are strongly encouraged to request any corrections to the database that might be necessary, including both errors and omissions. FiveThirtyEight will honor any and all requests for corrections, provided that the following two criteria are met:

a. Documentation is provided for the correction, for example, a link to a news account of the poll, or a PDF of a press release. Reference to the poll in another secondary source, such as Pollster.com, PollingReport.com, or RealClearPolitics, will suffice as documentation, although pollsters should be aware that these databases themselves contain errors.

b. FiveThirtyEight is satisfied that the pollster has made a reasonable and good-faith effort to provide an unbiased list of discrepancies to FiveThirtyEight, i.e. the errors have been notated to FiveThirtyEight whether or not they would favorably affect the pollster’s rating. By sending a request for corrections to FiveThityEight, the pollster warrants that it has in fact made such an effort. Although FiveThirtyEight will give any benefit of the doubt to the pollster, if in its best judgement the list of corrections requested by the pollster was deliberately biased (i.e. cherrypicked), it may refuse to accept any correction requests from that pollster, it may publicly reprimand the pollster, or both.

3. Pollsters should not be surprised to discover that errors and omissions in fact exist in the database. More often than not, the errors and omissions will also exist in the original sources that FiveThirtyEight used to compile its database, but this will not be true in all cases. So far, four pollsters have been provided with their data, covering approximately 400 polls; they identified 3 errors in the margin that FiveThirtyEight had recorded between the candidates, and 10 improperly omitted polls. More common will be errors related to the sample size of the poll, as this information was not always available from the original sources. For approximately 5-10 percent of the database, sample sizes were estimated rather than taken from hard numbers. Errors and omissions are more likely to occur in Presidential primary polls prior to 2008, in polls for the U.S. House, and in polls for gubernatorial races, and less likely to occur in general election polls for President and the U.S. Senate and the 2008 Presidential primaries

4. FiveThrityEight would appreciate if the pollster carefully review our methodology for including polls prior to submitting correction requests. The review process is not intended to be the venue for debates about FiveThirtyEight's methodology.

5. In some cases, there is ambiguity as to how to credit the poll, e.g., between the fieldwork provider and the media partner. FiveThirtyEight appreciates any information that would help to clarify these ambiguities, but retains the final decision on how to classify any polls.

6. FiveThirtyEight will issue a new set of pollster ratings no earlier than July 15th, but no later than August 15th. All requests for corrections should be provided to FiveThirtyEight by Friday, July 9th.