United States: Using Daubert To Exclude Flawed Surveys

We typically think of expert testimony as providing an insight into the evidence in the case, or drawing a conclusion from the evidence, that requires knowledge beyond the ken of a typical judge or juror. But expert testimony also can be used as a substitute for evidence that a party cannot, or does not want to, present through traditional evidentiary methods. Although courts have allowed such expert testimony in certain contexts, there is cause for concern when a party offers an expert whose function is to fill a gap in the evidence.

Notable among this category of expert testimony are opinions offered during class-certification proceedings in an effort to show that a case can be efficiently managed on a class-wide basis. Such testimony often takes the form of surveys or other statistical sampling techniques designed to establish liability or damages on a class-wide basis without requiring adjudication of each individual claim.

In the past few Terms, the US Supreme Court has addressed the permissible role of such surveys under Federal Rule of Civil Procedure 23 in certifying and maintaining class actions. See Tyson Foods, Inc. v. Bouphakeo, 136 S. Ct. 1036 (2016); Wal-Mart Stores, Inc. v. Dukes, 131 S. Ct. 2541 (2011). Even when such gap-filling expert testimony is allowed, however, it still must pass muster under the rules governing admissibility of expert testimony.

A recent decision authored by Judge Charles Breyer of the US District Court for the Northern District of California addresses both the permissible uses of surveys under Rule 23 and the admissibility of those surveys under Daubert. Of interest to us here, the decision provides an evidentiary-based blueprint for excluding surveys that are often commissioned by plaintiffs' lawyers for litigation.

In In re: AutoZone, Inc. Wage and Hour Employment Practices Litigation, 2016 WL 4208200 (N.D. Cal. Aug. 10, 2016), the plaintiffs alleged that their employer, AutoZone, failed to provide rest breaks in accordance with California law. The district court initially certified a class on the premise that AutoZone had a facially invalid policy throughout the class period and that the defendant's records could establish whether individual employees were permitted to take rest breaks. As litigation progressed, however, it became apparent that AutoZone's policy changed during the class period, that the policy was applied inconsistently and—most importantly—that relevant plaintiff-specific records did not exist. These new developments prompted AutoZone to file a motion to de-certify the class.

The district court agreed that, in light of the evidentiary gaps in the record, the class did not satisfy Rule 23's requirements of predominance and manageability. In so concluding, the district court rejected the plaintiffs' argument that a survey commissioned for litigation purposes could fill those evidentiary gaps, holding both that it was an impermissible use of a survey under Rule 23 and that the particular survey offered by the plaintiffs was inadmissible under Federal Rule of Evidence 702 and Daubert. Without that survey evidence, the district court concluded, the plaintiffs could not maintain the class.

Before delving into the district court's reasoning, a few words about the survey are in order. The plaintiffs commissioned their survey of class members to help calculate damages. The plaintiffs attempted to use the survey to maintain the class and establish liability only after the evidentiary gaps in the record became glaring. The survey asked class members various questions concerning whether they were allowed to take rest breaks during their shifts. For instance, of the survey respondents who had worked shifts lasting between six and eight hours, 29 percent stated that they were not authorized and permitted to take two rest breaks, 53 percent stated that they were authorized and permitted to do so, and 17 percent stated that they did not know or could not remember.

Relying on Ninth Circuit precedent, the district court explained that, as long as there is a proper foundation for the survey and it is conducted using accepted principles, questions about the expert's methodology and the survey design generally go to the weight of the survey, not its admissibility. The district court noted, however, that it could exclude a survey if there were substantial deficiencies in its design or execution.

The district court identified five crucial flaws that undermined the admissibility of the plaintiffs' survey. First, the "survey had a woefully low response rate." The survey used a random sample of 10,000 individuals in the class, but only 343 usable responses were obtained—a 3.43 percent response rate. And even after excluding the 4,320 individuals whom the expert categorized as "nonreachable," the response rate was still only 6 percent. The district court noted that the Reference Manual on Scientific Evidence states that surveys with response rates below 75 percent should receive "greater scrutiny," those with response rates lower than 50 percent should be regarded with "significant caution," and those with response rates between 5 percent and 20 percent are "very unlikely" to "provide any credible statistics of the population as a whole." The district court concluded that the response rate for the plaintiffs' survey was too low, especially in light of case law holding that response rates of 5 percent and 8 percent were inadequate.

Second, and relatedly, the district court reasoned that the low response rate suggested a form of "nonresponse bias." In other words, the individuals who responded to the survey were materially different from the general population, and therefore the survey was an unreliable tool for developing information about the class as a whole. The district court found it particularly troubling that 572 individuals refused to participate in the survey, meaning that "refusals outnumbered surveys responded to by almost two-thirds." The district court further found that the expert failed to adequately explain or correct for these drastically different response rates.

Third, the district court concluded that the survey was plagued by the problem of self-interest bias: The survey's prompt informed individuals that the survey was being performed as part of a class action lawsuit. The court noted that this renders the survey inherently suspect because it leads to at least two biasing phenomena. On the one hand, those recipients who do not believe that they have an interest in the outcome of the class action because they were afforded their rest breaks would be less likely to respond, leading to a response sample that is biased in favor of those individuals who experienced a violation. On the other hand, those individuals who do respond may, consciously or unconsciously, skew their answers to advance their self-interest as potential beneficiaries of the class action.

Fourth, the district court faulted the survey for asking individuals to recall specific events that occurred between three-and-a-half and eleven years prior to the survey. As evidence that this type of recall-driven survey leads to unreliable results, the district court noted that some individuals' responses as to the number and types of shifts that they worked made no sense in light of the limited evidence in the record. For example, a number of respondents said that they were given their rest breaks a specific percentage of the time when they worked a particular type of shift when it turned out that they had worked that type of shift only once.

Fifth and finally, the district court determined that the survey was imprecise as to both its questions and its sample. The survey, for example, failed to exclude the possibility that individuals voluntarily chose not to take rest breaks in certain contexts even though the break would have been allowed. And the respondents to the survey included at least one managerial employee who should not have been part of the survey because the survey failed to adequately inform participants that the questions applied only to breaks taken while working as an hourly employee.

Given all of these flaws, the district court held that the "problems with the survey are fundamental and demonstrate that it is an unreliable means of measuring AutoZone's potential liability to individual employees" and "therefore exclude[d] it under Rule 702 and Daubert."

The district court's decision is a straightforward application of Daubert to exclude a seriously flawed study in its entirety. The district court properly focused on the survey's low response rate and the concomitant problems of bias that arise in such circumstances. The district court's reasoning is especially helpful as applied to studies commissioned solely for the purposes of litigation: The fact that survey respondents were told that the survey was being conducted for a class action lawsuit gave respondents a biased incentive to participate in the survey (or not) and answer questions in a way that might benefit them monetarily. Given the uncertain impact that Tyson Foods may have on Dukes with respect to the permissible uses of such surveys in class-certification proceedings under Rule 23, the district court's decision provides an important evidentiary means of excluding plaintiffs' surveys and thus defeating class certification, avoiding liability, and diminishing damages calculations.

Originally published on October 3, 2016

Learn more about our Consumer Litigation & Class Actions, Employment Litigation & Counseling and Supreme Court & Appellate practices.

Visit us at mayerbrown.com

Mayer Brown is a global legal services provider comprising legal practices that are separate entities (the "Mayer Brown Practices"). The Mayer Brown Practices are: Mayer Brown LLP and Mayer Brown Europe – Brussels LLP, both limited liability partnerships established in Illinois USA; Mayer Brown International LLP, a limited liability partnership incorporated in England and Wales (authorized and regulated by the Solicitors Regulation Authority and registered in England and Wales number OC 303359); Mayer Brown, a SELAS established in France; Mayer Brown JSM, a Hong Kong partnership and its associated entities in Asia; and Tauil & Chequer Advogados, a Brazilian law partnership with which Mayer Brown is associated. "Mayer Brown" and the Mayer Brown logo are the trademarks of the Mayer Brown Practices in their respective jurisdictions.

© Copyright 2016. The Mayer Brown Practices. All rights reserved.

This Mayer Brown article provides information and comments on legal issues and developments of interest. The foregoing is not a comprehensive treatment of the subject matter covered and is not intended to provide legal advice. Readers should seek specific legal advice before taking any action with respect to the matters discussed herein.

To print this article, all you need is to be registered on Mondaq.com.

Click to Login as an existing user or Register so you can print this article.

In association with
Related Topics
Related Articles
Related Video
Up-coming Events Search
Font Size:
Mondaq on Twitter
Mondaq Free Registration
Gain access to Mondaq global archive of over 375,000 articles covering 200 countries with a personalised News Alert and automatic login on this device.
Mondaq News Alert (some suggested topics and region)
Select Topics
Registration (please scroll down to set your data preferences)

Mondaq Ltd requires you to register and provide information that personally identifies you, including your content preferences, for three primary purposes (full details of Mondaq’s use of your personal data can be found in our Privacy and Cookies Notice):

  • To allow you to personalize the Mondaq websites you are visiting to show content ("Content") relevant to your interests.
  • To enable features such as password reminder, news alerts, email a colleague, and linking from Mondaq (and its affiliate sites) to your website.
  • To produce demographic feedback for our content providers ("Contributors") who contribute Content for free for your use.

Mondaq hopes that our registered users will support us in maintaining our free to view business model by consenting to our use of your personal data as described below.

Mondaq has a "free to view" business model. Our services are paid for by Contributors in exchange for Mondaq providing them with access to information about who accesses their content. Once personal data is transferred to our Contributors they become a data controller of this personal data. They use it to measure the response that their articles are receiving, as a form of market research. They may also use it to provide Mondaq users with information about their products and services.

Details of each Contributor to which your personal data will be transferred is clearly stated within the Content that you access. For full details of how this Contributor will use your personal data, you should review the Contributor’s own Privacy Notice.

Please indicate your preference below:

Yes, I am happy to support Mondaq in maintaining its free to view business model by agreeing to allow Mondaq to share my personal data with Contributors whose Content I access
No, I do not want Mondaq to share my personal data with Contributors

Also please let us know whether you are happy to receive communications promoting products and services offered by Mondaq:

Yes, I am happy to received promotional communications from Mondaq
No, please do not send me promotional communications from Mondaq
Terms & Conditions

Mondaq.com (the Website) is owned and managed by Mondaq Ltd (Mondaq). Mondaq grants you a non-exclusive, revocable licence to access the Website and associated services, such as the Mondaq News Alerts (Services), subject to and in consideration of your compliance with the following terms and conditions of use (Terms). Your use of the Website and/or Services constitutes your agreement to the Terms. Mondaq may terminate your use of the Website and Services if you are in breach of these Terms or if Mondaq decides to terminate the licence granted hereunder for any reason whatsoever.

Use of www.mondaq.com

To Use Mondaq.com you must be: eighteen (18) years old or over; legally capable of entering into binding contracts; and not in any way prohibited by the applicable law to enter into these Terms in the jurisdiction which you are currently located.

You may use the Website as an unregistered user, however, you are required to register as a user if you wish to read the full text of the Content or to receive the Services.

You may not modify, publish, transmit, transfer or sell, reproduce, create derivative works from, distribute, perform, link, display, or in any way exploit any of the Content, in whole or in part, except as expressly permitted in these Terms or with the prior written consent of Mondaq. You may not use electronic or other means to extract details or information from the Content. Nor shall you extract information about users or Contributors in order to offer them any services or products.

In your use of the Website and/or Services you shall: comply with all applicable laws, regulations, directives and legislations which apply to your Use of the Website and/or Services in whatever country you are physically located including without limitation any and all consumer law, export control laws and regulations; provide to us true, correct and accurate information and promptly inform us in the event that any information that you have provided to us changes or becomes inaccurate; notify Mondaq immediately of any circumstances where you have reason to believe that any Intellectual Property Rights or any other rights of any third party may have been infringed; co-operate with reasonable security or other checks or requests for information made by Mondaq from time to time; and at all times be fully liable for the breach of any of these Terms by a third party using your login details to access the Website and/or Services

however, you shall not: do anything likely to impair, interfere with or damage or cause harm or distress to any persons, or the network; do anything that will infringe any Intellectual Property Rights or other rights of Mondaq or any third party; or use the Website, Services and/or Content otherwise than in accordance with these Terms; use any trade marks or service marks of Mondaq or the Contributors, or do anything which may be seen to take unfair advantage of the reputation and goodwill of Mondaq or the Contributors, or the Website, Services and/or Content.

Mondaq reserves the right, in its sole discretion, to take any action that it deems necessary and appropriate in the event it considers that there is a breach or threatened breach of the Terms.

Mondaq’s Rights and Obligations

Unless otherwise expressly set out to the contrary, nothing in these Terms shall serve to transfer from Mondaq to you, any Intellectual Property Rights owned by and/or licensed to Mondaq and all rights, title and interest in and to such Intellectual Property Rights will remain exclusively with Mondaq and/or its licensors.

Mondaq shall use its reasonable endeavours to make the Website and Services available to you at all times, but we cannot guarantee an uninterrupted and fault free service.

Mondaq reserves the right to make changes to the services and/or the Website or part thereof, from time to time, and we may add, remove, modify and/or vary any elements of features and functionalities of the Website or the services.

Mondaq also reserves the right from time to time to monitor your Use of the Website and/or services.


The Content is general information only. It is not intended to constitute legal advice or seek to be the complete and comprehensive statement of the law, nor is it intended to address your specific requirements or provide advice on which reliance should be placed. Mondaq and/or its Contributors and other suppliers make no representations about the suitability of the information contained in the Content for any purpose. All Content provided "as is" without warranty of any kind. Mondaq and/or its Contributors and other suppliers hereby exclude and disclaim all representations, warranties or guarantees with regard to the Content, including all implied warranties and conditions of merchantability, fitness for a particular purpose, title and non-infringement. To the maximum extent permitted by law, Mondaq expressly excludes all representations, warranties, obligations, and liabilities arising out of or in connection with all Content. In no event shall Mondaq and/or its respective suppliers be liable for any special, indirect or consequential damages or any damages whatsoever resulting from loss of use, data or profits, whether in an action of contract, negligence or other tortious action, arising out of or in connection with the use of the Content or performance of Mondaq’s Services.


Mondaq may alter or amend these Terms by amending them on the Website. By continuing to Use the Services and/or the Website after such amendment, you will be deemed to have accepted any amendment to these Terms.

These Terms shall be governed by and construed in accordance with the laws of England and Wales and you irrevocably submit to the exclusive jurisdiction of the courts of England and Wales to settle any dispute which may arise out of or in connection with these Terms. If you live outside the United Kingdom, English law shall apply only to the extent that English law shall not deprive you of any legal protection accorded in accordance with the law of the place where you are habitually resident ("Local Law"). In the event English law deprives you of any legal protection which is accorded to you under Local Law, then these terms shall be governed by Local Law and any dispute or claim arising out of or in connection with these Terms shall be subject to the non-exclusive jurisdiction of the courts where you are habitually resident.

You may print and keep a copy of these Terms, which form the entire agreement between you and Mondaq and supersede any other communications or advertising in respect of the Service and/or the Website.

No delay in exercising or non-exercise by you and/or Mondaq of any of its rights under or in connection with these Terms shall operate as a waiver or release of each of your or Mondaq’s right. Rather, any such waiver or release must be specifically granted in writing signed by the party granting it.

If any part of these Terms is held unenforceable, that part shall be enforced to the maximum extent permissible so as to give effect to the intent of the parties, and the Terms shall continue in full force and effect.

Mondaq shall not incur any liability to you on account of any loss or damage resulting from any delay or failure to perform all or any part of these Terms if such delay or failure is caused, in whole or in part, by events, occurrences, or causes beyond the control of Mondaq. Such events, occurrences or causes will include, without limitation, acts of God, strikes, lockouts, server and network failure, riots, acts of war, earthquakes, fire and explosions.

By clicking Register you state you have read and agree to our Terms and Conditions