EDRM is a cooperative industry organization created to address the need for standards and guidelines in the e-discovery market.
The mission of the EDRM Metrics Project is to provide an effective means of measuring the time, money and volumes associated with e-discovery activities. In addition, the Metrics model group is responsible for creating tracking, reporting and measurement protocols to assess e-discovery activities, and, to the extent metrics data becomes available, begin analysis of that data.
Having a defined approach and understanding the numbers early in the e-discovery lifecycle provides four key advantages:
This study, developed by the Metrics working group, highlights appropriate metrics and best practices that can be used in three key phases of the Electronic Discovery Reference Model (EDRM); Collections, Processing and Review. The requirements are based on a hypothetical employment-related lawsuit developed by the EDRM Search working group to identify best search strategy practices. In this scenario, a Silicon Valley based company, Dynatronics, Inc., is sued by a former director, Patricia Perez, for gender discrimination and wrongful termination. Dynatronics retains Eagan & Escher as its counsel and outsources production and review to LitSpecialists, an e-discovery firm specializing in this area.
For each stage, Collections, Processing and Review, this study explains:
This study also provides examples of typical reports that would be used by the Dynatronics legal team.
Dynatronics counsel has approved collection of the first wave of potentially responsive ESI. They have selected a number of primary custodians and shared sources found during the Identification interviews and activities. Counsel needs ongoing and summary reports on the actual volume identified and in order to create rough cost calculations to support ongoing filings and negotiations on the scope. Tracking metrics can aid in the filings and scope defining process to:
The Dynatronics case focuses on employees and communication file types. It is decided to track the volumes by custodian by data type, except for any corporate-wide sources without custodial level retrieval systems. Useful metrics to track include elements such as:
An Excel spreadsheet based on the EDRM metrics codes is used.
The approach should follow a well defined methodology:
This will strengthen the defensibility of your collections if they ever come under scrutiny.
Dynatronics has collected ESI from all of the custodians and corporate shared sources. As these collections are processed, some items cannot be processed or converted to load in the review system for a number of reasons. Counsel has requested a running report on exception items and items that could not be reconciled. Processing and review (which may run concurrently) are the two longest and most expensive phases. Using metrics that describe the size and attributes of the collected dataset, coupled with the projected and actual processing rate (including the historic and actual exception rate), allows Dynatronics’ management to plan the remaining downstream processes, to minimize cost and legal risk.
Input metrics include the total size (in bytes) of the collected dataset (by custodian) and its breakdown by file type, generated by script during the load process into the staging area.
Once processing starts, the key in-process metrics required will be the time taken and processing rate for each batch of files, broken down by custodian and file type. The exception rate (the proportion of files which fail to be processed correctly) will also be required by type, file type and custodian. Exceptions, which will increase Dynatronics’ costs and delay the start of review, fall into two categories: non-fatal exceptions which may require additional, costly reprocessing, and fatal exceptions which represent files that cannot be processed and that will have to be accounted for to opposing counsel.
Output metrics include the numbers of files processed, the duplicate/near-duplicate count, the processing rate, the fatal and non-fatal exception rates, the number of pages in each file and the number of images produced (depending on the processing capability, the review method and the capability of the review platform). The output metrics should also include the responsiveness rate, i.e., the number of files identified (by custodian) as responsive in the searches of the processed data and which were flagged for subsequent review.
Processing metrics are generated by a number of different groups. Careful attention was paid by Dynatronics, LitSpecialists, Inc. and Eagan & Escher at the start of the project to the source and format of the metrics data and how reports from each different stage (Collections, Processing, Review and Production) would be coordinated and reconciled.
This report allows Dynatronics to assess the volume and types of exceptions in its collection, separating them into actionable vs. non-actionable (unresolvable) categories.
The report also allows Dynatronics to estimate the additional costs (say $30 per exception) and additional time needed to re-process the exceptions. Using historical rates allowed Dynatronics to make better cost estimates during the collection phase, providing them extra leverage at the “Meet and Confer” stage, and to limit costly reprocessing before it occurred.
Document review, generally acknowledged as the costliest component of e-discovery also involves the greatest coordination among a number of participants (in-house counsel, the outside law firm, the review platform vendor, and the staffing vendor). As the collected ESI has been processed and uploaded into the review system, LitSpecialists will start the review. Documents will be reviewed for their relevancy and coded as to responsiveness or reasons for being withheld entirely or in part from production. The client and Eagan & Escher are eager to assess the number and content of documents to be produced. The review is also under a complex schedule and a closely-watched budget. LitSpecialists will provide progress reports monitoring review rates, to manage expectations and to keep the review team on target, and document statistics, to project the time and cost of production.
The use of metrics in the Review Stage ensures deadlines are met, tracks the cost of the review and helps prepare for production. But beyond those short-term goals, consistent capture and use of review metrics can establish baselines for projecting timeframes and budgets, as well as preferred review platforms and review team composition.
The metrics for Review are derived from both the review platform and the reviewers.
There are three sources of data for review metrics: the review software; the reviewers themselves; and/or the review manager(s). The review software will embody the universe of documents – size of the files to be reviewed and attributes of those files. It can also show the totals for hours logged on, and number of files reviewed and coded. Beyond these fundamental metrics, the availability of certain data is dependent on the type of reports generated from software used; or calculating and tracking the numbers manually. Metrics for hours worked/billed and system downtime, would be acquired from the reviewers.
Review metrics are generated by a number of different groups. Here, all metrics are being recorded in Excel spreadsheets so that the phases can be combined to create a summary of the winnowing of document volume to relevance.
The Document Review Status Report below shows in some detail the volumes of documents that have and have not been reviewed, and how they have been coded.
As an ongoing report, LitSpecialists will use it to monitor the rate at which the review has been advancing and to report their progress to the client and counsel.
For Dynatronics and Eagan & Escher, it also shows the proportions of documents from the collection set as they were coded and that may be included in the production.
The surprise was the high level of foreign language documents reported here. The team immediately revised their review staff to include the appropriate foreign language speakers as well as updated their budgets to account for the additional costs.