Review - Planning the Review
From EDRM
Contents
|
Meet and Confer
Early in the litigation the case team must make a determination about the scope of the review: what is to be reviewed, how it is to be reviewed, and what is the intended outcome. To a great extent, the discovery order will direct what data is to be collected and reviewed. A meet-and-confer conference between the parties should be conducted early on in the planning stages to make these decisions. The proposed change to the Federal Rules of Civil Procedure 26(f) states that these initial meeting are to include discussions and decisions on matters pertaining to electronically stored information (ESI). Topics to be determined include preservation of ESI, the form of production for the ESI, and approaches for asserting privilege and work-product protection claims. Each of these decisions impact the planning of the document review.
Scope & Objectives of the Review
Historically, lawyers were obligated to review every document presented to them by the client or opposing counsel in the litigation for relevance, responsiveness to discovery requests, and privilege. With the proliferation of email and other electronic documents, reviewing each and every document is often unrealistic. While the scope of the review is usually dictated by the discovery request, a key strategy is to control the scope of the review through the use of technology or other aids. These techniques should be discussed and agreed upon during the initial planning meetings.
Techniques that can be useful to limit the scope include:
- Strategic collection of data – it may be possible to do a more selective collection of key custodian data rather than a broad collection. There may be agreed upon file types or directory locations.
- Strategic filtering of collected data – this may take the form of applying date limitations or keyword filters against the data that has been collected.
The decision and selection of a knowledgeable review manager to manage the review process is an essential first step in the planning phase. The review manager must assure that those selected to conduct the review have a clear understanding of the case objectives.
There should be clear documentation of the key issues and a distinction between issues of fact and issues of law. Providing examples of the kind of evidence the litigation team will need to effectively prosecute or defend the matter will make the review team aware of what to keep their eyes open for. An understanding of who the key players are and what roles each plays in the case will enhance the review teams' ability to hone in on the critical pieces of evidence to the litigation.
The review objectives may evolve as a case progresses through the different phases of litigation. The following represents a few types of review that may be undertaken:
- Determine the relevancy of the information or documents collected or produced
- Bibliographically code database fields to facilitate improved search and retrieval of the documents collected or produced
- Determine whether or not any privilege applies to the documents subject to produciton
- Determine to which request(s) for production the documents are responsive
- Identify documents that should be marked as “confidential” or have portions redacted
- Relate key documents to alleged facts or legal issues previously outlined in the case
- Relate key documents to key players who may testify about the documents
- Identify other subjective information
In a few circumstances, a single review may encompass all these objectives. In reality, they are generally done in phases and by a variety of different reviewers, each with a clearly identified role in the review.
(back to top)
Defining Review Protocols
A coding manual is typically prepared to define not only the objectives of the review but the rules of engagement for the review team. Historically, a coding manual defined the bibliographic fields that were to be captured by the reviewers into the review database. Today, with many reviews being done in digitized databases that already capture much of this information, the manual may take different forms depending on the type and format of the review.
Examples of information that may be captured in a coding manual include:
- Rules for adding additional bibliographic information to the database such as: type of document (memorandum, letter, email), entities named in the document, marginalia found on paper docs that have been scanned, etc.
- Publication of look-up table to aid in identification of key players and known entities
- Rules for identification and subjective coding/tagging of the responsive and producible documents with an outline of the issues to be identified and possibly sample documents that are representative of the issues at hand
- Rules for the handling of documents requiring redaction
- Rules for identifying and coding/tagging privilege documents
- Rules for identifying and handling confidentiality issues
- Rules for adding annotations to documents in the form of attorney notes and workproduct
- Rules for coding/tagging emails and their attachments and email threads
- Rules for the identification and tagging of non-responsive documents, spam and junk email.
- Rules for handling of unreadable, password protected or other faulty documents
(back to top)
Determining Review Methodologies
There are many different methodologies used for a document review. Sound choices usually are impacted by multiple factors:
- Agreements made by counsel during the meet and confer phase or as ordered by the court
- Time and cost constraints for the project
- Collection and restoration methods
- Functionality available within the review platform
- Available resources to support different formats
Physical Format
Determining the physical format for the review should be established during the initial planning stages. A review may take many forms and utilize several different formats:
- Manual paper review
- Internal review using an in-house review system
- Internal review using a hosted online system
- External review using temp attorneys set-up in a war room with the necessary review tools.
Review Assignments
In a typical scenario, the documents to be reviewed are divided up such that each reviewer is reviewing a specific numbered range of records that do not overlap with any other reviewers.
In more complex scenarios, the review sets may be divided based on the source of the records or a subset of records that meet search specifications for a particular key player, date range or case issue. The review assignments are then based on the importance of the source or key player, for example, with higher level attorneys reviewing the most critical key players’ records, and more junior level or paralegals reviewing less important ones.
To aid in an expedited privilege review, searches may be conducted for names of attorneys or law firms that would indicate a high probability of the result set containing attorney/client privileged communications. Documents meeting the privileged search criteria could be assigned to an attorney whose task is to review for privilege and mark the documents as such.
(back to top)
Establishing Project Timetable & Workflow
The design of a project timetable or workflow is critical to the review process and should be established at the outset of the project. The workflow should be designed by the review manager with the assistance of any vendor/consultant retained and the firm’s internal litigation support personnel. Often, those most familiar with the technology will be able to provide invaluable guidance on how to leverage that technology to maximum advantage.
Agreements made during the meet-and-confer process will influence the review objectives, techniques used, platform chosen and the overall project workflow and milestones. Discovery deadlines will also influence the review strategy. For instance, a DOJ second request with a 15-day turnaround might be handled differently than a standard litigation matter with a more long-term discovery schedule. Accordingly, there are different strategies which are determined by the nature of the request and the resources of your organization.
The review team should work with the electronic discovery vendor/consultant to develop the most efficient workflow for the project. Often, the vendor/consultant will be able to counsel the review team on the most efficient approach based on the strengths or limitations of the review tool. Do not assume that the workflow that worked in your last case using a different tool will work the same way on the next case and a different review platform. Work with the vendor/consultant to develop the methodology that makes the most sense given the needs of each specific case.
Other factors to consider regarding the creation of a project timetable include:
- Overall volume of data to be reviewed and complexity of the issues being reviewed
- Internal resources devoted to the review project
- Volume of redactions and the workflow provided to perform them
- Resource availability of the vendor of choice
- Ability of vendor to produce to requirements; a sample run should be performed well in advance of any deadlines
The timetable should include:
- Deliverable deadlines for the vendor/consultant for each phase of the project:
- Collection & restoration of the data
- Processing, searching, and hosting the data
- Production of responsive documents (as well as privileged documents if there is to be separate production)
- Daily, weekly, monthly document totals per reviewer
- Other metrics to be used to measure the progress of the review and their milestones
- Production due dates set forth by the document request or the court
Milestones may be tied to specific custodians who are important to the matter, or critical time periods, issues or concepts. Using the power of the review tool to manage the workflow by segregating documents along these lines can greatly increase the speed and efficiency with which the project is completed.
Another key to ensuring that the workflow proceeds smoothly is to determine who will create the work assignments and how and when the assignments will be disseminated to the review team members. Depending on how a project is staffed, work assignments may be created by the vendor/consultant, the review manager, a paralegal or someone in the firm’s litigation support department.
(back to top)
Review Objectives
A key step in planning for the electronic review is to make sure that those selected to manage and conduct the review have a clear understanding of what the case is about. All those involved in selecting the review platform and spearheading the review process should review copies or summaries of the petition/complaint and answer(s). There should be a clear delineation of the key issues that discern between issues of fact and issues of law. Providing examples of the kind of evidence the litigation team will need to effectively prosecute or defend the matter will make the review team aware of what to keep their eyes open for. An understanding of who the key players are and what roles each plays in the case will enhance the review teams ability to hone in on the critical pieces of evidence to the litigation.
All members of the review team should be provided copies or summaries of any discovery requests and answers thereto. Any special discovery orders or stipulations should also be summarized and available to the review team to make sure that the review is conducted in compliance with those orders or agreements.
The review objectives may change as a case progresses through the different phases of litigation. The following represent a few types of review that may be conducted throughout the litigation:
- Determine the relevancy of the information or documents collected or produced.
- Bibliographically code database fields to facilitate improved search and retrieval of the documents collected or produced.
- Determine whether or not any privilege applies to the documents subject to be produced.
- Determine which request(s) for production the documents are responsive to.
- Identify documents that should be marked as “confidential” or have portions redacted.
- Relate key documents to alleged facts or legal issues previously outlined in the case.
- Relate key documents to key players who may testify about the documents.
- Identify other subjective information.
In a few circumstances, a single review may encompass all these objectives. In reality, they are generally done in phases and by a variety of different reviewers, each with a clearly identified role in the review.
(back to top)
Review Techniques
Determining the format for the review should also be established during the initial planning stages. A review may take many forms and utilize several different formats: manual paper review; internal review using an in-house review system; internal review using a hosted online system; external review using temp attorneys set-up in a war room with the necessary review tools. The nature and needs of the matter as well as the following factors will help determine the format of the review:
- Agreements made by counsel during the meet and confer phase, or ordered by the court
- Time and cost constraints for the project at hand
- How the documents were collected
- Training required by each member of the review team
- Available resources to support different formats
The focus of our discussion is on electronic discovery, so the format decisions will typically be in-house vs. online and native vs. TIFF/PDF. See Review - Selecting a Vendor for a more in-depth analysis of the decision factors and pros/cons to these types of review platforms.
Additional techniques to be employed during the course of the review may include:
Culling irrelevant documents
A paralegal or junior level associate may go through the repository to identify records known not to be relevant to the matter at all, suppressing them from further review, or deleting them from the database altogether. They may engage the help of a litigation support specialist or electronic discovery vendor to automate the identification of known irrelevant documents so that the reviewers don’t have to look at every single document that have all the attributes as those previously identified as irrelevant. Many electronic discovery tools provide an index to all the documents in the collection, allowing for a review of file names, types, dates, etc. for the purpose of excluding information from being further processed. Bibliographic coding is the process of capturing objective data from documents, i.e. date, author, recipients, title etc., and entering it into database fields to facilitate the search and retrieval of documents. This level of review can be conducted by document processors, case clerks, paralegals, or outsourced to a coding shop. When paper documents are converted to images, this level of coding is often necessary to facilitate search and retrieval. There is some debate whether or not OCR technologies provide enough full-text search capability to eliminate this step. Most feel, however, that some objective coding, along with OCR text is the safest way to ensure effective searches.
When planning the review, it is important to address how much bibliographic coding is going to be required ahead of time. If the review team will be relying on a metadata and full-text search strategy, tests should be run on a selection of representative records to ensure its ability to meet the review objectives without further indexing. Certain documents lend themselves to an auto-coding process where the search engine has the ability to automatically recognize specific fields of information and will populate the appropriate field with very little, if any human intervention. How much coding is done, as well as the dependence on a degree of accuracy will depend again on volume, time constraints, and the case budget?
Identifying producible documents
After culling, suppressing irrelevant documents, and providing the necessary indexing to the document repository, the review team is ready to address the task of reviewing documents that need to be produced, (or that have been produced by opposing parties). In a large repository with several reviewers, this task is often divided in a logical way in order to meet case management deadlines.
The documents may be simply divided up by volume such that each reviewer is reviewing a specific numbered range of records that don’t overlap with any other reviewers. In more complex scenarios, the review sets may be divided based on the source of the records, or a subset of records that meet search specifications for a particular key player or issue. The review assignments are then based on the importance of the source or key player, for example, with higher level attorneys performing the most critical key players’ records, and more junior level or paralegals reviewing less important ones.
To aid in an expedited privilege review, searches could be conducted for names of attorneys or law firms that would indicate a high probability of the result set being attorney/client privileged communication. The documents meeting the privileged search criteria could be assigned to an attorney whose task it is to review for privilege and mark the documents as such. Documents that are responsive to a request for production can be marked for production in a specific field, or with a simple tagging feature. If possible, it is best to identify which request the document is responsive to, as it may be required to provide this information to opposing parties. Further, documents subject to production should be reviewed for confidentiality or privacy requirements. If there is specific information that needs to be redacted, such as social security numbers, names, addresses, and phone numbers, a team of trained redaction specialists can go through the database, reviewing for this kind of information, and apply the appropriate electronic redaction marks and confidentiality stamps. The team’s ability to perform electronic redactions is dependent on the platform and format of the documents being reviewed. The number of redactions and how they will be performed is an important piece of the planning process when selecting the review platform and format of the documents loaded onto it.
Subjective coding
If information on key issues and players has been developed, the review team can capture documents that relate to key players and known issues in the case. Look-up tables can be developed to aid the reviewer to quickly pick from a pre-populated list of options. Key players may also be identified by the bibliographic coding staff if they are asked to capture all names in a document. Technology can also be leveraged to provide an index of all the key words, revealing the names of key players that can quickly be drilled down on to review the documents containing the names of those key players. Often times, an index can reveal the name of a key player hidden in the metadata of an electronic document versus anywhere on the document itself. This is one argument for collecting and indexing as much information as you can in native format.
(back to top)
Use of Technology
There are many vendors and software applications which facilitate the management, review and production of electronic evidence. The use of these constantly evolving technologies and services should be considered for the following key phases of the review process:
De-duplication / Scope Reduction
Reducing the volume of data in a review collection translates directly into cost savings. One of the best methods of data reduction is a process known as De-duplication. For instance, an Email message sent to three people in an organization could potentially exist in more than 10 locations within the company’s systems (each user’s computer, the Email server, backup tapes of Email server, etc.). De-duplication technology allows duplicate Email messages and other files to be tracked and reduced to one item in the review set. Because the file only appears once, it is only reviewed once which dramatically reduces the billable hours spent on the review. De-duplication also facilitates a more consistent and accurate production as the item can only be marked one time as ‘privileged’ or ‘responsive’ without the risk of another reviewer marking a copy differently.
Data Conversion
As previously discussed, there are many ways to conduct a review of electronic data. Regardless of the format (native, HTML, paper or Tiff/PDF) the key is to get the review collection into a uniform state with the ability to move from item to item quickly and efficiently. Because the data gets processed, it can quickly be presented to the reviewer without the need to open the item in its native application. (See the EDRM Processing chapter). For instance, one could instantly go from an Email message (.MSG) to a Word document (.DOC) to an Excel spreadsheet (.XLS) with the click of a button. Converting, or ‘processing’ the data also makes it possible to ‘tag’ or annotate documents with the issues and subjective codes set forth in the review objectives. (See the Review Objectives section). Additionally, the use of data processing technologies facilitates redaction or confidential information and production / bates numbering of responsive documents. See the EDRM Production chapter.
Searching
Once processed, electronic data can be easily searched for names and terms. Search techniques can be applied to exclude, prioritize and organize data thus making the review more efficient. For instance, a search of attorney names would create a review set likely to be privileged. Rapidly evolving technologies such as concept searching, visualization and data grouping tools can help reviewers get through documents faster. The following examples illustrate the scope and benefits of these new tools:
- Concept/Context Searching - can bring issues to the surface and identify unknown items related to the case or bring similar documents together.
- Visualization Tools - show documents in a way to let the reviewer manage the information or groups of information in a more efficient way.
- Near Duplicates - indicates documents which are almost alike such as different versions of an excel worksheet.
- Email Thread Management Tools - lets the reviewer see all the threads of the same email chain and apply consistent coding across all iterations.
- Linguistic Experts - special companies who use advanced language analysis to find and group important documents.
New technologies continue to evolve and are becoming generally accepted throughout the industry. While they can potentially lead to cost savings, each situation needs to be evaluated to determine if it is the right solution. (See the Selecting a Vendor and Emerging Technologies sections in this chapter).
(back to top)
Selecting a Platform for Review
Choosing an appropriate platform for the review is an essential step in planning your review. There are many options available, in-house solutions, ASP or web-based solutions, native file reviews, TIFF/PDF file reviews, etc. The important thing is to understand your objectives and select a platform that will optimize your ability to reach those objectives. The following section, Selecting a Vendor provides an in-depth look at these options as well as questions you should pose to the vendors to properly evaluate their capabilities.
(back to top)
Understanding Production and Delivery Requirements
The proposed rule changes to Federal Rules of Civil Procedure 26(f) require that opposing parties meet and confer about the format of production early on in the planning stages. Agreements made during the meet and confer process will influence the review objectives, techniques used, platform chosen and the overall project workflow and milestones.
Both parties must have plans to manage (i.e. how to bates number/redact/confidential stamp native files) the agreed upon production format within the time constraints allowed by the court. See the EDRM Production chapter for more in-depth information around the production process.
Factors to consider regarding the production timetable during the planning stages:
- Overall volume of data to be reviewed and complexity of the issues being reviewed.
- Converting to tiff may require additional time once review is complete – are rolling productions acceptable?
- Volume of redactions and the workflow provided to perform them.
- Ability of vendor to produce to requirements. A sample run should be performed well in advance of any deadlines.
(back to top)
Time Considerations/Resource Availability
Discovery deadlines will greatly influence the review strategy. For instance, a DOJ second request with a 30 day turnaround might be handled differently that a standard run-of-the-mill litigation matter with a more casual discovery schedule. Accordingly, there are different strategies which are determined by the nature of the request and the resources of your organization.
Large law firms may have the resources to handle the demand for resources to setup a review and staff reviewers. These resources include networking staff, security, desk top support, training, staffing managers, litigation support staff, and human resource. Smaller firms might rely on third party hosting vendors and temporary staffing agencies to provide staff. (Internal capabilities may also drive the review platform and format. See Selecting a Platform for Review).
Third party hosting vendors also have resources issues. There are large “blue chip” EDD processing / hosting vendors and there are small ‘local’ vendors with varying degrees of competence. One might rely on a large vendor for a sizable project with a tight deadline yet expect to pay a premium price. Compare to a smaller vendor who might be more aggressive in pricing while struggling to find the resources to complete the project within the deadline. (See Selecting a Vendor.) One should also consider the production format as the time to print paper, or convert to TIFF need to be build into the review/production schedule.
(back to top)
Management Plan
Previously we discussed the need for staffing to do the review and if the process was going to be handled in house or contracted to a third party hosting company. (See Managing the Review Process.) Either way the areas the internal staff must consider and manage:
Review Process
- Technical staff for connectivity and performance. Does the firm have a strong IT / Litigation Support organization? Can a large team of reviewers be adequately supported during a round the clock / weekend review?
- Experts in the data being reviewed. Did the data collected get loaded correctly to the review platform? Who will answer questions about the data should there be a processing error or password protected file?
- Data management. Which custodian’s data should be prioritized? Who assigns review data to the reviewers? Who will decide search criteria for keyword culling or privilege grouping?
Review Progress
- Contract attorneys v. internal resources. Does firm have enough internal reviewers?
- Performance. Will the review be complete by the deadline? Are the reviewers going fast enough and meeting their quotas?
- Quality Control. Someone in the internal team needs to spot check the work being done by the reviewers. Are the privileged document tags consistent? Are non-responsive documents truly non-responsive?
(back to top)
Client Cost Considerations
The processing and review of electronic evidence can potentially amount to the highest percentage of litigation costs. Accordingly, it is critical to consider various review and processing options and the overall impact on the litigation budget.
In-House v. Outsourcing
Many firms now have the tools and expertise to handle moderate amounts of EDD processing (de-duplication and conversion to review platform). While internal processing may be a lower cost option, the requirements of the case must be considered. If the litigation support staff becomes overwhelmed, the entire review could get sidetracked. The other major cost consideration is staffing the review. Internal attorneys will cost more than outsourced temp attorneys. However, the quality of the review may not be the same. Finally, what level attorney or paralegal will conduct the review? Junior level associates may seem attractive based on lower billable rates, but the review might go faster, and have a better quality if mid to senior level attorneys perform the review.
Vendor Selection/Services
Should the project be outsourced to a vendor, there are many factors to consider which have a direct impact on the project costs. Primarily, how does the vendor charge? While the industry seems to be gravitating towards a ‘per gig’ pricing model, some small to mid-size vendors still charge per page. Be sure to understand the services included in the per gig/per page pricing. Is de-duplication included? Does the fee include hosting? Are there extra costs to produce numbered TIFF files after the review? Additionally, does the vendor have the internal expertise to consult with your team on ways to reduce costs?
Budgeting
Figuring out the processing costs based on per page/per gig pricing is relatively simple compared with estimating the billable time necessary for the review. As previously stated, this is the part of the litigation that will cost thousands, perhaps even millions of dollars. However, once the data has been processed and loaded to a review tool it becomes easier to calculate the costs, timeline and necessary resources based on simple math. For instance, if a reviewer can review 30 documents and hour and there are 50,000 documents, the review will take approximately 1670 hours. That would amount to 208 working days. If staffed with 10 reviewers, the review would take roughly 20 days and cost about $370,000 (1670 hrs X 220.00/hr).
Project Management
Based on the above example, it is critical to stay on track with the project budget and timeline. If the review progresses slower than projected, it is best to identify this early to additional review resources can be added, or review criteria can be modified. Additionally, the clients cost expectations may need to be managed along the way – “we estimated the review cost based on 30 documents per hour but due to the complexity of the material, we are really only able to do 20. Therefore, we need to add reviewer and the costs will increase…”
Emerging Technologies
New concept searching and grouping tools can be employed to ‘jump start’ a review by allowing a reviewer to see ‘like’ documents one after another. In fact, studies show that reviewing similar documents together can speed the review by 10x. Grouping the documents also allows the review to be prioritized. For instance, all documents likely to be privileged or everything likely to be non-responsive can potentially be assigned to individual reviewers based on their expertise. A brand new associate can be assigned ‘things likely to be garbage’ while a more senior reviewer might tackle potentially privileged docs. This grouping and resource allocation can have a dramatic impact on review costs.
Quality Control
Regardless of the staffing, vendor or technology decisions there is still a fundamental QC process that needs to occur. The goal of the review should be a consistent end result in which responsive, or privileged documents are correctly assigned. If the reviewers stray from their training and incorrectly ‘tag’ items, significant amounts of time are necessary to correct the erroneous work product. Constant sampling and QC review along the way facilitates a ‘touch once’ approach which translates directly into efficiencies and cost savings.
(back to top)
