Advertisement

Computer Assisted Review Reference Model (CARRM)

EDRM’s Computer Assisted Review Reference Model | Additional Resources

EDRM’s Computer Assisted Review Reference Model

Computer Assisted Review (CAR) is a process of having computer software electronically classify documents based on input from expert reviewers, in an effort to expedite the organization and prioritization of the document collection. The computer classification may include broad topics pertaining to discovery responsiveness, privilege, and other designated issues. CAR may dramatically reduce the time and cost of reviewing , by reducing the amount of human review needed on documents classified as potentially non-material.

The EDRM Search team has prepared a Computer Assisted Review Reference Model (CARRM) to document the steps of the process.  This model represents joint efforts of some of the best known providers in Computer Assisted Review – Automony, an HP Company; Daegis; Exterro; Falcon Discovery; FTI Consulting; kCura; KPMG LLP; Kroll Ontrack; NightOwl Discovery; Recommind; and UBIC – as well as leaders from Bowman & Brooke LLP; DLA Piper LLP (US); Littler Mendelson, PC; and Quarles & Brady LLP. Click here for a complete list of EDRM Search members.

Like the EDRM framework, the CARRM framework should be a useful reference for e-discovery practitioners at corporations, law firms and elsewhere; e-discovery services and software providers; and organizations evaluating e-discovery tools.

We are publishing this draft model for review and feedback.  Please send comments to mail@edrm.net or post them on the EDRM site.



The Major Steps in the CARRM Process

Set Goals

The process of deciding the outcome of the Computer Assisted Review process for a specific case. Some of the outcomes may be:

  • reduction and culling of not-relevant documents;
  • prioritization of the most substantive documents; and
  • quality control of the human reviewers.

Set Protocol

The process of building the human coding rules that take into account the use of CAR technology. CAR technology must be taught about the document collection by having the human reviewers submit documents to be used as examples of a particular category, e.g. Relevant documents. Creating a coding protocol that can properly incorporate the fact pattern of the case and the training requirements of the CAR system takes place at this stage. An example of a protocol determination is to decide how to treat the coding of family documents during the CAR training process.

Educate Reviewer

The process of transferring the review protocol information to the human reviewers prior to the start of the CAR Review.

Code Documents

The process of human reviewers applying subjective coding decisions to documents in an effort to adequately train the CAR system to “understand” the boundaries of a category, e.g. Relevancy.

Predict Results

The process of the CAR system applying the information “learned” from the human reviewers and classifying a selected document corpus with pre-determined labels.

Test Results

The process of human reviewers using a validation process, typically statistical sampling, in an effort to create a meaningful metric of CAR performance. The metrics can take many forms, they may include estimates in defect counts in the classified population, or use information retrieval metrics like Precision, Recall and F1.

Evaluate Results

The process of the review team deciding if the CAR system has achieved the goals of anticipated by the review team.

Achieve Goals

The process of ending the CAR workflow and moving to the next phase in the review lifecycle, e.g. Privilege Review.


Additional Resources

Resources from Academia

Resource
Author
Date
The Grossman-Cormack Glossary of Technology-Assisted ReviewMaura R. Grossman, Wachtell, Lipton, Rosen & Katz, and Gordon V. Cormack, University of Waterloo.2012/12

Resources from EDRM Members

Resource
Author
Date
Measuring and Validating the Effectiveness of Relativity Assisted ReviewDr. David Grossman, Ph.D., prepared for Relativity2013/02
Workflow for Computer-Assisted Review in RelativitykCura Corporation2012/07

Resources from Others

Resource
Author
Date
Predictive Ranking: Technology Assisted Review Designed for the Real WorldJeremy Pickens, Senior Applied Research Scientist, Catalyst Repository Systems2013/02
  • Electronically Stored Information or ESI is information that is stored electronically on enumerable types of media regardless the original format in which it was created.
  • Electronically Stored Information: this is an all inclusive term referring to conventional electronic documents (e.g. spreadsheets and word processing documents) and in addition the contents of databases, mobile phone messages, digital recordings (e.g. of voicemail) and transcripts of instant messages. All of this material needs to be considered for disclosure.
Please complete the following to download file
Your Name (required)

Your Email (required)

Subscribe me to mailing list (required):
 Yes No

I would like to join EDRM (required):
 Yes - As an Organization Yes - As an Individual Yes - Unsure Organization or Individual No I already am a member

After you select "Submit", a link to the file will be displayed below and a copy of the file will be sent to the email address you entered. If you do not receive the email message, please check your spam folder.

Your Organization (required)

Organization Type (required)
 Association Corporation Government Law Firm Provider None

Your Telephone Number

Your Position

Your Website (http://...)

Your Address

Add Comment Register



Leave a Reply