Beyond Statistics: Reading Between the Lines of Document Review Progress Reports

Overview

By Joe Pirrotta

Whenever I evaluate new discovery software, my focus is often on the tool’s reporting function. While this may seem innocuous to some, for the managed document review team at Integreon, reporting is an absolutely essential element of every project we do. Our daily reports typically include many basic statistics such as total number of documents reviewed, how many are responsive or relevant, what percentage are privileged and the number of documents identified for each issue code. We also work with our clients to create custom reports as each project requires; such as custodian-specific workflow data. But what some clients don’t always realize upon first glance, however, is just how much more a progress report says between the lines. The following are some examples of the types of insights that a progress report can provide.

For the Project Manager (PM):

When training new team leads or project managers, one of the tasks we give them is to shadow an experienced senior team member and to help them with the reporting. From this exercise a trainee can gain an intuitive understanding of the unspoken language of the review. The process of creating a report often requires a PM to mindfully consider the numbers being reported and what they really mean. These numbers can lead to a series of iterative questions that can help in spotting project issues that may reside just beneath the surface of these statistics, as well as lead the inquisitive project manager to an overall more accurate sense of where things are at.

For example, did your productivity fall off a cliff today? If so, why was that? Was it due to an issue of absences, or has the team been working through a particularly complicated set of data, perhaps from a specific custodian? Did responsiveness see a significant uptick? How about privilege? Could your searches be on the wrong track and thus yielding inaccurate reporting data?

Any anomaly that the PM can uncover from within a progress report will become an opportunity to address that issue and be better prepared to talk about it when presenting the results to the client. Reports ensure that issues are caught early on in the process in order to avoid a potentially costly re-review. They can also help to ensure that a team is properly sized to meet a deadline. Reports present a great opportunity to ask these important questions. In addition they help us increase our accountability to become better problem solvers, by thinking on our feet and anticipating issues before they arise. Remember, the report is not just a bunch of graphs and percentages. It’s a daily chance to visualize the development and direction of a project.

For the Internal Stakeholder:

When I am managing my projects, I approach reporting as a sort of punctuation mark at the end of each day. It’s the last thing I do before I leave, and it enables me to reflect on the results from each set of metrics. As a director of the PM team, I also use reporting as a means to make sure that the PMs on my team each have a reasonable workload. If a progress report hits my inbox outside of normal business hours or if the client has to ask for an update, that is often a signal that a PM is over-utilized. When this happens, I make sure to check in with the project manager the very next day to check that they have all the resources they need in order to hit their deadlines and deliver quality work. I also look at the insights provided by the PM in the body of their email updates that accompany each progress report. This is a very good way for me to judge how close a PM is to their data and their overall engagement with the project.

For the Client:

At Integreon we like to suggest to our clients that our daily report should be a jumping-off point for additional questions they may have along the way, not only in relation to the managed review team, but for the entire team assigned to their discovery project. While the basic stats offer general insights into the progress of the review, these same stats can also help the attorney to assess if adequate resources are allocated across the entire project. This is important such as when a law firm is performing the second level review. The report can help counsel ensure that they focus on any particularly significant issues and to know that they have enough staff to handle the data coming to them. If a sudden increase in responsiveness occurs, counsel can use the report to give the technical team enough time to prepare for a larger production. If there’s a sudden increase in the occurrence of foreign languages, counsel can use reports to work with Integreon or in-house resources to ensure that those foreign language documents are triaged and reviewed by the appropriate team members.

Making the most of reporting is a crucial step in the overall success of a document review project. When utilized correctly, a report can serve as so much more than just being some PDF attachment showing what percentage of documents are marked with an issue code. Regardless of what role you play in a project, it’s important to use reports that will answer not just the big picture questions, but also those questions that are hiding just under the surface.