Innovation Shorts: Acquiring Intelligence: What We Have Learned About AI

More weightlifting than chess

We are talking in the contracts space. Obviously, there is a certain well-known AI program that does play chess rather well.

Right now, where AI contracting software really excels is in scanning vast quantities of transactional information at high speed, then extracting and summarising selected areas from that information. This is more about brute processing force than the application of meaningful intelligence. Coupled with analytic software, the tools can generate penetrating reports on the selected topics to be reviewed. Let’s take some examples.

Your company has a global template for confidentiality / non-disclosure agreements (NDAs), with an approved standard duration over which the obligations apply. The general counsel has been asked to confirm how often that duration varies across the NDAs the company actually signs up to. He could ask Derek, a freshly onboarded paralegal, to shelve his plans for the year and get to work. Or he could bring in an AI tool, supported by a deployment team, to carry out the review and report the statistics. The AI tool will be capable of scanning hundreds of NDAs in minutes, not hours, finding the term / duration provisions, extracting the salient information and summarising the variance statistics with user-friendly charts. The technology will need a little help – we’ll come back to this point soon – but the speed and ultimate accuracy rates will be good enough to ensure that Derek still has time for his social media addiction.

Nobody wants to wade manually through a mountain of NDAs looking for the exceptions to a standard clause. But many commercial lawyers consider contract drafting to be their core competency. “Authoring”, as it is called in the software world, is another capability that can be augmented by artificial intelligence. Machine learning algorithms are being used in a variety of contract lifecycle management (CLM) solutions to accumulate drafting patterns from a data bank of existing contracts, with the ability to prompt for approved contract clauses and pre-defined key terms.

A critical feature of an AI-boosted CLM is its contextual analysis. For example, the tool could suggest the inclusion of a currency fluctuation provision for a sales contract that is being negotiated with a cross-border customer.

There is a but, of course. You knew it. Derek knew it. AI tools only have the ability to identify and put together contract clauses based on linguistic drafting parameters: they cannot, as yet, identify whether a clause is substantively different – in legal terms – from another clause on the same topic that is drafted differently. That effort still requires human judgment.

Still, the data weightlifting muscle-power can be a game changer. Take a legacy contract migration project – shifting more than 10,000 contracts from various outmoded repositories into a single CLM. As part of the exercise, the general counsel wants to capture up to 10 fairly standard data points from each of those contracts so that the legal department starts to build an understanding of its contracts landscape going forwards. Depending on the data points in question, the tool used and the resources deployed, such an exercise could take as little as two weeks. By contrast, a similar project reviewing only 5,000 contracts and using three human reviewers could take as much as six months.

Educate your children

Like a superhero or traffic cop, AI has great power but needs to learn how to use it. A number of vendors have contracting AI tools that offer out-of-the-box capabilities, minimizing the need for further “tutoring”. But in our experience, most clients have requirements that extend beyond the pre-programmed functionality, which means the tool will need to learn and have its errors corrected on an iterative basis.

In this context, volume is a key factor. The AI will need heavy datasets – i.e., preferably many hundreds or thousands of agreements – to “learn” from during this tutoring stage. So as a general rule, the lower the number of contracts in scope for AI analysis, the weaker the business case for deploying the tool.

And even where overall contract volumes are high, best practice suggests that an iterative approach will be required – starting with the out-of-the-box capabilities and then tutoring the tool for deeper and more sophisticated data trawls phase by phase. One reason for that best practice is the simple, ugly fact that in most companies contracting processes are, to quote former boyband crooner Paul McCartney³, “all over the place”. Contracts are held in different repositories and sometimes owned by various functions, from Legal to Procurement to Sales Ops.

An iterative approach to AI deployment is necessary for both logistical reasons and because of the need for quality control. Earlier on we mentioned accuracy rates. It is probably time to address that particular elephant⁴.

AI tools for reviewing contracts are not currently 100% accurate, even for the out-of-the-box functionality. This is why a deployment team is always required for tutoring / error-correction purposes. The fact that the tools suffer from error rates is disturbing for legal departments hoping to automate and accelerate contract support work using the technology. But it is worth remembering that, prior to implementing an AI system, most of those departments have zero statistics on error rates for the human-only review processes that preceded the system. Ask the Chief Legal Officer what the team’s historic error rates are and see the look of terror slowly take hold.

That is not to say that the old school approach has no quality control system in place. Of course, it does – the classic senior review protocol. This is how the CLO gets to sleep at night, apart from using Temazepam. Junior team members are coached by seniors before being released to handle contract reviews on their own. For higher risk / value transactions, the two-tier quality assurance process often stays in place regardless of the experience of the primary reviewer.

Viewed in that context, the need for ongoing tutoring and error-correction work in the case of an AI tool should not be considered a big compromise on the advantages of using the technology. But it does mean that support resourcing should be factored in.

It takes a village

Which neatly brings us to our third topic. Maybe it’s an exaggeration to say that it takes a whole village to raise an AI contracts tool. But certainly, a small team, and preferably with at least two characteristics: sufficient bandwidth, and relevant experience.

Let’s get specific. Say you need to review 5,000 documents and identify up to 25 data points. With an AI tool that can handle the task, you will likely require 2 members of staff (or contractors) working full time on the project for around 2 weeks – and that’s just to handle the tool training phase. You would require a minimum contract sample size of 100 to 200 agreements. Then assume you have a 30-day deadline in which to complete the review. Budget 3 to 4 resources, working pretty much full time, to handle the quality control work on the abstracted data. And those resources can be more or less efficient, depending on their background and training in such review work.

And this is the challenge for legal departments interested in deploying contracts support AI at its current stage of development. How do they free up their own staff to support such effort, when the whole theory is that the AI tool is going to alleviate the human workload?

Two Games in Town

As this currently stand, there are two contracting AI games in town. The short game is the one where all the action is:

• A compelling event with a looming deadline;
• Massive volumes of contracts to review; and
• A set of fixed and fairly common data points that need to be extracted from the contracts.

Here we are talking about events like a regulatory change – IBOR / LIBOR, say, or the UK’s famous experiment in reducing red tape by increasing it, known as Brexit. Also, corporate changes – restructuring, acquisitions, divestitures etc. M&A due diligence similarly springs to mind.

If the short game is about big, urgent projects, the long game is altogether more visionary. This is about weaving AI functionality into the fabric of the legal support function, particularly focused on the contracting lifecycle. For example, using AI to accelerate the process of reviewing contracts – both mark-ups of client paper received from counterparties, and first-pass reviews of counterparty paper. But also, at the back end of the lifecycle – using AI to extract contract obligations so that the department can track and manage SLA and other performance metrics.

The long game is visionary because the business case for deploying AI in this context is made over years, not this coming fiscal year. Accuracy rates and the need to build or contract-in deployment teams may reduce the immediate benefits of investing in an AI tool, but over time the product’s quality, range of functionalities and the breadth of use cases will justify the investment.

In this world of annual budget pressures and quarterly performance targets, patience and vision are hard qualities to maintain for legal leaders. But what we do know about AI is that it grows up fast. In January 1988, the best chess player in the world predicted that no computer program would be able to defeat a grandmaster at chess until at least the year 2000. His name was Gary Kasparov, and less than ten years later IBM’s Deep Blue supercomputer beat him.

More Innovation Shorts to follow. Next time – the Labours of Legal Ops

¹Look it up on YouTube, youngsters.
²Not really, we made this up.
³From “I Lie Around”, during Sir Paul’s Wings phase.
⁴Elephant in the room, for God’s sake.