Providing estimates of matter cost and duration is an essential part of the legal project manager role.

Whenever you are asked to come up with estimates, the question you should ask yourself is:

What method will give you, your colleagues and clients, most confidence in your estimates?

Reference Class Forecasting (RCF) should be part of your estimating arsenal.

Sometimes, RCF alone may be all you need.

Although combining RCF with other estimating methods is likely to provide you with most confidence in your estimations.

What is Reference Class Forecasting?

At its simplest, RCF is where

1. You collect data about similar projects to the one you are estimating, and
2. Then base your estimates on what you have learned about the similar projects.

Similar projects are the reference class.

I advise people on my training courses that, when looking to provide early estimates for legal matters, begin by using RCF if possible.

When you have your reference class (i.e., list of similar matters) look at the mean (i.e, the average matter) and then base your estimates regards costs and duration on that mean.

The concept of RCF is easy to understand and straightforward to apply.

RCF can also be quite accurate, providing you have defined the reference class properly and you have good data to work with.

Development of Reference Class Forecasting

The theoretical grounding of reference class forecasting was developed by Nobel Prize winners Daniel Kahneman and Amos Tversky.  Its practical application has been developed by Bent Flyvberg, originally at the request of the U.K Government.

It has since been adopted and applied around the world, especially where estimates are required for large and complex projects.  A quick Google search on Reference Class Forecasting will lead you to quite a few papers and studies about RCF.

But you don’t even need to do that.  I’d recommend instead reading the book ‘How big things get done: the surprising factors behind every successful project’ by Bent Flyberg and Dan Gardner.

The book is an entertaining read (yes, honestly) and although it draws mostly from very large construction projects, there are a lot of lessons that legal project managers, and others, can learn from it.  The explanation about RCF is one such lesson, and not the only one.

Overcoming Optimism Bias

The insight that we are over optimistic about our ability to get things done was one of the key drivers behind the development of RCF.

Whether in our private or professional lives we are usually over optimistic, including our ability to provide correct estimates and our ability to get things done within cost and on time.

Hence an antidote to this is to use data from earlier projects.

RCF is a way of estimating from ‘outside-in’.  You are not estimating by looking from within the project and considering project risks and break work down etc.

You are looking at your project from the outside and using other similar projects on which to base your estimates.

‘It’s one of those’

To do RCF properly you must avoid the temptation of thinking that your project is ‘special’ and, because of this specialness, what has happened before on similar projects is not applicable.

Yes, in terms of detail, the matter you are estimating will have unique attributes.  However similar matters will have been planned and delivered before.  While each will be unique in some way, they have enough in common to be considered similar.

As Bent Flyberg says in his book, you must accept that your project ‘is one of those’.  So why not look for more ‘of those’ to help provide baseline estimates?

You should also resist trying to adjust the estimates generated from the mean of the reference class.  Adjusting too much may mean you are still in the ‘specialness’ mindset.  Flyberg’s recommendation is to adjust only when there are compelling reasons to do so, and when data exists to support your proposed change.

Framing the Reference Class

For RCF to work well, you will need to define the reference class appropriately.  Classes which are too wide or too small are not helpful.

You will need to think about the right criteria to use when selecting your reference class of similar projects.

The criteria will vary from practice area to practice area.  The best way to come up with suitable criteria is to talk this through with the subject matter experts, i.e. the lawyers in your firm.

Interestingly, research shows that for the most accurate results, the projects in the reference class should be projects which your organisation has worked on previously.  Not other organisation’s – yours.

I think this finding is especially relevant to law firms.  There is, I suggest, a direct correlation between the effectiveness of legal matter management and the quality of the legal teams working on them.  Law is (still) a people business.  So how has a core legal team within a firm managed similar matters in the past?

Goodbye to ‘inside-out’ estimations?

Project teams must give considered thought to their estimates, and they should follow an estimating methodology.

Teams following a predictive approach to project management are likely to create bottom-up estimates in detail using a Work Breakdown Structure (WBS).  Teams following a more adaptive (or agile) project management approach are likely to apply techniques requiring less bottom-up detail.

But if RCF is so useful, does this mean that ‘inside-out’ approaches are no longer needed?

Not all.  Arguably combining a RCF with an ‘inside-out’ estimate is the best way of all to approach estimation.

How does this combined approach work?

Lets return to the RCF for a moment.  Rather than looking at actual cost and duration of earlier projects within the reference class, look at the cost and duration overrun of each project and express those as percentages.  Let’s now assume the mean of the reference class shows a cost overrun of 40%.

Now go back to your ‘inside-out’ estimate, however it was derived.  Following the RCF data, you would be well advised to increase your cost estimate by 40%.  This is because you now know that, on average, cost overruns on these kinds of projects amount to 40% compared to the original ‘inside-out’ estimate.

The importance of data governance

Law firms have a lot of data.  Think of all those time records alone, with every fee earner in the firm recording their work in 6-minute units.

Hence using a RCF approach should be attractive to legal project managers.

But this assumes the data is readily available, consistent, and meaningful.  Anyone who has worked with data in law firms will know that, generally, these assumptions are unsafe.

For these assumptions to become reliable, law firms must practice good data governance and data management.

During one of my alumni meet-ups recently (every month I invite all my training course alumni to an informal meet-up) we discussed data governance and data management with CJ Anderson of the Iron Carrot.

It became apparent that while law firms have become more aware of the need for good data governance over the last few years, there is still much work to be done in this area.

Everyone has a role to play in generating better quality data in law firms.

Legal project managers can play their part by, for example, recording all their matter estimates, comparing those estimates to matter outcomes and building their own RCF data for different matter types.

Ideally this should be done as part of law firm standard data management practice, resulting in data which can be trusted and accessed easily by those who need it.