To realise value from AI it needs to be integrated in some way into a customer experience and or associated business process. Achieving this in an organisation which isn’t experienced with using AI can be a significant operational challenge needing a range of new skills, technology changes and cultural acceptance

A critical first step in considering the implications is to understand what aspects of a task AI will do. Once trained, most machine learning algorithms use input data as a trigger to return a prediction. How we use this prediction to drive an action is an important consideration and key to embedding in a business process to realise value

Data - Prediction - Decision - Action - Outcome
Breakdown of a task

Taking a model from the Prediction Machines Book by Ajay Agrawal, we can look at any task as a series of discrete steps (I’ve removed the training and feedback processes from this example):

  1. Data is input into the model, triggering a Prediction
  2. The model comes up with a statistical prediction (or predictions) with a confidence for each
  3. A decision needs to be made, based on the prediction as to which action to take
  4. The action is undertaken
  5. An outcome results

When a human does this, in most cases the prediction and decision steps happen all at the same time, without any detailed consideration of the probability of the action being correct. Common sense and general intelligence allow us to quickly make decisions.

With an AI based system we need to codify the decision step to turn the prediction into an action. Doing this can be very challenging for an organisation new to AI. What level of confidence do you need to have? 80%, 90%, 99.9%. What if it goes wrong? What is the impact?

To aid this decision making process and to support a business led critical analysis it is helpful to use the confusion matrix, a tool used by data scientists to measure model performance:

The confusion matrix

Let’s use an example to illustrate this. Say for example we want to use a model to identify someone and authentic them to access their phone – like Face ID.

We would expect in most situations where it’s the true user we get a true prediction and when it’s a false user we get a false prediction. From an experience and security view we can then decide how well we want the model to perform for it to be useable in real life.

We also need to consider the other two situations, where we have a true user, but the model predicts false: This is a bad outcome as it means the user can’t access their phone, we want this to occur with a low frequency. We also need to consider the user journey in this situation and perhaps provide a failure path – in most cases they can try again or fall back to a pin. Not an optimal experience but recoverable by the user.

The final situation, a false user that is predicted true: This would allow access to the phone to someone other than the right person so is a bad outcome and we would want this to be a very rare situation.

We can also ask the data scientists to tune their model to support this set of outcomes and can quantify the probability and cost/impact of each outcome.