Trust is the most important objective in AI products

Fellow product and design leaders have been led by various metrics such as retention and churn over the last two decades of software. These commonplace metrics are the traditional compass of software product success. Leaders of AI products now need to think differently about metrics and objectives due to the vastly different nature of deterministic (non AI) vs probabilistic (AI) software. Trust is now the objective that will move the needle. Whether your users trust your product’s outputs, decisions and actions is the difference between winning and losing users. Trust is the moat.

Now, there are some important foundational questions to answer starting with: what does trust in an AI product actually mean? Well, a lot of AI trust work seems to treat it as a quantity to increase, which is incorrect in most cases and actually ends up creating misuse (I’ll explain this later on). Trust for AI is best defined as the relationship of what the user believes the system can do versus what the system can do. Most AI failures come from the user trusting in an expectation and the product not delivering, thus abusing the trust. Trust is a relationship with your users.

What you want to aim for is calibrated trust, which is the sweet spot of the user not overtrusting or undertrusting but just in the middle. “But surely users overtrusting us is great?” having users who overtrust your AI product is a ticking time bomb and will eventually cause a churn problem. When a user overtrusts a product, it means their expectations don’t align with the capabilities of the product so they are highly likely to experience an AI failure of some kind and then, boom, the trust is shot to pieces. This is called misuse.

Using our prior definition of trust, we can start to think about how to create trustworthy AI products which comes down to two things: 1) Improving the abilities of your AI product so it is more capable, safe and reliable 2) Being transparent with your users about your AI product’s capabilities without cognitively overloading them or preventing them from completing their tasks. This is where Human-centred AI design is invaluable in developing solutions that are safe, reliable and trustworthy by employing AI design patterns that cover several fundamental areas like autonomy and control, enrolment and mental-models and safety and error-prevention.

Your job as a product or design leader in an AI product org should be to aim for calibrated trust. To do this, you need to speak to users regularly (with structured interviews), gather usage data, have strong logging and AI testing infrastructure. These things can start to help you understand where your system is failing and what could be causing users to jump ship. Secondly, you need to properly implement Human-centred AI design: how to communicate AI states, how to fight against hallucinations with design, how to make outputs transparent, how to let users override the AI effectively, etc. These are the difference makers between AI products that win and lose in the long term. Trust and design are the moat. Learn them and use them.