Never ‘fix and forget’, but ‘learn and evolve’

If someone were to come up with an idea for an investment, they would try to test it with historical data or simulation. If this verification shows a promising return, they would implement the idea, likely as an automatic or a semi-automatic trading system, and then put real money into it. But the story does not end here. Because it is built on assumptions and heuristics, everyone knows any investment strategy will have an expiry date; we just never know it beforehand. Thus, the user of a trading algorithm will have to continuously monitor its performance. When it no longer meets the expectations, it will need some tune-ups, or may even face retirement.

While explainable AI is a research topic, machine learning or AI software components are black box technologies with decision rationales being opaque to humans.

The same should happen to machine learning components in a larger system.

While explainable AI is a research topic, quite sure the machine learning or AI software components in use today are ‘black box’ technologies with decision rationales being opaque to humans. We know it works because we tested it (like we tested the investment strategy with historical data). However, we do not know why it works. One day, if some subtle thing changes and the machine fails to recognise it, the technology will stop being intelligent. The problem is that we do not know what this inalterable thing is because the machine never told us.

Therefore, while AI and machine learnings are niches and cool building blocks for a software system, we should pay close attention and recognise that they are not the same as the other components. The formula for calculating compound interest will not change in the next thousand years, and therefore we can rely on it, as long as we are consistent in whether the interest rate is expressed as percentage points or fractions. A system to understand a press release or extract information from a company’s annual report, however, may cease to work if the language evolves or some new legislation causes changes in parts of these documents. Just like how an investment strategy operates, we need to continuously monitor the performance of these black box technologies and tune them up occasionally.

Building a system is no longer once and for all. We need to keep in touch with those who understand the system to fine-tune and re-train the ML components.

This has changed the way we need to manage a software system in practice. Besides monitoring for cybersecurity and performing backup regularly, we should check whether the input and output match and make sense from time to time. Building a system is no longer once and for all. We need to keep in touch with those who understand how the system is built so that they can help fine-tune and re-train the machine learning components when needed. Moreover, whether there is a privacy concern in sampling the record of the input and output in order to determine if the algorithm is still relevant should be agreed upon by the engineers and stakeholders.

We should, thus, not be surprised if we need regular maintenance for AI systems, as this is necessary, and should plan for it accordingly when adopting such a solution.

By Adrian SW Tam, Director – Data Science, Synechron