May 5th, 2020

Are All Your AI and ML Models Wrong?

RSS icon RSS Category: Machine Learning, Makers

We are living in unprecedented times. Our society and economy are experiencing shocks beyond anything we have seen in living history. Beyond the human cost, there is a data science and machine learning elephant in the room (hopefully 2 meters away): Are your predictive models still doing the job you expect them to do?

The challenge here is greatly complicated by:

  • Huge government spending already committed, with likely further fiscal stimulus in the future instantly altering market dynamics.
  • Short and long term shocks to the economy from social distancing measures. Will consumer behavior be altered permanently in some scenarios?
  • An uneven impact, from the types of businesses struggling to how people and their families are affected, varying by each state’s individual regulations. Us Victorians are starting to feel hard done by as other states start to relax their lockdowns.
  • The feedback delay. In this fast-moving environment, we won’t know immediately what has changed, which will make both accurate modeling and monitoring model performance challenging.
  • New data might look very different from past data as people’s behavior changes. Perhaps people looking for financial assistance or credit who never have in the past.
  • Australia’s last recession was 28 years ago. This lack of relevant data could create a small data problem and further difficulty building stable models.

What do you do? One option is do nothing, accept higher error rates in your predictive modeling and AI Systems. Curl up in a dark room and hope things return to “normal”. But this is a very risky approach. Just take a look at the media, and many are predicting a ‘new normal’.

It might be much better to be proactive and work out what we can do and what we can’t. Sure the devastating impact of COVID-19 is unprecedented, but we see similar changes on a smaller scale all the time. Like when Virgin Australia entered (rather than exited) the domestic flying market or more recently with the advent of the neo-banks and fintechs. What can we learn from these and other shocks to the system?

I think we need to look beyond performance on the test set, search ways to rapidly develop AI models, and apply model monitoring techniques at scale. You can check some of H2O Driverless AI’s AutoML capabilities here.

I am interested in hearing others’ thoughts. Are you rebuilding?

About the Author

James Orton

James is a Data Scientist based in Australia. Outside of all things AI, James likes to ride bicycles, sometimes with both his kids

Leave a Reply

Using AI to unearth the unconscious bias in job descriptions

“Diversity is the collective strength of any successful organization Unconscious Bias in Job Descriptions Unconscious bias affects

January 19, 2021 - by Parul Pandey and Shivam Bansal
H2O Driverless AI 1.9.1: Continuing to Push the Boundaries for Responsible AI

At H2O.ai, we have been busy. Not only do we have our most significant new

January 18, 2021 - by Benjamin Cox
Meet the Data Scientist who just cannot stop winning on Kaggle.

In conversation with Philipp Singer: A Data Scientist, Kaggle Double Grandmaster, and a Ph.D. in

January 15, 2021 - by Parul Pandey
Liqui.do Speeds Credit Scoring for Fair Lending with H2O.ai

Liqui.do is a technological and innovative company developing a platform for leasing equipment for small

January 12, 2021 - by Eve-Anne Tréhin
New Improvements in H2O 3.32.0.2

There is a new minor release of H2O that introduces two useful improvements to our

December 17, 2020 - by Veronika Maurerova
Introducing H2O Wave

For almost a decade, H2O.ai has worked to build open source and commercial products that

December 15, 2020 - by Jo-Fai Chow and Benjamin Cox

Join the AI Revolution

Subscribe, read the documentation, download or contact us.

Subscribe to the Newsletter

Start Your 21-Day Free Trial Today

Get It Now
Desktop img