How do we measure AI adoption? High level metrics

At Mitra Robot, we have developed three key tests to measure user adoption in AI, aiming to determine if the solutions are as impactful for end users as they are innovative for creators like us.

How frequently is a product used in AI scenarios? Moving beyond the traditional "toothbrush test"—which checks if a product is used every day—we propose the "snack test." This evaluates whether the product is used compulsively within the last hour. For AI to be truly successful, it's crucial that end users develop a form of addiction, akin to those seen with the web, mobile, and social media platforms. Currently, it seems that the addiction lies more with AI creators like ourselves rather than the users themselves.

Are users still excited about the product 100 days after their first encounter? Consider your favorite social network or smartphone; these typically pass this test with flying colors. But what about AI tools? Personally, while I use ChatGPT extensively, I’ve observed that the general public's interest wanes over time. Even simpler tools like coding assistants show a decline in usage after the initial few months. For example, after a year of using it, I canceled my Github Copilot subscription. I switched to Cody, but soon found I barely used it for production code beyond the first few days.

Is the purchase driven by vanity or sanity? A simple method to evaluate this is through the presence of measurable Key Performance Indicators (KPIs). If a KPI exists, there are clear expectations set for the output. Otherwise, it might just be a "nice-to-have." What KPIs are currently used with Large Language Model (LLM) applications? In 99% of the cases I researched, the discussion centers around toy benchmarks—such as passing college-level biology tests—rather than significant improvements to business processes.

As we continue to build the next generation of AI, it is crucial to ensure that end users can meaningfully engage with the products. Let me know if you agree or disagree with these assessments.

In 99% of the cases I researched, the discussion centers around toy benchmarks—such as passing college-level biology tests—rather than significant improvements to business processes.