Below Expectations
AI hype continues to dominate the technology press and there is little to slow down this speeding AI locomotive as one AI company after another quotes massive gains in the number of active users they garner each month. MAUs, or monthly active users, is a KPI (Key Performance Indicator) that is tossed around like it has monetary value, as if a MAU were a paying subscriber, but they are not. What a MAU actually is as follows:
MAU
When it really gets down to it, given the valuations that the VC market has placed on AI companies, paying customers and how sticky they are is the real metric and that data is much harder to come by, particularly as individual consumers are able to pay monthly and cancel at any time. Corporations are a different animal as they tend to look at AI from an organizational perspective. Large companies look at AI as a tool, hoping to see a ROI across a large number of company AI users, so they represent a more stable income source and a stickier one, as long as they see ROI. Corporations also pay for at least some AI infrastructure, particularly applications, while individuals tend to use AI on-line or through their phones, with little income generation outside of a monthly fee, if any.
So what really matters is the breakdown of income, which we believe makes MAU less relevant, but in the world we live in today, the average CEO has the ‘got to have it’ complex that says if you don’t have AI you have fallen behind your competition. It is almost irrelevant whether you are gaining anything by having it, but if a competitor says they are making great strides because of AI, your board members will be asking why you have been so slow to join the real AI world.
What makes this even more difficult is the subjective nature of AI evaluations. There are measurement tools that calculate cycle time reductions for various worker functions, Average Customer Handle Time tools, Accuracy Analyzers and even Content Creation Volume measurement tools, all trying to pinpoint gains generated by AI. We certainly consider these tools valid as they have (hopefully) no bias toward particular results, but it is very hard to measure the overall effect AI has on productivity and even harder to measure how those changes affect ROI. That is why we often look to surveys whenever possible for a more personalized look at AI ROI.
There are hundreds of surveys, white papers, and expert reports on the benefits of AI, and a never-ending stream of technical journals that point to how AI is fundamentally changing us as human beings, but like product reviews on Amazon (AMZN), we gravitate toward the ‘one-star’ reviews to see what it is that might upset the apple cart in this brave new Ai world we live in. We are not doing it to be glass half empty people, but to get a bit of a more realistic view of AI from those who either use it or pay for it.
A recent survey from Atlassian, an Ai company dedicated to removing corporate knowledge silos by creating AI-based enterprise search systems, revealed some interesting facts as to how AI is used at the enterprise level and its effectiveness and ability to generate a return on its investment.
The bottom line is that AI does improve productivity, but by focusing on personal productivity over innovation the survey indicates that the Fortune 500 could lose $98 billion in lost returns on their AI investments.
Here's the data:
The survey classifies respondents into 5 categories:
MAU
- A single person
- Counted once, no matter how many times they use the AI each month
- If they log-in or just visit the AI they can be counted, although this is a very weak metric
- If they send a prompt or query they are also counted, a much stronger indication of usage, as would be the case when using specialized Ai tools like image creators.
When it really gets down to it, given the valuations that the VC market has placed on AI companies, paying customers and how sticky they are is the real metric and that data is much harder to come by, particularly as individual consumers are able to pay monthly and cancel at any time. Corporations are a different animal as they tend to look at AI from an organizational perspective. Large companies look at AI as a tool, hoping to see a ROI across a large number of company AI users, so they represent a more stable income source and a stickier one, as long as they see ROI. Corporations also pay for at least some AI infrastructure, particularly applications, while individuals tend to use AI on-line or through their phones, with little income generation outside of a monthly fee, if any.
So what really matters is the breakdown of income, which we believe makes MAU less relevant, but in the world we live in today, the average CEO has the ‘got to have it’ complex that says if you don’t have AI you have fallen behind your competition. It is almost irrelevant whether you are gaining anything by having it, but if a competitor says they are making great strides because of AI, your board members will be asking why you have been so slow to join the real AI world.
What makes this even more difficult is the subjective nature of AI evaluations. There are measurement tools that calculate cycle time reductions for various worker functions, Average Customer Handle Time tools, Accuracy Analyzers and even Content Creation Volume measurement tools, all trying to pinpoint gains generated by AI. We certainly consider these tools valid as they have (hopefully) no bias toward particular results, but it is very hard to measure the overall effect AI has on productivity and even harder to measure how those changes affect ROI. That is why we often look to surveys whenever possible for a more personalized look at AI ROI.
There are hundreds of surveys, white papers, and expert reports on the benefits of AI, and a never-ending stream of technical journals that point to how AI is fundamentally changing us as human beings, but like product reviews on Amazon (AMZN), we gravitate toward the ‘one-star’ reviews to see what it is that might upset the apple cart in this brave new Ai world we live in. We are not doing it to be glass half empty people, but to get a bit of a more realistic view of AI from those who either use it or pay for it.
A recent survey from Atlassian, an Ai company dedicated to removing corporate knowledge silos by creating AI-based enterprise search systems, revealed some interesting facts as to how AI is used at the enterprise level and its effectiveness and ability to generate a return on its investment.
The bottom line is that AI does improve productivity, but by focusing on personal productivity over innovation the survey indicates that the Fortune 500 could lose $98 billion in lost returns on their AI investments.
Here's the data:
The survey classifies respondents into 5 categories:
Figure 1 - Which Best Describes How You Feel About AI? - Source: SCMR LLC, Atlassian
Figure 1 shows that those with a positive view of AI’s capabilities saw their attitude improve this year (Type 2 – Type 4), while those with a more negative view (Type 0 – Type 1) saw that negative view decrease y/y, for an all-around improvement in AI perception. However, while there was improvement y/y, what employees did with that extra time is telling, as shown in Figure 2 and even more so for executives in Figure 3. Efficiency is certainly improved but executives see little other improvement for themselves and similarly for the organization. In fact the detailed data shows that few executives see any transformational improvements due to AI.
- Only 3% of executives see an improvement in operational efficiency
- Only 2% of executives see an improvement in work quality
- Only 4% of executives see an improvement in innovation
- Only 4% of executives see an improvement in the ability to solve complex problems
Figure 2 - How Are You Reinvesting Your Time? - Source: SCMR LLC, Atlassian
Figure 3 - Executives - "AI Makes You More..." - Source: SCMR LLC, Atlassian
We were a bit surprised at the survey data’s negative take on AI across the organization, but we also understand that the objective of the study was to show how a singular focus on productivity improvement has limited results and bypasses how AI can help an organization become more innovative, something the company specializes in, so there is some inherent bias in the data. However we were very surprised at how little executive thought of the employee productivity gains, and that they did little to improve corporate level innovation and ROI.
All in, the pressure managements feel to use AI seems to lead to metrics that support AI as a way to improve employee efficiency but does little toward the creative side of the business, which is essential in such competitive markets. While the efficiency improvement is commendable, the lack of improvement at the corporate level changes the perceived ROI of large AI investments, at least currently. Perhaps AI is missing the tools needed to afford the spectacular improvement across the corporation that model vendors are promising, but as yet improvements have been a little less than expected.
All in, the pressure managements feel to use AI seems to lead to metrics that support AI as a way to improve employee efficiency but does little toward the creative side of the business, which is essential in such competitive markets. While the efficiency improvement is commendable, the lack of improvement at the corporate level changes the perceived ROI of large AI investments, at least currently. Perhaps AI is missing the tools needed to afford the spectacular improvement across the corporation that model vendors are promising, but as yet improvements have been a little less than expected.
RSS Feed