Earlier this week we noted that while OpenAI’s (pvt) ChatGPT is a general-purpose Natural Language Model (NLM), adding front-end code that references specific datasets can create systems that can be tailored to more specialized applications, opening up NLMs to the real world of data analysis. These applications can be specific to an industry, a product type, or customer information, but they all share one common issue and that is they must be reliable.
ChatGPT and Google’s (GOOG) Bard are subject to inconsistencies in their responses ranging from incorrect information, incorrect conclusions, and referencing information that does not exist. Data scientists are not sure why large NLMs have this issue, but we have faced such situations even in our limited use of NLMs. ChatGPT, given that its training ended in late 2021 is careful to point out that its data is historically limited to the date on which its training ended and might not have more current information, despite its insistence that it has been updated a number of times since then. Google’s Bard however indicates that it is constantly being updated and states that ‘My training is never finished’, leading one to believe that its data is up to date, yet we find inconsistencies in Bard’s responses that indicate a lack of self-checking that is essential for use in most business circumstances.
Here was our simple test:
Q: How many VR headsets were sold in 2022?
A: According to the International Data Corporation (IDC), global shipments of AR/VR headsets declined 20.9% year over year to 8.8 million units in 2022.
Q: Can you break that down by quarter?
A: Sure. Here is a breakdown of global shipments of AR/VR headsets by quarter in 2022:
Quarter Shipments (in Millions)
1Q 1.6
2Q 2.3
3Q 2.7
4Q 2.2
Q: Can you break that down by brand?
A: Sure. Here is a breakdown of global AR/VR headsets by brand in 2022.
Brand Shipments (in Millions)
Meta 4.6
HTC (2498.TT( 2.1
Sony 1.1
Others 1.0
Q: Can you break down the full year by quarter for each brand?
A: Sure. Here is a breakdown of global shipments of AR/VR headsets by brand and quarter in 2022.
Brand 1Q 2Q 3Q 4Q
Meta 1.4 1.8 1.7 1.7
HTC 0.6 1.0 0.5 0.1
Sony 0.3 0.3 0.3 0.2
Others 0.3 0.2 0.2 0.1
If we add the quarters in last set, the full year comes to 10.7m units and each brand’s yearly total is different from the full-year totals given in the previous questions. We asked Bard the following question to clarify:
Q: Why do the quarterly VR brand numbers differ from the yearly brand numbers?
A: There are a few reasons why the quarterly VR brand numbers differ from the yearly brand numbers.
One reason is that the VR market is still in its early stages of development, and there is a lot of volatility in the market. This means that the quarterly numbers can be affected by a number of factors, such as the release of new headsets, changes in the market, and economic conditions.
Another reason is that the quarterly numbers are based on shipments, while the yearly numbers are based on sales. This means that the quarterly numbers can be affected by factors such as inventory levels and the timing of shipments.
Finally, the quarterly numbers are often based on estimates, while the yearly numbers are based on actual sales data. This means that the quarterly numbers can be more volatile than the yearly numbers.
Despite these factors, the quarterly numbers can still be a useful tool for tracking the VR market. By tracking the quarterly numbers, you can get a better understanding of the overall trends in the market and the performance of individual brands.
Here are some additional factors that can affect the quarterly VR brand numbers:
If we had not asked the clarification question or fact checked the answers, we could have based a decision on information that was not correct. The system’s initial reasoning for the differences in its stated results (market volatility) was suspect as it had no bearing on why simple math produced different results from facts that the system presented originally. The 2nd explanation (some were shipments, some were sales) was inconsistent in that it was not specified in the initial answer, an absolute necessity, in both the actual questions and the system’s specific answers, which were always said to be shipments by the system for VR Only when questioned did the system indicate that some of the numbers were referencing sales rather than shipments.
Additionally the system seems to use an ‘answering’ technique often employed by politicians when asked questions they do not want to answer or cannot answer, that of adding a hefty supply of information that does not directly relate to the answer, hoping that the additional verbiage will mask the fact that the answer was not given The scary part of that theory is that such a technique is a very human trait that had to be programmed into the system or ‘learned’ by the system after seeing the practice used in its training data. Maybe we are anthropomorphizing Bard a bit too much, but picking up bad human characteristics is not something we hope for or expect from AI.
Regardless, no matter what the circumstance, a NLM is only as good as its training and inference algorithms and its training data, which leads us back to the point that no matter how large the data sets or how specific they are to the required answers, the accuracy, precision, and recall parameters (also known as the ‘F1’) is the key to understanding the reliability of the system. When humans make decisions based on information presented, they also have an F1, with successful humans having a high F1, while those with low F1s are less successful when it comes to decision making. As each human programmer has an F1, the accumulated F1’s of the programmed algorithms would be an integral part of the system’s F1, so we can’t blame the system for coming up with incorrect answers. Perhaps a year backpacking across Europe might help Bard to gain some needed perspective and get down to finding its own personality and consistency.
ChatGPT and Google’s (GOOG) Bard are subject to inconsistencies in their responses ranging from incorrect information, incorrect conclusions, and referencing information that does not exist. Data scientists are not sure why large NLMs have this issue, but we have faced such situations even in our limited use of NLMs. ChatGPT, given that its training ended in late 2021 is careful to point out that its data is historically limited to the date on which its training ended and might not have more current information, despite its insistence that it has been updated a number of times since then. Google’s Bard however indicates that it is constantly being updated and states that ‘My training is never finished’, leading one to believe that its data is up to date, yet we find inconsistencies in Bard’s responses that indicate a lack of self-checking that is essential for use in most business circumstances.
Here was our simple test:
Q: How many VR headsets were sold in 2022?
A: According to the International Data Corporation (IDC), global shipments of AR/VR headsets declined 20.9% year over year to 8.8 million units in 2022.
Q: Can you break that down by quarter?
A: Sure. Here is a breakdown of global shipments of AR/VR headsets by quarter in 2022:
Quarter Shipments (in Millions)
1Q 1.6
2Q 2.3
3Q 2.7
4Q 2.2
Q: Can you break that down by brand?
A: Sure. Here is a breakdown of global AR/VR headsets by brand in 2022.
Brand Shipments (in Millions)
Meta 4.6
HTC (2498.TT( 2.1
Sony 1.1
Others 1.0
Q: Can you break down the full year by quarter for each brand?
A: Sure. Here is a breakdown of global shipments of AR/VR headsets by brand and quarter in 2022.
Brand 1Q 2Q 3Q 4Q
Meta 1.4 1.8 1.7 1.7
HTC 0.6 1.0 0.5 0.1
Sony 0.3 0.3 0.3 0.2
Others 0.3 0.2 0.2 0.1
If we add the quarters in last set, the full year comes to 10.7m units and each brand’s yearly total is different from the full-year totals given in the previous questions. We asked Bard the following question to clarify:
Q: Why do the quarterly VR brand numbers differ from the yearly brand numbers?
A: There are a few reasons why the quarterly VR brand numbers differ from the yearly brand numbers.
One reason is that the VR market is still in its early stages of development, and there is a lot of volatility in the market. This means that the quarterly numbers can be affected by a number of factors, such as the release of new headsets, changes in the market, and economic conditions.
Another reason is that the quarterly numbers are based on shipments, while the yearly numbers are based on sales. This means that the quarterly numbers can be affected by factors such as inventory levels and the timing of shipments.
Finally, the quarterly numbers are often based on estimates, while the yearly numbers are based on actual sales data. This means that the quarterly numbers can be more volatile than the yearly numbers.
Despite these factors, the quarterly numbers can still be a useful tool for tracking the VR market. By tracking the quarterly numbers, you can get a better understanding of the overall trends in the market and the performance of individual brands.
Here are some additional factors that can affect the quarterly VR brand numbers:
- Seasonality: The VR market is cyclical, with sales typically peaking in the holiday season. This can lead to higher quarterly numbers in the fourth quarter of the year.
- Marketing campaigns: VR brands often launch marketing campaigns in the run-up to major events, such as the Consumer Electronics Show (CES). This can lead to higher quarterly numbers in the quarter leading up to the event.
- New product releases: VR brands often release new products in the first quarter of the year. This can lead to higher quarterly numbers in the first quarter of the year.
- Economic conditions: The VR market is sensitive to economic conditions. When the economy is doing well, people are more likely to spend money on discretionary items like VR headsets. This can lead to higher quarterly numbers in economic boom times.
If we had not asked the clarification question or fact checked the answers, we could have based a decision on information that was not correct. The system’s initial reasoning for the differences in its stated results (market volatility) was suspect as it had no bearing on why simple math produced different results from facts that the system presented originally. The 2nd explanation (some were shipments, some were sales) was inconsistent in that it was not specified in the initial answer, an absolute necessity, in both the actual questions and the system’s specific answers, which were always said to be shipments by the system for VR Only when questioned did the system indicate that some of the numbers were referencing sales rather than shipments.
Additionally the system seems to use an ‘answering’ technique often employed by politicians when asked questions they do not want to answer or cannot answer, that of adding a hefty supply of information that does not directly relate to the answer, hoping that the additional verbiage will mask the fact that the answer was not given The scary part of that theory is that such a technique is a very human trait that had to be programmed into the system or ‘learned’ by the system after seeing the practice used in its training data. Maybe we are anthropomorphizing Bard a bit too much, but picking up bad human characteristics is not something we hope for or expect from AI.
Regardless, no matter what the circumstance, a NLM is only as good as its training and inference algorithms and its training data, which leads us back to the point that no matter how large the data sets or how specific they are to the required answers, the accuracy, precision, and recall parameters (also known as the ‘F1’) is the key to understanding the reliability of the system. When humans make decisions based on information presented, they also have an F1, with successful humans having a high F1, while those with low F1s are less successful when it comes to decision making. As each human programmer has an F1, the accumulated F1’s of the programmed algorithms would be an integral part of the system’s F1, so we can’t blame the system for coming up with incorrect answers. Perhaps a year backpacking across Europe might help Bard to gain some needed perspective and get down to finding its own personality and consistency.
RSS Feed