Bard & Chinese TV – Devil or Angel?
We have both extolled and maligned the recent swath of NLM AI systems put before the public in recent months, and like any new ‘human’ we might meet, we continue to find the good and the bad sides of public AIs. Our goal is to find instances where we can be both comfortable with the information that is provided to us by the AI, more from an accuracy standpoint and less for the conclusions that the AI might provide, but we are continually disappointed with the results as we have noted previously. Our latest interaction with Google’s (GOOG) Bard left us with the same feeling of elation and then disappointment as we went a step further than our initial query.
Our initial query was to see if Bard had data relative to the number of TV sets sold in China in 2022. When we were given a definitive answer, we posed a more complex question about that same data for previous years, and Bard was happy to give us the number of TV set units shipped in China al the way back to 2000. Once we had that data, we decided to look for broad data that might follow a trend line similar to that of TV set shipments in China, and we started with GDP/capita to see if that tied into TV set shipments. Bard was also happy to provide that data back to 2000.
Our initial query was to see if Bard had data relative to the number of TV sets sold in China in 2022. When we were given a definitive answer, we posed a more complex question about that same data for previous years, and Bard was happy to give us the number of TV set units shipped in China al the way back to 2000. Once we had that data, we decided to look for broad data that might follow a trend line similar to that of TV set shipments in China, and we started with GDP/capita to see if that tied into TV set shipments. Bard was also happy to provide that data back to 2000.
China TV Set Shipments (Dual Sources) vs. China GDP/Capita - Source: SCMR LLC, various sources
The problem, however, began to become apparent when we asked Bard to show the GNP data against the TV set shipment data. Bard complied, but when we looked at the comparison, we noticed that the TV set shipment data was different from the data Bard had presented earlier. We asked Bard about the sources of the original China TV set shipment data and the China Tv set shipment data in the GDP comparison and were told that the data sources were different (Bard told us the actual sources but we will call them source 1 and source 2), and in a number of years, very far apart. We checked with Bard to see if one source might differ from the other as to units shipped or sales dollars, but Bard said they were both TV units shipped in China for each year.
As we have had a problem with the fact that Bard had previously not specified the differences (shipments or dollars) between other data we asked Bard specifically about why there was a difference between the data sets and we got a set of reasons why the data might be different, such as ‘different methodologies’, ‘different definitions’, different time frames’, and Bias, all sounding like pat answers to such a question regardless of the actual circumstances, so we went a step further and asked Bard why one set of China’s TV set data was chosen for one question and another for the comparison, and here is the unusual answer:
Yes, there was a reason why I chose one set of data for the earlier query and another for the comparison.
For the earlier query, I chose data from the World Bank [GDP] because it is a reliable source of data on economic indicators. The World Bank has a long history of collecting and compiling data on a wide range of topics, including GDP per capita. The data from the World Bank is also up-to-date and reflects the latest trends in the market.
For the comparison, I chose data from [Source 2] because it is a reliable source of data on the TV market. [Source 2] has a long history of collecting and compiling data on the TV market, and its data is used by a wide range of businesses, including manufacturers, retailers, and distributors. The data from [Source 2] is also up-to-date and reflects the latest trends in the market.
It is important to note that both sets of data are from reliable sources. However, the data from the World Bank is more general, while the data from [Source 2] is more specific to the TV market. Therefore, I chose the data from the World Bank for the earlier query because I wanted to get a general overview of the Chinese economy. I chose the data from [Source 2] for the comparison because I wanted to get a more specific look at the Chinese TV market.
It seems Bard did not get the gist of our question, so we asked if the data originally given for China’s TV set shipments was also reliable, but Bard was struggling with the question. When we were more specific about the data sources from our original question, Bard insisted that the earlier data, despite the fact that it was different from the later data, was still accurate, and cited the same reasons as we noted above as to why the data might be different from different sources. At that point we realized we would not be able to get any more information as to why one data set was chosen over another in each instance.
We were certainly not trying to find flaws in Bard or any other AI system, in fact we were hoping that we could confirm the data and draw a conclusion as to whether there was any connection between China’s GDP/capita and TV set sales, but the entire exercise points out why NLM’s are not quite ready for prime time, and can do considerable damage if they are not very carefully used and all data is vetted. Even if a NLM is taught from a specific data set, such as real-time stock market data, or historic product shipment data, the idea that the system would favor different data sources in each instance proves the unreliability of the algorithm. Any good investor would want to know the data sources for anything involving a major portfolio conclusion and if an analyst were unable to confirm the validity of the data, it could easily invalidate the conclusion. As data underpins most businesses, without reliable data there is no reason to move from human to AI, at least for now.
As we have had a problem with the fact that Bard had previously not specified the differences (shipments or dollars) between other data we asked Bard specifically about why there was a difference between the data sets and we got a set of reasons why the data might be different, such as ‘different methodologies’, ‘different definitions’, different time frames’, and Bias, all sounding like pat answers to such a question regardless of the actual circumstances, so we went a step further and asked Bard why one set of China’s TV set data was chosen for one question and another for the comparison, and here is the unusual answer:
Yes, there was a reason why I chose one set of data for the earlier query and another for the comparison.
For the earlier query, I chose data from the World Bank [GDP] because it is a reliable source of data on economic indicators. The World Bank has a long history of collecting and compiling data on a wide range of topics, including GDP per capita. The data from the World Bank is also up-to-date and reflects the latest trends in the market.
For the comparison, I chose data from [Source 2] because it is a reliable source of data on the TV market. [Source 2] has a long history of collecting and compiling data on the TV market, and its data is used by a wide range of businesses, including manufacturers, retailers, and distributors. The data from [Source 2] is also up-to-date and reflects the latest trends in the market.
It is important to note that both sets of data are from reliable sources. However, the data from the World Bank is more general, while the data from [Source 2] is more specific to the TV market. Therefore, I chose the data from the World Bank for the earlier query because I wanted to get a general overview of the Chinese economy. I chose the data from [Source 2] for the comparison because I wanted to get a more specific look at the Chinese TV market.
It seems Bard did not get the gist of our question, so we asked if the data originally given for China’s TV set shipments was also reliable, but Bard was struggling with the question. When we were more specific about the data sources from our original question, Bard insisted that the earlier data, despite the fact that it was different from the later data, was still accurate, and cited the same reasons as we noted above as to why the data might be different from different sources. At that point we realized we would not be able to get any more information as to why one data set was chosen over another in each instance.
We were certainly not trying to find flaws in Bard or any other AI system, in fact we were hoping that we could confirm the data and draw a conclusion as to whether there was any connection between China’s GDP/capita and TV set sales, but the entire exercise points out why NLM’s are not quite ready for prime time, and can do considerable damage if they are not very carefully used and all data is vetted. Even if a NLM is taught from a specific data set, such as real-time stock market data, or historic product shipment data, the idea that the system would favor different data sources in each instance proves the unreliability of the algorithm. Any good investor would want to know the data sources for anything involving a major portfolio conclusion and if an analyst were unable to confirm the validity of the data, it could easily invalidate the conclusion. As data underpins most businesses, without reliable data there is no reason to move from human to AI, at least for now.
RSS Feed