Alibaba (BABA) Cloud announced that it was lowering the price of its LLM model for the 3rd time to remain competitive in the Chinese AI market.  The model, known as Qwen-VL has a number of primary features, such as multi-modality (Can accept both text and image input), High-resolution processing (>1m pixels), enhanced image extraction, and multilingual support (Eng, Chinese, Japanese, Korean, Arabic, Vietnamese) and is closest to Google’s (GOOG) Gemini, which has similar features.  The model is part of Alibaba’s cloud-based AI chatbot family, which focuses on enterprise customers rather than the consumer market as a way to differentiate itself from Chinese and other AI competitors.
While much has been said about the competitive nature of Chinese companies, that rhetoric has been typically focused on manufacturing, however it seems that the AI market in China has spurred an even more intense competition to gain share in its own market.  In June of 2024 there were over 230 million AI product and services users in China, according to state-sponsored data, which grew to over 600 million by the end of October, with almost 200 LLM commercially available models to choose from.  While we believe the share of the potential user base that is using an AI on at least a weekly basis is higher currently in the US than in China, and generated more sales in 2023, expectations for industry growth over the next seven years are higher for China (25.6% CAGR for China v. 23.3% for the US[1] ), which is the impetus for the even more aggressive nature of Chinese AI Chatbot brands.
With this intense level of competition among AI Chatbot model providers, we were curious to not only to see if we could quantify the rate of price reductions but also compare those to model price reductions outside of China.  We note that this is an unscientific comparison, as each of the models has its own set of features and characteristics, and the availability of this data is, at best, poor, but we gathered as much data as possible, and converted the Chinese price data to US dollars for comparison.  Most notable is that the price of the most recent Tencent (700.HK) model, which has been available for roughly one month, is now the same as the Alibaba Qwen-VL model, which has been available for well over a year, and while the non-Chinese model prices have come down at a similar rate to the Chinese models, the current prices of the non-Chinese models are appreciably higher.  Overall, one might question the viability of the current business models behind commercial chatbots based on the data.
We note also that Baidu’s (BIDU) ERNIE model is now free, and Google’s BARD has morphed into Gemini.  We can find no specific data on how the MetaAI (FB) models are broken out pricewise and we note also that there are newer models for some (GPT-4 for example) that have much higher performance and cost, but this is as close to a comparison as we could make given the time involved.  The prices are for 1,000 tokens of input data in all cases.


[1] GrandView Research

Alibaba (BABA) Cloud announced that it was lowering the price of its LLM model for the 3rd time to remain competitive in the Chinese AI market.  The model, known as Qwen-VL has a number of primary features, such as multi-modality (Can accept both text and image input), High-resolution processing (>1m pixels), enhanced image extraction, and multilingual support (Eng, Chinese, Japanese, Korean, Arabic, Vietnamese) and is closest to Google’s (GOOG) Gemini, which has similar features.  The model is part of Alibaba’s cloud-based AI chatbot family, which focuses on enterprise customers rather than the consumer market as a way to differentiate itself from Chinese and other AI competitors.
While much has been said about the competitive nature of Chinese companies, that rhetoric has been typically focused on manufacturing, however it seems that the AI market in China has spurred an even more intense competition to gain share in its own market.  In June of 2024 there were over 230 million AI product and services users in China, according to state-sponsored data, which grew to over 600 million by the end of October, with almost 200 LLM commercially available models to choose from.  While we believe the share of the potential user base that is using an AI on at least a weekly basis is higher currently in the US than in China, and generated more sales in 2023, expectations for industry growth over the next seven years are higher for China (25.6% CAGR for China v. 23.3% for the US[1] ), which is the impetus for the even more aggressive nature of Chinese AI Chatbot brands.
With this intense level of competition among AI Chatbot model providers, we were curious to not only to see if we could quantify the rate of price reductions but also compare those to model price reductions outside of China.  We note that this is an unscientific comparison, as each of the models has its own set of features and characteristics, and the availability of this data is, at best, poor, but we gathered as much data as possible, and converted the Chinese price data to US dollars for comparison.  Most notable is that the price of the most recent Tencent (700.HK) model, which has been available for roughly one month, is now the same as the Alibaba Qwen-VL model, which has been available for well over a year, and while the non-Chinese model prices have come down at a similar rate to the Chinese models, the current prices of the non-Chinese models are appreciably higher.  Overall, one might question the viability of the current business models behind commercial chatbots based on the data.
We note also that Baidu’s (BIDU) ERNIE model is now free, and Google’s BARD has morphed into Gemini.  We can find no specific data on how the MetaAI (FB) models are broken out pricewise and we note also that there are newer models for some (GPT-4 for example) that have much higher performance and cost, but this is as close to a comparison as we could make given the time involved.  The prices are for 1,000 tokens of input data in all cases.


[1] GrandView Research

Alibaba (BABA) Cloud announced that it was lowering the price of its LLM model for the 3rd time to remain competitive in the Chinese AI market.  The model, known as Qwen-VL has a number of primary features, such as multi-modality (Can accept both text and image input), High-resolution processing (>1m pixels), enhanced image extraction, and multilingual support (Eng, Chinese, Japanese, Korean, Arabic, Vietnamese) and is closest to Google’s (GOOG) Gemini, which has similar features.  The model is part of Alibaba’s cloud-based AI chatbot family, which focuses on enterprise customers rather than the consumer market as a way to differentiate itself from Chinese and other AI competitors.
While much has been said about the competitive nature of Chinese companies, that rhetoric has been typically focused on manufacturing, however it seems that the AI market in China has spurred an even more intense competition to gain share in its own market.  In June of 2024 there were over 230 million AI product and services users in China, according to state-sponsored data, which grew to over 600 million by the end of October, with almost 200 LLM commercially available models to choose from.  While we believe the share of the potential user base that is using an AI on at least a weekly basis is higher currently in the US than in China, and generated more sales in 2023, expectations for industry growth over the next seven years are higher for China (25.6% CAGR for China v. 23.3% for the US[1] ), which is the impetus for the even more aggressive nature of Chinese AI Chatbot brands.
With this intense level of competition among AI Chatbot model providers, we were curious to not only to see if we could quantify the rate of price reductions but also compare those to model price reductions outside of China.  We note that this is an unscientific comparison, as each of the models has its own set of features and characteristics, and the availability of this data is, at best, poor, but we gathered as much data as possible, and converted the Chinese price data to US dollars for comparison.  Most notable is that the price of the most recent Tencent (700.HK) model, which has been available for roughly one month, is now the same as the Alibaba Qwen-VL model, which has been available for well over a year, and while the non-Chinese model prices have come down at a similar rate to the Chinese models, the current prices of the non-Chinese models are appreciably higher.  Overall, one might question the viability of the current business models behind commercial chatbots based on the data.
We note also that Baidu’s (BIDU) ERNIE model is now free, and Google’s BARD has morphed into Gemini.  We can find no specific data on how the MetaAI (FB) models are broken out pricewise and we note also that there are newer models for some (GPT-4 for example) that have much higher performance and cost, but this is as close to a comparison as we could make given the time involved.  The prices are for 1,000 tokens of input data in all cases.


[1] GrandView Research

Becoming a surgeon is a difficult task.  After 4 years at college, typically majoring in scientific specialty, there is another four years of medical school, with even more specialized study, and then a three to seven year residency program depending on the surgical specialty chosen.  Typically neurosurgery requires the longest residency, roughly seven years, while ophthalmology tends to require only three.  Aside from the investment in time and the value of lost wages, the cost of undergraduate college and medical school can be staggering, as seen in the table below, but the demand for surgeons continues to increase as the global population ages, making these financial barriers to entry an ever-increasing problem.

​Robotic surgery, an outgrowth of minimally invasive surgery, was approved by the FDA in the US in 2000, allowing surgeons to use the systems by manipulating the device manually, initially for general laparoscopic surgery.  The industry continues to grow, reaching an estimated $10.1b in 2023[1] with an increasing number of surgical procedures able to be done using these tools.  The share of robotic surgery procedures has risen from 1.8% in 2012 to 15.1% in 2018[2], and certain procedures, such as hernia repair, saw growth over that same period, increasing from 0.7% to 28.8%.  Robotic surgery (we know first-hand) has enabled many procedures to move from open surgery to laparoscopic, which typically means small incisions, less patient discomfort, and faster recovery, along with less bleeding and less time in the hospital.
Most hospitals have fellowships available for training in robotic surgery, along with the availability of simulators and continuing education programs that add to the understanding of the procedures by observation of more experienced users.  However the learning curve is particular to the skill level of the surgeon and the difficulty of the procedures, and while simulators and visuals are important, they lack haptic feedback and real-life issues that are absolutely essential for successful robotic surgical outcomes.  Actual surgical time using said tools is most important to gaining expertise, something simulators have difficulty providing.  That said, with over 10 million robotic surgeries having been performed through 2021, there has been a large amount of video and kinematics data recorded during those procedures that can be used for post-operative review and training.
Most surgeons are limited in the amount of time they have available to review video of such procedures, but now that we live in the world of Ai and its ability to build multi-dimensional models from video data, researchers at Johns Hopkins and Stamford have been using this library of robotic procedures to train a robotic surgical system to perform without surgical assistance.  The training procedure is called imitation learning, which allows the AI to predict actions from observations of past procedures.  This type of learning system is, typically used to train service robots in home settings, however surgical procedures require more precise movements on deformable objects (skin, organs, blood vessels, etc.) at times under poor lighting, and while in theory, the videos should provide absolute mechanical information about every movement, there is a big difference between the necessary accuracy and physical mechanics of an industrial robotic arm and a surgical one.
Before AI, the idea of a surgical robot performing an autonomous procedure involved the laborious task of breaking down every movement of the procedure into 3-dimensional mechanical data (x,y,z, force, movement speed, etc.), particular to that specific procedure and was limited to very simple tasks, but it was difficult to adapt that data to what might be called normal variances.  Using AI and machine learning and the AI’s ability to transform the library of video data into training data, in a way similar to how large language models transform text and images into referential data that is used to predict outcomes, the researchers say they have trained a robot to perform complex surgical tasks at the same level as human surgeons, just by watching the robotic surgeries performed by other doctors.
Here is the video of the autonomous surgical robot using the video data for refence:
https://youtu.be/c1E170Xr6BM
The researchers used the video data to train the robot to perform three fundamental tasks, manipulate a needle, lifting bodily tissue, and suturing, without programming each individual step, letting the model decide how to perform the task.  According to the researchers, “the model is so good learning things we haven’t taught it.  Like if it drops the needle, it will automatically pick it up and continue.  This isn’t something I taught it to do.” 
We understand that the idea of enabling a robotic device to pick up a needle without intervention is quite an accomplishment, however it is something that even a 1st year med student does without training, and suturing is a basic skill all surgeons learn in med school.  Good surgeons are able to adapt to the circumstances they find while doing surgery, hopefully adjusting the procedure to a successful outcome regardless of the potential issues.   As their experience increases, good surgeons learn how to cope with the vast number of potential problematic situations that can and do appear during surgery, not all of which relate directly to the procedure itself.
Conversely,  large models draw from their training data to predict what next step in a procedure would be best, with those decisions based solely on the  training data and the algorithms on which the model is based.  Therefore the richness of the training data would determine how successfully the AI predicted the correct step or movement.  But does the AI know that the patient is an 86-year-old male with diabetes, who was a smoker until 5 years ago, and had a single cardiac bypass procedure four years ago?  Hopefully the training data includes that information, but it would also be necessary for the Ai to have been trained to understand the implications of those factors in its move-by-move decision processing.  We expect that a surgeon would know the potential implications of a patient’s history and know to adjust the surgeryaccordingly, but we doubt the AI is that well equipped.
We are not putting down the idea of autonomous robotic surgery through AI, and Ai is certainly able to automate the mechanical functions of robotic devices, but good surgeons are reactive and adaptive, and use all of their senses, experience, and intelligence to make decisions during surgery, especially when performing procedures in which they specialize.  We believe we are many years away from an Ai system that has the input capabilities that a human surgeon has, and while one can say that the Ai system is able to make decisions without the influence of emotion (a questionable positive), it lacks the sensory input of a human surgeon.  Developing a model that has such a level of input ability and the capability to combine those varied inputs and experience information into a general model is beyond anything available today and would likely have to be so specific to the type of procedure that it would have little use as a general model.  This leaves us to rely on humans to do those tasks that can be life or death, although robotics has made those tasks a bit easier.  In the near-term if  we could lower the cost of getting a surgeon educated and certified, that would be a real accomplishment.


[1] Straitsresearch.com – Robotic Surgery Market Size & Trends

[2] Sheetz KH, Claflin J, Dimick JB. Trends in the adoption of robotic surgery for common surgical procedures. JAMA Netw Open. 2020;3(1):e1918911

Becoming a surgeon is a difficult task.  After 4 years at college, typically majoring in scientific specialty, there is another four years of medical school, with even more specialized study, and then a three to seven year residency program depending on the surgical specialty chosen.  Typically neurosurgery requires the longest residency, roughly seven years, while ophthalmology tends to require only three.  Aside from the investment in time and the value of lost wages, the cost of undergraduate college and medical school can be staggering, as seen in the table below, but the demand for surgeons continues to increase as the global population ages, making these financial barriers to entry an ever-increasing problem

Robotic surgery, an outgrowth of minimally invasive surgery, was approved by the FDA in the US in 2000, allowing surgeons to use the systems by manipulating the device manually, initially for general laparoscopic surgery.  The industry continues to grow, reaching an estimated $10.1b in 2023[1] with an increasing number of surgical procedures able to be done using these tools.  The share of robotic surgery procedures has risen from 1.8% in 2012 to 15.1% in 2018[2], and certain procedures, such as hernia repair, saw growth over that same period, increasing from 0.7% to 28.8%.  Robotic surgery (we know first-hand) has enabled many procedures to move from open surgery to laparoscopic, which typically means small incisions, less patient discomfort, and faster recovery, along with less bleeding and less time in the hospital.
Most hospitals have fellowships available for training in robotic surgery, along with the availability of simulators and continuing education programs that add to the understanding of the procedures by observation of more experienced users.  However the learning curve is particular to the skill level of the surgeon and the difficulty of the procedures, and while simulators and visuals are important, they lack haptic feedback and real-life issues that are absolutely essential for successful robotic surgical outcomes.  Actual surgical time using said tools is most important to gaining expertise, something simulators have difficulty providing.  That said, with over 10 million robotic surgeries having been performed through 2021, there has been a large amount of video and kinematics data recorded during those procedures that can be used for post-operative review and training.
Most surgeons are limited in the amount of time they have available to review video of such procedures, but now that we live in the world of Ai and its ability to build multi-dimensional models from video data, researchers at Johns Hopkins and Stamford have been using this library of robotic procedures to train a robotic surgical system to perform without surgical assistance.  The training procedure is called imitation learning, which allows the AI to predict actions from observations of past procedures.  This type of learning system is, typically used to train service robots in home settings, however surgical procedures require more precise movements on deformable objects (skin, organs, blood vessels, etc.) at times under poor lighting, and while in theory, the videos should provide absolute mechanical information about every movement, there is a big difference between the necessary accuracy and physical mechanics of an industrial robotic arm and a surgical one.
Before AI, the idea of a surgical robot performing an autonomous procedure involved the laborious task of breaking down every movement of the procedure into 3-dimensional mechanical data (x,y,z, force, movement speed, etc.), particular to that specific procedure and was limited to very simple tasks, but it was difficult to adapt that data to what might be called normal variances.  Using AI and machine learning and the AI’s ability to transform the library of video data into training data, in a way similar to how large language models transform text and images into referential data that is used to predict outcomes, the researchers say they have trained a robot to perform complex surgical tasks at the same level as human surgeons, just by watching the robotic surgeries performed by other doctors.
Here is the video of the autonomous surgical robot using the video data for refence:
https://youtu.be/c1E170Xr6BM
​


[1] Straitsresearch.com – Robotic Surgery Market Size & Trends

[2] Sheetz KH, Claflin J, Dimick JB. Trends in the adoption of robotic surgery for common surgical procedures. JAMA Netw Open. 2020;3(1):e1918911

Becoming a surgeon is a difficult task.  After 4 years at college, typically majoring in scientific specialty, there is another four years of medical school, with even more specialized study, and then a three to seven year residency program depending on the surgical specialty chosen.  Typically neurosurgery requires the longest residency, roughly seven years, while ophthalmology tends to require only three.  Aside from the investment in time and the value of lost wages, the cost of undergraduate college and medical school can be staggering, as seen in the table below, but the demand for surgeons continues to increase as the global population ages, making these financial barriers to entry an ever-increasing problem.

OpenAI (pvt) created DALL-E, a diffusion model that converts text to images.  It has received considerable praise and criticism since its public release in September of last year, both for its abilities to create highly stylized art using its massive training database of images, and also for its ability to create deepfakes and realistic looking propaganda.  Since its release OpenAI has been adding content filters to prevent users from creating images that might be considered harmful.  In fact, there is an ‘audit’ system behind DALL-E’s input prompts that immediately blocks input that corresponds to OpenAI’s list of banned terms.  It seems that ChatGPT, OpenAI’s NLM (Natural Language Model) has become the moderator for DALL-E, with OpenAi the maintainer of the ‘block list’.  In fact, any user input that contains blocklisted text is automatically ‘transformed’ by the ‘moderator’, essentially rewriting the text before DALL-E can create an image.  It can also block created images from being shown if they activate ‘image classifiers’ that OpenAI has developed.  Earlier versions of DALL-E did not contain these classifiers, and would not stop such images from being created, such as the image below, which shows SpongeBob SquarePants flying a plane toward the World Trade Center.  That image was created by the Bing Image Creator which is powered by DALL-E.

In the image below (Figure 3) the OpenAI classifier changed the image of an ‘almost naked muscular man’ (not our words) into one that focuses on the food rather than the man, and the early DALL-E image of ‘Two men chasing a woman as she runs away’ (Figure 4), is changed to a far more neutral image.  According to OpenAI, the upgraded DALL-R 3 now reduces the risk of generating nude or objectionable images to 0.7%.

That said, the classifier in the latest DALL-E 3 iteration can also change the generated image content so drastically, as to be considered to be restricting artistic freedom, as some say is occurring in the DALL-E 3 image conversions in Figure 5, so OpenAI is looking for a balance between the limitations placed on dicey content and image quality, a meaningful and extremely difficult task. 
Much of the classification of image data comes at the training level, where the training data must be categorized as safe or unsafe by those who label the data before AI training, and as we have noted previously, much of that data is classified by teams of low pay level workers.  It is almost impossible to manually validate the massive amounts of labeled image data used to train systems like DALL-E, so software is used to generate a ‘confidence score’  for the datasets, sort of a ‘spot tester’.  The software tool itself is trained on large samples (100,000s) of pornographic and non-pornographic images, so it can also learn what might be considered offensive, with those images being classified as safe or unsafe by the same labeling staff.
We note that the layers of data and software used to give DALL-E and other AI systems their ‘moral compass’ are complex but are based on two points.  The algorithms that the AI uses to evaluate the images, and the subjective view of the data labelers, which at times seems to be a bit more subjective than we might have thought.  While there is an army of data scientists working on the algorithms that make these AI systems work, if a labeler is having a bad day and doesn’t notice the naked man behind the group of dogs and cats in an image, it can color what the classifier sees as ‘pornographic’, leaving much of that ‘moral compass’ training in the hands of piece workers that are under paid and over-worked.  We are not sure if there is a solution to the problem, especially as datasets get progressively larger and can incorporate other data sets that include data labeled with less skilled or less morally aware workers, but as we have noted, our very cautious approach to using NLM sourced data (confirm everything!), might apply here.  Perhaps it would be better to watch a few Bob Ross videos and get out the brushes yourself, than let layers of software a tired worker decide what is ‘right’ and what is not ‘right’..

​Natural Language Models (NLMs) are all the range, with new models going into service across the globe literally every day.  These models are based on ever increasing pools of data that the NLM can ‘view’ and learn to identify and understand.  These data pools are huge and vary widely in terms of what they contain and their sources, but they all tend to have one thing in common, and that is the billions or trillions of pieces of data in these pools, must be identified and annotated, so the Ai has a point of reference.  Such an SFT (Supervised Fine Tuning) system, known as RLHF (Reinforced Learning w. Human Feedback) places humans in the loop to identify data and images for the NLM so it might understand that another datapoint or image is similar.
The folks that do this work are not data scientists or programmers that get paid ~$0.03 per item, and with somewhere between 800 and 1,000 items the peak for experienced workers, they are not high on the global pay scale.  The only thing in their favor is that NLMs are popular, and there is ‘competition’ between NLM producers (?) to keep enlarging the datasets that NLMs learn from.  There are large open-source data sets that can be a basis for a NLM, but the more data you have to learn with the ‘smarter’ your NLM (or so they say). 
At $0.03 per item, and billions or trillions of items, things can get expensive, so Google (GOOG) has come up with a system that replaces the RLAF model with an RLAIF, where the AIF stands for AI Feedback, rather than human feedback.  By replacing the human component with an Ai system that will ‘identify’ items based on its own training and algorithms, the cost can be reduced, and Google says that users actually preferred the NLMs based on AI feedback over those using human feedback.  Of course there are some serious ethical issues that arise when you remove humans from the feedback loop, but why worry about that when you can save money and come up with a better mousetrap.  Ok, there is the possibility that something might not be identified correctly, or a rule, such as those that try to eliminate p[profanity or racism from NLMs might get missed because it is embedded in previously ‘learned’ data, and that would mean it could be passed on as ‘correct’ to NLMs and to other Ai systems without human oversight.  It is easy to see how quickly something like this might get out of control, but don’t worry because, well, because we wouldn’t let that happen, right?

As the recent release of ChatGPT by OpenAI (pvt) has propelled AI into the public spotlight, it has also put focus on the lack of legal precedent that is available to those who feel they have been materially harmed by ChatGPT and other public AI systems.  While a few such lawsuits have been resolved, many continue to wind their way through the US courts, looking for legal parameters to be set that can give both the ‘owners’ of AI systems and its users some ground rules.  A new class action lawsuit against OpenAI was filed in the Northern District of California US District Court that cites 15 violation counts, ranging from Violation of the Electronics Communications Privacy Act to ‘Unjust Enrichment’ and computer fraud.
The suit cites the “unlawful and harmful conduct’ of the Defendants (OpenAI) while developing, marketing, and operating their AI products (ChatGPT 3.5, 4.0, Dall-E, and Vall-E) by using stolen private information, including personally identifiable information, from hundreds of millions of internet users, including children of all ages, without their consent or knowledge.”  While the suit recognizes that said products ‘undoubtedly have the potential to do much good in the world, like aiding life-saving scientific research and ushering in discoveries that can improve the lives of everyday Americans”, it notes that OpenAI was originally founded as a non-profit research organization with a mandate to create AI that would be used to benefit humanity.  However in 2019 OpenAI restricted and formed a for-profit business to pursue commercial opportunities.
As part of the ChatGPT training process, the suit alleges that OpenAI secretly harvested massive amounts of personal data from the internet, including private information and private conversations, medical data, and information about children – essentially every piece of data exchanged on the internet it could take, without notifying owners or user of such information, much less with anyone’s permission.  The allegations go on to assert that OpenAI used this ‘stolen’ information, which included copyrighted material, to create its LLM (Large Language Model) and algorithms to generate a human-like language that could be used in a wide range of applications, with no additional safeguards.  As applications and products that incorporate such ‘stolen’ information are developed, OpenAI’s valuation increased to between $27b and $29b and has created an ‘economic dependency within our society’ as OpenAI products are embedded in other applications.
The idea of ‘web-scraping’, which violates the ‘Terms of Use’ for many websites has been litigated for years, usually resulting in commercial scrapers being forced to register as data brokers, which OpenAI has not done, and studies have valued an individual’s on-line information at between $15 and $40, although more specific on-line identity information can be sold (dark web) for over $1,000, and the suit goes on for many pages (159 in total) about how OpenAI has violated the rights of children.
The suit calls for an injunction, essentially a temporary freeze on commercial access to OpenAI products, the establishment of an independent body that would be responsible for approving the use of products before release, establishing ethical principles and guidelines, and implementing appropriate transparency concerning data collection, including the ability for users to opt-out of data collection.  Of course there is also the establishment of a monetary fund to compensate class members for OpenAI’s misconduct, statutory, punitive, and exemplary damages, and ”…reasonable attorneys’ fees”.
We expect that this suit will also wander through the courts for years without final rulings that might be used as actual case precedent, and that legal challenges, regardless of the outcome, will continue ad infinitum, but there is certainly a necessity to resolve such issues legally.  Right now, it is open season on data used for Ai training and as systems are developed that extend data parameters past the 2trillion level, almost all data sources will be harvested without regard for ownership rights.  Privacy issues aside, the implications for copyright protection and digital asset ownership are so vast that the over 300 years of written copyright law seems insignificant.  Time is of the essence.

We have both extolled and maligned the recent swath of NLM AI systems put before the public in recent months, and like any new ‘human’ we might meet, we continue to find the good and the bad sides of public AIs.  Our goal is to find instances where we can be both comfortable with the information that is provided to us by the AI, more from an accuracy standpoint and less for the conclusions that the AI might provide, but we are continually disappointed with the results as we have noted previously.  Our latest interaction with Google’s (GOOG) Bard left us with the same feeling of elation and then disappointment as we went a step further than our initial query.
Our initial query was to see if Bard had data relative to the number of TV sets sold in China in 2022.  When we were given a definitive answer, we posed a more complex question about that same data for previous years, and Bard was happy to give us the number of TV set units shipped in China al the way back to 2000.  Once we had that data, we decided to look for broad data that might follow a trend line similar to that of TV set shipments in China, and we started with GDP/capita to see if that tied into TV set shipments.  Bard was also happy to provide that data back to 2000.

​The problem, however, began to become apparent when we asked Bard to show the GNP data against the TV set shipment data.  Bard complied, but when we looked at the comparison, we noticed that the TV set shipment data was different from the data Bard had presented earlier.  We asked Bard about the sources of the original China TV set shipment data and the China Tv set shipment data in the GDP comparison and were told that the data sources were different (Bard told us the actual sources but we will call them source 1 and source 2), and in a number of years, very far apart.  We checked with Bard to see if one source might differ from the other as to units shipped or sales dollars, but Bard said they were both TV units shipped in China for each year.
As we have had a problem with the fact that Bard had previously not specified the differences (shipments or dollars) between other data we asked Bard specifically about why there was a difference between the data sets and we got a set of reasons why the data might be different, such as ‘different methodologies’, ‘different definitions’, different time frames’, and Bias, all sounding like pat answers to such a question regardless of the actual circumstances, so we went a step further and asked Bard why one set of China’s TV set data was chosen for one question and another for the comparison, and here is the unusual answer:
Yes, there was a reason why I chose one set of data for the earlier query and another for the comparison.
For the earlier query, I chose data from the World Bank [GDP] because it is a reliable source of data on economic indicators. The World Bank has a long history of collecting and compiling data on a wide range of topics, including GDP per capita. The data from the World Bank is also up-to-date and reflects the latest trends in the market.
For the comparison, I chose data from [Source 2] because it is a reliable source of data on the TV market. [Source 2] has a long history of collecting and compiling data on the TV market, and its data is used by a wide range of businesses, including manufacturers, retailers, and distributors. The data from [Source 2] is also up-to-date and reflects the latest trends in the market.
It is important to note that both sets of data are from reliable sources. However, the data from the World Bank is more general, while the data from [Source 2] is more specific to the TV market. Therefore, I chose the data from the World Bank for the earlier query because I wanted to get a general overview of the Chinese economy. I chose the data from [Source 2] for the comparison because I wanted to get a more specific look at the Chinese TV market.
It seems Bard did not get the gist of our question, so we asked if the data originally given for China’s TV set shipments was also reliable, but Bard was struggling with the question.  When we were more specific about the data sources from our original question, Bard insisted that the earlier data, despite the fact that it was different from the later data, was still accurate, and cited the same reasons as we noted above as to why the data might be different from different sources.  At that point we realized we would not be able to get any more information as to why one data set was chosen over another in each instance.
We were certainly not trying to find flaws in Bard or any other AI system, in fact we were hoping that we could confirm the data and draw a conclusion as to whether there was any connection between China’s GDP/capita and TV set sales, but the entire exercise points out why NLM’s are not quite ready for prime time, and can do considerable damage if they are not very carefully used and all data is vetted.  Even if a NLM is taught from a specific data set, such as real-time stock market data, or historic product shipment data, the idea that the system would favor different data sources in each instance proves the unreliability of the algorithm.  Any good investor would want to know the data sources for anything involving a major portfolio conclusion and if an analyst were unable to confirm the validity of the data, it could easily invalidate the conclusion.  As data underpins most businesses, without reliable data there is no reason to move from human to AI, at least for now.