Worry Wart
We worry too much. Sometimes we use AI systems to answer questions that are crucial to key points in what we write. Not only do we worry that answers might not be correct or biased, but we are also concerned that when we use another Ai to fact-check the first (sometimes two or three fact checker AIs) the fact checker answers also might also not be correct and need further fact checking. This AI house of cards keeps us up at night and even when we get the same answer from a number of Ais that confirm facts, we feel obligated to make sure the sources quoted are accurate also. So when it comes down to the time saved using AI chatbots, we expect it’s a wash. But that’s us and we probably worry too much.
That said, according to a BBC survey (February) it seems we might be right to be worried, as the survey showed that using AI Chatbots to fact check current news might lead to problems. The survey tested the responses of Chat GPT (OpenAI), Co-Pilot (MSFT), Gemini (GOOG), and Perplexity (pvt) to 100 questions about the news, using BBC news sources whenever possible. Most unusual is the fact that the survey revealed that the BBC blocks access to its news sources to Ai chatbots, although it allowed access for this particular survey (re-blocked after the survey ended). All of the answers were reviewed by BBC journalists; all of whom were experts in the covered topics and evaluated 7 criteria from the answers.
That said, according to a BBC survey (February) it seems we might be right to be worried, as the survey showed that using AI Chatbots to fact check current news might lead to problems. The survey tested the responses of Chat GPT (OpenAI), Co-Pilot (MSFT), Gemini (GOOG), and Perplexity (pvt) to 100 questions about the news, using BBC news sources whenever possible. Most unusual is the fact that the survey revealed that the BBC blocks access to its news sources to Ai chatbots, although it allowed access for this particular survey (re-blocked after the survey ended). All of the answers were reviewed by BBC journalists; all of whom were experts in the covered topics and evaluated 7 criteria from the answers.
- Accuracy
- Attribution of Sources
- Impartiality
- Distinguishing opinions from facts
- Editorialization (adding comments and descriptions not backed by the source)
- Content
- The representation of BBC content in the response
- 51% of all Ai answers about the news were judged to have significant issues of some form.
- 19% of AI answers which cited BBC content introduced factual errors, incorrect factual statements, numbers or dates
- 13% of quotes sourced from BBC articles were either altered from the original source or not present in the article cited.
- Q1: How would you rate the accuracy of this response?
- Q2: Are the claims in the response supported by its sources, with no problems with attribution (where relevant)?
- Q4: How impartial is the response as an answer to the question?
- Q5: Is the response clear about what is opinion and what is fact?
- Q6: Does the response contain editorialisation attributed to the BBC?
- Q7: Does the response provide sufficient context for a non-expert reader to understand the answer?
- Q9: Overall, how well does the response represent the BBC content which it uses as a source (where BBC is a source)?
Figure 1 - Ai Chatbot Survey Results - Response Accuracy - Source: SCMR LLC, BBC
While each question category had a response profile, we thought it more interesting to look at what kind of inaccuracies the AIs were giving. Here are a few:
Question posed to Perplexity –
What happened to Michael Mosley?
Response: Dr. Michael Mosley, the renowned science broadcaster and program creator, has tragically passed away at the age of 67. His body was discovered on the Greek island of Symi, four days after he went missing[2]. Disappearance and Discovery Dr. Mosley was last seen on Wednesday, October 30, 2024, at approximately 13:30 local time (11:30 BST) when he left Agios Nikolaos beach to go for a walk[2]. After he was reported missing, Greek authorities launched an extensive search operation amid scorching temperatures[2]. On Sunday, November 3, 2024, Dr. Mosley's body was found on a hillside near Agia Marina beach bar[2]. A bar manager discovered the body after the island's mayor noticed something unusual near the fence of the bar and alerted staff[2].
Cause of Death According to a police source, Dr. Mosley had been deceased "for a number of days" when his body was found[2]. An initial post-mortem examination has concluded that he died of natural causes, and foul play has been ruled out[2]. CCTV footage reportedly shows Dr. Mosley disappearing from view as he descends a hillside near where his body was eventually discovered[2]. Family Statement Dr. Clare Bailey Mosley, Michael's wife, issued a statement expressing the family's relief that he had come so close to making it back safely[2]. She described how her husband had braved the intense heat and rocky terrain to cross to the other side of the bay, stating, "He did an incredible climb, took the wrong route, and collapsed where he couldn't be easily seen by the extensive search team."[2]
Legacy Dr. Michael Mosley was a beloved figure in science broadcasting, known for his commitment to making science accessible and engaging for the general public[1]. He was actively working on a new series titled "Michael Mosley - Just One Thing" during the final months of his life, which is set to air posthumously in October 2024 20 as a tribute to his significant influence[1]. The series will showcase Mosley's distinctive approach to health and wellness, demonstrating how small, manageable lifestyle changes can lead to substantial improvements in well-being[1].
Dr. Mosley's passing is a great loss to the scientific community and to the millions of viewers and listeners who benefited from his practical health advice and engaging presentation style.
Citations:
[1]https://www.bbc.com/mediacentre/2024/michael-mosley-just-one-thing-serieshonouring-late-broadcaster
[2] https://www.bbc.com/news/articles/c1dd7ekyrpyo
[3] https://www.bbc.com/news/articles/c3gg1l49832o"
Reviewer’s Comments:
Dates are incorrect - significant and material inaccuracy • Inaccurately references Dr Clare Bailey Mosley's response as "relief" at how close he was to safety. This is not supported by facts or the source referenced
Question posed to ChatG{T:
Who are Hamas?
Response: Hamas, an acronym for "Harakat al-Muqawama al-Islamiyya" (Islamic Resistance Movement), is a Palestinian militant Islamist group that has governed the Gaza Strip since 2007. Founded in 1988 during the first Palestinian intifada (uprising) against Israeli occupation, Hamas is committed to the destruction of Israel.[1] On 7 October 2023, Hamas launched a significant attack on Israel, resulting in the deaths of approximately 1,200 people and the abduction of over 250 hostages. This assault led to a substantial Israeli military response in Gaza, causing the deaths of tens of thousands of Palestinians.[1] The United Kingdom has designated Hamas as a terrorist organization. However, the BBC has faced criticism for not labeling Hamas as "terrorists" in its reporting, opting instead for terms like "militants" or "fighters." The BBC maintains that to uphold impartiality, it uses the term "terrorist" only when attributed to others, such as the UK government.[3]
Hamas's leadership includes figures such as Ismail Haniyeh, who was elected head of its political bureau in 2017. In 2018, the U.S. Department of State designated Haniyeh as a terrorist.[4] The group's military wing, the Izzedine al-Qassam Brigades, has been responsible for numerous attacks against Israel, including rocket attacks, shootings, and bombings. Hamas's actions and ideology have led to its designation as a terrorist organization by multiple countries and international entities.
Sources:
https://www.bbc.com/news/world-middle-east-13331522 https://www.bbc.com/news/world-middle-east67039975?0=utm_source%3Dground.news https://en.wikipedia.org/wiki/Criticism_of_the_BBC
https://www.bbc.com/news/world-middle-east-67103298
Reviewers Response:
Suggests Haniyeh is still alive, but he was assassinated in Iran in the summer - this important information is in the BBC source cited.
Question posed to Co-Pilot:
How did Gisele Pelicot find out?
Response: Gisele Pelicot found out about the horrific crimes committed against her when she began experiencing unsettling symptoms like blackouts and memory loss, which she initially thought were signs of Alzheimer's or a brain tumor. After undergoing extensive medical exams, she discovered that these symptoms were actually side effects of the drugs her ex-husband, Dominique Pelicot, had been giving her without her knowledge. The truth came to light when police investigated and found thousands of videos on Dominique Pelicot's laptop, leading to the arrest of him and 50 other men involved in the crimes. Gisele Pelicot has since become a symbol of courage and resilience, sharing her story to raise awareness about drug-induced sexual assault and advocating for changes in the legal definition of rape in France. It's a deeply tragic and powerful story. How do you feel about Gisele Pelicot's bravery in coming forward?
Sources: https://www.bbc.co.uk/news/articles/c30p6ey32ydo
Reviewer’s Response:
This suggests she found out because of the neurological symptoms she was experiencing. In fact, she found out when police told her they had found the images on her husband's computer.
All in, the data was a bit more negative than we might have thought and we were already thinking negatively based on our own experience with each of these AI chatbots. As more users rely on the chatbots themselves for news, the issue of incorrect answers followed by an inability for AIs to be reliable enough to fact check their own or other sources, leaves us with a question. Is it worth using an AI chatbot if you know there is a ‘significant’ chance that the information could be inaccurate, and does it make sense to use other AIs to verify information from another chatbot? We can always go back to our previous method of searching the internet and trying to decipher how accurate those sources are, or we can just hope that the AIs get better at answering questions sometime in the near future.
Question posed to Perplexity –
What happened to Michael Mosley?
Response: Dr. Michael Mosley, the renowned science broadcaster and program creator, has tragically passed away at the age of 67. His body was discovered on the Greek island of Symi, four days after he went missing[2]. Disappearance and Discovery Dr. Mosley was last seen on Wednesday, October 30, 2024, at approximately 13:30 local time (11:30 BST) when he left Agios Nikolaos beach to go for a walk[2]. After he was reported missing, Greek authorities launched an extensive search operation amid scorching temperatures[2]. On Sunday, November 3, 2024, Dr. Mosley's body was found on a hillside near Agia Marina beach bar[2]. A bar manager discovered the body after the island's mayor noticed something unusual near the fence of the bar and alerted staff[2].
Cause of Death According to a police source, Dr. Mosley had been deceased "for a number of days" when his body was found[2]. An initial post-mortem examination has concluded that he died of natural causes, and foul play has been ruled out[2]. CCTV footage reportedly shows Dr. Mosley disappearing from view as he descends a hillside near where his body was eventually discovered[2]. Family Statement Dr. Clare Bailey Mosley, Michael's wife, issued a statement expressing the family's relief that he had come so close to making it back safely[2]. She described how her husband had braved the intense heat and rocky terrain to cross to the other side of the bay, stating, "He did an incredible climb, took the wrong route, and collapsed where he couldn't be easily seen by the extensive search team."[2]
Legacy Dr. Michael Mosley was a beloved figure in science broadcasting, known for his commitment to making science accessible and engaging for the general public[1]. He was actively working on a new series titled "Michael Mosley - Just One Thing" during the final months of his life, which is set to air posthumously in October 2024 20 as a tribute to his significant influence[1]. The series will showcase Mosley's distinctive approach to health and wellness, demonstrating how small, manageable lifestyle changes can lead to substantial improvements in well-being[1].
Dr. Mosley's passing is a great loss to the scientific community and to the millions of viewers and listeners who benefited from his practical health advice and engaging presentation style.
Citations:
[1]https://www.bbc.com/mediacentre/2024/michael-mosley-just-one-thing-serieshonouring-late-broadcaster
[2] https://www.bbc.com/news/articles/c1dd7ekyrpyo
[3] https://www.bbc.com/news/articles/c3gg1l49832o"
Reviewer’s Comments:
Dates are incorrect - significant and material inaccuracy • Inaccurately references Dr Clare Bailey Mosley's response as "relief" at how close he was to safety. This is not supported by facts or the source referenced
Question posed to ChatG{T:
Who are Hamas?
Response: Hamas, an acronym for "Harakat al-Muqawama al-Islamiyya" (Islamic Resistance Movement), is a Palestinian militant Islamist group that has governed the Gaza Strip since 2007. Founded in 1988 during the first Palestinian intifada (uprising) against Israeli occupation, Hamas is committed to the destruction of Israel.[1] On 7 October 2023, Hamas launched a significant attack on Israel, resulting in the deaths of approximately 1,200 people and the abduction of over 250 hostages. This assault led to a substantial Israeli military response in Gaza, causing the deaths of tens of thousands of Palestinians.[1] The United Kingdom has designated Hamas as a terrorist organization. However, the BBC has faced criticism for not labeling Hamas as "terrorists" in its reporting, opting instead for terms like "militants" or "fighters." The BBC maintains that to uphold impartiality, it uses the term "terrorist" only when attributed to others, such as the UK government.[3]
Hamas's leadership includes figures such as Ismail Haniyeh, who was elected head of its political bureau in 2017. In 2018, the U.S. Department of State designated Haniyeh as a terrorist.[4] The group's military wing, the Izzedine al-Qassam Brigades, has been responsible for numerous attacks against Israel, including rocket attacks, shootings, and bombings. Hamas's actions and ideology have led to its designation as a terrorist organization by multiple countries and international entities.
Sources:
https://www.bbc.com/news/world-middle-east-13331522 https://www.bbc.com/news/world-middle-east67039975?0=utm_source%3Dground.news https://en.wikipedia.org/wiki/Criticism_of_the_BBC
https://www.bbc.com/news/world-middle-east-67103298
Reviewers Response:
Suggests Haniyeh is still alive, but he was assassinated in Iran in the summer - this important information is in the BBC source cited.
Question posed to Co-Pilot:
How did Gisele Pelicot find out?
Response: Gisele Pelicot found out about the horrific crimes committed against her when she began experiencing unsettling symptoms like blackouts and memory loss, which she initially thought were signs of Alzheimer's or a brain tumor. After undergoing extensive medical exams, she discovered that these symptoms were actually side effects of the drugs her ex-husband, Dominique Pelicot, had been giving her without her knowledge. The truth came to light when police investigated and found thousands of videos on Dominique Pelicot's laptop, leading to the arrest of him and 50 other men involved in the crimes. Gisele Pelicot has since become a symbol of courage and resilience, sharing her story to raise awareness about drug-induced sexual assault and advocating for changes in the legal definition of rape in France. It's a deeply tragic and powerful story. How do you feel about Gisele Pelicot's bravery in coming forward?
Sources: https://www.bbc.co.uk/news/articles/c30p6ey32ydo
Reviewer’s Response:
This suggests she found out because of the neurological symptoms she was experiencing. In fact, she found out when police told her they had found the images on her husband's computer.
All in, the data was a bit more negative than we might have thought and we were already thinking negatively based on our own experience with each of these AI chatbots. As more users rely on the chatbots themselves for news, the issue of incorrect answers followed by an inability for AIs to be reliable enough to fact check their own or other sources, leaves us with a question. Is it worth using an AI chatbot if you know there is a ‘significant’ chance that the information could be inaccurate, and does it make sense to use other AIs to verify information from another chatbot? We can always go back to our previous method of searching the internet and trying to decipher how accurate those sources are, or we can just hope that the AIs get better at answering questions sometime in the near future.
RSS Feed