The Sirens
Google (GOOG) has just introduced what it calls “Deep Research” as part of its AI search system which already encompasses AI summaries that compile search results into natural language abstracts rather than a list of websites. While the Ai summaries were the first step in the evolution of internet search, deep research is the next step toward this new approach to internet search processes and the use of the internet for research.
The process was developed to address complex queries by breaking them down into smaller sub-topics and researching each sub-topic as its own search. The system them puts the sub-topic research into a pool and analyzes the results, creating a topical report which can contain bullet points, comparisons, summaries, and possible follow-up topics. The system is multi-modal in that the user can present the system with text, voice, or image input, and can even ‘opt-in’ allowing the model access to past searches, Gmail, and location history to get more tailored results.
We note that Perplexity (pvt) has been based on a similar search system since its release in 2022, although using its own proprietary search bots and index, which we logically assume is smaller than Google’s search index. Both systems are able to show ‘thoughts’, essentially the steps the Ai took to arrive at a conclusion, before and during the research. Here’s an example:
Thought Pattern
“How does the moon affect the economy of the earth?”
as described by Gemini. We have highlighted each sub-query.
1. Initial Exploration & Broad Understanding (What are the direct physical effects of the moon on Earth?)
The free version of Perplexity provides a ‘Deep Research’ mode, but the number of free searches is limited to a few each day. Google, at least for now, says they intend to include ‘Deep Research’ in the free Google AI Search tool, making it available to all users eventually. Right now it is available to ‘Power Users’.
As ‘power users’ we were offered Google’s AI Search capabilities and we posed the following question “What can you tell me about Google’s Deep Research and AI Mode?”. The result, using the Deep Research mode, produced a report, in this case one of over 5700 words after only a few minutes of research. We gave the model no other information, other than asking it to slightly reformat the document after it was created. The report, which we have attached in its reformatted form, contains the following (We asked Gemini to create a TOC for the report):
Table of Contents
1.Does the process of breaking down a single query into many sub-queries increase internet traffic?
Google anticipates increased overall search usage and already reports a 10% increase of Google queries that trigger AI Overview in major markets. However Google also indicates a significant decline of nearly 30% in click-through rates over the past year as users find AI overviews to be sufficient in answering queries. This, reduces the need to click directly on a link to external websites. While this does not directly answer the question, we know the answer is yes, as the use of deep research increases, the number of searches (including ‘sub-searches) will increase. This will serve to offset queries that can be answered directly from AI training data and show both traffic and search growth, but we also see a problem.
Sites work quite hard to optimize their search results in order to drive site traffic which comes from visits and, more importantly, click-throughs that keep folks on the site longer. If Ai search results are ‘enough’ to satisfy most queries, sites will see less internal traffic and could see less sales or generate less advertising revenue. Google will add agents to automate certain functions, like the purchase of tickets, making restaurant reservations, or filling out forms, which can drive traffic and sales to sites, but might not be enough to offset the click-through losses. While Ai search is new enough that the effects are small at this point, we expect Google will have to address that issue further in the future
.
2.Does including user feedback from Ai searches and deep research in training data updates potentially increase system bias?
If the user feedback collected reflects existing societal biases, the model can learn and amplify these biases. For example, if a large portion of users consistently "upvote" or positively reinforce content that subtly favors a particular demographic or viewpoint, the model might learn to prioritize such content, even if it's not objectively superior or fair. User feedback systems, especially those that personalize results based on past interactions, can create echo chambers. If a user consistently interacts positively with information aligning with a certain bias, the system might learn to echo that positive bias and show them more of that kind of information. This feedback loop can reinforce their existing biases and limit exposure to diverse perspectives, making the model itself appear more biased towards those specific views. Adversarial feedback can also be used by those bad actors that are looking to poison Ai results or promote a specific agenda.
Models rely on feedback from diverse groups and backgrounds to lessen bias skewed in a particular direction and actively solicits feedback, rather than relying only on passive feedback, which can come from underrepresented groups or those with an agenda and can statistically analyze feedback to see if certain groups are disproportionally representing a positive or negative view. The problem is that bad actors are constantly improving their biasing techniques, making it difficult for AI anti-biasing systems to keep up, leading to potential bias problems that become embedded in the AI’s internal data over time.
3. How is AI ‘confidence’ judged?
The answer Gemini gave to this question leaves a considerable amount to the imagination. It avoided specifics on how the system judge’s ‘confidence’ in its results, other than if it is able to support a deep research result with reliable sources. If queries are simple, it is understandable that the model will see typical search results as sufficient, but the criteria for judging reliability of sources and data remain part of Google’s secret sauce, and one either agrees to accept Google’s determination or does not.
4. How would one modify a website to take advantage of this new process?
Gemini indicated that following already established rules and practices for site SEO would not change under Ai search and deep research modes, but innumerable companies will try to find a way to game Ai search, as the have been doing with Google’s generic search since it became available. The Google suggestion is always the same, ‘good content’, but anyone with a website knows that there are many things that can help search results. Backlinks to popular sites, popular tags, lots of images and videos, and a litany of other tricks can help push up a site search ranking, so we know that while interesting content is the true objective, SEO is a big part of content creation and Google is very experienced at changing the rules enough to throw off those who try to game the system. Ai search and deep research will affect SEO but in ways we have yet to discover and Google is likely to do little to help folks figure them out, so in this case the choice is to take Google’s advice and keep producing high quality content.
Getting back to the report that the Gemini Deep Research system generated leaves us in an odd position. As ones who spend hours each day researching topics and information, it was astoundingly easy to create the report, so simple in fact that we wonder when the average Joe or Mary would choose not use it to improve productivity and keep the powers that be happy, but we see two drawbacks.
The obvious first would be fact checking. Assuming that all of the information in a large AI generated report is correct is a plan for disaster and the time needed to check each source, data reliability, and whether the conclusions are contextually correct is substantial, and the thought, “If the data was not correct, someone would have already noted that”, is also a mistake. As is well documented AI’s hallucinate and can make up sources, data, or draw conclusions that are factually correct but conceptually incorrect.
Example:
Premise 1 (Factually Correct): "In a vacuum, a feather and a bowling ball fall at the exact same rate."
Premise 2 (Factually Correct): "On Earth, a bowling ball falls significantly faster than a feather."
Conclusion (Factually Correct but Conceptually Incorrect): "Therefore, an object's mass is the primary determinant of its fall speed."
The conclusion states that a bowling ball does fall faster than a feather on Earth, which is a factual observation. It also implies that mass plays a role in the observed difference on Earth, which is true in the context of how air resistance interacts with different masses/shapes. The core conceptual error is attributing the primary determinant of fall speed to mass. The laws of gravity state that in a vacuum all objects accelerate at the same rate regardless of their mass, yet the reason the ball falls faster than the feather on Earth is due to air resistance, which has a much greater proportional effect on the feather's larger surface. As the AI does not have a conceptual understanding of the laws of gravity or physics and is answering by statistically matching word tokens, this type of error is not uncommon.
The second issue is a bit more ephemeral. Doing research is like walking a maze. Some turns lead to dead ends and others to a new path, but both result in learning, as it is just as important to know what trails lead to dead ends as which lead to a conclusion. If someone walks the path for you, you might know the conclusion but not how to get there, and a few moments of Q&A would reveal that lack of understanding. We are all for anything that can improve productivity without affecting the accuracy of results but we worry that the siren-like appeal of AI based deep research will lure many into a world devoid of intellectual insight and creativity. It is easy to be drawn in and even easier to justify when the process is so simple, but the result is a simplistic level of topic understanding and a lack of the cranial stimulation we all need to remember we are human. We still see Ai search and deep research as a glass half full scenario, but one that must be approach with a good deal of caution.
The process was developed to address complex queries by breaking them down into smaller sub-topics and researching each sub-topic as its own search. The system them puts the sub-topic research into a pool and analyzes the results, creating a topical report which can contain bullet points, comparisons, summaries, and possible follow-up topics. The system is multi-modal in that the user can present the system with text, voice, or image input, and can even ‘opt-in’ allowing the model access to past searches, Gmail, and location history to get more tailored results.
We note that Perplexity (pvt) has been based on a similar search system since its release in 2022, although using its own proprietary search bots and index, which we logically assume is smaller than Google’s search index. Both systems are able to show ‘thoughts’, essentially the steps the Ai took to arrive at a conclusion, before and during the research. Here’s an example:
Thought Pattern
“How does the moon affect the economy of the earth?”
as described by Gemini. We have highlighted each sub-query.
1. Initial Exploration & Broad Understanding (What are the direct physical effects of the moon on Earth?)
- Thought: "Before I can link the moon to the economy, I need to understand its fundamental physical influences on Earth. What are the most direct and undeniable effects?"
- Action (AI Query): "What are the primary physical effects of the moon on Earth?"
- Expected AI Response: Focus on tides (ocean and solid Earth), stabilization of Earth's axial tilt (affecting seasons/climate stability), and possibly slight gravitational effects.
- Thought: "Okay, tides are a big one. What industries or economic activities are directly impacted by tidal forces?"
- Action (AI Query - Follow-up 1): "How do ocean tides impact economic activities?"
- Expected AI Response: Navigation/shipping (tide charts for ports, safe passage), fishing (tidal patterns affect fish behavior, harvesting), renewable energy (tidal power generation), coastal tourism (beach access, recreational activities), coastal infrastructure (design, maintenance).
- Thought: "What about the moon's effect on Earth's tilt? How does that relate to the economy?"
- Action (AI Query - Follow-up 2): "What is the economic significance of the moon stabilizing Earth's axial tilt and climate?"
- Expected AI Response: Agriculture (stable seasons for planting/harvesting, predictable weather), tourism (seasonal activities), resource management (water availability, energy demands tied to climate). This is a more indirect but fundamental impact.
- Thought: "Beyond the obvious, are there any other, perhaps less direct, ways the moon influences economic sectors or human behavior that has economic consequences?"
- Action (AI Query - Follow-up 3): "Does the moon's phase affect human behavior or specific industries, like retail or healthcare, in any measurable economic way?" (This might be a more speculative query, but part of deep research is exploring less common hypotheses).
- Expected AI Response: Likely very limited or anecdotal evidence for direct economic impact on retail/healthcare. Might mention cultural events tied to lunar cycles (festivals, holidays) which do have economic impact, or specific agricultural practices (e.g., biodynamic farming).
- Thought: "For the impacts identified (e.g., tidal energy, shipping, agriculture due to climate stability), how are these economic contributions or costs quantified? What metrics are used?"
- Action (AI Query - Follow-up 4): "How is the economic value of tidal energy production measured?" or "What economic models are used to assess the impact of climate stability on global agriculture?"
- Expected AI Response: Details on GDP contribution, job creation, investment in specific technologies, cost savings from predictable climate, etc.
- Thought: "Now I have a good grasp of the direct and indirect economic impacts. How can I summarize this concisely, highlighting the most significant points and acknowledging any areas where the impact is less clear or direct?"
- Action (AI Query - Synthesis): "Summarize the primary economic impacts of the moon on Earth, distinguishing between direct and indirect effects, and provide key examples."
- Expected AI Response: A structured summary covering the key points from the previous steps, emphasizing the critical role of tidal forces in marine industries and renewable energy, and the foundational importance of climate stability for agriculture and global economies. It would likely conclude that while the moon's influence isn't always directly visible in GDP figures, its foundational physical effects are indispensable to several key economic sectors.
The free version of Perplexity provides a ‘Deep Research’ mode, but the number of free searches is limited to a few each day. Google, at least for now, says they intend to include ‘Deep Research’ in the free Google AI Search tool, making it available to all users eventually. Right now it is available to ‘Power Users’.
As ‘power users’ we were offered Google’s AI Search capabilities and we posed the following question “What can you tell me about Google’s Deep Research and AI Mode?”. The result, using the Deep Research mode, produced a report, in this case one of over 5700 words after only a few minutes of research. We gave the model no other information, other than asking it to slightly reformat the document after it was created. The report, which we have attached in its reformatted form, contains the following (We asked Gemini to create a TOC for the report):
Table of Contents
- Executive Summary
- Introduction: The Dawn of a New Search Era
- Understanding Google's AI Search: From SGE to AI Mode
- The Evolution of Google's Generative AI in Search
- Key Distinctions and Official Rollout Timelines
- Core Functionalities: What Google's AI Mode Delivers
- AI-Generated Summaries and Snapshots
- Conversational Mode and Follow-Up Capabilities
- Multi-Step Reasoning, Planning, and AI-Organized Results
- Multimodal Capabilities (e.g., Google Lens Ask with Video)
- Under the Hood: The Technology Powering AI Mode
- The Role of Google Gemini and its Customization for Search
- The "Query Fan-Out" Technique and its Implications for Deeper Web Exploration
- Integration with the Traditional Search Index and Data Sourcing
- User Experience: A Paradigm Shift in Information Discovery
- How AI Mode Enhances User Interaction and Efficiency
- Examples of Complex Queries and Direct Answer Provision
- Impact on User Satisfaction and Search Behavior
- Strategic Implications: Impact on Digital Marketing, SEO, and Content Creation
- Changes in Click-Through Rates (CTR) and Organic Visibility
- New Opportunities for Content Optimization and Topical Authority
- The Evolving Role of Structured Data and Technical SEO
- Impact on Paid Advertising and SERP Real Estate
- Conclusions
- Sources
- Does the process of breaking down a single query into many sub-queries increase internet traffic?
- Does including user feedback in training data updates potentially increase system bias?
- How is AI ‘confidence’ judged?
- How would one modify a website to take advantage of this new process?
1.Does the process of breaking down a single query into many sub-queries increase internet traffic?
Google anticipates increased overall search usage and already reports a 10% increase of Google queries that trigger AI Overview in major markets. However Google also indicates a significant decline of nearly 30% in click-through rates over the past year as users find AI overviews to be sufficient in answering queries. This, reduces the need to click directly on a link to external websites. While this does not directly answer the question, we know the answer is yes, as the use of deep research increases, the number of searches (including ‘sub-searches) will increase. This will serve to offset queries that can be answered directly from AI training data and show both traffic and search growth, but we also see a problem.
Sites work quite hard to optimize their search results in order to drive site traffic which comes from visits and, more importantly, click-throughs that keep folks on the site longer. If Ai search results are ‘enough’ to satisfy most queries, sites will see less internal traffic and could see less sales or generate less advertising revenue. Google will add agents to automate certain functions, like the purchase of tickets, making restaurant reservations, or filling out forms, which can drive traffic and sales to sites, but might not be enough to offset the click-through losses. While Ai search is new enough that the effects are small at this point, we expect Google will have to address that issue further in the future
.
2.Does including user feedback from Ai searches and deep research in training data updates potentially increase system bias?
If the user feedback collected reflects existing societal biases, the model can learn and amplify these biases. For example, if a large portion of users consistently "upvote" or positively reinforce content that subtly favors a particular demographic or viewpoint, the model might learn to prioritize such content, even if it's not objectively superior or fair. User feedback systems, especially those that personalize results based on past interactions, can create echo chambers. If a user consistently interacts positively with information aligning with a certain bias, the system might learn to echo that positive bias and show them more of that kind of information. This feedback loop can reinforce their existing biases and limit exposure to diverse perspectives, making the model itself appear more biased towards those specific views. Adversarial feedback can also be used by those bad actors that are looking to poison Ai results or promote a specific agenda.
Models rely on feedback from diverse groups and backgrounds to lessen bias skewed in a particular direction and actively solicits feedback, rather than relying only on passive feedback, which can come from underrepresented groups or those with an agenda and can statistically analyze feedback to see if certain groups are disproportionally representing a positive or negative view. The problem is that bad actors are constantly improving their biasing techniques, making it difficult for AI anti-biasing systems to keep up, leading to potential bias problems that become embedded in the AI’s internal data over time.
3. How is AI ‘confidence’ judged?
The answer Gemini gave to this question leaves a considerable amount to the imagination. It avoided specifics on how the system judge’s ‘confidence’ in its results, other than if it is able to support a deep research result with reliable sources. If queries are simple, it is understandable that the model will see typical search results as sufficient, but the criteria for judging reliability of sources and data remain part of Google’s secret sauce, and one either agrees to accept Google’s determination or does not.
4. How would one modify a website to take advantage of this new process?
Gemini indicated that following already established rules and practices for site SEO would not change under Ai search and deep research modes, but innumerable companies will try to find a way to game Ai search, as the have been doing with Google’s generic search since it became available. The Google suggestion is always the same, ‘good content’, but anyone with a website knows that there are many things that can help search results. Backlinks to popular sites, popular tags, lots of images and videos, and a litany of other tricks can help push up a site search ranking, so we know that while interesting content is the true objective, SEO is a big part of content creation and Google is very experienced at changing the rules enough to throw off those who try to game the system. Ai search and deep research will affect SEO but in ways we have yet to discover and Google is likely to do little to help folks figure them out, so in this case the choice is to take Google’s advice and keep producing high quality content.
Getting back to the report that the Gemini Deep Research system generated leaves us in an odd position. As ones who spend hours each day researching topics and information, it was astoundingly easy to create the report, so simple in fact that we wonder when the average Joe or Mary would choose not use it to improve productivity and keep the powers that be happy, but we see two drawbacks.
The obvious first would be fact checking. Assuming that all of the information in a large AI generated report is correct is a plan for disaster and the time needed to check each source, data reliability, and whether the conclusions are contextually correct is substantial, and the thought, “If the data was not correct, someone would have already noted that”, is also a mistake. As is well documented AI’s hallucinate and can make up sources, data, or draw conclusions that are factually correct but conceptually incorrect.
Example:
Premise 1 (Factually Correct): "In a vacuum, a feather and a bowling ball fall at the exact same rate."
Premise 2 (Factually Correct): "On Earth, a bowling ball falls significantly faster than a feather."
Conclusion (Factually Correct but Conceptually Incorrect): "Therefore, an object's mass is the primary determinant of its fall speed."
The conclusion states that a bowling ball does fall faster than a feather on Earth, which is a factual observation. It also implies that mass plays a role in the observed difference on Earth, which is true in the context of how air resistance interacts with different masses/shapes. The core conceptual error is attributing the primary determinant of fall speed to mass. The laws of gravity state that in a vacuum all objects accelerate at the same rate regardless of their mass, yet the reason the ball falls faster than the feather on Earth is due to air resistance, which has a much greater proportional effect on the feather's larger surface. As the AI does not have a conceptual understanding of the laws of gravity or physics and is answering by statistically matching word tokens, this type of error is not uncommon.
The second issue is a bit more ephemeral. Doing research is like walking a maze. Some turns lead to dead ends and others to a new path, but both result in learning, as it is just as important to know what trails lead to dead ends as which lead to a conclusion. If someone walks the path for you, you might know the conclusion but not how to get there, and a few moments of Q&A would reveal that lack of understanding. We are all for anything that can improve productivity without affecting the accuracy of results but we worry that the siren-like appeal of AI based deep research will lure many into a world devoid of intellectual insight and creativity. It is easy to be drawn in and even easier to justify when the process is so simple, but the result is a simplistic level of topic understanding and a lack of the cranial stimulation we all need to remember we are human. We still see Ai search and deep research as a glass half full scenario, but one that must be approach with a good deal of caution.
RSS Feed