What is Google’s AI Mode
On March 5th, Google launched a new generative AI integration in its search engine, called AI Mode, available only in the US and as part of a beta that users can voluntarily access.
AI Mode consists of a separate tab from other results, offering a chatbot experience powered by an LLM, very similar to Copilot within Bing, Perplexity, or ChatGPT Search.
Despite being in Spain, I signed up for the beta waitlist, and since yesterday, March 7th, I’ve been able to access the AI Mode beta by using a VPN and browsing as if I were in the US.
In this post, I’ll share my impressions after a few hours testing this new AI Mode, comparing it with rivals like ChatGPT and Google’s other generative AI experience, AI Overviews.
Interface and Usability
First, Google’s AI Mode is a 100% chatbot-style product, not a small space dedicated to inserting a chat response within a normal SERP, which is what AI Overviews has offered until now. Additionally, there’s no inclusion of typical Google modules or SERP Features. For now, it’s just text and discrete «cards» to highlight the websites used as information sources.
Therefore, the AI Mode interface is very different from the traditional Google interface, but at this point, it will feel familiar to almost any user, as it’s very similar to ChatGPT’s, but with the addition of the usual tabs found in any Google SERP.
Compared to the usual Google experience, you’ll notice some latency. While the AI Mode response is generating, the rest of the screen is empty, something we’re not used to with Google. But I don’t think the complete response takes longer to produce than in ChatGPT or Perplexity.
I do see two clear usability issues:
- Follow-up questions aren’t suggested; instead, the user must think of their next question and type it directly into the search box. This is a disadvantage (for both Google and the user) compared to Perplexity, which by default always shows five related questions at the end of the current response. ChatGPT is similar to Google AI Mode in this regard, as it doesn’t display suggestions to continue the conversation by default.
- The sources module doesn’t follow the user’s scroll by default. Since responses are usually long, this is negative for users who have scrolled down a lot and need to go back to the beginning of the response to view these sources. And logically, it’s bad for the cited content creators, who have fewer opportunities for clicks. To pin the references module so it follows you while scrolling through the response, you need to click «Show All.» At least in this aspect, AI Mode doesn’t perform worse than its rivals, as Perplexity’s highlighted sources also disappear when scrolling down (but its responses tend to be shorter), and compared to ChatGPT, which would be the worst of all since you have to click on Sources for a sidebar with cited sources to appear (although once expanded, it stays fixed).
On the other hand, Google’s chat mode has UX advantages: links to sources usually take you exactly to the relevant paragraph for the answer (using a scrollTo: fragment), and each successive question unfolds a new references module.
How AI Mode Handles Different Search Types
The response length varies greatly, as does the number of cited sources. It seems that if AI Mode detects, based on user intent, that a short answer is sufficient or even better than a long one, it will offer a much shorter and more direct response. This is very likely related to the Google API discovered by Mark Williams-Cook that classifies queries by intent, where «short answer» is one possible type.
These short answers offer a much lower number of sources than usual (1 in this case, which by the way doesn’t give a completely correct answer).
In my opinion, AI Mode shines more and offers something beyond the current Google when you deepen the conversation, that is, when you continue asking questions after the initial search.
For example, if it gives an incorrect answer and you correct it, AI Mode acknowledges its mistake and corrects its previous response, in the same way a chatbot like Gemini or ChatGPT would.
This is an improvement over AI Overviews, which if it gives a wrong answer, generally doesn’t offer a way to point out its error or refine the question to consider more data. Initially, AI Overviews did have follow-up capability, but Google later removed that functionality.
Medical and Legal Queries
When answering YMYL topics, such as health and legal issues, AI Mode always offers a disclaimer recommending you consult an expert. This is an advantage over Perplexity, which doesn’t do this. Meanwhile, ChatGPT almost always includes a similar warning for these searches.
Local Queries
Despite accessing through a VPN, Google seems able to draw from my account history (I’m still logged in with my personal account) and adapts its response to my usual location.
Regarding sources, here we see Google favoring its own services by including Google Maps links. Nothing surprising, since ChatGPT does the same, and also usually includes Google Maps links in searches with a local component.
When you make a clearly local search, such as «restaurants near me,» AI Mode responds with a list highlighting each business’s average reviews and address. It’s like adapting the Local Pack to a chat format. But surprisingly, they don’t include the map module as such, which is something even Gemini does (text list plus inserted map). Perhaps they’ll add modules later.
News Queries
I may have asked for the most difficult task here, but for a recent or real-time news item on a topic outside the US, AI Mode fails miserably.
In the case of «floods in Madrid,» it’s not able to see that the news requires News Stories treatment or even possible emergency coverage, as normal Google does (with a super useful widget).
Even for US news, I think the experience is worse than what Google already provides.
Transactional Queries
If you want to buy something, the AI Mode response doesn’t seem like an improvement over current Google, which already offers a product module with updated prices.
I believe that the further we move away from informational searches, especially complex ones, the less necessary it becomes to use AI Mode.
Queries in Other Languages
As expected, AI Mode currently only provides answers in English and to questions in English. If you enter a question in another language, it won’t respond.
Differences with AI Overviews
I’ve noticed that in some cases, even when responding in a brief and direct format, AI Mode’s answers can differ in form, nuances, and cited sources from those of AI Overviews.
But logically, the biggest difference between the two Google products is seen in searches that can accept long answers with nuances or various approaches. In those cases, AI Overviews might stay on the surface or give an insufficient answer, while AI Mode provides a much more complete result that truly serves as a starting point to delve into various questions from the initial search.
This is my subjective opinion, but I believe that if Google can refine the flaws in this beta, in the long run, it will have a much more useful complement to its traditional search engine than AI Overviews is today. I think the best approach would be to offer traditional responses for local, transactional, and navigational intents, and within informational intent for «short answer» or «yes/no» type searches, while providing something like what AI Mode currently offers for complex informational searches.
Web Sources
As mentioned, the number of cited sources varies greatly depending on the search, ranging from a single source to 36, which is the maximum I’ve found so far. The relevance of the included sources also varies.
In fact, I’ve found cases where sources can include results irrelevant to the search, an error probably caused by an exact keyword match on an authoritative site (Wikipedia in this case), but within a page that discusses another topic or doesn’t address the same search intent.
If it’s as I think, this flaw will probably be corrected as the tool’s testing phase progresses.
Links to Third-Party Websites
Apart from the sources module, we have the links that appear within the chatbot’s text response itself.
These links can appear in two ways: with an icon at the end of each sentence or statement (the usual form), and less frequently, as underlined anchor text within the text.
Links to third-party websites are generally quite numerous within the responses; there’s not much to criticize Google for here. But I’m afraid that the system itself, as already happens with ChatGPT and with AI Overviews themselves, most of the time doesn’t encourage users to click through to a third-party website.
Therefore, I fear that, as certain studies on AI Overviews have already revealed, the CTR to these links won’t be high, at least not as high as in a traditional «10 Blue Links» response.
There is an opportunity for users to click, it’s true, but there isn’t always an incentive, since in many cases the key data has already been provided by the chatbot’s response.
This will need to be verified over time and if AI Mode eventually becomes widespread, but I believe more traffic will be directed to third-party websites in cases where users are likely to want to expand on their own, and not in searches where a few paragraphs of text already serve to satisfy the answer.
Conclusion: AI Mode Shows Potential
Accepting that this is an early beta and that the product will evolve, and that my tests so far have been limited, I think AI Mode has the potential to be more useful to users than AI Overviews, which has always given me the impression of being shoehorned into the search engine, as if they wanted to show that Google also provides AI-generated answers, but without worrying about user experience and the usefulness of displaying them or not.
If the goal is to leverage the possibilities of generative AI, AI Mode makes more sense, and giving users control over whether they want to interact with this type of search engine or not is clearly the way to go (this is currently the case in the beta; we’ll see later).
As always, the question remains of how it will affect the web ecosystem, from which Google takes the answers it generates. In this, Google shows good intentions by displaying numerous links to sources, but I think it needs to go a step further and improve the UX to minimize friction when clicking through to a third-party website, at least in cases where it makes sense to expand.
And one more unknown, which if Google isn’t concerned about, I won’t worry about more than they do: how to integrate advertising within this type of results without negatively impacting the user experience. A problem that, by the way, Google has not yet solved for AI Overviews, even though they’ve been active for almost a year, and neither has OpenAI for the free version of ChatGPT (I understand it can’t continue to be free indefinitely and that at some point they’ll have to add ads).
Who will bell the cat of advertising in AI chatbots? We shall see.
