OpenAI’s generative AI chatbot, ChatGPT, has made a lot of noise since its release this past November. But nothing as loud as the recent to-do over Microsoft’s announcement, at its February 7th management meeting, that it would be incorporating ChatGPT into its proprietary search engine Bing. Not to be outdone, and shortly thereafter (one day actually), Google, the putative king of search, pronounced it would also be launching a generative AI product named Bard, but not quite yet. Google’s Bard would need more time for testing before its incorporation into Google search. And that’s probably a very good thing. For Google, and for us.
There’s been much commentary on the heels of these announcements, much of it a healthy critique of the deficiencies of generative AI in answering search queries. Most of the criticism focusing on the inaccuracy of responses, and the inability of generative AI to explain why it provides the responses it does. Fair enough. But when it comes to online search in its purest sense, there’s a massive difference in the quality of the query responses between the AI outputs and Google’s traditional algorithmic results. The quality isn’t close, even though it may appear that way. And here’s why.
The generative AI programs that both tech giants are deploying now use deep learning, recurrent neural networks, the type of neural networks that spit out an answer based on the probability of it being correct. This type of AI algorithm is used to make predictions, and in the chatbot use case, to predict the most likely response based on the original query. The same process is also responsible for making the chatbot ‘chatty’. The natural language processing component – the ability of the chatbot to ‘read’ your query and write a response in a manner that suggests it is a real person – is also a product of probabilistic processing. The algo reads and responds based on the words used, their syntax, and their sequencing in the massive training data sets it mines through (basically, everything that’s published online) to determine what’s likely being asked of it, and using pattern recognition, association, context and frequency to build and construct what it statistically deems to be the best response. Though these programs are extremely powerful, and as evidenced by both Google and Microsoft’s recent product demos, pretty damn good at producing quality responses wrapped up in slick, human-like ‘voice’, at the end of the day, the output is still entirely based on probability.
For some, this may not be problematic. For instance, if the query is simple enough, let’s say I ask “what’s the best recipe for prime rib dinner”, the likely outputs will be responses built from the association, syntax and frequency of “prime rib dinner” plus ”best” plus “recipe”. And this will likely suffice to get the job done. But, is that the answer I was really searching for? I wanted an answer for the best recipe for prime rib dinner. Did the chatbots give me the response I wanted, or did they give me a response that works?
They gave me a response that works.
To understand why Google, at least for now, will remain the king of search, and much more likely to continue to return the search results I want, it’s important to understand the basic objective of its search algorithm, and to do this, it’s helpful to take a closer look at Google’s origin story.
The concept for Google’s search engine can be traced back to Eugene Garfield, who in the mid-1950’s, with degrees in chemistry and library science, launched a print service which tracked the references of scientific, medical, and engineering journals. The Science Citation Index allowed the subscriber to be able to look up which papers were cited by other papers, how many times, and by whom¹. This organization and indexing of data, with sufficient information to rank the quality of the citations by how often they were made, and by the reputation of the scholar who referenced them, is the intellectual underpinning for Google’s PageRank algorithm. Garfield is even referenced in Google’s patent for PageRank, Patent #6,285,999 “Method for Node Ranking in a Linked Database”.
This concept has become much more sophisticated since 1997 (obviously), but it’s still the preeminent force that drives Google’s search engine – peer review ranking of the best results as a function of the amount of backlinks to the reference web page, that are also weighted for the quality of the source of the backlink (the usefulness of the source content, and the reputation of the content producer). This information feeds into google’s ‘authority’ score which then feeds into the ranking of query results. Thus, when one queries Google’s search engine, they’re much more likely to get the response they wanted – a response that has been quantified and ranked by its utility, based on an electronic ‘voting’ mechanism that reflects the perceived value to human information seekers. The response is the output of a bottom-up, informed judgement of millions of the most powerful parallel processing supercomputers in the world (the same ones that are allowing you to read this right now).
So for now, generative AI chatbots are no substitute for Google’s PageRank algorithm. And though there’s a case to be made that ChatGPT will generate greater traffic on Microsoft’s Bing search engine, it will likely be more the product of the entertainment value and novelty, and not because Microsoft has designed a better search product.
Long live the king.
¹The Efficiency Paradox – What Big Data Can’t Do, Edward Tenner, 2018
