BIG ML | EXPLORING CAPABILITIES
Can GPT-3 replace Google for finding knowledge?
GPT-3 and other AI models replacing search is an interesting supposition; but can it really happen? To really compare the two we’ll need to break down what search is, how it is used, and what the actual outputs are. It’s not exactly apples to apples here, as GPT-3 is a product that is sold. Google search on the other hand is not quite a product; Google sells ads, not search.
How is search used?
Searches can be broadly categorized by the user’s intent, or what the user is trying to accomplish with their query. Here are common intents with an example of each:
Navigational – Example search query: Netflix. Desired outcome: Navigate to the Netflix website.
Informational – Example search query: Command to find a file on Linux. Desired outcome: A command which can be used to find a file on a machine running a Linux distribution.
Commercial – Example search query: Best radar detectors. Desired outcome: A page discussing the main features of radar detectors and showing an ordered ranking of various models, which features they have, and what the price is.
Transactional – Example search query: RTX 3090 price. Desired outcome: A page showing the price of an RTX 3090 graphics card at each major retailer.
As you can see there are vast differences in the types of information desired when searching. But they all do have something in common: the desired outcome is the final page you land on, and not the search pages in the middle.
Search engines take you for a ride
Regardless of the type of search, the destination is (almost) always some page on the internet. And searching is like getting a free ride to that destination. Of course, there is a cost to maintaining the infrastructure to give you those rides, and the driver should be expected to make something for providing the service. And they did; It started simply enough, with the driver casually mentioning things related to the place you were heading. It wasn’t intrusive, the driver was profitable, and all was well.
But after a while the driver wanted to make more, as they tend to do. So they hired a psychologist, started doing A/B testing, and learned that even if two people are headed to the same place, they may want different things. And that’s when things started to get weird. Because now they had to start taking into account context, or additional information that is external to the your ride. Things like: where you’ve been before, what you usually buy, what your hobbies are, and more. WAY more.
It’s a grim situation really. While searching you get tracked, then you get taken to a website which might also be selling ads from the same network, and is probably also stuffed with affiliate links to Amazon. And what do they give you? In 2022 it’s likely to be AI-written text which has been proofread by outsourced editors.
Can GPT-3 really replace Google? Navigational and Informational Queries
First, let’s get Navigational queries out of the way first. This is an area where the destination is specific, and you want to ensure you end up at the right place and not a cleverly cloned phishing site. This is one area where search engines win, because they are always scraping and updating their database. GPT-3 for example is only trained on data through 2021, so any changes to the data set since then won’t be included in queries today. They’re essentially frozen in time.
There is also the convenience of it returning clickable links, which GPT-3 does not do on its own just using the playground; you would need to use some kind of client or front-end to do that.
Informational queries however are another story. And this is one area where GPT-3 really shines. If it’s a topic that exists in Wikipedia then it may make sense to go there, but for some search queries you can get endless blog spam. Blogs that are hastily written, possibly by AI, and crammed chock full of keywords, ads, and affiliate links.
This is where I think, if I’m going to get low quality content when I search, I will just use GPT-3 instead.
We’ll compare the results between GPT-3 and Google in the following queries:
- How to make slime
- How to register a domain
- Steps to make a simple paper airplane
- How do I become a member of the National Honor Society?
Query: How to make slime
The first query gets some interesting results from both Google and GPT-3. Google returns a page where the process is described in a ‘featured snippet’, and there are also links to the Home Depot website showing pictures and a video of the process.
GPT-3 returns this text:
There are a few ways to make slime, but the most common method is to mix together glue and Borax. To do this, simply mix together equal parts glue and water, then add a few drops of food coloring if desired. Next, add 1/4 teaspoon of Borax to the mixture and stir until it is fully dissolved. Your slime is now ready to play with!
This is one where it’s close, but I think Google wins in returning the most useful results. The information provided is rather comprehensive here.
Query: How to register a domain
This is one where Google performs quite poorly. The first 4 results in a search are ads to domain registration sites, and none of them describe the process. The 5th result is not an “Ad”, but it links to Google’s domain registration page, and also doesn’t describe the process. Finally after you scroll past that there is an on-page list of questions where one answers ‘How do I register my own domain?’ and has this as the answer:
GPT-3 does better here, but may take a few generation attempts, or perhaps a better prompt than the one I chose. The prompt I used was simply “How to register a domain:” and it got me these results:
There are a few steps you need to take in order to register a domain. First, you need to find a domain registrar. A domain registrar is a company that manages the reservation of Internet domain names. Once you have found a domain registrar, you will need to choose a domain name. Once you have chosen a domain name, you will need to pay the registration fee. After the registration fee has been paid, the domain registrar will add the domain name to the registry.
Query: Steps to make a simple paper airplane
With this query in Google the top result takes me to 6 seconds into a YouTube video. Having a video is nice because it shows me exactly how to do it. But of course, the video had an ad overlay on it.
GPT-3 can’t make videos of course and is limited to textual responses, but this is what you get:
-Fold a sheet of paper in half lengthwise.
-Fold the top corners of the paper down to the center crease, making a triangle.
-Fold the triangle in half, then unfold.
-Fold the top corners of the paper down to the center again, making a smaller triangle.
-Fold the triangle in half, then unfold.
-Fold each side of the paper toward the center, making a wing shape.
-Fold the paper in half along the center crease, then unfold.
-Your paper airplane is now complete!
It tries and does give a coherent response, but this would be a lot harder to follow than the video. In this case Google wins in my book.
Query: How do I become a member of the National Honor Society?
In this query, Google provides a featured snippet showing the eligibility requirements for applying:
But it seems to get that information from a PDF on a school’s website, and not from the NHS results. The NHS page itself is the first result and shows the guidelines for applying. Notably, the requirements on the NHS page are different than the ones shown in the featured snippet, and may only apply to that particular school or NHS chapter.
GPT-3 seems to agree with there being varying requirements between chapters:
The first step is to make sure that you meet the minimum requirements for membership. These requirements vary from chapter to chapter, but typically include having a minimum GPA and being in at least the sophomore year of high school. Once you have verified that you meet the minimum requirements, the next step is to fill out an application. The application will ask for information about your academics, extracurricular activities, and community service. Once you have submitted your application, it will be reviewed by a committee of NHS members and advisers. If you are selected for membership, you will be notified and will be asked to participate in an induction ceremony.
I think this is one where GPT-3 wins, as the results seem to be more accurate from a cursory look at now NHS membership works.
Other types of search queries
So far we compared Google and GPT-3 on navigational and informational queries. Google wins for navigation, but it’s close to GPT-3 for informational queries. It is very situational. However, there seems to be room for improvement on the side of the big ML models, so we could see future iterations of GPT get even better.
We’ll get into other kinds of queries in the next article of this series, coming this Friday 10/14.