Can Google’s AI Overview Be Trusted for DIY Project Advice?

Since it was released in May 2024, Google’s AI Overview has developed into a fairly useful tool, but is it a one-stop shop for all your DIY needs? Not if you ask ChatGPT, which churned out this answer to the question in the title of this post:

• “AI results can be useful but should be taken with a grain of salt. Double-checking and cross-referencing is key for successful DIY outcomes!”

As someone who has been researching and writing DIY material since 2010 and who has published more than 10,000 posts, articles and projects through various online media outlets, I say “Amen” to that.

Although I generally find Google AI helpful, I’ve also found that it can and does hallucinate, which means it supplies misleading or downright incorrect answers based on incomplete, conflicting or misunderstood data. We’ve all heard the stories of chatbots suggesting eating one small rock per day or adding glue to pizza, and while these suggestions are easy to identify as bonkers, some others are more subtle.

If you need help with appliance repair or maintenance, a DIY project or an interior design decision, Google Overview is a good place to start. Actually, you can’t avoid it if you use Google search, because it appears at the top of the results. But you could be making a mistake by relying on it exclusively, as this post will show.

Comparing Pears and Oranges

Ask Google whether pears or oranges are sweeter, and the AI Overview responds:

• “Generally, oranges are considered to be sweeter than pears.”

That’s news to me, because I’ve eaten ripe pears that were practically dripping with sweetness. Disagreeing with the answer gave me three options: Number one, I could simply disregard it, based on personal experience, and move on, or — number two — I could surrender and accept that Google Overview knows more than I do. But the third option, which is the one I chose, is to do as ChatGPT suggests and fact-check the answer by following the links supplied with it.

When I did that, I found out there is, indeed, data from Medical News Today to back up Overview’s answer, but when you look at it closely, the data isn’t conclusive. It states that a large orange contains 17.2 g of sugar while a cup of cubed pears contains 15.7 g. The problem is that a large orange (the data doesn’t say how large) and a cup of cubed pears (how big are the cubes?) aren’t necessarily equivalent quantities.

So Google’s answer isn’t exactly wrong, but given what it’s based on, it isn’t convincing, and it didn’t change my opinion. I still think pears are sweeter.

Google Overview Flubs an Electrical Question

While doing research for a recent post, I asked Google whether plugging multiple devices into a single electrical outlet slows charging. The Overview replied:

• “Yes, plugging multiple devices into a single electrical outlet can potentially slow down the charging speed of each device, as they are all sharing the same power source, which can lead to a reduced current available for each individual device.”

If that sounds to you like a flawed understanding of how electricity gets distributed in a residential circuit, I agree with you. After confirming my view with electrical experts, I explained in the post that each electrical charging device draws its prescribed current load regardless of how many other devices are plugged in, and charging rate isn’t affected…unless there’s a problem with the circuit (or the circuit is overloaded, which would trip the breaker).

In other words, the answer should be “no,” with qualifiers.

Interestingly, that post — published about two weeks ago — now appears as a trending reference next to Google’s answer, even though it contradicts it. The takeaway is that Google Overview, being a bit like Wikipedia on steroids, is a work in progress, and it’s constantly changing as new material becomes available. That’s great, but it means that accepting its answers at face value without fact-checking is risky. You could be absorbing and spreading misinformation — not to mention messing up whatever it is you’re working on.

Can DIYers Trust Google Overview?

Yes and no. DIYers search Google for all sorts of home improvement advice, ranging from the simple to the complex. Over the years, hundreds of humans like myself have covered the most simple DIY issues multiple times, and since Google Overview draws from this published content, its answers can be fairly complete and reliable. So if you want a quick answer to a simple problem, it will usually give you what you need.

Things aren’t as straightforward when questions are complex and involve judgement calls (such as choosing the best building materials for a particular project or designing a kitchen) or code issues, which vary with locality. For such questions, Overview functions best as a starting point for further research, not as a final answer. You have to follow the links.

It’s especially important to remember that AI doesn’t have its own opinions, so if Overview sounds like it’s expressing one, you need to research the human sources on which the opinion is based.

What about AI how-to instructions?

When it comes to actual procedures, there is often enough online material available for Overview to distill into step-by-step instructions. They may be a mishmash drawn from various sources, however, so following them verbatim is risky, especially if it’s a complex procedure like building a deck or installing a roof. Remember that Overview scans online forums and chat rooms as well as curated content, and it may equate half-baked procedures with expert ones. It’s safer to follow links — especially video links — so you know who is providing the instructions and their level of expertise.

Sources

• Medical New Today: Which fruits have the highest amount of sugar?; (2023)
• IBM: What are AI hallucination?; (2023)
• MIT Technology Review: Why Google’s AI Overviews get things wrong; (2024)