HubSpot chatbot considered harmful: results from our experiment

Are you considering adding a chatbot to your B2B website? Read this before you do.

Our friends at HubSpot are very keen on 'conversational marketing'. And, a majority (57 percent) of businesses agree that chatbots deliver larger return on investment with minimal effort, according to this Accenture study.

Well, up to a point.

There may be ROI in the case of transactional bots or bots designed to improve internal business productivity. But, when we added an informational customer service chatbot to our HubSpot website, we found the rewards were not all they were cracked up to be.

What pitfalls did we encounter? Let's start at the beginning of our experiment.

Chatbot experiment design

We designed a simple chatbot without human live chat, emulating the multiple choice model we had seen on HubSpot's own website. Quite deliberately, we didn't design it to have live chat because - like many of our clients - we felt that it would be hard to staff up a consistent response.

Instead, it allowed visitors to:

• Book a meeting with the sales team
• Contact us - send a message, as if on the contact page
• Go to the webinars page
• Learn about marketing - a short cut to our top content

The chatbot launched on the 13th of June and ran until the 13th of August 2020 - two months. We had high hopes.

Chatbot lead generation

To measure the results of the chatbot, we created a filtered contact list of people who converted through the chatbot at any time.

Apart from test traffic, there were just nine contacts in the list, two of whom had used obviously bogus contact details ('big bum' was one!) and all had used disposable, nondescript webmail addresses. They were immediately disqualified as potential leads.

The result for lead generation

Over the two-month trial period we had around 500 conversions of all kinds on our website, so the chatbot underperformed generally in terms of conversion rates: it was 0.2 percent on chat compared with about 0.5 percent conversion for the site as a whole. Chat delivered less than two percent of our new contacts during the trial.

Chatbot engagement in HubSpot

Lead generation isn't everything so we also looked at whether existing contacts were using the chatbot. We did this using a smart list that included every contact for whom the 'Conversation by bot' property is true.

It's worth noting that, unfortunately, it wasn't possible to put a date range on that check so this list includes a number of previous shorter-lived experiments. But that flatters the results rather than diminishes them.

The result for engagements

Excluding internal test traffic and the nine contacts generated by the chatbot, there were 24 contacts on the list. Of this list only a small handful had non-webmail addresses and none were qualified leads. So chat wasn't really doing a lot for pre-existing contacts either.

Worst of all, it hit our site performance... badly

As a marketing agency that builds websites, we are proud of creating sites that are fast, optimised and that convert well (HubSpot software willing!). So it's hardly a good look if our own site doesn't live up to the standards we set when delivering websites to clients.

Overall, adding HubSpot chatbot to our site - as measured on our homepage using Google PageSpeed Insights and Pingdom:

• Increased the page weight by a hefty 25 percent;
• More than doubled the load time overall, and
• Put the Google PageSpeed score for mobile in the red.

With Chatflow:

Without Chatflow:

These data imply a severe impact on user experience and site performance, especially for users on mobile devices. By any measure, it is a high price to pay for the limited results we achieved.

As a result, we have switched off the chatbot.

Are chatbots worth it?

It's true, enterprises are making positive headway with chatbots. 40 percent of large businesses – those with more than 500 employees – expect to implement one or more chatbots over this year. (Although Spiceworks made this prediction before 2020, so we suspect this may have changed.) There are all sorts of exciting developments happening with predictive AI and natural language processing that, as tech enthusiasts, we are fascinated by. We do really want chatbots to succeed!

However, as a small B2B business that relies almost entirely on our inbound marketing efforts, the impact on site performance was not worth the results we saw. In fact, its presence likely harmed our other lead generating efforts. Not great given that, with the massive amount of data they have to hand, the folks at HubSpot have shown that, in recent months, business activities have been suppressed across a number of sectors. In short, this is a risk we took, so you don't have to.

Ambitious businesses beware: the chatbot may not kill all humans, but it could kill your website performance.