How does ChatGPT compare to copywriters?

A diamond is forever. Just do it. Where’s the beef? Melts in your mouth, not in your hands.

Before these taglines entered the zeitgeist, they were among several other ideas that were crafted, considered, tested and ultimately discarded in favor of what became a brand’s iconic slogan. Of course, most marketing taglines do not need to become so iconic that they resonate with the public for decades. A basic tagline just needs to communicate the brand’s message to consumers. Yet, copywriters must still consider a large number of ideas before arriving at one that works, regardless of the marketing campaign’s size or desired impact. 

The nature of this approach might be changing, thanks to the arrival of generative AI.

“Most companies take their advertising needs to an agency and pay a lot of money for a lot of iterations of different ideas when they can basically get that kind of world-class performance from generative AI,” said Joseph Redden, the Aloysius and Eleanor Nathe Collegiate Professor of Marketing at the Mendoza College of Business at the University of Notre Dame.

Redden isn’t suggesting that companies should fire their creative teams and leave everything to AI, but he does believe that marketing teams and agencies can take a strategic approach to using models such as OpenAI’s ChatGPT. In the paper, “Test for the Best: Using ChatGPT to Create Effective Ad Taglines,” published in the Journal of Current Issues & Research in Advertising, he performs two empirical studies to test the capabilities and limitations of using ChatGPT to create taglines for magazine ads.

As a marketing professor, Redden encourages his Notre Dame MBA students to use ChatGPT for assignments just as they would in a professional setting in order to gain a sense for how it works. He also personally uses AI to help make purchasing decisions, develop research ideas or design professional figures for PowerPoint presentations.

“It seemed like it was offering good ideas, but was it? I wondered if I should be teaching it in my class,” he said. Unable to find any research studies that were as simple as what he wanted, he decided to perform his own. “The idea originally was more for my personal edification.”

Previous studies have explored how well generative AI can craft marketing communications, but they have used hypothetical brands or were limited to single product categories. Redden hoped for more generalizable findings and decided that the creative challenges behind crafting taglines made for a compelling test case.

In practice, copywriters cannot rely on a formulaic approach since ads usually must resonate within the predominant cultural context, elicit emotion or curiosity, and conform to the brand’s image and message. Because of this, he predicted that ChatGPT might underperform compared to the practicing experts.

For both tests, Redden pulled recent magazine ads for common products from 10 nationally recognized brands: Blue Buffalo, Capital One, Dell, Glad, Heinz, Home Depot, Honda, LaraBar, Prilosec and the U.S. Postal Service. In the first study, he provided the leading model at the time, ChatGPT-3.5, with the published ads and instructed it to generate 10 alternative taglines per ad. The prompt was simple, only requesting that the AI create likeable taglines that fit with the brand.

He recruited study participants from CloudResearch Connect, an online survey platform for researchers. They rated each generated tagline for likeability as well as the tagline from the original ad, which served as a benchmark for comparison.

“I found that, on average, ChatGPT-3.5 wasn't that good, the taglines were less effective,” Redden said of the AI’s ratings compared to the benchmark. “The average suggestions were not very good, but even the highest-rated tagline was still rated lower than the benchmark the agency experts had developed.”

As he was wrapping up the first test, OpenAI released ChatGPT-4, allowing him to run a second test with an improved AI model. He also updated the prompt to include more specific guidelines to consider the target audience and imagery used in the benchmark ad. This time, the study participants rated the AI-generated taglines more likeable than before, with the best-performing AI tagline eliminating the gap between it and the benchmark.

“Unless you're working on a massive brand with a massive scale where every little difference is amplified, or you’re trying to dramatically change your brand strategy, you should be using GenAI to create ideas,” said Redden. “But then you have to go test them to find that hidden gem.”

He explained that, on average, ChatGPT-4’s taglines were still not exceptional, but its ability to generate numerous taglines at virtually no cost to the marketer means that at least a few of them will perform as well as the benchmark. This will only improve with each new version of ChatGPT. Potentially, this could mean brands can save or reallocate significant funds while freeing up human experts to work on more complicated tasks.

“I think that basic copywriting tasks don't need a human to do them as much anymore. It'll be much faster to use AI,” Redden said. “I think the real skill now is in interpreting what it gives you, testing what it gives you, building on its theme and then persuading your organization to leverage it effectively.”

Along with copywriting, he expects this shift to occur in other fields such as graphic design or customer support, since generative AI is especially good at following specific rules and iterating on solutions. He does not, however, think any company will successfully switch entirely to automation. Instead, this could be an opportunity for these positions to shift “mindless tasks” to AI.

Redden is also curious to see how generative AI will change the customer buying journey and brand awareness in general. Consumers are increasingly turning to AI, including ChatGPT, to ask for guidance before making purchases instead of relying on internet searches.

“It's still a black box how these are trained, and it's not like Google where I can go buy a search word and make sure I'm placed higher in search engines,” he said. “Marketers don’t yet know how they can make sure their brands are represented and recommended by GenAI.”

Generative AI is also a new, beguiling technology, and not everyone is on board with using it. When and how people choose to trust it remains to be seen. “It's very nebulous,” he said. “For whatever reason, some people still seem a little leery when it suggests products versus other sources that we know are biased.”

Following this study, Redden continues to tell his MBA students to use AI, encouraging them to become "connoisseurs” of the technology. This way, they will better understand what specific questions to ask in order to elicit the best answers.

“It's got a lot of potential. If I think of something I’m curious about, then I'll study it and see how it performs,” he said. “That's the luxury of working in academia. We get to do that.”