A weekly Google Hangout dedicated to discussing content marketing, search marketing, SEO and more.
Topic: The problem with attribution models for marketing
Attribution modeling, in all of its messed up, inaccurate glory is on the rise. Attribution modeling, specifically multi-channel attribution modeling, simply put, is a way to assign “credit” to various channels for their role in creating conversions. And not all models are created equally.
Steve Farnsworth joins Erin to discuss the problems and challenges associated with measuring success with attribution modeling.
Erin Robbins, President at GinzaMetrics
Steve Farnsworth, CMO, The Steveology Group
FULL VIDEO TRANSCRIPT
Erin: Hey, welcome to Get Found, our weekly web series on digital marketing, search, and all things interesting in the marketing space. I’m joined today by Steve Farnsworth, who I will introduce in a minute, but today I want to talk a little bit about the problem with attribution models for content marketing, whether there actually is a problem and all that fun stuff.
According to the Content Marketing Institute (CMI), proving ROI on content marketing efforts is becoming less of a challenge. I’m not sure I’ve actually heard anybody say that. I think what might be more accurate is that people are investing in content marketing, there are lots of metrics for content marketing, and that people believe that content marketing actually works. But I don’t believe that assigning value in ROI to content and things like that is any less challenging today than it was a year ago.
What does seem to be on the rise is attribution modeling and conversations around it. Attribution modeling – specifically multi attribution modeling – simply put is a way to assign credit to various marketing things for their role in creating conversions and not all models are created equally or accurately.
Today, we’re going to talk about some of the challenges associated with attribution models. Let’s kick this thing off.
Steve, give us an introduction about who you are, why you’re awesome, and your thoughts on attribution models because you said you could tell us everything about attribution modeling in this 15-minute segment. The floor is yours.
Steve: That I knew. I’m Steve Farnsworth from the Steveology group, we do content marketing for high-tech B2B, and you can find me on Twitter at @Steveology. In terms of attribution models – and all I meant by that comment is everyone I’ve heard talk about this – this is a bane.
I was at a table full of CMOs last Wednesday night. We were talking about attribution models and I said, “I’d love to hear what you guys’ thoughts are,” and everybody just groaned, because they’re all fighting with the same thing. We had first touch attribution models for a long time and last touch attribution models. Most companies who are able to track down are tracking both because there is always a set of people who have a dog in the fight for the late stage stuff and the same is true for early stuff, so nothing there.
I think the reality is people kind of moving into having two kinds of attribution models that need to happen. One is multi-touch attribution models, which make a lot of sense. I think there are some issues regarding how you weigh each of the interactions, and I think there are some good reasons for that.
There are also time decay multi-touch attribution models, where you look at the activities but you wait the latest stage converting activity the heaviest. So I think there’s a lot of use in there.
The other one which I think is really good is cohort attribution models. Let’s say the item you’re looking at is when somebody signed up for your service, how many people maintained, who signed up in January, then how many people signed up in February, kind of going down the list, but across is how many of those people are retained.
You can put that kind of analysis in campaigns or all kinds of materials that you want to measure, but what it allows you to do – at least on a month-to-month – is to look at cost per closed lead. And cost per closed lead is huge: how much can you spend and still make money?
I think that’s kind of where attribution models are. Cohort, multi-touch attribution and time decay is what everyone has to be doing. They may not be doing it yet, but I think that’s fundamentally the state of the art.
Erin: One of the things that’s crazy with attribution modeling is that there are all these different things, everybody is doing them differently and everybody is calling them different things. First touch and last touch, these are actually two of my least favorite ways to do attribution modeling only because it’s so presumptuous in its nature of what actually caused something. Some people say it’ll eventually all equal out because some things will get last touch win other things were part of it, but then the next time, it’ll be the other thing. I think that’s wrong.
In Occam’s Razor, Avinash points out that what we first need to figure out is what somebody is talking about when they’re talking about attribution modeling, because often they actually don’t know what they’re referring to when they say attribution modeling.
I like the three categories that he breaks it out into, because then with each category, he goes into what types of models are best for each. Kind of a custom model is my favorite thing, but the first one is online to store. This is the attempt to try to understand the offline impact like brand value, butts in seats, phone calls and things like that, driven by online marketing and advertising directly to a store. This would be like online campaigns that actually get people to go to a Sprint or whatever store and actually sign up and activate a phone.
Then, there’s across multiple screens. This would be something like TV, desktop, tablet, smartphone, a lot of different things that you may see from different screens. That is obviously a really complex situation.
The last one would be across digital channels. This is stuff where we’re actually talking about social display, YouTube, referral, e-mail, search, all these different things.
All of these are actually different types of attribution modeling. Are we talking about online things to offline purchases, offline things to online purchases? Are we looking at different screens that you may see, and are we looking at different digital channels?
The first two are things that are probably not something relevant to address on the show necessarily and also are basically impossible. It’s almost impossible, especially on an extended basis, to figure out everything that somebody would have seen across anything. That would be assuming that you could Big Brotherly track somebody seeing a billboard or somebody else’s TV. If you go over to somebody’s house for the Super Bowl and you’re watching their TV, and you’re watching commercials on their TV, they don’t know the difference between you and the actual person you hung out with, so that’s a pain.
What we’re probably most looking at is across digital channels. How do you assign real credit to each channel? One of the things that I’ve noticed – and I’ve had conversations with actual brands – is that people are actually getting multiple credit. What they’ve adopted is a model where everybody who contributes along the conversion path gets credit, which means your total sum credit for a single customer or single cohort equals more than 100% because a lot of people are taking credit for a lot of percentages of conversions.
What are you seeing when people are looking to assign credit? Is there too much or not enough credit being given?
Steve: It depends. In a multi-touch attribution model, regardless of the number of touch points, you’re not necessarily pre-defined in terms of the percentage of the value that they have. The total percentage of the value is divided up amongst all the points. The downside to that is that you don’t necessarily know which ones have the greatest impact. You can see where maybe one item tends to have a better attribution to it.
There’s also a thing called position-based attribution model, which probably makes the most sense for folks. It does have a problem, but it’s basically both time decay and multi-touch attribution model. You might give the very first touch 40% and the last one 20%, and then divide the difference between the ones between.
The only problem is that you might be giving the very first touch way too much credit, maybe too high a credit, so there’s always going to be a fight about how much credit you give. I think split completely equally probably doesn’t make the most sense. Weighing front and back a little differently does make sense.
Erin: When we’re looking at people getting credit for what they’re doing on channels, I think one of the reasons that we’re all having such a problem – and we talked about this last week with account-based marketing – is that instead of working together to say, “We really care about what’s going on with assigning or wining accounts as a whole or creating these things as a whole,” what we’re all concerned with is, “Did I do a good job? Did my department get this, and can we have budget for next year to keep doing more of this? And how do I prove that?”
So I think that if we’re looking at it as more of an ecosystem, at which point everyone can’t survive without the other people, and even – let’s say that for each customer we do use this thing where it’s like 40%, 16%, 12%, 22%, whatever, and get all these numbers together, even if you said that, “Okay, this channel only contributed 12%,” if you take that 12% away, there’s not a conversion because you don’t make it to 100%. You can’t say that something is not important just because something is contributing 12%.
I think that’s where attribution modeling kind of hits some snags for me. It takes away the idea that even if something only contributes a little, you still wouldn’t have reached the goal line. You would have missed the goal by an inch. Whether you miss it by an inch or miss it by a mile, you miss.
Steve: I think one of the things that you kind of floated when you first started talking about that is that people have different agendas in terms of first touch, last touch, often because it’s a budget thing. Either credit for the business or they want to fight for budget issues, so they have a vested interest in looking at that data, and whilst it’s interesting to track and I would ask those questions, it’s probably not that data I would rely on in terms of attribution models, so I think it’s problematic.
John Miller, co-founder of Marketo actually had an interesting take on the future of marketing, and I really agree. Marketo is a marketing automation company, but what he talked about is that a greater portion of marketing people’s salary should be based on the business that’s closed, that they should assume a much greater risk-reward place.
The thinking is not that you don’t need salespeople, but if marketing truly helped nurture that and they are paid and rewarded on closed business or it’s a big piece of their thing – marketing has other things than just closed business that it has to do, but closed business is a big part of their risk/reward – that’s what marketing should be, that they’re focused with their eyes on the prize.
I’ve always thought marketing should be about closing business. The past has been kind of, “Well, there’s so much marketing to do.” “Yes, but we need closing business as one of our first priorities in that line of stuff.
Erin: I guess that depends on what the organizational structure is as to how complicated a situation like that is. If marketing was responsible for closed business, you would probably get everybody to be more on the same team, right? I guess it could lead to even more fighting, but you’d hope that what it would do is to really band everybody together and be like, “We all win. We all sink or swim as a team here.”
The thing that I think is complicated there is that it depends on whether you have something where people can do direct purchasing. Can you market to someone and then have somebody directly go in and purchase it? Like Coca-Cola did some marketing and advertising, I went into a 7/11 and bought a Coca-Cola, versus Oracle where I saw some Oracle stuff. I knew I needed it, I compared and contrasted it, but then I had to talk to a salesperson, I had to go through a process, I had to do all these things.
There are some places where marketing could probably take credit or figure out a way to be assigned specifically that. But then I feel like you’re going to end up with a similar battle between what some people would say is marketing and some people would say is advertising, although I would like to just say that it’s probably all one thing because now we have so much distinction between paid and organic. Do we really need to call those things different things?
Steve: I think it does make sense, only because somebody has to be totally responsible for that piece. So marketing qualified leads, leads that the marketing has decided are worthwhile to nurture, to try to bring along and incentivize to engage and do that kind of stuff, and sales gets to decide what is a sales qualified lead, a lead that they feel has a high likelihood of closing. The only time I there’s a real issue is if everyone doesn’t have the same conversation with the same metrics.
When does an MQL become an SQL, and when will sales openly accept it? And what will they do once they accept it? What will happen? And will it go back to the MQL pile if they don’t close it? Do you have that process? I think if you actually have all the conversations, everybody gets to agree. It has to be universal, everybody has to sit down and agree on the same metrics for what becomes an MQL to an SQL, and what happens afterward. I think you avoid a lot of that confusion about whose it is and what it is. Now it becomes, “We’re really clear about when this is yours and when this is mine, and what my responsibilities are.”
Erin: So then, is the advocacy for marketers to take ownership over not necessarily closed business but over MQLs generated? Because I think a lot of people are already kind of doing that. Over the years, I’ve listened to a number of sales organizations and marketing organizations fight, fuss and generally complain about each other. Sales always saying marketing didn’t drive enough marketing qualified leads, sales qualified leads, they didn’t do enough of their stuff, and marketing complaining that they were being held responsible for business closed and revenue, and that they handed off tons of leads and that sales just didn’t bother to close them.
Steve: I think that that happens, but it shows that there’s a fundamental problem. First of all, I think that marketing people should be held responsible for closed business, and I think that sales should be held responsible for doing a certain set of things on accepted SQLs.
The reality is that Brian Kardon, who is the former CMO for Eloqua, when he would hand out leads, if they didn’t call on them, he would take his leads and give them to one of the junior salespeople. Needless to say, the salespeople got really irritated. What was [15:50 inaudible] “If you’re not calling on them in the first 24 hours after I give them to you, you’re not going to move on them, and I’m giving it to this guy who is going to move on them.”
So I think that you have to have that kind of stuff, and if marketing is saying, “You’re giving us crap,” then they really haven’t had an MQL-SQL conversation, because nothing should be getting to them that isn’t ready to go, that everybody agrees on. So this getting over there, there’s a breaking of that process. I would say anybody who told me that, I’d say you either haven’t done or you haven’t done it seriously, or marketing came up with all of the specs and didn’t ask sales to be a major part of that decision.
Erin: And I think too often, that is the case. It’s this lack of time spent setting things up, digging in and figuring out what the real parameters are, what we’re really looking for, how we know, and whether these are the right things.
A lot of times, you’ll have things that marketing will do, which is one of marketers’ favorite things to do, which is like cohort analysis or targeted personas, all these different things where they’re like, “We have all these audience profiles.” Did you vet these with sales? Did you actually ask who these people are? Or even better, not just with sales, but with whoever is doing customer retention and customer success. What do successful customers in your organization look like? How are you going to go after them, and what do those people look like?
Because to me, when we talked about account-based marketing, this is one of the ways you determine what those targeted accounts even are. What does a successful account look like? How do you know the difference between people who tend to churn? Is it your failing, or is it that they’re just not a good match for your organization? Are there specific characteristics of companies that tend to do really well? Are you looking at things geographically? Are you looking for certain team makeup?
All these things are really important when you’re trying to determine what something looks like, but if you’re not talking to the people who actually end up having to close the deal and have these conversations, and the people who actually have to be on the phone, providing support after this customer is onboarded, then you’re missing such a huge chunk that there’s no way you can be doing it well.
Steve: Yes, I think the more marketing is tied to the sales and successes of sales, the better it’s always going to be. I think one of the huge things that comes down with content marketing and I think is true with the introduction of account-based marketing is that you see that marketing should be involved in sales enablement, which his providing the kinds of content and the tools that help sales succeed most.
Usually, salespeople know what that is. Marketing can bring a lot, not only create new materials that help accentuate that, but also make sure those materials are available to other people and share successes, and also to look at how we can continue bringing what we’re doing right now into onboarding.
So even after somebody has done sales enablement, somebody comes onboard, they’re thinking: are these the same automated marketing? Are there things dripping out, working with customer success people? Articles about best practices, maybe send something the first day, two days later, and then every week for six months, whatever. There are all these things that marketing should be doing and should be responsible for.
I’ve actually talked to a CMO just a few weeks ago who’s had a bunch of folks who were only incentivized on basically creating MQLs. There was actually a sales function, but they were to create MQLs which they can then turn into SQLs. If marketing actually came to them with a bunch of leads, leads that they had generated from other marketing things, said, “Here are all these leads you can follow up on,” and this team wouldn’t follow up on them because they were not incentivized on leads that they didn’t generate directly themselves.
Those kinds of breakdowns – and sales department had no interest in fixing that problem, which is ironic. They had to give away the leads to partners instead of giving them to their own salespeople to close. I think when you see these kind of problems, it really shows that there’s a breakdown. Management is not talking to each other, not being tied together by the same goals and metrics.
Erin: And it’s interesting that we see this so frequently. I’d like to say that what you just described is the first time I’ve ever heard anybody have that problem, but it’s not. It’s the ten millionth time I’ve heard someone have that problem, and it’s why we got into this conversation during ABM, which is talking about handing off the baton. The idea is that there is no handoff. We’re all kind of running with the baton together forever because it’s account-based marketing and you should technically always be involved with the account. You should continue to nurture these people and provide them with information, and you should be able to continue to upsell and prevent churn. That involves sales, marketing and continuing education.
The problem is that somehow, along the way – and this is a systemic problem in upper management, because that’s where it starts – is that what we’re going to do is pit departments against each other and we’re going to make them fight it out. This is the problem. At a granular level, when we’re talking about attribution modeling – because this is like org level attribution modeling, not even channel related – we’ve decided that channels and things should compete with each other as opposed to rewarding people for working together, which is, in my mind, the fundamental problem with attribution modeling, that you’re not doing the right thing with it.
The idea of attribution modeling is not to say, “Social did more than e-mail, and specifically Facebook did more than YouTube and Twitter, and e-mail gets this, and then now Facebook gets this.” That’s not the point of the attribution modeling idea. If that’s what you’re using it for, you’re being a jerk.
The idea of the attribution model is that you understand how things are working within the ecosystem, where it’s most likely that certain conversions are taking place for certain audiences, and then maximizing conversions based on the mix of those things and the types of target audiences you have, like leveraging messaging strategies and channels together across things.
Same thing within your organization. Within your organization, the idea is not that “Marketing did this, sales did this, customer success did this,” and then based on those things, somebody gets more money than somebody else. The idea is, “At what point does everybody play a certain role? How do we maximize each of those roles to close as many customers as possible, and keep them here as long as we can?” Not to assign blame and things like that.
I think that’s where it really came in. You have boards, you have all different kinds of people, and at the end of the day, people don’t make numbers or something doesn’t go right, and the blame game happens. That’s how we got to this dysfunctional system.
Steve: Yes. For all those reasons, I think Eloqua were hesitant to put a stake in the sand. They’re afraid that they’re going to be judged and found wanting. I sure get that. I totally understand that, especially since historically, when marketing was a dark art, it’s really hard to say, “Well, how many leads is PR going to close for me?”
Now we can actually put a stake in the ground, and even just measuring against ourselves – I think by and large, benchmarks are complete BS.
Erin: Thank you. I was about to say that.
Steve: But you can benchmark against yourself and you can say, “Here’s what we’re doing. We want to try to turn these dials.” There are other considerations sometimes, but once you know where all your leads are coming from and you’ve mapped that out – and it’s shocking how people don’t have that already dialed in. It should be your number one priority: where are the leads coming from, what are they costing, what are they doing, what’s going to happen to them if they don’t close?
When you’re doing that, you kind of start saying, “Where’s the biggest thing, and can I turn that dial up?” That’s what you should be focused on, turning the biggest dials up, and then looking to figure out which of those channels are working best and working together.
Erin: So true. People love benchmarking, across all facets of life. People want to benchmark one relationship against another, they want to benchmark their weight, their salary, everything. They don’t even want to just compare it to themselves, they want to compare it against everybody else they know. It’s kind of a dangerous game to play, and I think maybe benchmarking and some of the weird things with that would be a good conversation for the future.
I do know that we’re out of time for this week, but we will catch you guys again next week. It is Halloween, so happy Halloween! I hope that everybody has a good day. Steve, thank you for joining, and I will see you next week.
Steve: Always a pleasure.