By Kieran Luke
Modern marketers do all the same things traditional marketers have historically done to promote brands and drive sales, like conducting customer research, choosing target segments, positioning products, and designing marketing campaigns.
However, the digital era introduces a new set of constantly evolving channels and provides much more information about campaign performance than traditional advertising channels like television and print. To keep up, marketers need to develop problem-solving skills.
In our entry-level digital marketing skills assessment, Digital Marketing Level 1 (DM1), we represented these skills in three question categories:
1. Conceptualize: Understand the terminology, tools, and strategies in digital marketing.
2. Calculate: Use data to measure how campaigns are performing.
3. Interpret: Draw conclusions from data to optimize campaigns.
DM1 tackles each of these problem types, which means we can show you average scores across each type. In our exclusive report The State of Skills: Digital Marketing 2018, we analyzed a sample of 10,000 professionals across major Fortune 500 companies, growth-stage startups, and everything in between, who took the DM1 assessment between May 2016 and September 2017.
Among our top takeaways was that the skills gap in digital marketing is driven by missing data skills. Here’s how our test-takers did:
Average Score by Problem Type
The Calculate questions were clearly the most difficult, with test-takers only getting about two out of the six problems correct, on average.
We thought that perhaps four of the six questions were really hard or confusing, so we took a look at the success rate of each question in the Calculate section.
Success Rates for Calculate Questions on DM1
Our finding? It wasn’t a few really hard questions bringing the average down. Questions 4 and 6 both required the test-taker to complete more than one calculation, which could classify them as harder. However, people didn’t do well on the other questions, either. Question 1, which asked test-takers to calculate a single conversion rate, was the easiest, but the success rate of 45% is on par with the average success rate for Conceptualize and Interpret questions.
In short, test-takers struggled with the Calculate questions, revealing a shortcoming in processing data.
To ensure the Calculate questions were “good” and didn’t yield the low scores due to poor design, we did two things. First, we rechecked whether the questions were clearly written and unambiguous. Second, we looked at how top scorers performed compared to other test-takers on each question. Well-crafted questions help top scorers differentiate themselves from the rest of the pack, and we can measure this by looking at the relative success of test-takers in each quartile.
So let’s go ahead and look at the relative success of each quartile on the Calculate questions.
Relative Success on Calculate Questions by Quartile
In this chart, the more teal, the better the question was at differentiating Quartile 1, the top 25% of scorers, from the rest of the data set. We can see that every Calculate question helps quartiles 1 and 2 differentiate themselves as stronger performers than quartiles 3 and 4. In fact, questions 4, 5, and 6 were the top three most differentiating questions of the entire assessment. In each of these, Quartile 1 accounted for at least 50% of the right answers. Bottom line: These are “good” questions.
So, what are people actually getting wrong? Let’s dig into the easiest Calculate question (Question 1, with a 45% success rate) to find out. This question is about click-through rates, and doesn’t give the test-taker any multiple-choice options to plug in and try. Since click-through rates are a key metric in display advertising, savvy marketers should have familiarity with calculating them.
To solve the problem, test-takers have to understand that the click-through rate is the number of clicks (250) divided by the number of impressions (25,000), which yields an answer of 1.00%. We also accepted the answer as a decimal (0.01), in European notation (1,00%), and with fewer decimal places (1%, 1.0%). Not difficult! But getting it wrong means a shortcoming in math or understanding metrics, both of which impede ability to manage a successful digital marketing campaign.
Here are the most common ways test-takers got this question wrong:
1. They calculated the wrong rate, confusing the methodology for determining click-through rate with other formulas. Examples of alternate answers provided include:
- Cost per impression ($1,000 ÷ 25,000 = 0.04)
- Cost per click ($1,000 ÷ 250 = 4)
- Clicks per dollar (250 ÷ $1,000 = 0.25)
2. They got the order of magnitude wrong when expressing percentage, e.g., 10% versus 1% versus 0.1%.
3. They expressed the answer as a decimal (0.01, which would be correct) but then added a percent symbol (0.01%, which is incorrect).
We saw these patterns across the other Calculate questions, too.
The most common mistake individuals made was not understanding how metrics relate to the various steps in a marketing funnel. For example, in email marketing, the funnel includes the following milestones: email sent, email received, email opened, click from email, conversion to lead, and, ultimately, conversion to sale. Misinterpreting the meaning of metrics at any of these key points can lead to misunderstanding how a campaign is performing down the line, culminating in the potential to make poor decisions with marketing budget, next steps, and beyond.