Testing and analysis for maximum benefit

OTOi, Marketing Service Team | One To One Interactive
February 13, 2002

Overview

In any well-planned marketing process, testing and analysis of marketing programs are key elements to ensure marketing success. Sometimes new marketing programs are never actually "tested", but executed in rollout (large-scale) quantities instead of smaller, more manageable test quantities. When untested new programs are immediately rolled out with large quantities and budgets, the company's risk is substantially increased. In other cases where small tests are planned, the testing component may be short-changed due to the rush to make an immediate impact or confusion about what issues a given test design can and cannot address. When this occurs, the back-end analysis is often limited and, therefore, the expected benefits — new learnings and customer insights — do not materialize. Thus, in order for the marketer to get the most from testing initiatives, it is important that he or she becomes familiar with the important considerations that must be made when developing a testing strategy. The objective of this article is to discuss these important considerations, as well as to help the marketer determine how best to incorporate testing strategy into marketing campaigns.

Why to Test

Testing allows the marketer to determine which marketing programs, creative, offers, etc. are optimal prior to incurring the expense of a full-scale rollout. This translates into increased cost-effectiveness of the campaign, including higher profit or ROI for the program once it is sent.

Goals of Testing

Specifically, testing should involve the following goals:

• Hypothesis testing:
  For example, are response rates for two marketing programs equal? Is the expected order size for one program greater than for another? Is the test program with the highest retention rate, in fact, superior (with respect to retention rate) to the other programs, or are the differences merely due to sampling error?
• Substantive understanding about relationships between customer characteristics and marketing program results.
  In other words, how do different customer characteristics affect response behaviors to a given marketing program?

When to Test

Typically testing is most appropriate in the following situations:

• When new creative is developed
• When looking to introduce a new product or product feature
• When looking to fine-tune various elements of a successful mailing to achieve even better results
• When the marketing mix changes (i.e., price, offer)
• When the cost per order is not what was anticipated
• When the goal is to expand the market to a wider list
• In order to test validity of segmentation

What to Test

What to test depends on various factors including budget, mailing/sample size, marketing objectives and willingness to assume risk. Typical elements included in testing initiatives often include offer, target, creative, logistics/contact strategy, message, medium and, in the case of e-mail, subject line.

The best factors to test are those elements that will have the greatest impact on response rate. Typically, the greatest difference in results can be expected from changes in the product or product positioning, changes in offers or the selection of different lists, publications or sites. In addition, when testing e-mail campaigns, the subject line is an easily tested item that costs very little to alter and can have a substantial impact on open rates.

How to Reduce the Number of Test Cells

It is often the case that certain marketing program combinations may have little to no appeal to any customer group. It therefore makes eminent sense to weed these combinations out prior to live market testing. If only five of fifty-four product combinations merit serious consideration, then only these five need to be tested.

Primary research methods can be deployed to assess which combinations merit live testing. All such methods ask individuals to rank order the combinations. The popularity of each combination is analyzed and determined and only combinations with a high percent of top choices are selected for field-testing. Alternatively, a more formal approach, based on conjoint analysis, may be used to identify combinations with the greatest appeal for different prospect groups. These most preferred combinations should then be selected for field-testing.

The utilization of a partial factorial test design is another way to simplify the execution of the test, but not the back-end of the analysis that follows. In this case, the test design becomes an "experimental design", so called because multiple factors are varied simultaneously, thereby increasing complexity. For example, if a marketer has 54 possible combinations to be tested, in the case of partial factorial design, a smaller number of combinations are tested and the results are generalized via a statistical technique that predicts the back-end performance of the untested cells.

Setting Sample Size

Determining proper sample sizes for testing and back-end analysis depends on several factors. In order to properly estimate sample size requirements, the following key factors need to be provided:

• Expected response rate:
  This can be based on learnings obtained from previous campaigns, or in the case of no prior knowledge, can be an educated guess by the marketer. Generally, very small response rates require larger samples.
• Desired confidence level:
  Typically, companies want to achieve at least 85%-90% confidence that any differences observed in test cell performance are "real," or would occur again with another population under the same competitive and seasonal circumstances. Higher confidence levels require larger sample sizes.
• Desired sensitivity or level of tolerance:
  In other words, the difference we want to detect between response rates. If we want to detect very small performance difference (i.e., the difference between a response rate of 1% and 1.5%, larger sample sizes are required.

Benefits of Testing

Whether a simple or more complex test design is employed, a standard process where all new marketing programs are routinely and randomly tested in smaller quantities before being rolled out en masse accomplishes three objectives:

• Limits potential losses
• Allows identification of the idea consumer profiles (targeting) for optimal matching of marketing treatments and customers in future program rollouts
• Facilitates better demand forecasting in future program executions

Most companies find that these benefits of testing more than pay for the relatively small investment required to design and execute tests. Companies that incorporate a regular testing program give themselves an advantage in ensuring an efficient and profitable marketing process.

back to White Papers