Ray worked with B-2-B and Consumer clients throughout the world ... including USA, Canada, Mexico, Asia, the South Pacific, Europe, the Middle-East, Central & South America, Africa.

This website is a compilation of Ray's 10 years on the Web.


Power Direct Marketing: The Book

How Big Is a Test?

Let’s have a little dialogue about how big a test should be.

How large is large—how small is small? And what difference does it make?

Vin Jenkins, principal in his own direct response agency in Australia, has gone to great lengths to demonstrate that an honest and valid test of lists, an offer, a package—anything with direct mail—can be done with fewer than 2,000 units. If you do all the right things right and have the sophisticated math models to measure same, I’m sure Vin is right. It can work.

With another view is Al Mercer of Canada. Al talks about a 10% rule. A rule of thumb which says that you always test 10% of a list to predict how it is most likely to perform in a rollout. No matter the size of the list universe, you always test 10%.

And then there is me. I more or less disagree with both these fine, experienced direct marketing pros. Why? Because I sold lists for a number of years and have bought hundreds of lists for scores of clients since.

I’ve been involved with direct response programs that used mail, space, telephone, and allied collateral and support materials. My comfort level is different because of my experience. It is different from both Vin and Al for testing of lists, for testing offers of any type, for testing creative—for testing anything.

Why do list owners, managers, and brokers have minimum order numbers for lists? One reason is the volume required for computers to be cost efficient. But a more important reason is that unless you put enough numbers into the marketplace, you will not have a measurable response to analyze.

It is not uncommon for lists to be available in a minimum 5,000 or 10,000 quantity. The list owner/broker wants the list to work just as much as you do. Without a sufficient test you may never be sure. Even if you hit it lucky and get a high response off a small number mailed, there is no assurance you’ll experience the same results with the rollout.

Personally, I’m not comfortable with less than 2,500 names in any one test cell. Nothing less than 2,500 in an offer test—a test of price, a premium, a limited time offer, anything. And I really prefer 5,000 names on all list tests. As a minimum.

The key in testing, and then being able to measure and analyze what happens, is not how many prospects hear, see, or read about you—it is how many respond! Response is the key, the trigger. Not the number out the door, but the number that come back.

When you test direct mail packages, layout and design, different copy approaches, various package sizes, offers of any and all types, what you are seeking is a winning package. And, as the title of this chapter suggests, "winning isn’t everything, but who cares about all that other stuff!"

Television and space are a little different because your audience reach is different. Here you’re probably talking to thousands, maybe millions, with a single effort. And even though the media of print and broadcast are certainly different from direct mail and each other, you will get a "feel" of the success quickly. Maybe with one insertion of the ad or a short flight of broadcast you’ll learn sufficient from your analysis to decide direction.

What’s best for you? Beats me! Broken record, but "I don’t know" and "it depends" are still the correct answers. What you test and the size of the test, even if you test, very likely will be influenced, if not decided, by the size of your marketplace.

If you’re a consumer marketer, you probably have tens of thousands to millions of prospects. If you’re business-to-business, you may have only a handful. You decide what’s best for you. You decide.

Top of This PageReturn to Previous Page

Contents by ROCKINGHAM*JUTKINS*marketing, all rights reserved.
Design by William F. Blinn Web Design, all rights reserved.