Austin E W., Pinkleton B.E. Strategic Public Relations Management. Planning and Managing Effective Communication Programs

Подождите немного. Документ загружается.

214 CHAPTER 10

for a smaller number of systems, they likely will experience delays in data

collection and even higher maintenance costs.

In addition, CAPI requires participants to have at least a minimal level

of computer literacy so that they can complete a questionnaire accurately,

which raises concerns about the ability of researchers to use a representative

sample. Even though most people now use computers on a regular basis,

CAPI would potentially present researchers with difﬁcult challenges if they

were attempting to complete a general population survey.

As a ﬁnal note, some researchers have raised concerns that CAPI is

needlessly slow. Although this is not always the case, there is a time–cost

tradeoff for researchers to consider. Using more computers reduces the

time it takes researchers to collect data, but it costs more money. CAPI

has clear beneﬁts in speciﬁc interview situations and some researchers

are enthusiastic about its potential, but its limitations have hindered its

development as a primary form of personal interviewing to date.

Mall Intercept Surveys

Mall intercept surveys, or simply mall intercepts, are a common method

of collecting data in a personal interview format. Well-dressed interview-

ers, typically wearing matching-colored blazers and carrying clipboards,

are a common sight in most malls today. Researchers also use different

locations for intercept studies, including downtown areas, college cam-

puses, and other areas that attract large numbers of people. When orga-

nizations sponsor mall intercepts, they position interviewers at strategic

locations throughout a mall and orally administer a survey to shoppers.

Interviewers may ask shoppers to ﬁll out a survey in a manner similar to

group-administered surveys. They also may provide an inducement for

participation.

Mall intercepts offer several beneﬁts that have made them relatively

popular, particularly among market researchers. Perhaps most important,

researchers can complete a study relatively quickly if necessary, and mall

intercepts are inexpensive relative to other research methods. Combined,

these two beneﬁts make mall intercepts an attractive research method given

the time and budgetary constraints that hinder many research projects.

In addition, the method is relatively uncomplicated and provides some

ﬂexibility in data collection.

Mall intercepts have some key limitations that researchers need to con-

sider before undertaking a project, however. Perhaps most important is

that although a probability-based sampling method is possible in an in-

tercept study, it is difﬁcult as a practical matter. Even if researchers used a

probability-based sampling method, the results would generalize to shop-

pers at a single location. For these reasons, researchers normally use non-

probability sampling methods that, although convenient, limit the external

validity and reliability of a study’s ﬁndings. In addition, many shoppers

SURVEY RESEARCH 215

avoid mall interviews, raising further concerns about the validity and re-

liability of study results. Interviews typically suffer from poor interview

conditions because of noise, foot trafﬁc, and other distractions, and re-

searchers must keep questionnaires short and to the point. Although inter-

cept studies offer beneﬁts in certain circumstances and are popular among

research practitioners, researchers must use them carefully and remember

that their results suffer from limited reliability and generalizability.

Personal Interview Critique

Researchers use personal interview surveys for several reasons, many

of which concern data quality. Historically, researchers conducting face-

to-face surveys have achieved high response rates, which are enhanced

through prenotiﬁcation letters and the ability of interviewers to interact

with sample members. In addition, interviewers usually can establish a

strong rapport with survey participants. This provides them with an op-

portunity to probe respondents’ answers when they are inadequate or

unclear. Personal interviews may take place in sample members’ homes,

which allows them to consult personal records or locate other information.

The home also is a comfortable interview environment and tends to make

respondents less sensitive to questionnaire length. In addition, sampling

frame bias is low when researchers use Census Bureau data as a basis for

the sampling frame because all individuals in a population have a chance,

at least in theory, of being included in the sample. As a result of these

characteristics, researchers can use personal interview surveys to produce

high-quality data.

Personal interview surveys also offer interviewers a high degree of

ﬂexibility in obtaining research information. One-on-one interviews lend

themselves to questions requiring high degrees of depth or detail, and the

interviewer can use visual aids or other interview devices as necessary.

Interviewers also can explain questionnaire items that are confusing to re-

spondents, particularly if the respondent misunderstands the intent of a

question or is confused by a response category. Researchers can even esti-

mate some information, although they should do this sparingly because of

the potential to introduce error into survey results. The ﬂexibility of one-

on-one interviews also allows interviewers to use complex questionnaires

with difﬁcult skip patterns. The ﬂexibility and quality of data produced

through personal interview surveys have historically made this research

method highly desirable to researchers.

Unfortunately, personal interview surveys suffer from signiﬁcant draw-

backs that have limited researchers’ use of this survey method. Probably

the greatest drawback to personal interview surveys is cost. Personal inter-

views generally are the most expensive survey research method. Although

cost comparisons are not always available, Czaja and Blair (1996) estimated

that a national personal interview survey would cost more than twice as

216 CHAPTER 10

much as a similar telephone interview survey. These costs result from the

need for ﬁeld supervisors and extensive interviewer training, as well as

travel expenses and other costs. In addition, personal interviews generally

take longer than telephone interviewing because of logistical complexities.

The administration of a personal interview survey is costly and complex

when sample members are geographically dispersed. These are signiﬁcant

limitations given the importance of time and cost factors in many research

decisions.

Another signiﬁcant disadvantage in the use of personal interviews is the

possibility of interviewer bias. The physical appearance, age, dress, race,

sex, and verbal and nonverbal communication skills of the interviewer may

inﬂuence respondents to provide answers that do not accurately reﬂect

their feelings. In addition, participants may be hesitant to report highly

personal behavior. The resulting bias ruins the reliability and generaliz-

ability of research results. A high degree of careful interviewer training

and ﬁeld supervision is required to avoid interviewer bias, which also

contributes to survey cost and length. Finally, some samples are difﬁcult,

or even dangerous, to access for a personal interview. Busy work schedules

make it difﬁcult to schedule interviews with many adults. In other cases,

sample members live in areas with high crime rates. Each of these sample

members is important as potential respondents and researchers must in-

terview them if possible, but reaching them presents a special challenge

when conducting personal interview surveys.

FINAL THOUGHTS

Survey research is an indispensable part of organizations’ attempts to mon-

itor their internal and external environments, solve complex problems, and

plan and evaluate communication campaigns. Despite what some have

suggested, there is no best method of survey research, and the potential

advantages and disadvantages of each method do not apply equally, or

even at all, to every research situation. The best method for a project de-

pends on various situation-speciﬁc factors, and because of this, project

managers must consider the use of a survey research method in relation to

the needs and constraints of each situation.

It is important for practitioners to understand each survey research

method because of this. Practitioners serve their own interest by under-

standing the key issues associated with the use of one survey method over

another in a research situation, as well as other critical aspects of survey

implementation such as issues associated with the use of probability ver-

sus nonprobability sampling. This knowledge allows practitioners to make

informed research decisions and engage in sophisticated problem-solving

and campaign-management activities.

Making Research Decisions:

Questionnaire Design

Chapter Contents

Understanding Reliability and Validity

Levels of Measurement and Why They Matter

Types of Questions and the Information Each Type Provides

Ensuring Clarity and Avoiding Bias

Questionnaire Layout and Design

Handling “Don’t Know Responses

Design Features That Affect Response Rate

Final Thoughts

Writing a questionnaire seems simple. As the famous survey researcher

G. W. Allport once said, “If we want to know how people feel: what they

experience and what they remember, what their emotions and motives are

like, and the reasons for acting as they do—why not ask them?” (Selltiz,

Jahoda, Deutsch, & Cook, 1959, p. 236).

Unfortunately, just asking them is not as easy as it sounds. To avoid

obtaining misleading results, questions must be clear, must elicit honest

and reliable answers, and must keep the respondentinterested in providing

answers. The construction of questions must differ according to whether

they are being read or heard, and the researchers can ask only as many

questions as respondents have the time and energy to answer. It is easy

to write bad questions and difﬁcult to write good ones. Guidelines for

questionnairedesign typically focus on the importance of clarity, simplicity,

217

218 CHAPTER 11

and objectivity. Other important considerations include making questions

interesting and the questionnaire logical so that respondents feel motivated

to answer carefully.

Given the myriad of details that can make or break a questionnaire,

the best questionnaires often turn out to be those you would have written

once it is too late and you have the answers to the one you already used.

To avoid giving postsurvey regrets, questionnaires need to be pretested

with attention to every detail and with members of the intended sample

of respondents. Important issues to consider when writing questionnaires

include the following:

Validity and reliability concerns as they relate to the sample, the topic,

and the client

Levels of measurement and why they matter

Ways to ensure clarity and avoid bias

Types of questions and how the information each type provides differs

Questionnaire layout and design to ensure logical ﬂow and visual

clarity

UNDERSTANDING RELIABILITY AND VALIDITY

What is a “fast” city? If you are the communication manager for the lo-

cal chamber of commerce, would you want your city rated as “fast” or

as “slow”? National and regional rankings of cities, universities, corpora-

tions, and other organizations are published all the time. Depending on

your interpretation, “fast” could mean exciting or it could mean stress-

ful. Meanwhile, “slow” could mean boring or it could mean mellow and

comfortable. When Psychology Today published its feature on fast cities, the

things that made a city fast included the length of time it took to be waited

on in a bank, the speed at which people walked down the street, the num-

ber of people who wore watches, the speed with which people talked, and

the rates of coronary heart disease in the city population. A ranking of 1 on

this list probably would not make city leaders happy. But how can ratings

based on objective data be refuted effectively (Sidebar 11.1)?

The answer is that measures need to be valid and reliable. A valid measure

is one that seems to representa particular idea in a convincing way. If people

generally can agree that the things used to measure something such as a fast

city are appropriate, the measures are considered valid. A reliable measure

has consistency. If virtually anyone can replicate the study using the same

measures and come out with similar answers, the measures are considered

reliable. The fast city measures could be attacked as invalid by arguing

that coronary heart disease has no relationship to bank teller speed, except

perhaps for the bank teller.

SIDEBAR 11.1

Measuring a City’s Pace

To see if there is any relationship between a city’s characteristic pace and its

rate of coronary heart disease, we looked at four indicators.

Walking speed: We clocked how long it took pedestrians to move 60 feet along

relatively uncrowded streets. To eliminate the effects of socializing, we timed

only people walking alone. We also excluded children, pedestrians with large

packages or obvious physical handicaps, and window shoppers.

Working speed: We timed how long bank clerks took either to give change, in

set denominations, for two $20 bills or to give us two $20 in return for change.

Talking speed: In each city we tape-recorded how long it took postal clerks to

explain the difference between regular mail, certiﬁed mail, and insured mail.

We then calculated their actual “articulation” rates by dividing the number of

syllables in the response by the total time it took.

The watch factor: As a simple measure of concern with clock time, we counted

the percentage of men and women who were wearing wrist watches.

Individually, each of these measures has its weaknesses: They all tap into spe-

cial groups, not the city’s general population; the second two are confounded

by skill and efﬁciency; and the last is affected by fashion as well as concern

with time.

Finally, we created an index of the overall pace of life in each city by giving

the four scores equal weight and adding them together. The chart below shows

how the cities ranked, from 1st to 36th, in each category.

FAST CITIES, SLOW CITIES, HOW THEY RANK

Overall Walking Bank Talking Watches

City Pace Speed Speed Speed Worn CHD*

Boston, MA 1 2 6 6 2 10

Buffalo, NY 2 5 7 15 4 2

New York, NY 3 5 7 15 1 1

Salt Lake City, UT 4 4 16 12 11 31

Columbus, OH 5 22 17 1 19 26

Worcester, MA 6 9 22 6 6 4

Providence, RI 7 7 9 9 19 3

Springﬁeld, MA 8 1 15 20 22 7

Rochester, NY 9 20 2 26 7 14

Kansas City, MO 10 6 3 15 32 21

St. Louis, MO 11 15 20 9 15 8

Houston, TX 12 10 8 21 19 36

Paterson, NJ 13 17 4 11 31 1

Bakersﬁeld, CA 14 28 13 5 17 20

Atlanta, GA 15 3 27 2 36 33

(Continues)

219

220 CHAPTER 11

SIDEBAR 11.1 (Continued)

Overall Walking Bank Talking Watches

City Pace Speed Speed Speed Worn CHD*

Detroit, MI 16 21 12 34 2 11

Youngstown, OH 17 13 18 3 30 6

Indianapolis, IN 18 18 23 8 24 22

Chicago, IL 19 12 31 3 27 13

Philadelphia, PA 20 30 5 22 11 16

Louisville, KY 21 16 21 29 15 18

Canton, OH 22 23 14 26 15 9

Knoxville, TN 23 25 24 30 11 17

San Francisco, CA 24 19 35 26 5 27

Chattanooga, TN 25 35 1 32 24 12

Dallas, TX 26 26 28 15 28 32

Oxnard,CA27303023734

Nashville, TN 28 8 26 24 33 14

San Diego, CA 29 27 34 18 9 24

East Lansing, MI 30 14 33 12 34 29

Fresno, CA 31 36 25 17 19 25

Memphis, TN 32 34 10 19 34 30

San Jose, CA 33 29 29 30 22 35

Shreveport, LA 34 32 19 33 28 19

Sacramento, CA 35 33 32 36 26 23

Los Angeles, CA 36 24 36 35 13 28

*Lower numbers indicate faster speeds, more watches worn, higher CHD rates. CHD

indicates rates of coronary heart disease, adjusted for the median age in each city.

From “Type A Cities and Your Heart,” by R. Levine, 1989, Psychology Today. Copyright

1989 by Sussex Publishers, Inc. Reprinted with permission.

For measures to be valid, the concept or idea they represent must be clear

and the operationalizations, the actual measures themselves, must seem ap-

propriate. To a great extent, determining validity is an exercise in persua-

sion. Reliability is easier to verify objectively.

Threats to validity and reliability sometimes are subtle, making it impor-

tant to think carefully about the context in which surveys will be answered

and interpreted. The measure itself must seem appropriate, and the scale

used to measure it must seem appropriate. For example, operationalizing

a concept such as a “livable” city might seem fairly straightforward. U.S.

News and World Report publishes a feature on this issue every year focus-

ing on characteristics such as crime rates, housing costs, and employment

rates. Most people would agree that a more livable city would feature a

lower crime rate, lower housing costs, and higher employment rates. But

wait: rising housing costs make a city more attractive, not less attractive,

to those investing in real estate. So a high score on housing costs could

QUESTIONNAIRE DESIGN 221

mean less livable to some people and more livable to others. Meanwhile,

some might argue that the most appropriate measure of livability would be

people’s perceptions of safety and satisfaction, rather than more objective

measures of crime rates, numbers of parks, and so on.

Moreover, the measures chosen to operationalize livability might be

criticized as insufﬁcient to measure the concept appropriately. Perhaps

factors such as average commute times, numbers of cultural events, and the

quality of public schools (as indicated by standardized tests? Availability

of special services?) also need to be included to provide a valid measure.

The three-quarters of the population who do not have school-age children,

however, may not consider public school quality a primary determinant

of a city’s livability. In addition, some people might consider other factors

such as nearby access to parkland as critical to quality of life, whereas

others might consider a booming nightlife a higher priority. And what

about weather? Is a warm temperature an advantage or a disadvantage? It

depends on whether you prefer to snow ski or water ski. Thus, to measure

a fairly simple idea such as a livable city, the measures chosen must be the

following:

Sufﬁcient in number to represent enough about the concept

Appropriate as indicators of the concept

Unambiguous, so that a high score on a measure clearly represents a

high level of the concept

Reliability, too, is an important characteristic for measures, with two

primary components. First, the indicator of a concept must be replicable,

that is, reusable with a similar result. The second component of reliability

is how consistently the various operationalizations of a concept measure

it. The operationalizations are the ways the researcher measures an idea,

such as by counting the number of adults employed in the community

during a single year. Consistency of the measures is important because ob-

servers tend to ﬁnd a group of measures more convincing than any single

measure. For example, if a city that scores highly on employment rates

and cultural events also scores highly on individuals’ reports of perceived

safety and happiness with the town, these measures as a group represent-

ing “livability” can be called reliable. Some people may think employment

rates are more important than personal reports of happiness, others may

consider personal reports more important, but they all may accept the oth-

ers’ measures if convinced that the measures “hang together” consistently.

This type of reliability can be measured statistically.

Rankings such as “most livable cities” can have major repercussions

for organizations that score well or poorly or just better or worse than

expected. According to Monks and Ehrenberg (1999), educational insti-

tutions that receive a less favorable rating from the U.S. News and World

222 CHAPTER 11

Report annual rankings of colleges and universities end up with a lower-

quality pool of applicants and have to use bigger ﬁnancial incentives to

attract desirable students. As shown in Table 11.1, rankings of universi-

ties can vary dramatically depending on the methods used to develop the

ranking scores. As Gater (2002) has asserted, it is nearly impossible to iden-

tify, quantify, and measure characteristics that fairly compare colleges and

universities because they have wide-ranging sizes, scopes, missions, and

disciplinary emphases. Nevertheless, many parents of potential students,

students themselves, faculty considering potential employers, and donors

considering potential beneﬁciaries look for rankings to help them in their

decision making.

As Table 11.1 shows, rankings change depending on the types of insti-

tutions included in the analysis, how the organizations collect data about

“quality” or “value,” and when they collect the data. Some raters, for ex-

ample, analyze institutional-level data such as admission and graduation

rates. Some include surveys of university administrators. Some include

surveys of alumni or current students. In addition, some surveys rely on

volunteer samples rather than representative samples. Table 11.1 shows

that even the same organizations rank universities differently depending

on whether they focus on a particular issue (e.g., academic excellence or

cost) or include a range of issues important to most applicants (e.g., aca-

demic excellence and cost) in the ranking criteria. Meanwhile, other orga-

nizations may focus on criteria important only to some, such as political

activism. In addition, rankings can change depending on whether they fo-

cus exclusively on schools that meet criteria for “national” universities or

mix public and private colleges along with public and private universities.

Communication managers at educational institutions must understand va-

lidity and reliability issues to deal effectively with these rankings, which

frequently receive a great deal of news coverage and attention from impor-

tant constituents. Universities that wish to publicize rankings that make

them look good while de-emphasizing those that portray them less pos-

itively have to defend the validity and reliability of one survey credibly

while attacking the validity and reliability of another.

LEVELS OF MEASUREMENT AND WHY THEY MATTER

Survey questions fall into four general levels, shown in Table 11.2, that

dictate how the questions can and should be analyzed. It is important to

know your level of measurement because the level dictates the types of

statistical tests you can perform on the data. Usually, clients hope to deter-

mine how certain beliefs or behaviors relate to other beliefs and behaviors.

If measures have been collected at a low level of measurement, this will

seriously limit the analyst’s ability to investigate relationships of interest.

TABLE 11.1

Rankings of Universities Based on Varying Criteria From Different Organizations

U.S. News &

Peterson’s Mother Jones Princeton Review Princeton Review World Report

Kiplinger Peterson’s “Academics “Activist “Most Politically “Academic “Top National

“Best Value” (2003) “Solid Value” (2004) Plus” (2004) Campuses” (2003) Active” (2004) Experience” (2004) Universities” (2004)

1. UNC Chapel Hill 1. Barry University 1. Clemson 1. U of Puerto Rico, 1. Claremont McKenna 1. U of Chicago 1. Harvard U

Mayaguez College

2. U of Virginia 2. Bryan College 2. William & Mary 2. UCLA 2. US Naval Academy 2. Marlboro College 2. Princeton U

3. College of William 3. Burlington 3. SUNY Maritime 3. Spelman College 3. Hampden-Sydney 3. Reed College 3. Yale U

& Mary College College College

4. U of Georgia 4. Carson-Newman 4. US Air Force 4. California Commu- 4. Sarah Lawrence 4. St John’s 4. U of Pennsylvania

College Academy nity Colleges College College (MD)

5. U of Florida 5. Centenary College 5. US Military 5. Rutgers U 5. Georgetown U 5. Swarthmore 5. Duke U

of LA Academy College

6. New College of Florida 6. Eastern 6. UC Berkeley 6. Suffolk U 6. US Air Force 6. Haverford College 6. Massachusetts Institute

Mennonite U Academy of Technology

7. Georgia Institute of 7. Freed-Hardeman U 7. UC Irvine 7. U of Massachusetts 7. Eugene Lang College 7. St. John’s College 7. Stanford U

Technology (NM)

8. U of Illinois at 8. Lenoir-Rhyne 8. UC San Diego 8. Mount St. Mary’s U 8. Hampshire College 8. Harvey Mudd 8. California Institute

Urbana, Champaign College College of Technology

9. Truman State 9. Rollins College 9. UC Santa 9. Arizona State U 9. The George Wash- 9. Sarah Lawrence 9. Columbia U

University Barbara ington U College

10. Virginia Polytechnic 10. Shorter College 10. U of Mary 10. UC Berkeley 10. Wesleyan College 10. Carleton College 10. Dartmouth College

Institute and State U Washington

Sources

Peterson’s database Alumni feedback Alumni feedback Editors’ analysis of Convenience- and Combination of Perceptions of

and institutional and institutional news reports volunteer-sample convenience- and administrators

data data surveys of students volunteer-sample (61% response rate) and

surveys of students, institutional data

counselors, parents,

and educators

staff opinions

223