Assignment Instructions
Research in the workplace solves a problem. Tasked with a problem in the workplace, you
may be asked gather the information necessary to fully understand the issue at hand, solve
that problem (or offer potential solutions), prove that your solutions are viable, and/or test
your solution(s). Doing this work requires different types of research that go beyond simply
querying a library database or using Google. You often will need to speak directly to target
populations and audiences, and directly contact resources and experts in different professions
and in the community. You also may need information in addition to or instead of scholarly
resources. Local and national journalism may add context and perspective. Professional
experts, government agencies, state and local authorities all may be relevant sources, as are
individuals in target populations. Essentially, research in the workplace requires you to think
critically and creatively about

The type of information you need; and
The best way to get that information.
Your job as a researcher is to address, explain, and/or solve a problem using the most relevant
and applicable methods and resources. If a resource can supply information you need, then it
is the right resource for the job.
It’s also important when thinking about a problem your researching to keep in mind that you
probably aren’t the first person or organization to deal with this issue. Look at other
organizations, groups, or communities negotiating the same or similar issue. Research how
those groups describe and deal with the problem. The perspective of experience is invaluable
to your work.
This project asks you to do workplace research into a local problem impacting UCF or the
surrounding community. Your goal for this project is to describe the problem in detail using
as much information as you can gather from as many different sources as are useful. That
means you are looking at research gathered by others (e.g., government agencies, non-profit
organizations, professional and academic experts), but also you will gather your own data by
contacting experts and asking impacted population for their perspective. You will produce a
memo that reports your findings, giving readers a robust understanding of the problem you
have researched.
Upon successful completion of this module, you will be able to:

Engage with a problem-based scenario similar to those found in the workplace
Develop research skills appropriate for addressing a specific problem
Integrate research into a project deliverable
Develop professional and technical writing and editing skills
Practice writing for a specific purpose and audience
Produce a specific genre of professional discourse (informational report)
To complete this project, you will choose a local problem occurring at UCF or in the
community. You can select a problem from the list below, or pick your own problem of a
similar nature.
Sample Problems:

What is the optimal register layout for Starbuck’s at UCF to reduce lines and wait time?
Hint: Tapingo and grub hub

Background information: Put the problem in context. What does the audience need to
know to understand why the problem is a problem?
Explanation of the problem: Describe the problem in detail. What is happening?
Causes of the problem: Describe the factors contributing to the problem’s occurrence.
Why is the problem happening?
Impacted Population: Describe the people most directly impacted by the problem. Who
is the problem happening to?
To conduct your research you may wish to use any of the following methods and resources,
or anything else that helps you explain the scope of your problem:

Newspapers (local, university, national)
Reports from government agencies, universities, and/or NGOs
Scholarly research
Facts and statistics compiled by government agencies and/or NGOs
Interviews with experts and/or impacted individuals
Surveys (social media makes doing surveys easy)
Major Deliverable

A report in memo format that describes your problem in detail using all the research you
have conducted. The memo should have the following section headings:
o Background Information
o Explanation of the Problem
o Causes of the Problem
o Impacted Population
o Works Cited
Supplemental Deliverables

Research Plan: A memo that identifies the problem you have selected and your plans for
researching it, including research methods and potential sources for each required topic
listed above.
Research Progress Report: An update that describes a) the research you’ve done to date;
b) which topics that research applies to; c) what you have left to do before completing a
Please address the following questions:

What is research?
What purpose does research serve? Why do research?
How is research used in your writing?
There are no right or wrong answers. Address the questions based on your experience with
research in the academic and professional contexts in which you’ve encountered it.
*Analysis of Research-Based Report
Please find the report on attachments (Research_CleanAir)
As you skim the report, focus on the research included in each section. As you do, answer
the following questions:

What is the purpose of the report?

What types of research do you see in the report? (Think back to Ch. 2 from Just Enough
How is the research being used? How does the research contribute to the achievement of
the report’s purpose?
Which sections have the most research? Why do you suppose that is?
Your analysis of the report should develop a sense of the role of research in professional and
technical documents, which will prepare you for conducting that type of research as you
complete your own work.
Please submit your analysis here as a Word document
*Research Plan (Preliminary Draft)
Now that you have a topic, it’s time to plan your research strategy.
Here you will submit a memo that identifies the problem you have selected, and outlines
your plans for researching it, including research methods and potential sources for each
required headings of your final deliverable (see below).
Research Summary Headings

Background Information
Explanation of the Problem
Causes of the Problem
Impacted Population
The Basics
Research is a discipline with many applications. This chapter
introduces the core practices and fundamental ideas and techniques you will use repeatedly in many situations. We’ll cover
who should do research, different types of research and when
to use them, and roles within each set of research activities.
To help counter any skepticism about the business value of
research, we’ll also review some common objections and how
to overcome them.
Who should do research? Everyone!
Ideally, everyone who is on the design team should also participate in the research.
If you are a sole practitioner, well, that’s easy. You will have
excellent direct experience and can tailor the process and documentation to suit your needs. (Just be particularly mindful of
your personal biases.) If you work with other people, involve
Just Enough Research
them from the start. Presenting them with the world’s most
stunning report will give them a terrific reference document,
but it’s far less likely to inspire them to approach their work
differently. (Do you disagree? Perhaps you are an economist.)
When you find yourself making a case for a skeuomorphic,
bronze astrolabe interface based on the research you’ve all
done together, you’ll be able to spend less time explaining the
rationale and more time focused on the merit of the conclusion.
“As you saw in the interviews, we found that our target group
of amateur astronomers exclusively uses nineteenth-century
equipment for stargazing….”
People who have a hand in collecting the insights will look
for opportunities to apply them. Being the smart person is more
fun than obeying the smart person, which is how the researcher/
designer dynamic can feel if designers are merely the recipients
of the analysis.
At my first design agency job, the research director was a
charming PhD anthropologist with a penchant for vivid, striped
shirts. Despite being fresh out of academia, he was much more
of a scout troop leader than a fusty professor. Interviews and usability tests were scavenger hunts and mysteries with real-world
implications. Unlike heinous, contrived team-building activities—rope courses and trust falls—doing research together actually did make our team more collaborative. We were learning
interesting, valuable new things, and everyone had something
different to contribute. The content strategist would notice the
vocabulary real people used and the developer had good questions about personal technology habits. The visual designer was
just really into motorcycles, and that helped sometimes too.
Someone needs to be the research lead—the person who
keeps everyone on track and on protocol and takes ultimate
responsibility for the quality of the work. If you take this on it
might mean that you’re the primary researcher, gathering the
data for others to help you analyze, or you could have more of an
ensemble approach. The most important thing is that everyone
involved knows the purpose or goal of the research, their role,
and the process.
The Basics
Find your purpose
One of our maxims at Mule is that every design project ultimately amounts to a series of decisions. What are the most
important features? What is the best navigation scheme? How
big should the logo be?
For any given project, we include only the research activities
that support the specific decisions we anticipate. If the client has
only identified an audience and wants to explore ways to better
serve them (“What can we offer of value to high school science
teachers?”), our research will be more open-ended than if the
design problem is already well defined (“How can we get high
school science teachers to download and use our lesson plans?”).
This has been playing out on the fields of “mobile first.” Many
organizations are seeing a significant increase in their mobile
traffic. They know they have to do something different for users
on mobile devices, but aren’t quite sure what. So, they’re looking for ideas, or should be. It’s too soon to jump to fine-tuning
solutions. For example, should the Fantastic Science Center,
our fictional museum client, rewrite all of the exhibit descriptions for a mobile audience, or build a native event reservation
app, or encourage school group students to post exhibit photos
to Facebook from their phones? Organizational research will
tell you which interactions benefit the museum most, while
user research will indicate which are most plausible and the
circumstances under which they will take place. Maybe you
will discover that school district policy prohibits students from
using their phones on field trips, but parents are likely to take
photos of family visits to share with their Facebook friends.
In that case, parents are the ones to target with a social media
marketing campaign.
There are many, many ways of classifying research, depending on who is doing the classification. Researchers are always
thinking up more classifications. Academic classifications may
be interesting in the abstract, but we care about utility, what
helps get the job done. Research is a set of tools. We want to
make sure we can find the right one fast, but we aren’t too concerned with the philosophy of how the toolbox is organized.
Just Enough Research
To choose the best research tool for your project, you’ll need
to know what decisions are in play (the purpose) and what
you’re asking about (the topic). Then you can find the best ways
to gather background information, determine the project’s goals
and requirements, understand the project’s current context, and
evaluate potential solutions.
Generative or exploratory research: “What’s up with…?”
This is the research you do before you even know what you’re
doing. It leads to ideas and helps define the problem. Don’t think
of this as just the earliest research. Even if you’re working on an
existing product or service, you might be looking for ideas for
additional features or other enhancements, or new products you
could bring to an audience you’re already serving.
Generative research can include interviews, field observation,
and reviewing existing literature—plus feeling fancy about saying “generative research.”
Maybe the museum is trying to decide how to allocate that
grant money and has discovered that a lot of parents who recently had their first child are coming to the website and you
want to figure out what else you can offer them. Your question
might be, “What’s up with new parents anyway?” Your goal
would be to see the new parent experience from their eyes, to
understand what they do and what they need. Your generative
research activities might include interviewing new parents on
the phone, following new parents around on a typical day, or
looking at the questions new parents ask on social websites.
Once you’ve gathered information, the next step is to comb
through it and determine the most commonly voiced unmet
needs. This sort of research and analysis helps point out useful
problems to solve. Your thinking might lead to a hypothesis,
such as “Local parents of young children would value an app
that offers ideas for science events and activities based on their
child’s developmental milestones.” Then you can do further (descriptive) research on how parents recognize and commemorate
those milestones.
The Basics
Descriptive and explanatory: “What and how?”
Descriptive research involves observing and describing the characteristics of what you’re studying. This is what you do when
you already have a design problem and you need to do your
homework to fully understand the context to ensure that you
design for the audience instead of yourself. While the activities
can be very similar to generative research, descriptive research
differs in the high-level question you’re asking. You’ve moved
past “What is a good problem to solve?” to “What is the best
way to solve the problem I’ve identified?”
At Mule, we’ve done a lot of design work for eye health organizations. Despite the fact that several of us have really terrible
vision (and very stylish glasses), none of us had any expertise
beyond whether the chart looks sharper through lens number
one or lens number two. The Glaucoma Research Foundation
offered a clear design problem to solve: how to create useful,
accurate educational materials for people who had been newly
diagnosed with an eye disease. So, a round of descriptive research was in order.
To inform our design recommendations, we interviewed
ophthalmologists and patients, and reviewed a large quantity of
frankly horrifying literature. (Please, have your eyes examined
regularly.) By understanding both the doctor and patient priorities and experiences, we were able to create online resources
full of clear information that passed clinical muster and didn’t
provoke anxiety.
For the Fantastic Science Center, descriptive research comes
into play once we’ve identified a design problem, such as providing an online robotics course for students around the world.
Maybe this supports the organizational goal to create a global
robot army. It would be important to understand how online
learning would best fit into the lives of the target students. For
example, do they have their own equipment or do they share?
How do target users find out about new online activities? How
do the needs of students who only have mobile devices compare to those who have access to a laptop or desktop? Which
activities are they already engaged in that might compete with
or complement such a course?
Just Enough Research
Evaluative research: “Are we getting close?”
Once you have a very clear idea of the problem you’re trying
to solve, you can begin to define potential solutions. And once
you have ideas for potential solutions, you can test them to
make sure they work and meet the requirements you’ve identified. This is research you can, and should, do in an ongoing and
iterative way as you move through design and development. The
most common type of evaluative research is usability testing, but
any time you put a proposed design solution in front of your
client, you really are doing some evaluative research.
Causal research: “Why is this happening?”
Once you have implemented the solutions you proposed, and
have a website or application up and running out in the world,
you might start noticing that people are using it in a certain way,
possibly a way that isn’t exactly what you’d hoped. Or perhaps,
something really terrific is happening and you want to replicate
the success in other parts of your operation. For example, you’ve
noticed that ever since the Fantastic Science Center redesign
launched, tickets for the Friday evening science-loving singles
event are selling better, but ticket sales have completely dropped
off for the Sunday afternoon film program. You need to do some
causal research.
Establishing a cause-and-effect relationship can be tricky.
Causal research often includes looking at analytics and conducting multivariate testing (see Chapter 9). This means reviewing
your site traffic to see how visitors are entering and moving
around the site and what words they might be searching for,
as well as trying design and language variations to see which
ones are more effective. Causal research might show that your
film program traffic all came from one referring website that
no longer links to you. Or, you might have to expand beyond
looking at site performance to see what else is going on. Maybe
a competing organization started an event with a very similar
name and confused your target audience.
The Basics
As long as you’re clear about your questions and your expectations, don’t fret too much about the classification of the
research you want to undertake. Remain open to learning at
every stage of the process. And share this love of learning with
your team. Your research will benefit from a collaborative approach that includes assigning specific responsibilities to different people.
Research roles represent clusters of tasks, not individual people.
Often one person will take multiple roles on a study, or a single
role can be shared.
The author plans and writes the study. This includes the problem statement and questions, and the interview guide or test
The interviewer is the person who interacts directly with the
test participants.
The coordinator plans how time will be used during the study
and schedules sessions, including arranging times with the
The notetaker is responsible for capturing the data during a
test session. It is best that the interviewer and notetaker be two
separate people so that the interviewer can devote full attention
Just Enough Research
to the participant. If this is impossible, the interviewer can arrange to record the session as unobtrusively as possible. Having
both written notes and a recording makes data analysis easier.
The recruiter screens potential participants and identifies the
respondents who would be good subjects. The recruiter and
scheduler can easily be the same person.
The analyst reviews the gathered data to look for patterns and
insights. More than one person should have this role.
The documenter reports the findings once the research study
is complete.
Often it’s useful for clients or other available team members to
watch the research in progress. This is appropriate as long as
the presence of the observers will not influence the research
itself. As a substitute, you can make raw recordings available.
You can change roles with each set of activities if that works
best, or develop a routine that allows you to focus on the information gathering. Just as with design and coding, every time
you complete some research, you’ll have ideas for how to do it
better next time and you’ll have found new ways to incorporate
learning into your work.
Listen. Be interested. Ask questions. Write clearly. And practice. Whatever your day job is, adding research skills will make
you better at it.
The Basics
The research process
We’ll cover ways to organize research activities in extensive detail in Chapter 3. For the purposes of this section, what matters
is that everyone working together has a shared understanding of
how the work will proceed. This can be as simple as a checklist.
In addition to organizing the efforts of your immediate team,
you might need to get approval to do research at all, either from
the client or from decision-makers in your organization. Handle
this as early as possible so you can focus on the work rather
than defending it.
Overcoming objections
Sometimes the people you’re working with or for will consider
research somewhere between a threat and a nuisance.
You might have to get a certain amount of what they call
organizational buy-in to proceed. This could start with agreement from your immediate team, but the whole point of doing
research is to have a stronger basis for decision-making, so if
another level of decision-making, such as executive fiat, trumps
the research, you will have wasted your time. Get ready to advocate for your research project—before you start it.
The objections you will hear
Here is a handy list of objections and responses.
We don’t have time
You don’t have time to be wrong about your assumptions. What
are your key assumptions? What if they’re all wrong? How much
work would you have to redo? How long would that take?
We don’t have the expertise, or the budget
You have what it takes, thanks to this book! Even with little
or no budget, you can usually locate some related research
online, wrangle representative users to interview, and do a
Just Enough Research
little usability testing. Applying some critical thinking to your
assumptions costs nothing, but a change in your habits can offer
tremendous returns.
The CEO is going to dictate what we do anyway
You’re going to fight to change that dictatorial culture. Reasonably
accurate and well-documented research has been known to
sway even the most magnificent and well-fed egos. And if the
leadership really does have a “damn the facts, full speed ahead”
attitude, get a different job.
One research methodology is superior (qualitative vs.
What you need to find out determines the type of research you
need to conduct. It’s that simple.
You need to be a scientist
This isn’t pure science we’re talking about here. This is applied
research. You just need to have (or develop) a few qualities in
common with a good scientist:
• Your desire to find out needs to be stronger than your desire
to predict. Otherwise you’ll be a mess of confirmation bias,
looking for answers that confirm what you already assume.
• You need to be able to depersonalize the work. There are no
hurt feelings or bruised toes in research, only findings.
• You need to be a good communicator and a good analytical
thinker. Otherwise questions and reports get muddy, and results will be worse. This is just a set of skills that most people
can develop if they have the right attitude.
You need infrastructure
I suspect you own or can borrow a laptop and have access to
the internet. That is all you need.
The Basics
It will take too long
Upfront research can provide a basis for decision-making that
makes the rest of the work go much faster. Nothing slows down
design and development projects as much as arguing over personal opinions or wasting effort solving the wrong problem.
And you can start small. A couple of weeks can mean very little
to your overall schedule while adding significantly to your potential for success.
You can find out everything you need in beta
There are a lot of things you can find out in beta: what functionality is working, whether users have a hard time finding
core features. But there are also a lot of things that are very
helpful to know before you start designing or coding at all, and
you can find those out pretty fast: what your target audience is
doing right now to solve the problems your product or service
purports to solve, whether people want this product at all, and
what your organization has to do to support it.
Again, it’s a matter of where you want to invest and what you
have to lose. Don’t waste anyone’s time or effort on untested
assumptions if you don’t have to.
We know the issue/users/app/problem inside and out already
Unless this knowledge comes from recent inquiry specific to
your current goals, a fresh look will be helpful. Familiarity
breeds assumptions and blind spots. Plus, if you are very familiar with your users it will be very easy for you to find some to
talk to.
And who is the “we” in this case? In the absence of a mind
meld, the client’s experience with the users or the business
problem doesn’t transfer to the designer. Shared understanding is key.
Just Enough Research
Research will change the scope of the project
It’s better to adjust the scope intentionally at the start than be
surprised when new information pops up down the road to
amend your plans. Research is an excellent prophylactic against
unexpected complexity.
Research will get in the way of innovation
Relevance to the real world is what separates innovation from
invention. Understanding why and how people do what they
do today is essential to making new concepts fit into their lives
Actual reasons behind the objections
At the root of most of these objections is a special goo made up
of laziness and fear.
I don’t want to be bothered
Unless you are naturally curious about people, research can
seem like annoying homework at first, but once you get into it,
you’ll find it totally fun and useful. A little knowledge opens up
a whole world of new problems to solve and new ways to solve
the problems at hand. That makes your work more rewarding.
If research is one more thing tossed on your already overfull
plate, then someone needs to ask the “Who should be doing
this?” question again—but the problem is you being too busy,
not research being unimportant. Research needs to be integrated
into process and workflow or it will get shoved in a corner. If
your project has a project manager, talk with them about finding
ways to make it work.
I am afraid of being wrong
The cult of the individual genius designer/developer/entrepreneur is strong. In certain “rockstar knows best” cultures,
The Basics
wanting to do research can come across as a sign of weakness
or lack of confidence. Fight this.
Information and iteration are the keys to a successful design.
Research is just one set of inputs.
I am very uncomfortable talking to people
You are creating a system or a service actual people are going to
have to use. This system will be talking to people on your behalf,
so it’s only fair that you talk to people on its behalf. That said,
some people on your team will have more comfort and skills
when it comes to interacting with your research subjects, so
consider that when you’re deciding who does what.
Having to respond to challenges and objections before you can
get to work might feel like a waste of time, but it can be very
useful in its own right. Describing the goals and potential of your
research to people who aren’t sold on the value will actually
help you focus and better articulate what you hope to uncover.
These discussions will give you a better understanding of the
environment you’re working in. Research is all about context.
Research in any situation
“Poor user experiences inevitably come from poorly informed
design teams.”
—Jared M. Spool, founder of User Interface Engineering
Design happens in context. And research is simply understanding that context.
Research happens in a context as well. Your professional
environment should inform the research activities you choose
and how you go about them.
The following contexts and situations aren’t mutually exclusive. You might be in some that overlap. Just remember that
no matter what situation you’re in, you can do or participate in
some useful research.
Just Enough Research
On the one hand, as a freelancer, you can do whatever the hell
you want as part of your design and development process as
long as you deliver what the client expects when the client
needs it. On the other hand, if someone is hiring you as a solo
web designer, they may balk at paying for something that falls
outside of their concept of that gig.
If you are doing work, you need to get paid for it. “Just tossing
in the research” is a terrible mistake designers who want to do
good work make all the time. Instead, you have to commit to
research personally as part of how you work, make your case
for it, and be sure to include it in your fee. Research will make
your design stronger and enhance your ability to defend your
decisions to the client.
If you’re being brought in as a contractor to work as part of
an internal team, make sure you have access to all of the information and insight required to do your job. Contractors run the
risk of being pigeonholed. You’re the designer, why do you need
to bother with research? When information is distributed on a
“need-to-know” basis, you’re the best judge of what you need
to know to get the job done.
At a client services agency
If you are at an agency, you have the opportunity to improve
your process with each new project. A certain amount of research is built in simply because you have to scope the work to
bid on the job and understand the client’s needs (or do an awesome job of faking it) to land the gig. The better you do these
things, the better time you’ll have doing the work.
At Mule, and at many other agencies, the first phase of work
on any new project is an intensive period including all the
research that’s useful and practical. We talk to stakeholders,
interview users, review competitors, and sometimes conduct a
round of benchmark usability testing. Sometimes we do this in
a matter of a few weeks. We need to get up to speed quickly, so
working collaboratively is essential.
The Basics
As with freelancing, coming in from the outside puts us at a
definite advantage because we are external to existing processes
and political considerations. Some clients bring us in to ask the
questions they know need to be asked, but are not in a position
to ask themselves.
Falling back on ignorance can be a position of strength.
Asking naive questions can cut right to the heart of assumptions
and open people up to thinking about problems in a new way.
“How does that benefit the business?” and “Why do you do it
that way?” are a couple of terrific questions that can be very
tricky for someone on the inside to get away with.
In-house at an established company
In an established organization of any size, politics are a huge
consideration. Challenging the assumptions of those in power
can be incredibly threatening to those people. It can also be the
best thing you can do to ensure that your work succeeds—if you
don’t get fired. (See Chapter 4 for more on introducing even the
most stubborn organization to the joys of research.)
If you’re at an organization that genuinely embraces critical
thinking and clear communication in the design process, that’s
terrific. I hope you’re also taking very good care of your unicorn
desk mate. Otherwise, proceed with open eyes and discretion.
An existing product means that a glorious data trove exists:
customer service! Customer service is where actual, individual
human needs and expectations crash headfirst into reality. If you
have ready access to the customer service representatives, talk
to them. You will make friends. Customer service staff have so
much expertise and often get very little respect within an organization. And they have to communicate with unhappy people
all day, every day. A conversation with someone who respects
and values their contribution is likely to be a good time for all.
In addition to, or instead of, direct access to the customer service people, get hold of the inbound support requests. This will
be a fantastic source of insights into the ways different types of
customers think about their needs and the language they use to
describe them. You can also practice seeing through what people
say they want to what they actually need. Steve in Louisville
Just Enough Research
may be asking for a more informative error message, but what
he really wants is to be able to reorder his usual pizza and salad
with a different credit card and no error message.
You don’t just want insights; you also want a way to put those
insights back into the product.
It’s very helpful to have a clear idea of how product and marketing decisions are made in your company. Do you have a truly
customer-centered culture? When leaders talk about research,
are they talking about market research, or do they have a more
holistic view? Is there a market research group? Is design or user
experience research already part of your process?
In-house at a startup
When you have a small startup team, you don’t have to worry
too much about understanding your own organization beyond
knowing what you have to do to keep the lights on. In the early
stages it should be easy to share information with the team, as
long as you aren’t growing so fast that people and perspectives
start getting left out.
You do need to have some clarity around your audience and
the business context you’re operating in. You’re trying to introduce something new into the world. Who needs it and what is
important to those people? When you’re discussing the initial
design and development of your product, discuss the role of
research with the team. Document and review assumptions to
identify the areas in which doing some research might be the
most beneficial. Get some early agreement on how research
will be involved, keep track of your assumptions, and adopt a
skeptical point of view.
The approach and biases of the founder and the investors
might dominate, so if you aren’t one of those, you will have to
be very clear about the value of research to your endeavor and
savvy about how to make your case.
Working with an agile development team
Agile is a popular software development philosophy with the goal
of building better software faster in a productive, collaborative
The Basics
working environment. Many short iterations of two or three
weeks replace the traditional approach of multi-month or multiyear projects broken into distinct phases.
On the surface, agile seems antithetical to design. The agile
manifesto explicitly values “responding to change over following
a plan.” Design is planning. However, any work with complex
ideas and dependencies requires holding some ideas outside the
development process. You can’t cave in completely to the seductive solipsism that agile offers, or you’ll be tunneling efficiently
and collaboratively toward the center of the earth. While flexibility and responsiveness are certainly virtues that many project
teams could use more of, let’s not discount the importance of
having some sort of plan.
From a user experience perspective, the primary problem
with Agile is that it’s focused on the process, not the outcomes.
It doesn’t offer guidance on what to build, only how. Perhaps
your team is more efficient and happier making a lot of stuff
together, but how do you know that stuff is the best it could be,
meeting real user needs and fit to compete in the marketplace?
If you’re always reacting without a framework, you need
some guiding mandates. Which customers do you listen to and
why? Which user stories do you prioritize? What are you ultimately building toward?
Research is not antithetical to moving fast and shipping constantly. You’ll need to do some upfront work for background
and strategy and the overall framework. Then, as the work
progresses, do continual research.
It might sound counterintuitive, but the most effective approach may be to decouple the research planning from the
development process—that is, don’t wait to start coding until
you’ve answered all your research questions. Once you have
some basic tools and processes in place, such as observation
guides, interview guides, recording equipment, and questions
for analysis, you can take a Mad Libs approach and fill in your
actual questions and prototypes on the fly.
Jeff Patton describes this continuous user research process in
his article “Twelve Emerging Best Practices for Adding UX Work
to Agile Development” ( He offers a tidy
three-point summary:
Just Enough Research
• Aggressively prioritize the highest-value users.
• Analyze and model data quickly and collaboratively.
• Defer less urgent research and complete it while the software
is being constructed.
In other words, focus only on the essential user types, deal
with your data as soon as you get it, involve your team in the
analysis, and do the less important stuff later.
This of course opens up the questions of who are the highestvalue users and what are the more or less urgent research activities. Prioritize those user types whose acceptance of the product
is critical to success and those who least resemble the software
developers on your team. Go learn about them.
Recruiting and scheduling participants is the most difficult
part, so always be recruiting. Set up windows of time with different participants every three weeks. When you have them,
you can either conduct an ethnographic interview (see Chapter
5) to understand their behavior before the next round of development or do some usability testing on the current state of the
Use what you learn from the initial user research and analysis to create personas that inform high-level sketches and user
stories. Then, when the team is working on a feature that has a
lot more engineering complexity than interaction design complexity, you can fit in additional evaluative research.
Throughout the development cycle, the designers can use
research to function as a periscope, keeping an eye out for new
insights about users and competitive opportunities while doing
usability testing on whatever is ready.
Just enough rigor
Professional researchers are not unlike journalists. While many
people have sufficient skills to observe, analyze, and write, it’s
allegiance to a set of standards that sets the pros apart. In addition to being professional and respectful in your work, there are
just a few responsibilities to keep in mind.
The Basics
Cover your bias
Wherever there is research there is bias. Your perspective is
colored by your habits, beliefs, and attitudes. Any study you
design, run, or analyze will have at least a little bit of bias. Your
group of participants will be imperfectly representative. Your
data gathering will be skewed. Your analysis will be colored by
selective interpretation.
Don’t give up!
You can’t eliminate it completely—but the simple act of noting potential or obvious bias in your research process or results
will allow you to weigh the results more appropriately. In lieu
of a trained eye, use the following bias checklist, or make your
own. Grade hard.
Design bias
Design in this case refers to the design of the studies themselves,
how they are structured and conducted. This is the bias that
creeps into studies when you don’t acknowledge bias, or if you
include or leave out information based on personal goals or
Sampling bias
Since we’re talking about quick and dirty qualitative research,
sampling bias is almost unavoidable. Counter it by being mindful
in the general conclusions you draw.
If your app for science-minded new parents is intended to
serve men and women in equal numbers but all your subjects
are women, that’s a biased sample.
Interviewer bias
Conducting unbiased interviews is difficult. Inserting one’s
opinions is easy. Make sure that interviewers remain as neutral
as possible.
Just Enough Research
This is something to watch out for particularly at the beginning of interviews when you are trying to establish rapport.
Maybe the interviewer is super enthusiastic about one aspect of
the museum. Practice interviews and critiques with an internal
team are the best way to develop a neutral interviewing style.
Sponsor bias
This is one of the biggest issues with onsite lab usability tests,
because going onsite feels special and can be exciting or even
daunting to a participant. If the Fantastic Science Center is inviting you in to their facility, offering you snacks, and writing you
a check, it is very possible you will be gentler in your evaluations. To decrease sponsor bias without being deceptive, use a
general description of the organization and goals of the study
without naming the specific company until and unless it appears
in materials you are evaluating. (Once you get to the point of
showing a website design featuring the Fantastic Science Center
logo, the secret will be out.)
For example, begin a phone interview with “We’re interested
in how you select and plan activities for your family,” rather
than “We want you to tell us what would entice you to visit the
Fantastic Science Center.”
Social desirability bias
Everyone wants to look their best. People want to be liked. It can
be hard to admit to an interviewer that you don’t floss or pay off
your credit card bill every month, so participants will sometimes
give the answers that put them in the best light. Emphasize the
need for honesty and promise confidentiality.
The Hawthorne effect
The behavior of the people you are studying might change just
because you are there. Staff who typically goof around and chat
during the day might clam up and shuffle files if you’re hanging
The Basics
about to observe their workflow. Do your best to blend into the
background and encourage research participants to go about
their normal day.
The ethics of user research
What harm can come of asking people how they decide what to
have for dinner or how they use their phones to find directions?
We aren’t talking about clinical trials of dangerous, new cancer
drugs, but all research that includes people and their personal
information should be conducted ethically and conscientiously.
It’s our responsibility as professionals to proceed without deceiving or injuring any of the participants.
Below is a starter set of ethical concerns you should keep in
mind whenever you are doing research. (For more thorough
guidelines, take a look at the ICC/ESOMAR Code on Market and
Social Research, which is available in fifteen languages: http://
The project as a whole
Maybe this goes without saying, but it is worth saying nevertheless. Is your overall goal, the project that the research supports,
ethical? Will your success lead to harm for others? If it will, don’t
participate in it. Designers have a role to play as gatekeepers.
You should be intentional about your position. Conducting a
completely above-the-board study on women to induce them to
buy a diet aid with dangerous side effects doesn’t make it right.
The goals or methods of the research
A certain amount of user research and usability requires keeping certain facts from the participants. Usually this is benign,
such as hiding the name and description of the product you’re
designing, but sometimes it’s a problem. Will concealing these
facts lead those users to participate in anything they might not
otherwise agree to? Are you tricking them or setting some unrealistic expectation about the real world? Are you presenting
false information as true?
Just Enough Research
Consent and transparency
Informed consent is the rule. This means that participants must
understand and agree in advance to the overall goals of any study
and how their information will be recorded, used, or shared.
Let them know if they are being watched by unseen observers.
Make sure that research participants are of sound mind and able
to give consent to participate. This means that working with
underage research participants is very tricky, and requires the
parents’ consent.
Safety and privacy
Ensure that participants know what is required of them in advance and will be comfortable and not fatigued. Verify that your
presence in a home or workplace will not lead to any risks or
danger. For example, if you’re observing someone taking care
of small children, make sure that your actions don’t distract in
any way that would interfere with proper care.
And for the love of all humanity, never, ever agree to do
telephone interviews when anyone involved is driving. Not
participants, not interviewers, not passive observers. No one.
As soon as you learn that someone is on the phone while driving, end the call, and follow up by email or another means to
reschedule if necessary.
Be a skeptic
Get in the habit of asking a lot of questions. Question all your
assumptions and determine whether you need to check your
facts. If you’re constantly on the lookout for threats and potential points of failure, you and your products will be stronger.
This is a type of critical thinking that will serve you well at all
times. You need to be aware of how much you don’t know and
what that means.
Awareness of your own limits will allow you to be as effective
as possible within them.
The Basics
Best practices
There are many good reasons why people get master’s degrees
and PhDs and become professional analysts and researchers, and
there are plenty of reasons why companies benefit from hiring
those people. Specialized, educated, and trained researchers
cultivate a deep curiosity, have a broad base of relevant knowledge, and gain academic and professional experience conducting ethical and methodical studies. As a designer or developer,
you might have good reasons to avoid DIY and hire a trained
These include:

A large, complex project.
A large, complex organization.
Complex or sensitive subject matter.
A very specialized or challenging user base, such as children
or neurosurgeons.
• Heinous organizational politics.
• Lack of team members with the time or inclination to acquire
additional skills and duties.
Skilled, trained professional researchers have rigor. They can
apply precise critical thinking in the face of common distractions
and pressures, such as the enthusiasm of their team or their
manager’s personal preferences. The best researchers are like
Mr. Spock, with just enough humor and humanity to temper
their logical thought processes and allow them to roll with imperfect circumstances. You want rigorous, not rigid.
In the absence of a trained professional, how do you ensure
you are being sufficiently rigorous? You’re an amateur attempting these stunts on the open road instead of a closed course; how
do you make sure you and your work don’t go up in flames?
You borrow the methods of America’s greatest amateur,
Benjamin Franklin: discipline and checklists.
Discipline requires you to be ever-watchful for bad habits,
shoddy thinking, and other human frailties that will undermine
your efforts. Checklists substitute the experience of others for
Just Enough Research
your own. Discipline also requires that you don’t deviate from
the checklists until you have sufficient experience yourself.
Here is the first checklist, that of best practices. Go over these
again and again until you know them by heart, and then post
them visibly so you never have to rely on memory.
1. Phrase questions clearly
This refers not to the questions you’re asking, but the big question you’re trying to answer. Unless you know and can clearly
state what you’re trying to find out and why, applied research
is a pointless exercise.
2. Set realistic expectations
A successful study is preceded by expectation-setting for everyone involved, including the questions to be answered, the
methods to be used, and the decisions to be informed by the
findings. This is particularly important if you need to request
time or budget especially for the work. If your research work
doesn’t meet the expectations of the stakeholders, they will treat
you like you’ve wasted time and money. Ask team members and
managers what they hope for. Tell them what to expect.
3. Be prepared
Research is like kitchen work: the better you prep, the faster
and cleaner the work goes. If you don’t prepare, you end up
with a huge mess and a kitchen on fire. Get your process and
materials in order before you start. Set these up so they’re easy
to reuse as needed.
4. Allow sufficient time for analysis
You need a little time for things to click into place. After doing
the research, it’s tempting to just forge ahead to solutions without giving yourself enough time to digest. Again, a bit more time
here can save lots later on.
The Basics
5. Take dictation
Notes or it didn’t happen. Effective research requires effective
reporting, and sharing your results and recommendations with
others. A good report doesn’t have to be arduous to compile
or read. It needs to be sufficiently informative and very clear
to anyone who needs to make decisions based on the research.
You may be doing your own research to save time and money,
but be honest with yourself and your team about your capacity
for maintaining this level of rigor. Otherwise you risk wasting
both time and money, as well as spreading misinformation and
decreasing the overall reputation of research as a necessary
input into the work.
Can you commit?
Good. Then onward.
How much research is enough?
“There are things we know that we know. There are known
unknowns—that is to say, there are things that we now know we
don’t know. But there are also unknown unknowns—there are
things we do not know we don’t know.”
—Donald Rumsfeld, former US secretary of defense
Avoiding unnecessary research
In addition to offering the clarity and confidence necessary to
design, research is essential to reducing your risk—the risk you
incur by relying on assumptions that turn out to be wrong or
by failing to focus on what’s most important to your business
and your users. However, some assumptions are higher-risk
than others.
To make the best use of your time and truly do just enough
research, try to identify your highest-priority questions—your
assumptions that carry the biggest risk.
Ask this question: given our stated business goals, what potential costs do we incur—what bad thing will happen—if, six
months from now, we realize:
Just Enough Research
• We are solving the wrong problem.
• We were wrong about how much organizational support we
have for this project.
• We don’t have a particular competitive advantage we thought
we had, or we didn’t see a particular competitive advantage
before our competitor copied us.
• We were working on features that excited us but don’t actually matter that much to our most important customers.
• We failed to reflect what is actually most important to our
• Our users don’t really understand the labels we’re using.
• We missed a key aspect of our users’ environments.
• We were wrong about our prospective users’ habits and
If there is no risk associated with an assumption—for example, if you are working on a technical proof of concept that
really, truly doesn’t have to satisfy any real-world users—then
you don’t need to spend time investigating that assumption.
On the other hand, maybe the success of the new design for
the Fantastic Science Center’s online store depends on the assumption that many people who shop online value the ability
to publicly share their transactions. You could conduct research
to understand the social sharing practices and motivations of
people who shop online before diving into design and development. Or you could go ahead and design based on an optimistic
assumption, then see what happens. At risk are the time and
money to design and build the functionality, as well as the organization’s reputation. (“They just told everyone on the internet
about the robot I bought my kid for her birthday. Not cool!”)
Better understanding of online shoppers mitigates the risk by
validating the assumption and informing your design with real
user priorities. In addition, you might uncover opportunities to
provide something of even greater value to that same audience.
All it takes to turn potential hindsight into happy foresight is
keeping your eyes open and asking the right questions. Failing
isn’t the only way to learn.
The Basics
That satisfying click
No matter how much research you do, there will still be things
you wish you’d known, and there are some things you can only
learn once your design is out there in the world. Design is an
iterative process. Questions will continue to crop up. Some of
them you can answer with research and some you can only
answer with design. Even with research, you’ll need to create a
few iterations of the wrong thing to get to the right thing. There
is no answer to the question of enough, other than the point at
which you feel sufficiently informed and inspired. The topics
in this book can only offer a starter kit of known unknowns.
That said, one way to know you’ve done enough research is
to listen for the satisfying click. That’s the sound of the pieces
falling into place when you have a clear idea of the problem you
need to solve and enough information to start working on the
solution. The click will sound at different times depending on
the problem at hand and the people working on it.
Patterns will begin to emerge from the data. Those patterns
will become the answers you need to move forward. This will
be very satisfying on a neurochemical level, especially when
you start out with a lot of uncertainty. Since human brains are
pattern recognition machines, you might start seeing the patterns you want to see that aren’t actually there. Collaborating
with a team to interpret the data will reduce the risk of overly
optimistic interpretation.
If you don’t have enough information, or what you’re finding
doesn’t quite hold together, the pieces will rattle around in your
head. Ask a few more questions or talk to a few more people.
Talk through the results. The pieces will fall into place.
Learn to listen for that click.
Just Enough Research
The Benefits and Costs of the
Clean Air Act from 1990 to 2020
U.S. Environmental Protection Agency
Office of Air and Radiation
March 2011
The Benefits and Costs of the Clean Air Act from 1990 to 2020: Summary Report
The study was led by staff from the US Environmental Protection Agency Office of Air and Radiation, with support
provided, under contract to EPA, by the organizations participating on the Study Team.
The full integrated report and this summary report were reviewed by the EPA Science Advisory Board’s Advisory
Council on Clean Air Compliance Analysis (hereafter the Council) and its three technical subcommittees. The
individual detailed reports that focus on each of the key analytical components of the overall study were also
reviewed by the Council and/or one or more relevant subcommittees.
The study was greatly improved by the ideas and expertise of the individuals and firms participating on the Study
Team, and by the rigorous and thoughtful expert review by the external review panels. However, responsibility for
the study’s results, the analytical decisions leading to those results, the interpretations reported herein, and the
recommendations made for future efforts, rests with the Environmental Protection Agency.
Study Team
Study Review Panels
US EPA Office of Air and Radiation
Industrial Economics, Incorporated
E.H. Pechan & Associates
ICF International
Research Triangle Institute
Stratus Consulting
Sonoma Technologies
Advisory Council on Clean Air Compliance Analysis
Air Quality Modeling Subcommittee
Health Effects Subcommittee
Ecological Effects Subcommittee
For further information
This document is an abridged version of a longer report which evaluates the benefits and costs of programs
implemented pursuant to the 1990 Clean Air Act Amendments. The longer report in turn summarizes and
integrates a series of technical reports documenting particular analytical tasks, such as estimation of compliance
cost and projection of air quality changes. Data presented in this summary report are documented in the full
integrated report and/or the supporting technical analyses.
Electronic copies of this summary report, the full integrated report, and all publicly available supporting technical
documents can be downloaded at:
Paper copies of this summary report can be obtained by submitting a request indicating the number of copies
required to: CAAA.Benefit‐Cost‐
For information about the technical aspects of the study, contact Jim DeMocker, Senior Policy Analyst, Office of Air
and Radiation, US EPA at
For information about the peer review of the study, contact Stephanie Sanzone, Designated Federal Official for the
Council at
The Benefits and Costs of the Clean Air Act from 1990 to 2020: Summary Report
Summary of Findings and Recommendations
This study evaluates the benefits and costs of programs implemented pursuant to the 1990
Clean Air Act Amendments, relative to a hypothetical baseline which assumes control programs
established under the 1970 Clean Air Act and 1977 Amendments stayed fixed at their 1990 levels of
scope and stringency. The study applies the framework and principles of benefit‐cost analysis to
estimate significant beneficial and costly effects of these programs, express these effects where feasible
and appropriate in dollar value terms to
facilitate comparison of disparate effects, and
then calculate the overall net economic
benefits (benefits minus costs) of the changes
in Clean Air Act‐related programs resulting
from the 1990 Amendments.

Based on the scenarios analyzed in this
study, the costs of public and private
efforts to meet 1990 Clean Air Act
Amendment requirements rise
throughout the 1990 to 2020 period of
the study, and are expected to reach an
annual value of about $65 billion by
Though costly, these efforts are
projected to yield substantial air quality
improvements which lead to significant
reductions in air pollution‐related
premature death and illness, improved
economic welfare of Americans, and
better environmental conditions. The
economic value of these improvements is
estimated to reach almost $2 trillion for
the year 2020, a value which vastly
exceeds the cost of efforts to comply
with the requirements of the 1990
Clean Air Act Amendments.
Exhibit 1. Primary Central Estimates of direct benefits and
direct costs for the 2000, 2010, and 2020 study target years.
(In billions of 2006 dollars). The graph shows the extent to
which benefits exceed costs throughout the study period.
Because of inflation, the value of a US dollar varies from year to year. In this study, dollars are defined according
to the value they held in the year 2006.
The Benefits and Costs of the Clean Air Act from 1990 to 2020: Summary Report

The extent to which estimated benefits exceed estimated costs and an in‐depth analysis of
uncertainties indicate that it is extremely unlikely the costs of 1990 Clean Air Act Amendment
programs would exceed their benefits under any reasonable combination of alternative
assumptions or methods identified during this study. Even if one were to adopt the extreme
assumption that air pollution has no effect on premature mortality –or that avoiding such effects
has no value—the benefits of reduced non‐fatal health effects and visibility improvements alone are
more than twice the total cost of compliance with 1990 Clean Air Act Amendment requirements.

Economy‐wide modeling was also conducted to estimate the effect of the 1990 Amendments on
overall U.S. economic growth and the economic welfare of American households. When some of
the beneficial economic effects of clean air programs were incorporated along with the costs of
these programs, economy‐wide modeling projected net overall improvements in economic growth
and welfare. These improvements are projected to occur because cleaner air leads to better health
and productivity for American workers as well as savings on medical expenses for air pollution‐
related health problems. The beneficial economic effects of these two improvements more than
offset the costly effects across the economy of expenditures for pollution control.

The most significant known human health effects from exposure to air pollution are associated
with exposures to fine particles2 and ground‐level ozone pollution. Many of these effects could be
quantified for this study; but other health effects of fine particles and ozone, health effects
associated with other air pollutants, and most air pollution‐related environmental effects could be
quantified only partially, if at all. Future improvements in the scientific and economic information
needed to quantify these effects would be expected to further increase the estimated benefits of
clean air programs.
Particle pollution, also known as “particulate matter” or PM, is a term used to describe a broad class of extremely
small solid particles and liquid droplets suspended in the air. Particle pollution can include one or more different
chemical components, including acids (such as nitrates and sulfates), organic chemicals, metals, and soil or dust
particles. The size of particles has been linked to their potential for causing health problems since it is easier for
smaller particles to bypass protective mechanisms in the nose and throat and enter deeply into the lungs. The
number which sometimes follows the term PM refers to the aerodynamic diameter of particles expressed in units
of microns (millionths of a meter); so PM2.5, for example, refers to a mixture of aerosol particles which are less
than or equal to 2.5 microns. EPA classifies particle pollution into two main categories: (1) “inhalable coarse
particles” such as those often seen near roadways and dusty industrial activities, which are larger than 2.5 microns
but smaller than 10 microns, and (2) “fine particles” such as those found in smoke and haze, which are 2.5 microns
and smaller. The terms PM2.5 and “fine particles” therefore refer to the same fraction of particle pollution.
The Benefits and Costs of the Clean Air Act from 1990 to 2020: Summary Report
The findings of this study have potentially significant implications for policy, programs, and
research related to air pollution in the U.S. The recommendations presented below focus on research
needs and the expansion and refinement of future studies.

Clean Air Act programs address a wide variety of air pollutants beyond the fine particle and
ozone pollution which emerged as the primary focus of this study’s quantitative results. The
data and modeling tools needed to estimate the health and environmental consequences of
these other pollutants, however, are limited. There is an ongoing need for investment in
research to improve the coverage of potentially important effects in benefit‐cost studies of air
pollution control programs. Additional research is also needed to reduce uncertainties in the
estimates of effects already incorporated in benefit‐cost studies, especially relatively significant
effects such as those associated with fine particle‐ and ozone‐related premature mortality and
the economic value of avoiding those outcomes.

Programs to reduce key Clean Air Act pollutants through national ambient concentration
standards such as those for fine particles and ozone, programs to address air pollutants with
more localized affects such as toxic compounds and heavy metals, and programs and policies
which reduce emissions of greenhouse gases may impose various requirements on a given
source of emissions. Future air pollution program assessments would be more useful to
policymakers and the public if they were designed to provide insights on the combined effects
of programs to address these different categories of air pollution.

Typical macroeconomic modeling tools and practices tend to focus on assessment of effects
across the economy of compliance expenditures while ignoring the economy‐wide benefits of
cleaner air. Consideration should be given to improving macroeconomic modeling of major
environmental programs so their benefits as well as their costs are reflected in projections of
how these programs affect the overall economy and the economic welfare of American
The Benefits and Costs of the Clean Air Act from 1990 to 2020: Summary Report
About this Report
This report is the third in a series of EPA studies
which estimate and compare the benefits and
costs of the Clean Air Act and related programs.
The first report was called the Retrospective
Study, and was published in 1997. This first
study estimated the benefits and costs through
1990 of programs implemented pursuant to the
1970 Clean Air Act and the 1977 Amendments,
and included an analysis of the benefits and
costs of phasing out leaded gasoline.
The second report was called the First
Prospective Study. Published in 1999, it
evaluated the incremental benefits and costs of
the 1990 Clean Air Act Amendments and
associated programs through the year 2010,
relative to controls in place as of 1990. In
addition to evaluating the effects on human
health, the economy, and the environment of
Titles I through V of the Amendments,3 the First
Prospective Study analyzed the benefits and
costs of phasing out stratospheric ozone
depleting chemicals such as chlorofluorocarbons
(CFCs) under Title VI.
The current report is called the Second
Prospective Study. This new study updates and
expands the First Prospective Study by using
new and better data and modeling tools. The
new study also looks further out into the future
by evaluating the costs and benefits of 1990
Clean Air Act Amendment programs through the
year 2020.
ANALYSES (as amended, in part):
(a) The Administrator…shall conduct a comprehensive
analysis of the impact of this Act on the public health,
economy, and environment of the United States…
(b) In describing the benefits of a standard described
in subsection (a), the Administrator shall consider all of
the economic, public health, and environmental
benefits of efforts to comply with such standard…
The Administrator shall assess how benefits are
measured in order to assure that damage to human
health and the environment is more accurately
measured and taken into account…
(c) [T]he Administrator shall consider the effects…on
employment, productivity, cost of living, economic
growth, and the overall economy of the United States.
(e) [T]he Administrator…shall appoint an Advisory
Council on Clean Air Compliance Analysis
of…recognized experts in the fields of the health and
environmental effects of air pollution, economic
analysis, environmental sciences, and such other fields
that the Administrator determines to be appropriate.
(g) The Council shall‐
(1) review the data to be used for any analysis
required under this section and make
recommendations to the Administrator on the use of
such data;
(2) review the methodology used to analyze such
data and make recommendations to the
Administrator on the use of such methodology; and
(3) prior to the issuance of a report…review the
findings of such report, and make recommendations
to the Administrator concerning the validity and utility
of such findings.
Exhibit 2. Clean Air Act Section 312 statutory language
(abridged) as amended by Section 812 of the 1990
Amendments. The text of the law defines Congress’
direction to EPA regarding the scope and review of these
The Clean Air Act is comprised of a number of statutory titles. Title I requires attainment of national air quality
standards for designated pollutants such as ozone, Title II focuses on mobile source control programs, Title III
addresses hazardous air pollutants, Title IV establishes programs to address acid deposition and related effects,
Title V establishes permitting requirements, and Title VI focuses on protection of the stratospheric ozone layer.
The Benefits and Costs of the Clean Air Act from 1990 to 2020: Summary Report
The Second Prospective Study focuses on evaluating the significant changes made over the last decade
in the implementation of Titles I through IV. Readers interested in benefit and cost information related
to Title V (permits) and Title VI (stratospheric ozone protection) are referred to the First Prospective
Study and subsequent EPA Regulatory Impact Analyses.
The effects of the 1990 Clean Air Act Amendments estimated herein reflect actions and partnerships
across multiple levels of government, private organizations, households, and individuals. This combined
effort involves federal standard setting and implementation, state and local programs to meet federal
standards, and expenditures by private entities to achieve the requisite emissions reductions.
Goals and Objectives of the Study
During the legislative efforts leading up to enactment of the 1990 Clean Air Act Amendments, members
of Congress working on the Act’s reauthorization made it clear they wanted more and better
information from EPA about the economic, health, and environmental effects of air pollution control
programs. To ensure this improved information was available to support future policymaking, Congress
added statutory language which required EPA to conduct periodic studies to evaluate the benefits and
costs of the Clean Air Act itself. Enhanced credibility and continual improvement in data and methods
were promoted by requiring that the design, implementation, and results of each study would be
reviewed by a multidisciplinary panel of outside experts.
To meet Congress’ goals for the third study in this series of Clean Air Act benefit‐cost analyses, EPA
defined a central objective and three supplementary objectives. Consistent with the central objectives
defined for the two preceding studies, the current study was designed to estimate the direct4 costs and
direct benefits of the Clean Air Act as a whole, including the major federal, state, and local programs
implemented to meet its requirements. The present study focuses on estimating the incremental
effects of the 1990 Amendments in particular, and covers the period from 1990 –when these most
recent Amendments were passed—through the year 2020.
A second, subsidiary objective of the study was to gauge the economy‐wide effects of the 1990 Clean Air
Act programs, including evaluation of the Act’s effects on the overall growth of the U.S. economy and
the economic well‐being of American households.
In this study, “direct” costs or benefits refer to first‐order economic effects of pollution control programs. For
example, the expenditure of funds to purchase, install, and operate pollution control equipment is considered a
direct cost of a pollution control program. Similarly, the reduction in risk of a pollution‐related health effect is a
direct benefit of the reduction in emissions achieved by the use of that equipment. Indirect effects are those
which emerge as consequences of the direct effect, such as the higher cost of producing steel if the direct cost to
an electric utility of installing pollution control equipment leads to an increase in electricity prices paid by a steel
plant. An example of an indirect benefit is the improvement in worker productivity achieved when the direct
benefit of avoiding pollution‐related illness helps workers avoid sick days. The present study focuses on evaluation
of direct benefits and costs but also, to a limited extent, assesses indirect effects through economy‐wide modeling.
The Benefits and Costs of the Clean Air Act from 1990 to 2020: Summary Report
EPA also sought, as a third objective, to be as comprehensive as possible –subject to practical limitations
imposed by budget and information constraints—by considering a wide range of human health, human
welfare (i.e., quality of life), and ecological effects. While some of these effects may contribute only
minimally, if at all, to the quantitative estimates of benefits and costs generated for this study, looking
at a broad range of effects was intended to ensure that (a) effects of concern to various stakeholders
were included and (b) EPA and outside researchers could obtain additional insights about any
deficiencies in the scope and quality of current information.
A fourth and final objective of the current study was to assess its limitations and uncertainties to identify
opportunities for improving data and methods, and to explore the need for refining the scope and
design of future air pollution benefit‐cost studies. External peer review by the outside experts serving
on the Council was a critical aspect of efforts to meet this objective, as well as the other objectives of
this study.
Study Design
The current study is similar to the previous two in its fundamental design. To isolate the effects of Clean
Air Act programs, the study configures and compares two alternative states of the world: one with the
1990 Clean Air Act Amendments, and one which assumes the 1990 Amendments were not passed.
In particular, the first scenario was built to reflect the actual history of post‐1990 Clean Air Act
implementation, including known programs already established, and future programs and control
strategies anticipated in the later years of the study period. This scenario was called the “with 1990
Clean Air Act Amendments scenario,”
or With‐CAAA case for short, and it
represents a world of lower emissions
but higher costs following enactment
of the 1990 Clean Air Act Amendments.
The With‐CAAA case is represented by
the lower line in Exhibit 3, which
depicts a not‐to‐scale schematic
illustrating the scenarios analyzed.
The second, contrasting scenario
reflects a hypothetical world which
assumes federal Clean Air Act and
related programs were frozen as of
November 1990, the month the
Amendments were signed into law.
Therefore, 1990 serves as the “base
year” of the analysis when the two
Exhibit 3. Second Prospective Study scenarios conceptual
schematic. This exhibit is a schematic depiction of the scenarios to
illustrate their timing and conceptual foundations. The differences
in emissions between studies and between years are not to scale
and should not be viewed as a comparison of emission reductions
achieved between studies or between years.
The Benefits and Costs of the Clean Air Act from 1990 to 2020: Summary Report
scenarios are initially set as equal but then begin to diverge. The counterfactual scenario was called the
“without 1990 Clean Air Act Amendments scenario,” or Without‐CAAA case. The hypothetical Without‐
CAAA case is represented in Exhibit 3 by the upper 1990 to 2020 trend line showing the higher emissions
which would result if standards stayed fixed but the economy and the population of the U.S. grew over
the 1990 to 2020 period.
Once they were configured, the With‐CAAA and Without‐CAAA scenarios were processed through a
series of economic and physical effects models, and their differences were estimated and compared.
Specifically, each scenario was analyzed using a sequence of models to estimate what pollution control
measures were (or might be) taken by government, private industry, and individuals; and what the
effects of those measures might be in terms of economic and environmental change. The sequence of
modeling steps followed to analyze the two scenarios is shown in Exhibit 4. Detailed descriptions of
each analytical step –including the particular data, models, and methodologies used and their attendant
uncertainties—are provided in the full integrated report and supporting technical documents.
One consequence of this sequential modeling approach is that the scenarios were defined early in the
study. As such, this study reflects a particular snapshot in time with respect to known and anticipated
control programs, especially those incorporated in the With‐CAAA scenario. Several important
programs, however, have been initiated or revised since the analytical scenarios were locked for the
study in late 2005. For example, the With‐CAAA scenario reflects the Clean Air Interstate Rule (CAIR)
which had been recently promulgated when the scenarios
were set, but this rule is now being replaced by a different rule
designed to address the problem of long‐range atmospheric
transport of air pollution. Information about the estimated
benefits and costs of recent rules is available in the relevant
EPA Regulatory Impact Analyses.
To ensure high‐quality, credible results, the study used the
best available data and state‐of‐the‐art modeling tools and
methodologies. Most important, the design of the study,
many of the intermediate methodological choices and
findings, and the final results and their interpretation were all
reviewed by the Council and its three technical
subcommittees. The specialized expert review of the
emissions and air quality, human health effects, and ecological
effects study components by the three technical
subcommittees complemented and supported the Council’s
broad expertise, which included substantial expertise in
Exhibit 4. Analytical sequence of the
Second Prospective Study. This flowchart
shows the order of the major analytical
steps followed to conduct the study.
The Benefits and Costs of the Clean Air Act from 1990 to 2020: Summary Report
Primary Results
Direct Cost
Compared to the baseline scenario without
the 1990 Clean Air Act Amendments and
related programs, the With‐CAAA scenario
adds controls across five major categories
of emission sources. All significant
emissions sources are assigned to one of
these five major source categories. Two of
these categories cover stationary point
sources of emissions, two cover mobile
sources, and the fifth category covers
smaller sources dispersed over wide areas.
The categories are:
1. Electricity generating units (e.g., coal‐
fired power plants)
2. Non‐utility industrial sources (e.g.,
industrial boilers, cement kilns)
3. Onroad vehicles and fuel (e.g., cars,
buses, trucks)
4. Nonroad vehicles and fuel (e.g.,
aircraft, construction equipment)
5. Area sources (e.g., wildfires,
construction dust, dry cleaners)
Electricity generating units
Non‐utility industrial
Onroad vehicles & fuel
Nonroad vehicles & fuel
Area sources
Local controls
Additional controls to
meet national standards
Exhibit 5. Year 2020 direct cost of compliance by source
category. (In billions of year 2006 value dollars). The first five
darker orange bars show how compliance costs compare for
the five major categories of emissions source. Additional
controls applied to these five source categories at the local
level for the purposes of meeting air quality standards are
shown by the sixth darker orange bar. The seventh, lighter
orange bar also reflects additional local controls but these are
shown separately because their costs are significantly more
The costs incurred to reduce emissions
from these sources under the 1990 Clean
Air Act Amendments are estimated to rise
steadily throughout the 1990 to 2020 study
period. By 2020, the study target year
when differences between the With‐CAAA and Without‐CAAA scenarios are at their greatest, additional
annual compliance expenditures are estimated to be about $65 billion (in year 2006 value dollars).
As shown in Exhibit 5, these incremental costs of compliance did not fall evenly across the five major
source categories. Almost half of the year 2020 direct costs are to meet requirements for onroad
vehicles and the fuels used to operate them. About 40% of the $28 billion in onroad expenditures is to
meet fuel composition requirements and the rest is incurred to meet tailpipe standards and implement
vehicle inspection and maintenance programs.
The Benefits and Costs of the Clean Air Act from 1990 to 2020: Summary Report
Electric utilities account for the second largest area of expenditure, with costs in the year 2020 equal to
a little over $10 billion. The programs leading to the bulk of these expenditures include the Title IV acid
rain sulfur dioxide allowance trading program, the Clean Air Interstate Rule, programs targeted at
reducing nitrogen oxide emissions (e.g., the NOx SIP Call), and controls required to meet the national
ambient air quality standards for fine particles and ozone.
Implementation of federal and regional control programs to meet the national fine particle and ozone
standards accounts for much of the cost incurred by the five major emissions source categories.
However, for many local areas, emissions reductions achieved by these programs are not sufficient to
reach attainment with national air quality standards. Under the Clean Air Act, these local areas are
required to implement additional controls tailored to their particular needs and opportunities for the
further emission reductions needed to improve air quality and attain the national standards.
Expenditures for local controls which could be identified as both suitable for a given location and cost‐
effective to implement were estimated to reach about $6 billion by 2020.
By the year 2020, reaching the 8‐hour National Ambient Air Quality Standard (NAAQS) for ozone in some
locations appears to be a significant challenge. Some of these locations are assumed under the With‐
CAAA scenario to apply all controls identified as technologically feasible and cost‐effective for their
location yet still show modeled ozone concentrations higher than the 8‐hour national standard. The
With‐CAAA scenario therefore assumes additional emissions reductions are achieved using “unidentified
controls” of unknown cost and/or technological availability and applicability. Since the particular control
strategies for each of these locations cannot currently be identified, their costs are highly uncertain. The
With‐CAAA scenario assumes that the additional emissions reductions achieved by unidentified controls
will cost $15,000 per ton. The $15,000 per ton assumed value could turn out to be too high or too low
depending on local circumstances and the prospects for near‐term improvements in control
technologies and cost, although there is some evidence that local areas would be reluctant to
implement measures that cost more than $15,000 per ton. The total incremental cost of these
additional local controls using unidentified technologies is estimated to be $13 billion. Given the
relatively high level of uncertainty in this component of Clean Air Act program compliance costs, it is
reported as a subtotal separate from the identified control measures subtotal of $52 billion.
Emissions Reductions
The controls applied across the major categories of emissions sources under the With‐CAAA scenario
achieve substantial reductions in emissions contributing to ambient concentrations of fine particles,
ozone, and other air pollutants. As shown in Exhibit 5, the total costs of control from some sectors –
such as electricity generating units and onroad vehicles and fuels—were high relative to other source
categories, but these sources also achieved the greatest reductions in emissions. For example, onroad
vehicles and fuel represent 46% of total control costs in 2020 but they also contribute 41% of the year
2020 reduction in total NOX emissions. The full range of emissions reductions estimated under the With‐
CAAA case and the breakdown by source category are described in the full report, but the overall
The Benefits and Costs of the Clean Air Act from 1990 to 2020: Summary Report
In addition to directly‐emitted fine
particles,5 three other pollutants
designated for control under the Clean
Air Act contribute to increases in
ambient concentrations of fine
particles through secondary formation
and transport in the atmosphere. 6 For
example, gaseous sulfur dioxide can
be transformed in the atmosphere to
particulate sulfates. Volatile organic
compounds (VOCs) and nitrogen
oxides are also key pollutants
contributing to the formation of
ground‐level ozone.
Millions of tons
reductions in pollutants which
contribute most to changes in fine
particles and ozone are highlighted in
Exhibit 6.
The estimated With‐CAAA scenario
emissions reductions depicted in
Exhibit 6 are large because they reflect
both absolute reductions relative to
1990 base year conditions and
avoided increases in emissions which
Exhibit 6. Year 2020 key pollutant emissions under the With‐CAAA
result under the Without‐CAAA case
and Without‐CAAA scenarios. (In millions of short tons). The
when standards stay fixed at 1990
difference in height between the orange and blue bars for each
pollutant shows the estimated reduction in that pollutant achieved
levels but economic activity increases
from 1990 to 2020. Approximately 75 by 1990 Clean Air Act Amendment programs.
percent of the 2020 emissions
reductions are attributable to improvements relative to 1990, while the remaining 25 percent is
attributable to avoiding increases in emissions that could result if Clean Air Act standards stay fixed
while population and economic activity grow.
Fine particle pollution already in a solid or liquid aerosol state at the point of emission from a tailpipe or
construction site is commonly referred to as “directly emitted fine particles,” or sometimes “primary particles.” In
contrast, fine particles which form in the atmosphere later from gaseous precursors, such as sulfur dioxide, are
referred to as “secondary fine particles.”
In recent years the importance of ammonia in secondary formation of fine particle air pollution has become
clearer. However, unlike the other pollutants shown in Exhibit 6, ammonia is not currently a designated air
pollutant under the Clean Air Act, and there are no explicitly assumed differences in control requirements for
ammonia between the With‐CAAA and Without‐CAAA scenarios.
The Benefits and Costs of the Clean Air Act from 1990 to 2020: Summary Report
Most of the reduction in volatile organic compounds is achieved by controls on evaporative emissions
from area sources such as household solvents, controls on vehicle and nonroad engine tailpipe and
evaporative emissions, and controls on non‐utility industrial sources.
For nitrogen oxide emissions, all five major source categories achieve emissions reductions under the
With‐CAAA scenario; but the most substantial contributions to lower emissions are attributable to
tailpipe standards for onroad vehicles and reductions achieved by utilities subject to cap‐and‐trade
programs and/or the Clean Air Interstate Rule. Requirements related to the national standards for fine
particles also reduce nitrogen oxides emissions.
Electricity generating units such as coal‐fired power plants are the source category which achieves the
most significant reductions in sulfur dioxide emissions, accounting for about 75 percent of the total
reduction projected in 2020. Cap‐and‐trade programs, the Clean Air Interstate Rule, and other control
programs implemented pursuant to the national fine particle standards account for most of the
estimated difference in sulfur dioxide emissions between the With‐CAAA and Without‐CAAA scenarios.
About 40 percent of the year 2020 reduction in directly‐emitted fine particles is achieved by controls on
area sources such as construction dust and residential woodstoves. Reductions from utilities and from
nonroad and onroad sources also contribute toward meeting the requirements of the national ambient
air quality standards for fine particles.
Air Quality Improvements
The substantial reductions in emissions which contribute to ambient concentrations of ozone and fine
particles lead to significant differences in modeled air quality conditions under the With‐CAAA and
Without‐CAAA scenarios. Air quality modeling results for all pollutants and all target years analyzed in
this study are available in the full report, though the estimated change in fine particle concentrations is
highlighted here because reductions in exposure to this pollutant are responsible for the vast majority of
benefits which could be evaluated in economic terms for this study.
Exhibit 7 shows that reductions in fine particle concentrations by 2020 are large and widespread, as
demonstrated by the pervasive blue colors indicating improvement in air quality. The most significant
reductions occur in California and the Eastern U.S., especially the Ohio Valley region, primarily due to
sulfur reductions from electric utilities and industrial facilities combined with mobile source reductions
concentrated around heavily‐populated metropolitan areas. Because these areas had relatively high
fine particle concentrations in the 1990 base year, the modeling results imply that 1990 Clean Air Act
Amendment programs were effective in targeting high emissions sources in and around locations where
improvements in air quality would benefit the greatest number of people. There are a few locations in
the West where fine particle concentrations are estimated to be slightly higher in 2020 under the With‐
CAAA scenario due to localized effects related to electrical generating unit dispatch or fuel choice.
These localized disbenefits, shown by the isolated spots of orange color in Exhibit 7, are negligible
The Benefits and Costs of the Clean Air Act from 1990 to 2020: Summary Report
Exhibit 7. Difference in annual average fine particle (PM2.5) concentrations between the With‐CAAA and
Without‐CAAA scenarios: With‐CAAA minus Without‐CAAA for 2020. (In micrograms per cubic meter). The
map shows the change in concentrations of fine particles in the atmosphere achieved by 1990 Clean Air Act
Amendment programs. The darker the blue color, the greater the improvement in air quality. The few spots
of orange on the map are isolated locations where the air quality model projected slightly higher fine particle
concentrations under the With‐CAAA scenario than under the Without‐CAAA scenario.
compared to the large and widespread overall reductions in fine particle pollution under the With‐CAAA
Ozone concentrations are also significantly lower overall under the With‐CAAA scenario relative to the
Without‐CAAA scenario. As shown by maps provided in the full report, the patterns of air quality
improvements for ozone are similar to those observed for fine particles with widespread regional
improvements across the East and improvements in the West occurring predominantly in areas
influenced by Southern California population centers.
The Benefits and Costs of the Clean Air Act from 1990 to 2020: Summary Report
Health Improvements
The steady improvements in air quality estimated under the With‐CAAA case from 1990 to 2020 period
lead to increasing health and environmental benefits over the entire study period. By 2020, the
differences in air quality and human health outcomes between the With‐CAAA and Without‐CAAA
scenarios are considerable.
Fine Particle and Ozone
The largest reductions in fine
particle concentrations are
achieved in areas with relatively
poor air quality and/or high
population density (see Exhibit
7). This result is due in large part
to the effective design of federal,
state, and local programs aimed
at meeting ambient air quality
standards in ways which
maximize public health
improvements. The
effectiveness of these programs
in achieving well‐ targeted
reductions in exposure means
that the differences in health
outcomes between the With‐
CAAA and Without‐CAAA
scenarios are substantial, even
For example, as early as 2000,
annual average exposures7 to
Health Effect Reductions
(PM2.5 & Ozone Only)
Year 2010
Year 2020
PM2.5 Adult Mortality
PM2.5 Infant Mortality
Chronic Bronchitis
Acute Bronchitis
Acute Myocardial Infarction
Asthma Exacerbation
Hospital Admissions
PM, Ozone
Emergency Room Visits
PM, Ozone
Restricted Activity Days
PM, Ozone
School Loss Days
Lost Work Days
Ozone Mortality
Exhibit 8. Differences in key health effects outcomes associated with fine
particles (PM2.5) and ozone between the With‐CAAA and Without‐CAAA
scenarios for the 2010 and 2020 study target years. (In number of cases
avoided, rounded to 2 significant digits). The table shows the reductions
in risk of various air pollution‐related health effects achieved by 1990
Clean Air Act Amendment programs, with each risk change expressed as
the equivalent number of incidences avoided across the exposed
“Average exposure” in this case refers to “population‐weighted annual average exposure,” which is calculated by
dividing the total population exposure over the course of a year by the total number of people in the exposed
population. This measure provides a helpful summary indicator of overall exposures and exposure changes, in this
case across all people living in the 48 states; though people living in particular locations may experience much
higher or much lower exposures or exposure changes than people in other locations. To illustrate, consider a
population of three people where two people experience a change in exposure from 30 to 10 micrograms per
cubic meter (µg/m3), and a third person’s exposure change is from 25 to 20 µg/m3. The change in population‐
weighted average exposure would be (20+20+5) divided by 3 = 15 µg/m3. While the 15 µg/m3 change provides a
The Benefits and Costs of the Clean Air Act from 1990 to 2020: Summary Report
fine particles among the U.S. population are lower by an average of 5 micrograms per cubic meter under
the With‐CAAA scenario. By 2020, the average exposure difference between the scenarios increases to
an estimated 9 micrograms per cubic meter, all as a result of programs related to the 1990 Clean Air Act
Amendments. This 9 microgram per cubic meter reduction is tantamount to cutting exposures almost in
half, because the population average exposure for 2020 under the Without‐CAAA scenario is slightly
higher than 19 micrograms per cubic meter.8
The large reduction in risk of premature mortality associated with fine particles is the most significant
outcome among those listed in Exhibit 8. Ozone health studies also indicate there is a separate, additive
contribution to reduced premature mortality risk from this pollutant beyond the premature mortality
effect associated with fine particle exposures. This study’s estimates for these incidence reductions are
based on a strong and extensive foundation of peer‐reviewed epidemiological literature. The
methodologies used to apply these epidemiological studies to the estimation of reduction in population
risks from fine particle and ozone exposure have also been extensively peer‐reviewed.
In addition to reductions in incidences of premature mortality,9 reductions in exposure to fine particles
and ozone are also estimated to achieve major reductions in serious diseases such as chronic bronchitis
and acute myocardial infarction, as well as fewer hospital admissions, emergency room visits, lost work
days, and lost school days.
Hazardous Air Pollutants
Controls on emissions of hazardous air pollutants, including heavy metals and toxic gases, are known to
reduce adverse health effects, though data and tools to quantify the full extent of the reductions in
health risks from these pollutants are limited. A case study assessing the effects of the 1990 Clean Air
Act Amendments in reducing benzene emissions and exposures in the Houston area was conducted as
part of this study. The study found a significant cancer‐reducing benefit overall in the region, but also
found that 1990 Clean Air Act Amendment programs led to the most substantial reductions in those
areas with the highest baseline cancer risks. These results are described in detail in the full report and in
a separate technical report documenting the Houston benzene case study.
useful measure of the shift in overall population exposure, it may obscure the fact that the third individual
experienced a significantly smaller improvement and is left with a significantly higher residual exposure.
For perspective, this level of population‐wide annual average fine particle exposure is about the same as that
experienced by people living in Los Angeles in the year 2000. (See Text Box 4‐1 of the full report.)
The term “incidence” is not intended to represent premature mortality of a particular known individual, but
rather small reductions in risk experienced by many people that sum to an aggregate change in population risk
numerically equivalent to one avoided premature mortality.
The Benefits and Costs of the Clean Air Act from 1990 to 2020: Summary Report
Other Clean Air Act Pollutants
Reductions in ambient concentrations of other Clean Air Act pollutants such as carbon monoxide also
confer health benefits, though many of these benefits are difficult to quantify for various reasons. For
example, in the case of carbon monoxide, available health studies are not well suited to isolating the
incremental contribution of carbon monoxide reductions to improved health when significant
reductions in other pollutants, such as fine particles, are modeled at the same time. Furthermore,
health effects of some pollutants can be quantified in physical terms but economic studies supporting
valuation of the changes in physical outcomes are unavailable. Whether the limits on quantification of
these other criteria pollutant10 effects emerge at the physical effect or economic valuation step, the
result is that these effects are not reflected in the primary estimates of health improvements presented
in this report.
Other Benefits to People and the Environment
Beyond the direct health benefits of Clean Air Act programs, a variety of other improvements to human
well‐being and ecological health are assessed in this study. Efforts to evaluate these other “non‐health”
effects were motivated by the study’s goal of providing ins…
Purchase answer to see full

Why Choose Us

  • 100% non-plagiarized Papers
  • 24/7 /365 Service Available
  • Affordable Prices
  • Any Paper, Urgency, and Subject
  • Will complete your papers in 6 hours
  • On-time Delivery
  • Money-back and Privacy guarantees
  • Unlimited Amendments upon request
  • Satisfaction guarantee

How it Works

  • Click on the “Place Order” tab at the top menu or “Order Now” icon at the bottom and a new page will appear with an order form to be filled.
  • Fill in your paper’s requirements in the "PAPER DETAILS" section.
  • Fill in your paper’s academic level, deadline, and the required number of pages from the drop-down menus.
  • Click “CREATE ACCOUNT & SIGN IN” to enter your registration details and get an account with us for record-keeping and then, click on “PROCEED TO CHECKOUT” at the bottom of the page.
  • From there, the payment sections will show, follow the guided payment process and your order will be available for our writing team to work on it.