This report is a joint project conducted by the Project for Excellence in Journalism (PEJ) funded by the Pew Charitable Trusts and the Joan Shorenstein Center on the Press, Politics and Public Policy at the Kennedy School of Government at Harvard University.

PEJ and Shorenstein designed the study, analyzed the findings and wrote the report in conjunction. The content analysis was conducted by PEJ staff with the financial support of both the Shorenstein Center and the Pew Charitable Trusts. Professor Marion Just was the lead researcher from The Shorenstein Center.

Sample Design

The content was based on media coverage originally analyzed for PEJ’s weekly News Coverage Index (NCI) from January 1-March 9, 2008. 

Each week, the NCI examines the coverage from 48 different outlets in five media sectors, including newspapers, online news, network TV, cable TV, and radio. Following a system of rotation, 35 outlets each weekday are studied as well as 7 newspapers each Sunday.

For this particular study of campaign coverage, ABC and CBS radio headlines were excluded. Therefore, the 46 media outlets examined for this campaign study were as follows:

Newspapers (13 in all)
The New York Times was coded every day

Coded two out of these four every day
The Washington Post
Los Angeles Times
USA Today
The Wall Street Journal

Coded two out of these four every day
The Boston Globe
Star Tribune (Minneapolis)
Austin American-Statesman
Albuquerque Journal

Coded 2 out of these 4 every day
The Sun Chronicle (Attleboro, MA)
Star Beacon (Ashtabula, Ohio)
Chattanooga Times Free Press
The Bakersfield Californian

Web sites (Five in all, Mon-Fri)

CNN.com
Yahoo News
MSNBC.com
Google News
AOL News

Network TV (Seven in all, Mon-Fri)

Morning shows
ABC – Good Morning America
CBS – Early Show
NBC – Today

Evening news
ABC – World News Tonight
CBS – CBS Evening News
NBC – NBC Nightly News
PBS – Newshour with Jim Lehrer

Cable TV (Fifteen in all, Mon-Fri)

Daytime (2:00 to 2:30 pm) coded 2 out of 3 every day
CNN
Fox News
MSNBC

Nighttime CNN – coded 3 out of the 4 every day
Lou Dobbs Tonight
Situation Room (6 pm)
Out in the Open/CNN Election Center
Anderson Cooper 360

Nighttime Fox News – coded 3 out of the 4 every day
Special Report w/ Brit Hume
Fox Report w/ Shepard Smith
O’Reilly Factor
Hannity & Colmes

Nighttime MSNBC – coded 2 out of the 4 every day
Tucker (6 pm)
Hardball (7 pm)
Countdown w/ Keith Olbermann
Live with Dan Abrams

Radio (Six in all, Mon-Fri)
NPR Morning Edition every day *
*From January 1 to 11, we coded the first half-hour of “Morning Edition” (5:00-5:30 am); from Jan 14 on, we coded following a rotation between the first half-hour (5:00-5:30 am) and the first 30 minutes of the second hour (6:00-6:30 am).

Talk Radio
Rush Limbaugh was coded every day
One out of two additional conservatives each day
Sean Hannity
Michael Savage
One out of two liberals each day
Ed Schultz
Randi Rhodes

From that content, the study included all campaign-related stories:

  • On the front page of newspapers
  • In the entirety of commercial network evening newscasts.
  • The first 30 minutes of network morning news, the PBS evening news, and all cable programs
  • The first 30 minutes of the talk radio programs and a 30 minute segment of NPR’s Morning Edition
  • The top 5 stories on each website at the time of capture

The basic NCI codebook codes for topic at three different levels, and includes the following variables: date coded, Story ID number, story date, source, broadcast start time, broadcast story start timecode, headline, story word count, placement/prominence, story format, story describer, broadcast story ending timecode and lead newsmaker. Since January 1, 2008, three additional variables were analyzed for campaign stories in the NCI routine coding. These included the variables campaign lead newsmakers, significant presence and presidential campaign topic. The complete methodology for the weekly NCI and CCI, a sub-set of NCI, has further details on the coding system and intercoder reliability.

Sample Selection


Stories from January 1 through March 9, 2008

To arrive at the sample for this particular study of campaign coverage, we began by pulling all the stories from January 1 through March 9, 2008, that were originally coded as campaign stories, meaning that 50% or more of the story was devoted to discussion of the ongoing presidential campaign. From that group, we selected stories that focused on at least one of the five candidates (Mitt Romney, John McCain, Mike Huckabee, Hillary Clinton, Barack Obama) for 25% or more of the time or space of the story.

All newspaper and online stories were then included. For broadcast, stories 30 seconds or less was removed from the sample. For the stories from cable TV and the three major networks (ABC, CBS, and NBC) further sampling was conducted by selecting every other campaign story by outlet. This was done by listing the stories from each show in chronological order and randomly selecting the first story. We then selected every-other story within each outlet to arrive at the final sample of television stories.

This process resulted in the following sample size for the stories from 2008: 2,590 total stories, including 413 newspaper stories, 281 stories from news websites, 540 stories from network TV, 984 from cable TV, and 372 from radio programs. Out of these stories, we discovered and coded 5,374 total assertions.

Stories about John McCain from 2007

This study also examined John McCain’s major narrative themes throughout 2007. To gather the stories focused on McCain in 2007, we selected all the campaign stories in the weekly News Coverage Index sample that were about the campaign and included McCain as 50% of the story or more.

For all of 2007, 239 stories were selected that focused on McCain and the campaign, which resulted in 199 McCain narrative assertions.

Coding Design

A coding protocol was designed for this project based on previous studies by PEJ and Professor Just and the particular aims of the PEJ-Shorenstein study.

For each of the five major presidential candidates (Barack Obama, Hillary Clinton, John McCain, Mitt Romney and Mike Huckabee) a survey of campaign coverage was conducted and the coding team, Professor Just, and senior PEJ staff arrived at four or more common themes that were appearing frequently within the media coverage about each candidate. These themes became the assertions that the coders were looking to identify within each campaign story in the study’s sample. In some instances, new master themes emerged after coding had begun and were added to the analysis.


Unit of Analysis
The unit of analysis for this study was the assertion (or thread/theme). Any time a coder came across an assertion within one of the stories that were coded, he or she noted the assertion and coded the following variables for each assertion. Because the unit of analysis was the assertion and not the story, many stories had multiple assertions within them, and some stories may not have had any relevant assertions for this study.


Variables
In addition to certain existing variables from the NCI and CCI projects, the variables included in this study were the following: theme/message, affirming the narrative thread or rebuttal, significance of the statement within the story, and variable source of statement.


Theme/message are the overarching themes that the press, candidates, and operatives focus on and are usually centered on a candidate’s character. As noted above, in most cases the master themes about each of the five candidates were identified and tested prior to coding. This variable measures which of the themes the statement addressed.


Affirming the narrative thread or rebuttal captures whether the assertion being coded is affirming the narrative thread or refuting it.


Significance of the statement within the story indicates whether the thread was the central theme of the story or whether it was just a passing reference in a story about something else. For a thread to be a significant part of a story, 50% or more of the story must relate directly to that thread.


Source of statement designates the person who is making the statement. This is not necessarily the author of the piece, but the person who expresses the particular statement. For example, if a newspaper story quotes an unnamed voter as asserting that, “Senator Obama represents change,” that means the source of the statement is the voter and not the journalist writing the piece.

Coding Team & Process

Using the existing data in the Index and adding the codes for new variables, the team responsible for performing the content analysis on this particular study was made up of five trained coders, a coding administrator, and a senior research methodologist on the PEJ staff.

We have tested all of the variables derived from the regular weekly Index coding and all the variables reached a level of agreement of 80% or higher. For specific information about those tests, see the methodology section for the NCI.

During coder training for this particular study, intercoder reliability tests were conducted for all the campaign-specific variables. There were two different tests conducted to assure reliability. The first test was to assure that coders could identify the assertions within campaign stories. Each coder was given the same group of stories and asked to identify where threads appeared in those stories. The agreement between all the coders on this task was above 80%, however, some further training was conducted with each coder to make sure that the instances of disagreement were identified and understood prior to the start of the actual coding for the study.

The second test consisted of each coder being given a list of assertions and asked to code each of the campaign specific variables for those threads.

From that test, the specific levels of agreement for the variables in this study were as follows:
Thread/statement: 90%
Affirming narrative thread or rebuttal: 95%
Significance of statement within the story: 80%
Source of statement: 86%

Additional Analysis

The results of PEJ’s coding were twinned with a companion survey of public attitudes about the candidates by the Pew Research Center for the People and the Press. The survey was conducted twice, on February 8-11 and April 18-21, 2008, to gauge shifts in public resonance about the candidates. Together, the two studies allowed us to explore how much these press messages were shaping public opinion of the candidates. Learn more about the News Interest Index.