Focus of the Study

This Portrait From Iraq Study is a special report based on additional analysis of content already aggregated in PEJ’s weekly News Coverage Index (NCI).

In our weekly analysis of press coverage, PEJ identifies three different storylines of the Iraq war: events on the ground, the policy debate (usually focused in Washington, DC), and the homefront (impact of the war on those in the United States).

For this study, we wanted to closely examine one of the three areas of coverage: namely, coverage of events inside Iraq.

The time period studied was January 1 through October 31, 2007.

Sample Design

For this study of Iraq war coverage, 40 of the 48 outlets included in the weekly NCI reports were studied further. Talk radio programs (Rush Limbaugh, Ed Schultz, Randi Rhodes, Sean Hannity, and Michael Savage) and news radio headlines (ABC News radio and CBS news radio were excluded. See full list of outlets below.)

To create the universe sample, we first identified all stories in those outlets coded as “Iraq war, events on the ground” from January 1 through October 31, 2007.

According to the weekly coding from the NCI, between January 1 and October 31, there were 3,477 stories that were primarily about events on the ground in Iraq. Stories are coded as being about events on the ground in Iraq if 50% or more of the story is devoted to that topic.

From there, we selected one third of the 3,477 stories for additional analysis.  To identify the stories for additional analysis, we first selected a random set of days for each medium out of the days we had already coded for in the Index (which includes our rotation of outlets). After selecting the days for each outlet, we would then code any stories that were coded as being about events on the ground in Iraq that appeared on those particular days.

To arrive at our days to consider for each outlet, we randomly selected either the first, second, or third day of the year (beginning in January) that we had coded that material. We then proceeded to include every third day of that outlet for the rest of our time sample.

We did this same method for each of the outlets in our sample with the exception of the New York Times. Since the New York Times is the only outlet we code six days a week, selecting every third day would have meant that we would have only had stories from the Times that appeared on the same days of the week throughout (Sundays and Wednesdays, for example). Instead, we selected two days out of the first week of January that were three days apart, and rotated those days of the week each subsequent week. For example, on the first week we randomly selected Monday and Thursday, and then the next week we rotated and included Tuesday and Friday.

From this random selection of stories, we arrived at a sample size of 1,106 stories. In this selection of stories, there were no stories that had appeared on the O’Reilly Factor on Fox News. According to our overall Index coding, the O’Reilly Factor had only run 3 stories focused on events on the ground in Iraq during the first 30 minutes of the shows we coded in our rotation. Because of the large viewership of the O’Reilly Factor, we decided to include all of these 3 stories into our overall Iraq study sample despite the fact that they did not appear in our randomly selected days. With these O’Reilly stories, the final number of stories in the study was 1,109.

Outlets Examined:

The media outlets examined are as follows:

Newspapers (Twelve)

NY Times
Washington Post
Los Angeles Times
USA Today
Wall Street Journal
The Boston Globe
Star Tribune (Minneapolis)
Austin American-Statesman
Albuquerque Journal
Star Beacon (Ashtabula, Ohio)
The Chattanooga Times Free Press
The Bakersfield Californian

Web sites (Five)

CNN.com
Yahoo News
MSNBC.com
Google News
AOL News

Network TV (Seven)

Morning shows

ABC – Good Morning America
CBS – Early Show
NBC – Today

Evening news

ABC – World News Tonight
CBS – CBS Evening News
NBC – NBC Nightly News
PBS – Newshour with Jim Lehrer

Cable TV (Fifteen)

Daytime

CNN
Fox News
MSNBC

*From Jan. 1 to March 16, we coded from 1-1:30 p.m. EST; from March 19 on, we coded from 2-2:30 p.m. EST

Nighttime CNN

Lou Dobbs Tonight
Situation Room
Paula Zahn Now
Anderson Cooper 360

Nighttime Fox News

Special Report w/ Brit Hume
Fox Report w/ Shepard Smith
O’Reilly Factor
Hannity & Colmes

Nighttime MSNBC

Tucker
Hardball
Countdown w/ Keith Olbermann
Scarborough Country

Radio

NPR Morning Edition every day

From that content, the study included stories devoted primarily to events in Iraq:

  • On the front page of newspapers
  • In the entirety of commercial network evening newscasts.
  • The first 30 minutes of network morning news, the PBS evening news, NPR’s Morning Edition, and all cable programs
  • The top 5 stories on each website at the time of capture

The basic NCI codebook codes for topic at three different levels, and also includes date coded, Story ID number, story date, source, broadcast start time, broadcast story start timecode, headline, story word count, placement/prominence, story format, story describer, lead newsmaker, and broadcast story ending timecode. The complete methodology for the weekly NCI has further details on the coding system and inter-coder reliability.

Coding Design

In addition to the existing variables in the NCI, the Iraq study included a variety of variables designed to probe about the coverage of the events on the ground. These included dateline, Iraq sub storyline, range of stakeholders, predominant source, and theme.

Dateline designates the province, city, or country that was given as the dateline of the story.

Iraq sub storyline measures the broad Iraq-war related topic, or what the story is about “on its face.” The list of topics for this variable was determined through a combination of the questions that appeared in the PEJ survey of journalists who have reported from Iraq and also from observations made by PEJ staff over the past ten months.

Range of stakeholders measures the effort of the news organization to present viewpoints and opinions from varying groups of people involved in the Iraq conflict. Examples of these stakeholders are groups such as Iraqi civilians, members of the U.S. military, representatives from the White House, and members of the Iraqi parliament.

Dominant source identified which group was the predominant source in such stories where the views expressed came primarily from one specific group.

Theme designates the existence of evaluations of five main storylines within any given news story. This variable measured whether a story had any explicit language that indicated a positive or negative development in regard to the following five issues: the success of the U.S. policy in Iraq, troop morale, the stability of the Iraqi government, the views of Iraqis about the presence of U.S. troops, and the stability of the country as a whole. A story did not have to be entirely about these topics to be coded as a theme, however, if several different evaluations about themes were mentioned, we coded for the theme that was given the most attention in the story. If none of the above storylines were evaluated in the story, this variable was used to determine whether the story was an account of violence that did not address any of the five themes, a straight recitation of facts regarding some topic other than violence, or whether the story evaluated themes other than the five being tracked in this specific study.

Coding Team & Process

Using the existing data in the Index and adding the codes for new variables, the team responsible for performing the content analysis was made up of the four trained coders, a coding administrator, and a senior research methodologist on the PEJ staff.

During coder-training, inter-coder reliability tests were conducted for all variables. For the variables derived from PEJ’s weekly Index, the average level agreement was 96%. For the new variables, initial tests found levels of agreement of 80% or greater for all but three variables (Iraq sub storyline, range of stakeholders and theme). For those three variables, clarifications were made to the codebook and additional training implemented. An additional test was conducted and all three variables reached a level above 80% agreement. In total, testing was conducted on 5% of the sample. In addition, the coding administrator monitored coding throughout the process to ensure levels were maintained.

The specific levels of agreement for the variables in this study were as follows:

Dateline: 95%

Lead newsmaker: 87%

Iraq sub storyline: 84%

Range of stakeholders: 87%
    
Dominant source (when there was agreement on the range of stakeholders): 89%

Theme: 83%