A number of members of the PEJ staff assisted in the production of this report, “When Technology Makes Headlines: The Media’s Double-Vision About the Digital Age.”
The team leaders on the project were research analyst Kenny Olmstead and Senior Researcher Paul Hitlin. Sovini Tan, Mahvish Khan and Josh Appelbaum aided in the research, including coding and content analysis of 437 stories about technology.
Other staff members who made substantial contributions to the report were: weekly News Index manager Tricia Sartor, analyst/coder Laura Houston Santhanam, deputy director Amy Mitchell and director Tom Rosenstiel.
When Technology Makes Headlines: The Media’s Double-vision About the Digital Age was conducted in three parts. The first was made up of coding from PEJ’s weekly News Coverage Index (NCI). That sample analyzed data from June 1, 2009 through June 30, 2010 and is referred to here as the broad sample. It consists of all news stories including those that were about subjects unrelated to technology.
The second part of the study includes a closer examination of a sub-section of technology-related stories as they were originally coded in the NCI. This sample covers the time period of June 1, 2009 through June 30, 2010 and is referred to here as the technology-focused sample. Details of that process are below.
The third part of the study involves data collected about social media as part of PEJ’s New Media Index. Details of the social media sample are also below.
Details of each of the three analyses follow:
The broad sample included all the outlets that are part of PEJ’s regular News Coverage Index. The complete methodology of the News Coverage Index is available here.
At the beginning of each year, PEJ reexamines the outlets including in the NCI sample and makes changes in order to keep up with circulation and viewership trends in the media.
Because this particular study includes data from both 2009 and 2010, the makeup of the sample differs slightly during those two years.
In 2009, these outlets, along with the methods of rotation, were as follows:
Newspapers (Sun-Fri)
Coded every day
NY Times
Coded two out of these four every weekday and Sunday
Washington Post
USA Today
Los Angeles Times
Wall Street Journal
Coded two out of these four every weekday and Sunday
Kansas City Star
Pittsburgh Post-Gazette
San Antonio Express-News
San Jose Mercury News
Coded 2 out of these 4 every weekday and Sunday
Herald News (MA)
Anniston Star (AL)
Spokesman-Review (WA)
Meadville Tribune (PA)
Web sites (Coded 6 of 12 each day, Mon-Fri)
CNN.com
Yahoo News
MSNBC.com
Google News
AOL News
Foxnews.com
USAToday.com
Washingtonpost.com
ABCNews.com
BBC News (international version)
Reuters.com
NYTimes.com
Morning Network TV (Mon-Fri)
ABC – Good Morning America
CBS – Early Show
NBC – Today
Evening Network TV (Mon-Fri)
ABC – World News Tonight
CBS – CBS Evening News
NBC – NBC Nightly News
PBS – NewsHour with Jim Lehrer (rotated daily between the first 30 minutes and the second 30 minutes of the hour-long broadcast)
Cable TV (Fifteen in all, Mon-Fri)
Daytime (2:00 to 2:30 pm) coded 2 out of 3 every day
CNN
Fox News
MSNBC
Nighttime CNN – coded 2 out of the 4 every day
Situation Room (6 pm)
Lou Dobbs Tonight
CNN Prime Time/Campbell Brown: No Bias, No Bull
Anderson Cooper 360
Nighttime Fox News – coded 2 out of the 4 every day
Special Report w/ Bret Baier
Fox Report w/ Shepard Smith
O’Reilly Factor
Hannity
Nighttime MSNBC – coded 2 out of the 4 every day
The Ed Show/1600 Pennsylvania Ave
Hardball (7 pm)
Countdown w/ Keith Olbermann
Rachel Maddow
News Radio (Mon-Fri)
NPR Morning Edition every day (rotated daily between the first 30 minutes of the first hour and first 30 minutes of the second hour)
ABC Radio headlines at 9am and 5pm
CBS Radio headlines at 9am and 5pm
Talk Radio (Mon-Fri)
Rush Limbaugh every other day
1 out of 2 additional conservatives each day
Sean Hannity
Michael Savage
1 out of 2 liberals each day
Ed Schultz
Randi Rhodes
In 2010, the outlets, along with the methods of rotation, changed to the following:
Newspapers (Sun-Fri)
Coded two out of these four every weekday and Sunday
The New York Times
Los Angeles Times
USA Today
Wall Street Journal
Coded two out of these four every weekday and Sunday
The Washington Post
The Columbus Dispatch
Tampa Tribune
Seattle Times
Coded 1 or 2 out of these 3 every weekday and Sunday
The Day (CT)
Rome News Tribune (GA)
Ventura News (CA)
Web sites (Coded 6 of 12 each day, Mon-Fri)
Yahoo News
MSNBC.com
CNN.com
NYTimes.com
Google News
AOL News
FoxNews.com
USAToday.com
WashingtonPost.com
ABCNews.com
HuffingtonPost.com
Wall Street Journal Online
Morning Network TV (Mon-Fri)
ABC – Good Morning America
CBS – Early Show
NBC – Today
Evening Network TV (Mon-Fri)
Code 2 out of 3 each weekday
ABC – World News Tonight
CBS – CBS Evening News
NBC – NBC Nightly News
PBS
Code two consecutive days, then skip one
Newshour with Jim Lehrer
Cable TV (Fifteen in all, Mon-Fri)
Daytime (2:00 to 2:30 pm) coded 2 out of 3 every day
CNN
Fox News
MSNBC
Nighttime CNN – coded 2 out of the 4 every day
Situation Room (6 pm)
John King, USA
CNN Prime Time
Anderson Cooper 360
Nighttime Fox News – coded 2 out of the 4 every day
Special Report w/ Bret Baier
Fox Report w/ Shepard Smith
O’Reilly Factor
Hannity
Nighttime MSNBC – coded 2 out of the 4 every day
The Ed Show
Hardball (7 pm)
Countdown w/ Keith Olbermann
The Rachael Maddow Show
News Radio (Mon-Fri)
NPR
Code 1 out of 2 every weekday
Morning Edition
All Things Considered
Radio Headlines
ABC Radio headlines at 9am or 5pm
CBS Radio headlines at 9am or 5pm
Talk Radio (Mon-Fri)
1 out of 2 conservatives each day
Rush Limbaugh
Sean Hannity
1 liberal every other day
Ed Schultz
Story Inclusion
For the broad sample, PEJ analyzed all stories with a national or international focus that appeared as follows:
• On the front page of newspapers
• During the first 30 minutes of network morning news, cable programs, and talk radio shows
• During a thirty minute segment of NPR’s Morning Edition and PBS’ NewsHour with Jim Lehrer
• As one of the top 5 stories on each Web site at the time of capture
• During the entirety of the commercial network evening newscasts and syndicated news headlines segments on ABC radio and CBS radio
The technology-focused sample constitutes the main analysis for this report. It was made up of stories coded as technology stories in the broad sample from June 1, 2009 through June 30, 2010. First, we gathered the stories that were coded with a broad topic of Technology or Technology and Culture. In addition to these two broad topic areas we also included content identified as technology-related through the more specific variable of “storyline.” The storyline variable tracks specific events in the news. At times, technology-related events fell under different broad topic areas such as business or crime. This two-layer approach ensures that we cast as wide a net as possible in identify the content to analyze.
The list of technology-related storylines was as follows:
Cyberspace Issues
Apple
Video game consoles
Microsoft/Yahoo Search Agreement
MySpace Suicide Case
Google Launches Chrome
Digital TV Switch
LA Train Crash
Craigslist Killer
Texting while driving
Phoebe Prince Suicide/Cyber Bullying Case
General Google News
In all, 953 stories were identified as technology-related.
From this group of 953 stories, we next selected every other story to examine in greater detail through additional content analysis. Before coding, though, a trained coder examined each of the selected stories to make sure that it applied to the subject matter in this study.
In all, the technology-focused sample included 437 stories.
Capture and Retrieval
All outlets are captured and included in PEJ’s media archive.
For newspapers that are available in print in the Washington, D.C. area, hard copies are used. For newspapers that are not available for delivery, digital editions of the paper are retrieved either through the newspaper’s own Web site, or through the use of digital delivery services such as pressdisplay.com and newsstand.com. When necessary, the text of articles are supplemented by the archives available in the LexisNexis computer database.
Radio programs are captured through online streams of the shows. Using automated software, we record several local affiliates that air the program in various markets throughout the country. The purpose of this method is to ensure that we have a version of the program in case one of the streams is unavailable on a particular day, and so that we record the show in a manner that represents the way a typical listener would hear the program with commercials and newsbreaks.
Online websites are captured manually by a member of PEJ’s staff. The capture time is rotated daily between 9-10 am ET and 4-5 pm ET. The home pages and pages with the top articles for all sites are saved so that when we reference the material, the format is the same as it appeared online at the time of capture.
Finally, all television shows are recorded digitally and archived for coding purposes. PEJ is a subscriber to DirectTV satellite service and all programs are recorded onto multiple TiVo recording units before being burned onto DVDs for archival purposes.
All television and radio programs are then coded by a member of PEJ’s staff who watches or listens to the archived version of the program.
Coding Team & Process for Weekly Index Coding
The data derived from PEJ’s regular Index coding was conducted by PEJ’s team of 15 trained coders. We have tested all of the variables contained in the regular weekly Index coding and all the variables reached a level of agreement of 80% or higher. For specific information about those tests, see the methodology section for the NCI.
Additional Coding of Technology-focused stories
For the stories from the technology-focused sample additional coding was conducted for six specific variables.
• Tech storyline refers to particular storylines that occurred often in news media during the time period under study
• Tech area involves the issue or larger subject being covered
• Lead company designates the company that is the main focus of the story
• Presence of companies identifies whether any of the major tech companies tracked (Apple, Google, Twitter, Facebook, and Microsoft) where present in at least 25% of the story
• General technology thread refers to the concepts or impressions that form around technology as a whole in a given story
• Company threads refers to concepts or impressions that form around a particular company in a given story
Coding Team & Process for the Additional Coding
A team of four of PEJ’s experienced coders worked with a senior researcher in order to complete the additional coding for this particular study.
In addition to the main intercoder testing conducted on all NCI variables, supplemental testing was conducted on the additional variables used in this portion of the study. For the following codes, 30 randomly selected stories were coded by all members of the coding team.
The percent agreement for each variable was as follows:
Tech storyline: 91%
Tech area: 88%
General topic thread: 80%
Lead company: 91%
Apple presence: 99%
Google presence: 98%
Twitter presence: 96%
Facebook presence: 96%
Microsoft presence: 100%
Specific company threads
Google is innovative and its products are superior/better designed: 97%
Google makes content more searchable/findable: 98%
Google is a pretty virtuous company, really does try to follow the slogan “don’t be evil”:100
Google has too much information/power: 100%
Google steals others’ content and ideas: 100%
Google creates business competition for other companies: 98%
Microsoft is innovative and its products are superior/breakthrough/better designed: 100%
Microsoft is a dangerous manipulative monopoly: 100%
Microsoft products have too many bugs/viruses: 100%
Microsoft is behind the innovation curve: 100%
Apple is innovative and Apple’s products are superior/better designed: 96%
Apple’s products are user friendly and that’s why they are popular: 95%
Apple is anti-competitive and too controlling with its products and services: 99%
Apple creates products people don’t really need/planned obsolescence: 97%
Apple has a loyal/rabid fan base unlike other companies: 99%
Apple’s products are overhyped and overpriced: 96%
Twitter makes dissemination of information easier and connects people: 98%
Twitter is pointless, harms creativity, is self centered etc.: 100%
Facebook brings people together: 96%
Facebook fosters communication: 97%
Facebook doesn’t protect its user’s privacy/is an unsure network: 100%
The sections in this report about social media are based off the data collected from PEJ’s New Media Index. The NMI is a weekly report that captures the leading commentary of blogs and social media sites focused on news and compares those subjects to coverage in the mainstream press.
This study aggregates the weekly data collected for the News Media Index from June 1, 2009, through June 25, 2010.
Universe
To study new and social media, PEJ wanted to be able to include as wide a range of outlets as possible. For unlike the traditional press, blogs and social media pages reach into the millions and change daily as new ones emerge and other dissolve. In exploring various options, we saw value combining the work of some sites that specialize in tracking these outlets continuously with our own coding scheme and analytics.
Two prominent Web tracking sites, Technorati and Icerocket, monitor millions of blogs and pieces of social media, using the links to articles embedded on these sites as a proxy for determining what these subjects are. The website Tweetmeme uses a similar method to monitor the popular links on the social networking site Twitter.
Each of these sites offers lists of the most linked-to news stories based on the number of blogs, tweets, or other pages that link to them. PEJ does not determine what constitutes a “news” story (as opposed to some other topic), but rather relies on the classifications used by each of the tracking sites.
A PEJ staff member manually captured the lists from each site every weekday between 9 and 10 am ET. From those lists, the top five linked to articles were captured for further analysis by PEJ staff (SEE BELOW).
Through July 3, 2009, PEJ captured information about blogs from both Technorati and Icerocket. However, the relevant component of Technorati’s site stopped working in early July and has been down ever since. Therefore, the NMI reports beginning the week of July 6-10, 2009, only included blog data from Icerocket.
Coding Procedures
Once the lists of articles were compiled, PEJ staff conducted a content analysis of the subject matter of these linked-to news articles in a similar manner to the News Coverage Index.
Almost all of the codes and rules are the same as with the NCI. The variables coded in both projects include story date, source, story word count, story format, story describer, big story and broad story topic.
In order to meet high standards of reliability, these variables are all included as part of PEJ’s continuing intercoder testing involving 15 coders and reached levels of agreement above 80%.
For more details about PEJ’s intercoder testing procedures for these codes, refer to the detailed methodology about the News Coverage Index.
The only additional variables used in the NMI were identifying the original outlet of the news story and tracking the number of links aimed at each story included in the sample. Technorati, Icerocket, and Tweetmeme each provided the number of links within their lists.
Calculations
The priorities of the bloggers are measured in terms of percentage of links. Each time a blog or social media page adds a link to its site directing its readers to a news story it suggests that the author places at least some importance on the content of that article. The user may or may not agree with the contents of the article, but they feel it is important enough to draw the reader’s attention to it.
The calculations for the NCI have a different base. That Index measures the time (in seconds) or space (in words) of each story. That is then used to calculate the percent of newshole devoted to each topic.
The reason that the New Media Index uses a different measure, links rather than newshole, is because the nature of online media is different from other traditional forms of media. First, there is no limit to the amount of space that can be devoted to a specific story. In a newspaper, there is a limited amount of space on a front page, for example, and a television newscast is limited by its allotted amount of time. Web sites have no such limits.
Second, PEJ determined that in this procedure, the number of blogs that link to a news article are a far greater measure of the significance of that article online than the length of the story. A particular article might be quite long in terms of number of words, but if only a few blogs link to it, that article would have only a small influence in the new media environment. A short story that gets linked to many times has a far greater influence.
The percent of links for each big story is determined by taking the total number of links in the sample and then dividing that number by the number of links devoted to each specific big story. The percentages are then ranked in order to discover the five storylines that were most present in online commentary.
Differences from the NCI
In addition to the base calculation, there are three differences between the NMI and the NCI to note:
1. The capture times for the Web sites included in the News Coverage Index rotate each day. In the New Media Index the times are the same each day. Since these lists compile the number of links to stories over a 48-hour window, rotating the time of capture would result in different increments of times between each capture. Through testing, PEJ has discovered that the stories on the lists change significantly more over a 24-hour period than they do over a 12 or 16-hour period. Thus it is more methodologically sound to capture at the same time each day.
2. The News Coverage Index is comprised of primarily U.S.-based media outlets, but the aggregators of blogs and social media include both U.S. and non-U.S. blogs. In addition, stories that are linked-to can be from non-U.S. sources.
3. PEJ’s weekly News Coverage Index includes Sunday newspapers while the New Media Index is Monday through Friday.
