This week a couple of my Sciblings have been abuzz about an article published in some journal I'd never heard of... a minor impact journal...the Journal of Who Gives a Fuck Science Communication. Bora has a great break down of some of the major criticisms. Drugmonkey, one of the subjects of the "analysis" in this article, is also displeased and critical of the author's conclusions.
Video 1: A current favorite at the Isis house. When emailed this video, PhysioProf replied, " Couldn't they afford to animate some fucking legs on those fuckers?" I have always wondered why Mr. Lunt has no eyes.
But, for those of you who are still interested, here's the run down...
Inna Kouper, a graduate student in Library and Information Science at Indiana University, somehow magically chose 11 blogs to study, one of which was Pharyngula. Now, I'm not hating on Pharyngula. PZ plays an important role in the blogosphere and, while I think that sometimes his commenters get out of control, he's got a unique voice and an uncanny ability to rally the troops. No one can deny that the climate at Pharyngula is not necessarily reflective of the entire blogosphere. Still, the fact is that Inna Krouper sampled 11 blogs. There are 80 blogs currently at ScienceBlogs, 8 more at Discover Blogs, and a bazillion independent and network blogs indexed by the Nature Network. Yet, somehow Inna chose these 11 blogs as representative of the genre and one of them was motherfucking Pharyngula. Then, she did this:
A combination of quantitative and qualitative techniques of content analysis has been used in this study. The qualitative analysis involved iterative close reading of posts and comments with the purpose of identifying common types of statements and activities conceptualized as modes of participation. This conceptualization was informed by the speech act theory and the pragmatics perspective yet it was purposefully left rather loose and open to allow for the categories to emerge from the data. Each time a mode of participation was identified, it was entered into a catalog, and then a post or a comment was assigned a corresponding code. Along with the modes of participation, the posts were coded for topics and sources of the post; the comments were coded for the reader's identification elements (e.g., a nickname, first name, full name, link to blog, or blog author). Subsequently all codes were counted and the analysis proceeded with the examination of the most frequent and rare patterns and their groupings.
Translation?
Figure 1: Inna sat down one night, read some blogs, and then wrote some shit. She must really be itching to finish the ole thesis.
I mean, I truly am baffled by these methods, especially when the author brags that "it is necessary to analyze current practices of science blogging. To date no attempts have been made to do that. The present study is the first step in this direction." This article is a step alright...
Figure 2: Problem is, none of realize where that step is taking us until it is too late to unlearn the stupid.
I'm just plain ole disappointed by the "methodology." This author could have taken the opportunity to perform a carefully controlled study with randomly-selected non-scientists. She could have shown them blinded content and administered questionnaires. Instead she wrote 10 pages of opinion and passed it off as science.
After pages upon pages of presenting cherry-picked content, Inna concludes this:
Science blogs examined in this study are very heterogeneous. They provide information and explain complicated matters, but their evaluations are often trivial and they rarely provide extensive critique or articulate positions on controversial issues... It appears that science blogging can also be characterized as relying on reductive analysis and dependent reporting and drawing caustic and petty commentary. These characteristics may as well be applied to the newspaper and magazine science communication, but with the newer science communication outlet such as blogging they indicate that the potential of blogging to do something differently, e.g., to provide informed expert and citizen commentary, is not realized. In their current multiplicity of forms and contents science blogs present a challenge rather than an opportunity for public engagement with science. Lack of genre conventions, which for the audience translates into broken expectations and uncertainty, impedes the development of stable readership and participation from the larger public, which may also be very heterogeneous. The "neighborhood bar" or "water cooler" commentary creates a sense of community with shared context and culture, but at the same time it creates a barrier that prevents strangers and outsiders from joining the conversation. As a community of scientists or individuals close to science, the existing readers may enjoy the entertaining nature of science blogs and not need science blogs to serve as a place for discussion and rational debate. Relying on such community of readers, bloggers may reduce their interpretive activities and resort to copying, re-distributing, and re-packaging of the existing information, which is still quite rewarding given the background of the majority of current readers and yet requires much less time and effort. This study provides further evidence that blogging as a web tool has no magic properties on its own. Without a concerted effort of different social actors involved it will not solve any problems...
Reading this, I realize that I did my PhD in the wrong damned field. I would be a much more prolific publisher if I had entered a field where I could have written whatever bullshit moved me on any given day and called it "research."
I also wonder how many of you feel like you simply add "caustic and petty commentary"? I question how Inna can conclude that blogs pose a barrier to the conversation. That's a difficult statement to take seriously, knowing that Inna had no access to traffic data for any of the blogs she read. For me, I know that a single blog will be read by 1000s more non-scientists than any original scientific article I publish in a peer-reviewed journal. And, she certainly wouldn't have found the analysis trivial if she had read some of Ed Yong or Carl Zimmer's work, not that I find any of the blogs she included trivial. Then again, I think it is the diversity of voice is what makes the blogosphere so beautiful.
My sample size = 1 is probably no better than Inna's sample size = 11, but I can at least offer my experience to the data set. I get many letters a week from young people interested in science careers and soliciting advice on graduate school, fields of study, and professional development. The number of people who have come to my office in person to have these conversations is trivial in comparison. Thus, these data would lead me to conclude that my blog presence has lowered the barrier to engagement with this audience.
I'll also never forget one of the occasions, quite a while ago, that I wrote about some novel research. It was a topic semi-related to my expertise. One of you then went to PubMed (or some other search engine) and called bullshit on me by citing some of my lab's own work. It was glorious and I was proud of my scrappy little muffins for months.
Inna does not take in to account the benefit of blogging for the blogger. First and most importantly, blogging is hilarious. I have met some tremendous people in the blogosphere who have become valuable resources. I use a lot of the folks here to bounce career ideas off of and I have met scientists that are using techniques that I might not have considered. I suspect that it is because of blogging that I am now totally bff with my academic society. I might never have been noticed if I hadn't taught Marty Frank the word "cocknozzle." Finally, blogging has allowed me to forge some professional relationships that might not otherwise have been available to me. I feel fortunate that MRU will eventually be a stop on Rebecca Skloot's book tour. Many of the people in the administration here were both thrilled and surprised that anyone had contact with her. I only know her because we're both kick ass, totally hot bloggers.
I'm not going to take this article too seriously. I am, however, going to challenge academic researchers to think more critically when they assess the blogosphere. As scientists, we have access to an extensive toolbox and a multitude of metrics that could be used to evaluate the impact of the blogophere, without having to rely on Inna Kouper's poorly-organized, half-assed ramblings as the first "scientific analysis."
Surely, we can do better than that. I mean, even this is a more honest analysis..
Figure 3: Joseph Hewitt's comic interpretation of the ScienceBloggers. That's Dr. Isis in the top panel putting the choke hold on some dude and calling him a "muppethugger." For the full-sized image, hilarious commentary, and clues to who's who in the 3rd panel, click over to Joseph's website. Man, Ed Brayton is one ripped, muscley dude!
Inna Kouper (2010). Science blogs and public engagement with science: practices, challenges, and opportunities Journal of Science Communication, 9 (1)
Information Science
Education
Comments
I also wonder how many of you feel like you simply add "caustic and petty commentary"?"
I try, I try.
Posted by: beccaMcSnarky | March 15, 2010 12:07 AM
Caustic, sure. But, petty?
I've learned so much from the blogs I follow here, either directly from the bloggers, clicking on links to studies or other info from a comment, or simply looking up info to support (or sometimes refute) my arguments.
I found this usage amusing:
"Digital information and communication technologies (ICTs) are novelty tools that can be used to facilitate broader involvement of citizens in the discussions about science."
NOVELTY tools?! Like whoopee cushions and dribble glasses? It can be fun at ScienceBlogs, but it never occurred to me it could be slapstick. Nyuck, nyuck, nyuck.
Wait, was that petty?
Posted by: The Gregarious Misanthrope | March 15, 2010 12:35 AM
Ah, content analysis, fun fun. I should probably say something about how we should respect other disciplines' methodologies, but I've always found Comm's version of "the department of obvious studies" freaking annoying, so I won't.
Posted by: Sisyphus | March 15, 2010 2:21 AM
Hmmm, I may be missing something, but where is the data ? The only thing remotely looking like it (I admit I read quickly) is a keyword cloud... I'm no expert in visual representation of data but this method looks kinda thin... At least, it feels bloggy and all...
Posted by: BdN | March 15, 2010 3:06 AM
Just forgot :
Because it's really important readers can form their own opinion about the shrews' nectar consumption... and they can't get any other source of information to compare... and readers are so stupid they can't understand what is a comparison. It's not like the author wrote that shrews drink nectar produced by bertam palm... Oh, wait, he did...
Posted by: BdN | March 15, 2010 3:18 AM
Preach it, Isis! I'm so enamored with science blogging that I'm designing a course for undergrads using blogs (reading and writing them!) as an educational tool. I guess we'll see how the administration feels about that when I have my interview for the teaching fellowship...
Also, I totally agree with this: "I feel fortunate that MRU will eventually be a stop on Rebecca Skloot's book tour. Many of the people in the administration here were both thrilled and surprised that anyone had contact with her. I only know her because we're both kick ass, totally hot bloggers." The assistant dean of my graduate school is sooo psyched about Rebecca's visit to our campus. She told me that she called the director of my graduate program and raved to him about how awesome I am for getting her to come here. I count that as a major win, all because of science blogs!
Posted by: Laura | March 15, 2010 8:05 AM
Dr. Isis, I'm afraid that unless you went to a very, very bad school indeed, there's no program in the humanities or social sciences that would have put up with this bullshit either. Seriously, even in literary criticism you have to build your analysis on something. This reads like a freshman comp paper - and not a good one.
What I don't get is why people are paying so much attention to this. Ok, some grad student wrote a paper. Woohoo.
Sharon
Posted by: Sharon Astyk | March 15, 2010 8:20 AM
It's unfortunate that you feel the need to take down a newly minted PhD because they dare to use a legitimate non-bench science method to critique your friends. Whatever happened to supportive environment for new scientists? Shame on you!
Posted by: hypatia cade | March 15, 2010 9:10 AM
Legitimate non-bench science? Where did you see that?
I am also curious about why you think I'm not supportive of new scientists. Surely, you don't expect people to offer unwarranted praise under the guise of support. Some of the best science-shaping criticism I've gotten has come in the firm of some pretty blunt reviews. This has nothing yo fo with anyone being friends with anyone. Sharon nailed it when she pointed out that if the best you can do in terms if an analysis is a word cloud, then it's really not science.
Posted by: Isis the Scientist | March 15, 2010 9:27 AM
That video is pure baby crack...and I can't stop watching it.
Posted by: Dr. O | March 15, 2010 10:05 AM
Oooo, Kouper (reminding me very much of Koopa Koopa, but that's beside the point)peeves me. Yes, some posters (the minority, by far) do petty things to get noticed, but it's generally against the grain of the comments. I'm also shocked that she concluded that science blogs serve no real purpose and do not cause discussions on important and diverse topics. She also claims that they're not informative (has she never heard of Starts With A Bang?!?!?), and are closed communities (they're totally open-that's why you can go back to the beginning and read all the prior posts!!!!).
Also, I kind of want to change my major, so I can use the word "magic" in my thesis and not get laughed at (granted, everyone does put fairy dust in their PCRs to get them to work, and gnomes steal lab equipment, but we don't tell outsiders about that...).
Posted by: ktbug ladydid | March 15, 2010 10:43 AM
Welcome to the world of qualitative research. As part of my hobby (faculty development) I have been introduced to this world, both through reviewing manuscripts and as parts of my own attempt to document the usefulness of various activities I am involved in on campus. For the latter part, I actually read a book on the topic (it may replace Ambien) and decided that I had trouble getting my brain around the whole concept. Others had better design and perform that part of the study.
The whole idea of identifying terms, attitudes, etc that repeatedly occur within a framework (for example, when interviewing patients who are late for clinic appointments) is a valid research area. This field is harder than it sounds, and it provides important information for the development of more sciencey-sounding sociobehavioral research. You can't design a potentially valid survey instrument until you know what to look for! And while you may think you know what to look for, you may miss the best question if you don't do something qualitative first.
Qualitative research does have to follow some statistical standard to be valid. My major concern with the present paper was how the sample size was determined and how the blogs were chosen. Are they really representative of the genre? Or were they chosen because the commentariat was large? I guess I could try reading the original paper [see paragraph 1 about my own issues with this type of research.]
This is the type of preliminary study that qualitative researchers do to the develop their catalog for a study. The catalog contains the concepts or memes that will be examined, and lists potential synonyms. For an interview study of patients late to appointments, one might have a theme of transportation that would include lack of funds, bus running late, car broke down, and other related phrases. Another theme might be ancillary services: the lab or xray did not finish my studies on time, the registration desk was backed up, etc.
So, yes, this sort of thing looks like a word cloud, because it basically is. This type of research exists to identify word and idea clusters, usually as a guide to future studies. I still have concerns about the study, but this is a legitimate form of research. I have a friend getting a PhD in this area who could maybe say pithier things about it.
Posted by: Pascale | March 15, 2010 10:57 AM
Pascale, sister, you've just got to read the paper. When I said "word cloud," I literally meant word cloud. There's a word cloud in the paper. That's the extent to which this qualitative paper enters the quantitative.
I think you're right that there is probably some very important sociological research being done using cataloging. This just ain't it.
Posted by: Isis the Scientist | March 15, 2010 11:13 AM
OK, I am taking the paper to clinic with me this afternoon. I have also sent it to my qualitative research buddy, who reminded me (for the eleventy-billionth time) that qualitative research is not generalizable. He, too, will look it over and tell me if it is as FOS as we hard-sciency types believe.
The critical piece in this type of work is the way the terminology becomes more focused (in other words, how the words in the cloud cluster to become the big ones).
Posted by: Pascale | March 15, 2010 1:31 PM
Wow, I shouldn't have let myself take a look at this post, because now I'm not going to get any work done for the next hour or so! As my undergraduate honours work, I've been examining how social media is or isn't used to facilitate/coordinate environmental activism. I developed a survey and now, having collected and analyzed the responses, I can see where, if I were to follow up, I would change a number of things about the survey - but I feel like I at least tried a bit harder than Inna Kouper to ask questions that would let me draw some meaningful conclusions. And I'm only getting a BA for this. It's frustrating, because I'm interested in some of the same issues that she is, though I'm more interested in communication about environmental issues, and I think it's a legitimate research area, but not if the research is conducted like this.
Even more so than the size of her sample, I'm curious about why she doesn't note that some of the blogs are actually associated with professional media outlets, while others are not. To me that's the obvious explanation for her "finding" that the Wired Science blog used a "reporting" model almost exclusively. I also don't understand how, based on posts within a very short time span for a very small sample of science blogs, she can possibly claim to have adequately for the emergence of new genre conventions. She concludes that science blogs are heterogenous, but it seems like that should have been obvious from the beginning, and it should have been discussed at far more length in the paper. A more useful study might have noted the different starting points of the science bloggers and explored how their approaches to communicating science differ. Conflating a small and highly varied group with different starting points and a potentially varied audience is not going to lead to useful information about science communication.
Posted by: tariqata | March 15, 2010 1:37 PM
I thinks everyone's expectations of the quality of papers in the area of Communications are too high. This is probably an average to above average paper in a low-to-no impact electronic journal. I doubt it is even indexed. SciFinder does not find articles published in JCOM.
Posted by: Rob | March 15, 2010 1:56 PM
#3, 4, 7, 11. There is nothing wrong with Qualitative research. It can be good, informative and useful. What I had a problem with this paper is that it was essentially an Opinion/Perspectives piece posing as Quantitative research.
#16. I like JCOM. Which is why I was so dismayed when they published this crappy paper. Which is why I blogged about it and fisked it line-by-line (go see, Isis linked to it on the very top of her post). They have published some excellent stuff before (and I know some is coming up soon). Which is why I agreed to publish there AND to, recently, join the editorial board.
Now, the paper I published in JCOM never pretended to be research - it was definitely a Perspectives kind of piece where I hope I was cogent, informed and somewhat provocative, but never pretended I did anything close to a systematic study, even less crunched numbers.
In my suggestions to JCOM (they are having a big meeting soon) one of my suggestions was to clearly differentiate (categories, tags, perhaps even separate sections of each issue) between Opinion/Perspectives papers, Qualitative research papers, and Quantitative research papers. If implemented, this should make readers' expectations clearer.
Posted by: Coturnix | March 15, 2010 2:22 PM
The paper sounds like a lot of crap, but I look forward to Pascale's analysis of it. I didn't think 11 blogs was too small a sample size, but it depends on what conclusions someone might be making. And PZ allows a lot of "caustic and petty" commentary on his blog. As in, that blog that he and his commenters like to make snide comments about, the threats were elevated to threatening sexual violence on the writers of the other blog. I think that's offensive. I'm not sure what PZ's mentality is that he allows near-chaos and tolerates disgusting threats rather than real and reasoned insults and critiques, but it is his blog so he can do what he want. I've also seen other blogs where the comments were highly regulated to the point where anyone who came anywhere near their line was called a troll near immediately. So regulation is not the solution or real discussion doesn't happen.
Posted by: FrauTech | March 15, 2010 3:41 PM
Re Pascale @#14,
interesting observation. Almost makes it more important to critique the substance so that the questions are not misframed even before anyone gets going on better studies.
Posted by: DrugMonkey | March 15, 2010 6:10 PM
Go Pascale! I am very interested to hear what you think of the whole paper, and what your qualitative research colleague thinks. I'm looking forward to reading a thorough and informed critique.
Dr Isis, I bet I couldn't understand your methods sections!
In qualitative research, the sample size does not matter as it is not generalisable research. The method of sampling is important - it must be done in a way that will allow the research questions to be answered.
Two things that don't seem to be addressed in your excerpts of the paper that are very important in carrying out high quality qualitative research are saturation of data and triangulation of data and analysis.
Perhaps this is an initial part of a larger study.
And perhaps you should read this Dr Isis http://www.bmj.com/cgi/content/full/315/7110/740
Posted by: Nerida | March 15, 2010 9:16 PM
Surely, you are fucking with me with this statement.
Posted by: Isis the Scientist | March 15, 2010 9:24 PM
I've been looking into the blog selection issue in a (very) small way in a related post.
Posted by: DrugMonkey | March 15, 2010 9:49 PM
I found this post to be a shocking attack on a young academic. This isn't blunt criticism. This is an ad hominem attack.
It is perfectly legitimate to attack the merits of her work, within the context of her academic discipline. If you want to attack the methods of her academic discipline, then feel free. But that seems like a separate post.
Finally, I'm surprised that you are so offended by this article. The reason I'm surprised is that I've never considered this to be a science blog. Sure it is about academia. And it is written by a scientist. But it reads like the personal blog of a scientist, not a science blog.
I do read actual science blogs, and they are a different thing all together. You clearly aren't trying to communicate the results of research here. If that was your goal, no doubt you wouldn't be writing anonymously. For instance, I've never seen Carl Zimmer post about his shoes on The Loom.
That isn't to say that you aren't doing something important. As a young female social scientist, I really enjoy reading your perspective. I even consider reading this blog to be, for the most part, a professional development activity. But a science blog? I wouldn't have categorized it that way.
Posted by: juno | March 16, 2010 10:13 AM
Juno, do you even know the definition of ad hominem? Surely by now you've learned it on one of those real science blogs you're reading.
But, you're right. Zimmer doesn't talk about his shoes. Instead he posts about tattoos.
Personally, I consider myself just a little dumber for even bothering with your comment. - not an ad hominem. You clueless fuck - this is ad hominem
Posted by: Funky Fresh | March 16, 2010 10:23 AM
I'm surprised that you are so offended by this article
Nobody here is offended. Critical, yes. Offended, no.
Posted by: SKM | March 16, 2010 11:14 AM
and survey says...
Posted by: DrugMonkey | March 16, 2010 3:01 PM
I used to make an effort to put funny or interesting graphics (photos, etc.) in my blog posts. But I gave up long ago because no one, not even I, can come even close to Isis in this sort of embellishment.
I love those sheep!!!!
Posted by: Greg Laden | March 16, 2010 4:18 PM
Ping!
Posted by: Dr. Free-Ride | March 16, 2010 8:16 PM
If it's "not generalisable", then it ain't motherfucking science.
You clueless fuck - this is ad hominemNo, it's not; it's just an insult. Ad hominem would be: "Because you are a clueless fuck, what you just said must be wrong."
Posted by: Comrade PhysioProf | March 16, 2010 8:42 PM
As usual, Greg is right about the sheep.
Posted by: becca | March 16, 2010 11:49 PM
Comrade PhysioProf said:
If it's "not generalisable", then it ain't motherfucking science.
It may not be science, but it is still research.
Posted by: Nerida | March 17, 2010 12:44 AM
Googling which toilet paper is the softest on my ass is also research. Doesn't mean it should go in a peer reviewed publication.
Posted by: Funky Fresh | March 17, 2010 12:47 AM
P. Blecher, H.C. Korting (1995) Tolerance to Different Toilet Paper Preparations: Toxicological and Allergological Aspects Clinical and Laboratory Investigation. 191: 299 - 304.
Posted by: Southern Fried Scientist | March 17, 2010 2:43 PM
Southern Fried Scientist FTW!
Posted by: Coturnix | March 17, 2010 2:47 PM
If it's "not generalisable", then it ain't motherfucking science. Well no, but it probably is social science. And the difficulty with finding a sample you can generalize from is an issue for social scientists.
I've logged lots of hours doing content analysis or preparing documents for content analysis. You can do interesting (although not always generalizable) things with it. However, there are standards. You really aren't allowed to make stuff up. That article doesn't seem to meet most of them.
Posted by: katydid13 | March 19, 2010 3:26 PM
As a graduate student at Purdue, I would like to emphasize that this "research" was done at Indiana University.
Posted by: Sal | March 19, 2010 4:58 PM