I should not be writing this! I absolutely have better things to do by almost any measure. Yet I have been asked to clarify which of my publications are “refereed” and which not. Unless I decide simply to resign, which would among other things leave my current students in the lurch, I cannot entirely refuse to respond.
It will be easier to respond at some length than to answer straightforwardly as requested— although I suppose I could flip a coin for all the grey cases, which include around half of my scholarly output over the years. I grant that people with other specializations may have other experiences, but I know what I am talking about for my area, and I doubt that my case is all that unusual.
Gimme Shelter
Let’s begin by recalling a few things about tenure. In addition to it being the actually existing system that ensures basic job security and quality of life for teachers in academia (if there were actually-existing alternatives to tenure that did the same, we could discuss pros and cons of tenure in good faith) it was established in large part to give scholars a sort of shelter.
Behind this shelter, they had scope to engage in research, teaching, and writing that might not be considered fully legitimate as judged by certain outside-looking-in measures. For example, scholars might pursue inquiries that are inconvenient to religious orthodoxies, illegible or not-yet-proven in terms of reigning scholarly paradigms, politically undesirable as judged by ruling regimes, or simply perceived (as in the case of Socrates, who sadly did not enjoy tenure) to be “corrupting the minds of youth.” Satisfying academic classrooms, if allowed to unfold as they should with teachers who have scope to challenge students and students who have space to explore cultural critiques, are difficult to sustain without such shelter.
One of the pleasures of gaining tenure is that for years I have largely been able to disregard—although not without some risk, perhaps vaguely like Socrates with exceedingly lower stakes—the forced, misleading, counterproductive, anti-intellectual, and all-around bogus hard distinction between “refereed” and “not-refereed.” (Once again I disavow knowledge of every field but insist that I know about my own.) My choices have unfolded partly on principle but mainly as an evolving result of “following my nose” toward the best ways I could find to nurture and disseminate my research, teaching, and outreach.
My Most-Cited Article: Productive, Irrelevant, or Counter-Productive?
One example is my most-cited article, an analysis of Madonna that appeared in an edited book on religion and popular culture. It was polished and edited more extensively than almost any of my publications. But was it “refereed” (read: legible to bean-counters as unambiguous “productivity”)? I have never been certain that anyone who evaluated me could answer. In retrospect this article probably reads to most colleagues as at least in the ballpark of “real scholarship”—although articles that are book chapters don’t fully count as “articles”—since the book is from a top university press and has been widely cited. However, in real time this was ambiguous. Key colleagues thought I was wasting my time and were at least tempted to lump the work as “not-scholarly” and when evaluating me to discount it: not refereed, slightly embarrassing, and a terrible choice as measured in career-building opportunity costs.
I also know the following about this piece. Its earliest draft was not part of my core scholarly agenda—it was intended as a quick side project urging the Christian Century magazine not to be dismissive of a Madonna video. Likely because this journal was dismissive, exactly as I feared, my manuscript was rejected. (Let’s belabor it, I’m evoking what I said above about shelter to venture beyond what is legible in reigning scholarly paradigms.) So I expanded it into a “normal” article with footnotes, which garnered another rejection from a journal of feminist theology. Thus I failed both at both public popularization and refereed productivity. But I am thrilled that it wound up in the book (refereed or otherwise) because by the time I had rewritten it, with excellent editing, it was better and far more widely read.
Eight Valid Factors of Quality and Value That Are Not Measured by Refereeing
In my experience, such complexity is the rule, not the exception. Let me state the problem in general terms before I return with a few more examples. There are continua between all the following poles:
Writing of high intellectual quality—or not.
Writing that represents lengthy time investments and serious innovation—or not.
Journals with editorial boards (or functional equivalents) that are distinguished and/or full of interlocutors one should desire—or not.
Journals (or equivalents) that extensively vet and edit drafts of submitted work— or not. (Sometimes this is called refereeing—or not. Refereed journals by no means always edit.)
Journals (or equivalents) that have readerships that one would desire as dialogue partners—or not.
Journals (or equivalents) with a crossover readership from academia and various publics—or not. (This can be important or not. In fields where I work it is important.)
Publications with long word/page counts—or not. (Typically it takes more work, resulting in higher quality and value, to make something longer into something shorter. If I double the time spent on an article to cut it in half and polish its prose, I get a fourfold negative incentive for my trouble: no doubling of “productivity” but rather a 50% cut.)
Research undertaken in part for synergies with breadth of quality teaching (a clear virtue but rarely a good careerist move)—or not.
Not a single one of these continua corresponds to “refereed/not-refereed.” Every one includes a great deal of refereed work at the undesirable end of its continuum. Every one includes a great deal of not-refereed work or ambiguous cases at the positive end.
Common wisdom in our university—structuring the logic that determines who has a viable career—posits that a refereed-or-not continuum is a valid approximate composite measure of quality and value on these fronts. I am explicitly denying this. No doubt some scholars’ mileage will vary, but in my experience with dozens of journals, the distinction “refereed-or-not-refereed” is an independent variable that is close to random with respect to all the above continua.
A great deal of what I read that wastes my time—exhibiting low quality, low time investment by authors, low readership, non-optimum word counts (often grossly inflated due to a lack of editing, although often overly compressed too), and low value for teaching—is refereed. A great deal of what I read that has high quality, high investment… let’s just say “worthwhile” and worth emulating—is either explicitly “not refereed” or we circle back to the coin flip method for deciding ambiguous cases. To belabor again: a tenured scholar with the prerogative and responsibility of exercising judgment should frequently prefer the non-refereed.
Further Concrete Examples
Since I really do feel aggrieved to be repeating these points—I’ve been making them for years, enough to know that they will be ruled out of bounds and treated as an annoying waste of time—I will offer only a few more examples with fewer details.
Several of my most important and prestigious articles took the form of longish review articles in widely read journals. These journals would be deemed at first approximation “refereed”—or if not, then toward the valorized end of a “quality/prestige” continuum. My contributions are best understood as short articles with pointed scholarly interventions—often involving a great deal of work to make them short enough to fit into review article slots. On balance, I worked harder on these and am prouder of them compared to most of the articles (refereed!) that were spin-offs from my first book.
Bean-counters ask: aren’t these “mere reviews” (read: we will discount them if we count them at all)? Well, yes. Are they scholarly articles? Absolutely yes. Are they in refereed journals? (I already said generally yes, but now I add that some cases are clearer than others, but the quality, word counts, and dissemination are comparable across the board.) Are they refereed articles in these journals? (Time for the coin flip; it is unclear how to categorize invited review essays, except that it involves discounting.)
After we sort all this out, we can turn to other review articles—similar in quality and sometimes more widely read by more desirable interlocutors—but not placed in journals that rank high on a prestige continuum.
The Yo-Yo Effect
So here we have a huge bloc of work that is like a yo-yo—on the way up it is “real scholarship,” on the way down “he is doing mere reviews,” then up again to “real,” down to “non-refereed,” and so on.
Likewise we can play yo-yo with dozens of shorter reviews—clearly not articles yet an immense amount of work—that I defend passionately as scholarly contributions.
What about invited work for reference works or encyclopedias? I have written some such pieces that were quick and easy, either written for the occasion or spun out of my books. But also in the same category are several of my longest, most ambitious, most time-consuming, and in one case most prestigious, works on my resume. Pieces on this entire spectrum have often been counted roughly the same.
I agreed to write several of these because they could be building blocks for books. One such article is lengthy and not “encyclopedia-like” in form. Nevertheless it will continue to be rated a “mere encyclopedia article” (read: surely discounted, possibly ignored or even treated as a negative) until it becomes the heart of an introduction to a refereed book (the holy grail!). “Objectively” this will have been a counter-productive waste of my time until the book comes out—except that this is part of my process for writing the book.
Throwing Bad Energy After Good
So far I have been talking about throwing good energy after bad, as measured by the university’s incentive system. But there also is throwing bad energy after good, whether for benign motives or to game the system.
Some of my less important and least read articles—representing my smallest time investment and value added—were refereed. When my university insulted its faculty by imposing a one-size-fits-all, staggeringly ugly, and outrageously time-wasting straitjacket of a website for reporting productivity—with an apparent goal of making it easier for administrators to generate quantitative reports—they touted it as a time saving benefit to faculty. This was based on the idea that their website could (supposedly) search the relevant databases and import data automatically. Guess what such a search found for me, from more than 100 articles and reviews? An exceedingly small number drawn entirely from this last category of my least valuable work.
When a Day Counts More than a Year and Readers Are on the Internet
It is no exaggeration, but a straight literal fact, that I have sometimes seen lines on my resume that only cost me a couple afternoons of work being weighed by bean-counters “the same”—or more!—than lines that cost me years of effort. The most egregious cases hinged on whether to count things as refereed. On one side were the already-noted “spin it out in an afternoon” articles in a refereed column. On the other side were my two most ambitious book chapters—synthesizing massive amounts of research. It would have cost me less trouble to expand these latter two chapters into books than it did to cut their first drafts back to article length. Yet both could be categorized as “mere invited articles” in edited volumes which might be construed (read: discounted) as reference works.
Turning to on-line journals that are increasingly central to scholarly discussion—with boards that may or may not “referee” depending on semantics—similar themes tumble through similar ambiguities. I’ve written many short and simple online publications. I also I have many longish online essays that will be building blocks for my books. All this is before we turn to this blog—refereed by me and me alone!
Which of these are “refereed scholarship”? I predict that my deans’ numerical printouts will reflect zero of them—although there is no clarity about this and every reason to assume that that definitions will be applied inconsistently. However, some of these online pieces have been vetted, edited, and read more—not less—than my comparable pieces in standard (“legacy?”) journals. Some are on par in quality with the best things I have ever written, even though I exercised my judgment to place them online.
Exercising Judgment When “Refereed” is Random With Respect to Value
Such exercise of judgment—whether about publishing online, accepting invitations to write for edited books, spending time on “mere reviews,” etc.—is not solely my prerogative as a tenured scholar. It is my responsibility as a scholar in a land-grant public university.
So let’s restate the point I made at the start. On principle, as a tenured scholar, it is part of my intellectual and spiritual discipline to try (in most cases) to bracket and disregard the forced, misleading, counterproductive, anti-intellectual, and all-around bogus hard distinction between the “refereed” and the “not-refereed” in fields where I work. Rather I will continue to aim for a composite quality score on all the above-mentioned continua—amid which the refereed vs. not-refereed distinction does not quite reduce to a random variable, but comes close.
MBE standard notice: The time I spend on this blog is not in addition to a Twitter and FaceBook presence, but an alternative to it. If you think anything here merits wider circulation, this will probably only happen if you circulate it.