Evaluation guidelines: A regulatory genre informing reviewing practices

Review genres in academic discourse include both texts that are publicly accessible (e.g. book blurbs) and others that are more occluded (e.g. referees’ reviews). The latter may be freely drafted by the reviewers or follow editorial policies appearing in evaluation guidelines. These are of interest in two respects: they affect the content and structure of more public review genres, and they reveal acceptability criteria for research products, which tend to be gleaned only indirectly in anonymous reviews. This paper examines the textual organization, content and wording of 70 paper evaluation guidelines from various disciplines, mostly collected from the internet and partly obtained from academic journals’ editorial boards. Four main types of texts can be identified: lists of evaluative dimensions to be rated on a scale; descriptions of evaluative dimensions to be considered; sets of open-ended questions to be addressed; combinations of the above. The four text types mention similar evaluative criteria (i.e. current relevance of the issue addressed; thoroughness of literature review; originality of contribution; solidity of methodology; soundness of data; logic of argumentation; adherence to the discipline’s or journal’s standards; clarity of exposition; typographic accuracy) and encode objects of evaluation mostly through evaluative terms (e.g. adequacy, appeal, balance, clarity, comprehensiveness, conciseness, relevance, significance, suitability, usefulness) but also through descriptive ones (e.g. analysis, citations, contribution, English usage, interpretation, organization). The former identify the range of qualities a paper may display, but are not accompanied by indications of what they consist in. The latter identify components or aspects of a paper, whose evaluative dimensions, however, are left unstated (e.g. [detailed] analysis; [correct] English usage). Despite being lexically explicit, therefore, the guidelines are not extremely informative content-wise. The underlying assumption appears to be that their addressees, as experts in their fields, already know how to identify qualities specifically relevant to academic writing.