Peer review is the principal mechanism by which the quality of research is judged. Most funding decisions in science and the academic advancement of scientists are based on peer-reviewed publications.
Because the number of scientific articles published each year continues to grow, the quality of the peer-review process and the quality of the editorial board are cited as primary influences on a journal’s reputation, impact factor, and standing in the field.
Scientific journals publishing peer-reviewed articles depend heavily on the scientific referees or reviewers who typically volunteer their time and expertise. In most circumstances, at least 2 reviewers are solicited to evaluate a manuscript; some journals request 3 reviews. This may be required in situations where review by a statistician is needed. In cases of controversy or strong disagreement regarding the merits of the work, an additional review may also be solicited or one of the journal’s editors might give an evaluation. More than 3 reviewers are sometimes used if reviewers from several fields are needed to obtain a thorough evaluation of a paper.
In addition to fairness in judgment and expertise in the field, peer reviewers have significant responsibilities toward authors, editors, and readers.
Qualitative research is only one of the methods that are appropriate for our journal, but over the past several years we at AMJ have worked diligently to increase the number and quality of the qualitative research papers we review and publish. Just this year, one of our qualitative papers won the award “Best Paper in Organizational Behavior” from the OB Division of the Academy of Management (Margolis & Molinsky, 2008). Our efforts to increase high-quality qualitative work in the Journal stems, in part, from our mission to publish research that has the highest impact. Qualitative research certainly fits this bill, as work in this area has won multiple best paper awards in AMJ and Administrative Science Quarterly, and qualitative research was overrepresented (in terms of the total number of studies published) in AMJ’s survey regarding the most interesting management-related articles published in the past 100 years (Bartunek, Rynes, & Ireland, 2006).
Considerations of Research Quality in the Review
Theoretical validation concerns the fit between the social reality under investigation and the theory generated. Does the article clearly articulate / define the social reality (SR) / phenomenon under investigation in its naturea) and scopeb)?
Procedural validation concerns features of the research design that inherently improve the fit between the reality studied and the theory generated. Are the methodology and methods chosen inherently suited to elicit data and insight about the social reality of interest?
Communicative validation concerns the integrity of the interlocking processes of social construction with the relevant communication communities. Does the description of the data collection methods or instruments provide sufficient detail to indicate the data were authentically co‐constructed on participants’ own terms?
Pragmatic validation concerns the compatibility of theoretical constructs with empirical reality. Did the methods (framed by both theory and methodology) resonate in the research context and elicit data specific to the social reality of interest? (i.e., something that a generic line of inquiry would not have elicited?)
These guidelines are based on their research articles on research quality:
– Walther, J., Sochacka, N. W., & Kellam, N. N. (2013). Quality in interpretive engineering education research: Reflections on an example study. Journal of Engineering Education, 102(4), 626–659)
– Sochacka, N. W., Walther, J. and Pawley, A. L. (2018), Ethical Validation: Reframing Research Ethics in Engineering Education Research To Improve Research Quality. Journal of Engineering Education, 107(3): 362-379. doi:10.1002/jee.20222/
OTHER RESOURCES FOR PEER REVIEWERS
- The Role of Peer Review in Identity Development for Engineering Education Researchers – Anne Gardner & Keith Willey (2019) The role of peer review in identity development for engineering education researchers, European Journal of Engineering Education, 44:3, 347-359, DOI: 10.1080/03043797.2018.1500526
- Publons Reviewer Academy – Become a Master of Peer Review
- Wiley Reviewer Resources – Journal Reviewers
- SAGE Publishing – How to Review Articles
- AERA Reviewer Resources – Resources for AERA Journal Peer Reviewers
- Sage Connection Insights – More Free Scholarly Articles Discussing Peer Review
- Editor Resources – A Guide to Becoming a Peer Reviewer
- Science Magazine – Learning the Ropes of Peer Reviewing