We recently published a paper that describes a framework to highlight human and social aspects in empirical software engineering research. This framework brings elements from the Design Science research we have done, research my student Courtney Williams did for her Master’s thesis, the keynote I gave at ICSE 2019, and a framework developed by McGrath and Runkel back in the 70s. We apply our Who-What-How framework to two paper venues in our community: papers published at ICSE 2017 and in the Springer Empirical Software Engineering Journal in 2017. The following shows the main elements in our framework and below you can find a teaser of the results we found from our analysis. In a nutshell, what we found is that although many papers aim at supporting developers or improving developer productivity, most papers do not directly involve developers in their research but rely on data.

The Who What How Framework:

Title:

The who, what, how of software engineering research: a socio-technical framework

Authors:

Margaret-Anne Storey, Neil A. Ernst, Courtney Williams & Eirini Kalliamvakou

Abstract:

Software engineering is a socio-technical endeavor, and while many of our contributions focus on technical aspects, human stakeholders such as software developers are directly affected by and can benefit from our research and tool innovations. In this paper, we question how much of our research addresses human and social issues, and explore how much we study human and social aspects in our research designs. To answer these questions, we developed a socio-technical research framework to capture the main beneficiary of a research study (the who), the main type of research contribution produced (the what), and the research strategies used in the study (how we methodologically approach delivering relevant results given the who and what of our studies). We used this Who-What-How framework to analyze 151 papers from two well-cited publishing venues—the main technical track at the International Conference on Software Engineering, and the Empirical Software Engineering Journal by Springer—to assess how much this published research explicitly considers human aspects. We find that although a majority of these papers claim the contained research should benefit human stakeholders, most focus predominantly on technical contributions. Although our analysis is scoped to two venues, our results suggest a need for more diversification and triangulation of research strategies. In particular, there is a need for strategies that aim at a deeper understanding of human and social aspects of software development practice to balance the design and evaluation of technical innovations. We recommend that the framework should be used in the design of future studies in order to steer software engineering research towards explicitly including human and social concerns in their designs, and to improve the relevance of our research for human stakeholders.

Citation:

Storey, M., Ernst, N.A., Williams, C. et al. The who, what, how of software engineering research: a socio-technical framework. Empir Software Eng 25, 4097–4129 (2020). https://doi.org/10.1007/s10664-020-09858-z Our paper is online at this link.

Teaser of the results!