Assessing individual contributions to software engineering projects: a replication study

ABSTRACT

Background and Context

Assessing team members’ indivdiual contributions to software development projects poses a key problem for computing instructors. While instructors typically rely on subjective assessments, objective assessments could provide a more robust picture. To explore this possibility, In a 2020 paper, Buffardi presented a correlational analysis of objective metrics and subjective metrics in an advanced software engineering project course (n= 41 students and 10 teams), finding only two significant correlations.

Objective

To explore the robustness of Buffardi’s findings and gain further insight, we conducted a larger scale replication of the Buffardi study (n = 118 students and 25 teams) in three courses at three institutions.

Method

We collected the same data as in the Buffardi study and computed the same measures from those data. We replicated Buffardi’s exploratory, correlational and regression analyses of objective and subjective measures.

Findings

While replicating four of Buffardi’s five significant correlational findings and partially replicating the findings of Buffardi’s regression analyses, our results go beyond those of Buffardi by identifying eight additional significant correlations.

Implications

In contrast to Buffardi’s study, our larger scale study suggests that subjective and objective measures of individual performance in team software development projects can be fruitfully combined to provide consistent and complementary assessments of individual performance.

KEYWORDS:

• Team software development projects
• individual contributions
• GitHub log data analysis
• peer evaluation
• issues
• commits
• user stories

Acknowledgments

This research is supported by the National Science Foundation under grant no. DUE 1915196. We are grateful to all our student participants who consented to share their data for research purposes, and to Kevin Buffardi for supplying us with additional data from his study and helping us to better understand what he did.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the National Science Board [DUE 1915196]; National Science Foundation [DUE 1915196]; National Science Foundation [DUE 1915196]; National Science Foundation [DUE 1915196].