Replication of Controlled Experiments
in Empirical Software Engineering—A Survey
Here is a link to two pdf versions of my thesis:
My thesis was marked väl godkänd (high pass) and is published by the Department of Computer Science at Lund university.
- ISSN 1651-6389
- LU-DV-EX: 2006-75
It is also published electronically by Lund university and listed at the National Library.
Keywords
Empirical software engineering, replicated experiments, survey, replication, replicate,
replica, experiment series, external validity, construct validity, guidelines
Abstract
This survey studies 51 replicated experiments in empirical software engineering in 20 experiment series. These 51 experiments constitute all replicated experiments identifiable by searches of electronic databases of scientific publications.
To study the phenomenon of replication of experiments in software engineering, the report sets out with a detailed review of existing guidelines for experimentation and replication, both within the field of software engineering and from other disciplines.
Thereafter, the experiments and series of experiments are studied from both a quantitative and a qualitative perspective. The survey identifies the journals and conferences where experiment results have been announced, the researchers and institutions most active in replication, as well as the success rates of various types of replication.
The survey also analyses how researchers describe their experience with replication, with regards to access to original material (lab packages), motivations for replication and discussion about experiment subjects (students vs. professionals).
Finally, the conclusions of the survey are discussed, and proposals are made with regards to enhancing guidelines for experimentation and replications, as well as proposals for further work.