Reproducibility in M&S research: issues, strategies and implications for model development environments

References

• Alberts, B. (2010), ‘Editorial Expression of Concern’, Science, 327, 144, http://www.sciencemag.org/content/327/5962/144.1.full
   Google Scholar
• Altman, M., and King, G. (2007), ‘A Proposed Standard for the Scholarly Citation of Quantitative Data’, D-Lib Magazine, March/April 2007, http://www.dlib.org/dlib/march07/altman/03altman.html
   Google Scholar
• Anand , M , Bowers , S , McPhillips , T and Ludascher , B . 2009 . Exploring Scientific Workflow Provenance Using Hybrid Queries over Nested Data and Lineage Graphs . Proceedings of the Scientific and Statistical Database Management , : 237 – 254 .
   Google Scholar
• Annotea (2011). ‘The Annotea Project’. http://www.w3.org/2001/Annotea/
   Google Scholar
• Baggerly , K and Coombes , K . 2009 . Deriving Chemosensitivity from Cell Lines: Forensic Bioinformatics and Reproducible Research in High-Throughput Biology . Annals Applied Statistics , 3 : 1309 – 1334 .
   Web of Science ®Google Scholar
• Beck, K. (2003), ‘Test-Driven Development by Example’, Boston: Addison-Wesley, 2003
   Google Scholar
• Borgman , CL . 2007 . Scholarship in the Digital Age: Information, Infrastructure, and the Internet , Cambridge , MA : The MIT Press .
   Google Scholar
• Chang, G., Roth, B.C., Reyes, L.C., Pornillos, O., Chen, Yen-Ju, Chen P. A. (2006), ‘Retraction’, Science, 314, 1875, http://www.sciencemag.org/cgi/content/full/314/5807/1875b
   Google Scholar
• Creative Commons (2011), http://www.creativecommons.org
   Google Scholar
• Cummings, T.P., and Glotzer, S.C. (2010), Inventing a New America Through Discovery and Innovation in Science, Engineering, and Medicine, Baltimore, MD: World Technology Evaluation Center
   Google Scholar
• Data Cite (2011), http://thedata.org/citation/tech
   Google Scholar
• David, A.P. and Spence, M. (2003), ‘Toward Institutional Infrastructures for e-Science: The Scope of Challenges’, Technical Report, Oxford Technical Institute, May 2003
   Google Scholar
• Davis, P. and Anderson, A. R. (2003), Improving the Composability of Department of Defense Simulations, RAND Corporation
   Google Scholar
• De Roure , D , Goble , C and Stevens , R . 2009 . The Design and Realisation of the myExperiment Virtual Research Environment for Social Sharing of Workflows . Future Generation Computer Systems , 25 : 561 – 567 .
   Web of Science ®Google Scholar
• Donoho , D , Maleki , A , Rahman , I , Shahram , M and Stodden , V . 2009 . Reproducible Research in Computational Harmonic Analysis . Computing in Science and Engineering , 11 : 818
   Google Scholar
• Economist (2010), ‘The Clouds of Unknowing’, The Economist, March 18, 2010, http://www.economist.com/node/15719298
   Google Scholar
• Edwards , HS . 2003 . Improving Student Performance by Evaluating How Well Students Test Their own programs . Journal of Educational Resources in Computing , 3 : 1 – 24 .
   Google Scholar
• Fenton , EN and Bieman , J . 2010 . Software Metrics: A Rigorous and Practical Approach , London : Chapman and Hall .
   Google Scholar
• Fomel , S and Claerbout , J . 2009 . Guest Editors’ Introduction: Reproducible Research . Computing in Science and Engineering , 11 : 5 – 7 .
   Web of Science ®Google Scholar
• Fomel , S and Hennenfent , G . 2009 . Reproducible Computational Experiments using Scons . IEEE International Conference on Acoustics, Speech, and Signal Processing , 4 : 1257 – 1260 .
   Google Scholar
• Foster , I . 2005 . Service-oriented Science . Science , 308 : 814 – 817 .
   PubMed Web of Science ®Google Scholar
• Gentleman , R . 2005 . Reproducible Research: A Bioinformatics Case Study . Statistical Applications in Genetics and Molecular Biology , 4 : 123 – 137 .
   Google Scholar
• Kiczales , G , Des Rivieres , J and Bobrow , DG . 1992 . The Art of the Meta-object Protocol , Cambridge , MA : MIT Press .
   Google Scholar
• Leisch, F. (2003), ‘Sweave and Beyond: Computations on Text Documents’, in Proceedings of the 3rd International Workshop on Distributed Statistical Computing, March 20–22, Vienna, Austria, pp. 87–96
   Google Scholar
• Mesirov , J . 2010 . Accessible Reproducible Research . Science , 327 : 415 – 416 .
   PubMed Web of Science ®Google Scholar
• NSF (2006), ‘Simulation-Based Engineering Science, Revolutionizing Engineering Science Through Simulation’, National Science Foundation, Washington, DC, May 2006
   Google Scholar
• NSF (2011), ‘Changing the Conduct of Science in the Information Age’, Summary Report of Workshop Held on November 12, 2010, National Science Foundation, June 28, 2011, http://www.nsf.gov/pubs/2011/oise11003/
   Google Scholar
• Oinn , T , Addis , M , Ferris , J , Marvin , D , Senger , M , Greenwood , M , Carver , T and Glover , K . 2004 . Taverna: A Tool for the Composition and Enactment of Bioinformatics Workflows . Bioinformatics , 20 : 3045 – 3054 .
   PubMed Web of Science ®Google Scholar
• Overstreet, M.C., Nance, R.E., and Balci, O. (2002), ‘Issues in Enhancing Model Reuse’, in Proceedings of the First International Conference on Grand Challenges for Modeling and Simulation
   Google Scholar
• Peng , R . 2009 . Reproducible Research and Biostatistics . Biostatistics , 10 : 405 – 408 .
   PubMed Web of Science ®Google Scholar
• reST (2011). ‘reStructuredText: Markup Syntax and Parser Component of Docutils’, http://docutils.sourceforge.net/rst.html
   Google Scholar
• Saeki, M., Hiroi, T., and Ugai, T. (1993), ‘Reflective Specification: Applying a Reflective Language to Formal Specification’, in Proceedings of the 7th International Workshop on Software Specification and Design, December 6–7, IEEE CS Press, pp. 204–213
   Google Scholar
• Schwab , M , Karrenbach , M and Claerbout , J . 2000 . Making Scientific Computations Reproducible . Computing in Science and Engineering , 2 : 61 – 67 .
   Web of Science ®Google Scholar
• Sphinx (2011), ‘Python Document Generator’, http://sphinx.pocoo.org
   Google Scholar
• Stodden , V . 2009 . The Legal Framework for Reproducible Scientific Research: Licensing and Copyright . Computing in Science and Engineering , 11 : 35 – 40 .
   Web of Science ®Google Scholar
• Stodden, V. (2010), ‘The Scientific Method in Practice: Reproducibility in the Computational Sciences’, Technical Report, MIT Sloan Research, February 2010
   Google Scholar
• Wilensky, U., and Rand, W. (2007), ‘Making Models Match: Replicating an Agent-Based Model’, Journal of Artificial Societies and Social Simulation, 10, http://jasss.soc.surrey.ac.uk/10/4/2.html
   Google Scholar
• WTEC (2009), Panel Report on International Assessment of Research and Development in Simulation-Based Engineering and Science, World Technology Evaluation Center
   Google Scholar
• Yilmaz , L . 2005 . On the Need for Contextualized Introspective Simulation Models to Improve Reuse and Composability of Defense Simulations . Journal of Defense Modeling and Simulation , 1 : 135 – 145 .
   Google Scholar
• Yilmaz , L . 2011a . Reproducibility in Modeling & Simulation Research . Simulation , 87 : 3 – 6 .
   Google Scholar
• Yilmaz, L. (2011b), ‘Simulation Reproducibility: Implications for Theory, Methodology, and Model Development Environments’, Epistemology of Modeling and Simulation Conference, 1–3 April, University of Pittsburgh, http://www.modelingepistemology.pitt.edu/program/
   Google Scholar