We studied how accessible and reliable the computer code behind major climate models has been over time. By reviewing different phases of the Coupled Model Intercomparison Project, we found improvements in transparency and coding practices, but also gaps that limit reproducibility. Our work suggests practical steps to make future climate research more open, traceable, and trustworthy for scientists and society.
Añel, J. A.: The Importance of Reviewing the Code, Communication of the ACM, 54, 40–41, https://doi.org/10.1145/1941487.1941502, 2011. a
Añel, J. A.: Comment on “Most computational hydrology is not reproducible, so is it really science?” by Hutton et al., Water Resour. Res., 53, 2572–2574, https://doi.org/10.1002/2016WR020190, 2017. a
Añel, J. A.: Reflections on the Scientific Method at the beginning of the twenty-first century, Contemp. Phys., 1, 60–62, https://doi.org/10.1080/00107514.2019.1579863, 2019. a
Añel, J. A., Montes, D. P., and Rodeiro Iglesias, J.: Cloud and Serverless Computing for Scientists, Springer, ISBN 978-3-03-041783-3, https://doi.org/10.1007/978-3-030-41784-0, 2020. a, b
Añel, J. A., García-Rodríguez, M., and Rodeiro, J.: Current status on the need for improved accessibility to climate models code, Geosci. Model Dev., 14, 923–934, https://doi.org/10.5194/gmd-14-923-2021, 2021. a, b, c, d, e
Allison, D., Shiffrin, R., and Stodden, V.: Reproducibility of research: Issues and proposed remedies, P. Natl. Acad. Sci., 115, 2561–2562, https://doi.org/10.1073/pnas.1802324115, 2018. a, b
Arvanitou, E.-M., Ampatzoglou, A., Chatzigeorgiou, A., and Carver, J. C.: Software engineering practices for scientific software development: A systematic mapping study, J. Syst. Softw., 172, 110848, https://doi.org/10.1016/j.jss.2020.110848, 2021. a
Association for Computing Machinery: Artifact Review and Badging, https://www.acm.org/publications/policies/artifact-review-and-badging-current (last access: 31 May 2024), 2020. a, b
Backus, J.: The history of Fortran I, II, and III, 25–74, Association for Computing Machinery, New York, NY, USA, ISBN 0127450408, https://doi.org/10.1145/800025.1198345, 1978. a
Baxter, S. M., Day, S. W., Fetrow, J. S., and Reisinger, S. J.: Scientific Software Development Is Not an Oxymoron, PLOS Comput. Biol., 2, 1–4, https://doi.org/10.1371/journal.pcbi.0020087, 2006. a
Boehm, B. W.: A Spiral Model of Software Development and Enhancement, ACM SIGSOFT, 11, 14–24, https://doi.org/10.1145/358886.358895, 1986. a
Boulanger, A.: Open-source versus proprietary software: Is one more reliable and secure than other?, IBM Syst. J., 1, 239–248, https://doi.org/10.1147/sj.442.0239, 2005. a
Carlson, D., Eyring, V., van der Wel, N., and Langendijk, G.: WCRP's Coupled Model Intercomparison Project: A Remarkable Contribution to Climate Science, https://www.wcrp-climate.org/images/modelling/WGCM/CMIP/CMIP6FinalDesign_GMD_180329.pdf (last access: 31 May 2024), 2018. a
CMIP: Coupled Model Intercomparison Project, http://cmip-pcmdi.llnl.gov/ (last access: 31 May 2024), 2025a. a
CMIP: History of CMIP, https://pcmdi.llnl.gov/mips/cmip5/history.html (last access: 31 May 2024), 2025b. a
Costa, L., Barbosa, S., and Cunha, J.: Evaluating Tools for Enhancing Reproducibility in Computational Scientific Experiments, in: Proceedings of the 2nd ACM Conference on Reproducibility and Replicability, ACM REP '24, 46–51, Association for Computing Machinery, New York, NY, USA, ISBN 9798400705304, https://doi.org/10.1145/3641525.3663623, 2024. a
DoD CIO: Clarifying Guidance Regarding Open Source Software (OSS), Tech. rep., 6 pp., https://dodcio.defense.gov/Portals/0/Documents/FOSS/2009OSS.pdf (last access: 14 April 2024), 2009. a
Drake, J. B.: Climate Modeling for Scientists and Engineers, Society for Industrial and Applied Mathematics, ISBN 978-0898714778, 2001. a
Durack, P. J., Taylor, K. E., Gleckler, P. J., Meehl, G. A., Lawrence, B. N., Covey, C., Stouffer, R. J., Levavasseur, G., Ben-Nasser, A., Denvil, S., Stockhause, M., Gregory, J. M., Juckes, M., Ames, S. K., Antonio, F., Bader, D. C., Dunne, J. P., Ellis, D., Eyring, V., Fiore, S. L., Joussaume, S., Kershaw, P., Lamarque, J.-F., Lautenschlager, M., Lee, J., Mauzey, C. F., Mizielinski, M., Nassisi, P., Nuzzo, A., O’Rourke, E., Painter, J., Potter, G. L., Rodriguez, S., and Williams, D. N.: The Coupled Model Intercomparison Project (CMIP): Reviewing project history, evolution, infrastructure and implementation, EGUsphere [preprint], https://doi.org/10.5194/egusphere-2024-3729, 2025. a
Easterbrook, S. M.: Getting the source code for climate models, https://www.easterbrook.ca/steve/2009/06/getting-the-source-code-for-climate-models (last access: 31 May 2024), 2009. a
Easterbrook, S. M.: Climate Change: A Grand Software Challenge, in: Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research, FoSER '10, 99–104, Association for Computing Machinery, New York, NY, USA, ISBN 9781450304276, https://doi.org/10.1145/1882362.1882383, 2010. a
Righi, M., Andela, B., Eyring, V., Lauer, A., Predoi, V., Schlund, M., Vegas-Regidor, J., Bock, L., Brötz, B., de Mora, L., Diblen, F., Dreyer, L., Drost, N., Earnshaw, P., Hassler, B., Koldunov, N., Little, B., Loosveldt Tomas, S., and Zimmermann, K.: Earth System Model Evaluation Tool (ESMValTool) v2.0 – technical overview, Geosci. Model Dev., 13, 1179–1199, https://doi.org/10.5194/gmd-13-1179-2020, 2020. a
Fenton, N. E. and Bieman, J.: Software Metrics: A Rigorous and Practical Approach, CRC Press, https://doi.org/10.1201/b17461, 2014. a
García-Rodríguez, M.: FortranAnalyser, Zenodo [code], https://doi.org/10.5281/zenodo.5942943, 2022. a, b
García-Rodríguez, M., Añel, J. A., and Rodeiro-Iglesias, J.: Assessing and improving the quality of Fortran code in scientific software: FortranAnalyser, Software Impacts, 21, 100692, https://doi.org/10.1016/j.simpa.2024.100692, 2024. a, b, c
García-Rodríguez, M., Añel, J. A., and Rodeiro-Iglesias, J.: Source code and analysis results for CESM1 CMIP6 model, Zenodo [code, data set], https://doi.org/10.5281/zenodo.16933339, 2025a. a
García-Rodríguez, M., Añel, J. A., and Rodeiro-Iglesias, J.: Source code and analysis results for CFSv2 CMIP5 model, Zenodo [code, data set], https://doi.org/10.5281/zenodo.16932623, 2025b. a
García-Rodríguez, M., Añel, J. A., and Rodeiro-Iglesias, J.: Source code and analysis results for GEOS-5 CMIP5 model, Zenodo [code, data set], https://doi.org/10.5281/zenodo.16932661, 2025c. a
García-Rodríguez, M., Añel, J. A., and Rodeiro-Iglesias, J.: Source code and analysis results for GFDL CMIP5 model, Zenodo [code, data set], https://doi.org/10.5281/zenodo.16932683, 2025d. a
García-Rodríguez, M., Añel, J. A., and Rodeiro-Iglesias, J.: Source code and analysis results for GISS CMIP5 model, Zenodo [code, data set], https://doi.org/10.5281/zenodo.16932705, 2025e. a
García-Rodríguez, M., Añel, J. A., and Rodeiro-Iglesias, J.: Source code and analysis results for GISS CMIP6 model, Zenodo [code, data set], https://doi.org/10.5281/zenodo.16933370, 2025f. a
García-Rodríguez, M., Añel, J. A., and Rodeiro-Iglesias, J.: Source code and analysis results for HiRAM CMIP6 model, Zenodo [code, data set], https://doi.org/10.5281/zenodo.16932804, 2025g. a
García-Rodríguez, M., Añel, J. A., and Rodeiro-Iglesias, J.: Source code and analysis results for IPSL CMIP5 model, Zenodo [code, data set], https://doi.org/10.5281/zenodo.16932753, 2025h. a
García-Rodríguez, M., Añel, J. A., and Rodeiro-Iglesias, J.: Source code and analysis results for IPSL CMIP6 model, Zenodo [code, data set], https://doi.org/10.5281/zenodo.16933458, 2025i. a
Gardner, J. and GitHub staff: Linguist: Language Savvy, GitHub, https://github.com/github/linguist (last access: 31 May 2024), 2024. a
GMD Executive Editors: Editorial: The publication of geoscientific model developments v1.1, Geosci. Model Dev., 8, 3487–3495, https://doi.org/10.5194/gmd-8-3487-2015, 2015. a
Hunter-Zinck, H., de Siqueira, A. F., Vásquez, V. N., Barnes, R., and Martinez, C. C.: Ten simple rules on writing clean and reliable open-source scientific software, PLOS Comput. Biol., 17, 1–9, https://doi.org/10.1371/journal.pcbi.1009481, 2021. a
Kanewala, U. and Bieman, J. M.: Testing scientific software: A systematic literature review, Inform. Software Tech., 56, 1219–1232, https://doi.org/10.1016/j.infsof.2014.05.006, 2014. a, b
Kernighan, B. W. and Ritchie, D. M.: The C Programming Language, Prentice Hall, ISBN 978-0131103627, 1988. a
Knutti, R., Masson, D., and Gettelman, A.: Climate model genealogy: Generation CMIP5 and how we got there, Geophys. Res. Lett., 40, 1194–1199, https://doi.org/10.1002/grl.50256, 2013. a
Matthes, E.: Python Crash Course, No Starch Press, ISBN 978-1593279288, 2019. a
McCall, J., Richards, P., and Walters, G.: Factors in Software Quality, Proceedings of the Second International Conference on Software Engineering, 2, 155–160, 1977. a
McConnell, S.: Code Complete, Microsoft Press, ISBN 978-0-7356-1967-8, 2004. a
Metcalf, M., Reid, J., and Cohen, M.: Modern Fortran Explained, Oxford University Press, numerical recipes in fortran 90th edn., ISBN 978-0199601424, 2011. a
Moore, G. E.: Cramming more components onto integrated circuits, McGraw-Hill, 38th edn., https://doi.org/10.1109/JPROC.1998.658762, 1965. a
Nature: Does your code stand up to scrutiny?, Nature, 555, https://doi.org/10.1038/d41586-018-02741-4, 2018. a
Nguyen-Hoan, L., Flint, S., and Sankaranarayana, R.: A Survey of Scientific Software Development, in: Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM '10, Association for Computing Machinery, New York, NY, USA, ISBN 9781450300391, https://doi.org/10.1145/1852786.1852802, 2010. a
Perkel, J. M.: Containers in the Cloud, Nature, SIAM News, 1, 247–248, https://doi.org/10.1038/d41586-019-03366-x, 2019. a
Pipitone, J. and Easterbrook, S.: Assessing climate model software quality: a defect density analysis of three models, Geosci. Model Dev., 5, 1009–1022, https://doi.org/10.5194/gmd-5-1009-2012, 2012. a
Pressman, R. S.: Software Engineering: A Practitioner's Approach, McGraw-Hill, ISBN 978-0073375977, 2005. a
RealClimate.org: Sources of code and data related to climate science, http://www.realclimate.org/index.php/data-sources (last access: 31 May 2024), 2009. a
Riesch, M., Nguyen, T. D., and Jirauschek, C.: bertha: Project skeleton for scientific software, PLOS ONE, 15, 1–12, https://doi.org/10.1371/journal.pone.0230557, 2020. a
Seibel, P.: Practical Common Lisp, Apress, Berkeley, CA, ISBN 978-1-59059-239-7, 2005. a
Stodden, V., Guo, P., and Ma, Z.: Toward Reproducible Computational Research: An Empirical Analysis of Data and Code Policy Adoption by Journals, PLoS ONE, 8, e67111, https://doi.org/10.1371/journal.pone.0067111, 2013. a
Stodden, V., Seiler, J., and Ma, Z.: An empirical analysis of journal policy effectiveness for computational reproducibility, P. Natl. Acad. Sci., 115, 2584–2589, https://doi.org/10.1073/pnas.1708290115, 2018. a, b
Stouffer, R., Eyring, V., Meehl, G., Bony, S., Senior, C., Stevens, B., and Taylor, K.: CMIP5 Scientific Gaps and Recommendations for CMIP6, B. Am. Meteorol. Soc., 98, 95–105, https://doi.org/10.1175/BAMS-D-15-00013.1, 2017. a
Stroustrup, B.: The C++ Programming Language, Addison-Wesley, 4th edn., ISBN 978-0321563842, 2013. a
Trisovic, A., Lau, M. K., Pasquier, T., Crosas, M., and Broman, K. W.: A large-scale study on research code quality and execution, Sci. Data, 9, 60, https://doi.org/10.1038/s41597-022-01143-6, 2022. a
U.S. Code: 17 U.S. Code § Section 105. Subject matter of copyright: United States Government works, https://www.law.cornell.edu/uscode/text/17/105 (last access: 3 August 2026), 1976. a
van Wendel de Joode, R., de Bruijn, J. A., and van Eeten, M. J. G.: Protecting the Virtual Commons, T.M.C. Asser Press, ISBN 978-90-6704-159-1, 2003. a
Wieters, N. and Fritzsch, B.: Opportunities and limitations of software project management in geoscience and climate modelling, Adv. Geosci., 45, 383–387, https://doi.org/10.5194/adgeo-45-383-2018, 2018. a
Wilson, G., Aruliah, D. A., Brown, C. T., Chue Hong, N. P., Davis, M., Guy, R. T., Haddock, S. H. D., Huff, K. D., Mitchell, I. M., Plumbley, M. D., Waugh, B., White, E. P., and Wilson, P.: Best Practices for Scientific Computing, PLOS Biol., 12, 1–7, https://doi.org/10.1371/journal.pbio.1001745, 2014. a
Wilson, G., Bryan, J., Cranston, K., Kitzes, J., Nederbragt, L., and Teal, T.: Good enough practices in scientific computing, PLOS Comput. Biol., 13, e1005510, https://doi.org/10.1371/journal.pcbi.1005510, 2017. a
Wu, C., Chakravorti, T., Carroll, J. M., and Rajtmajer, S.: Integrating measures of replicability into scholarly search: Challenges and opportunities, in: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI '24, Association for Computing Machinery, New York, NY, USA, ISBN 9798400703300, https://doi.org/10.1145/3613904.3643043, 2024. a
Zaporozhets, D.: GitLab, https://about.gitlab.com/ (last access: 31 May 2024), 2011. a