Reproducibility in Systems Biology Modelling
Systems biology modelling involves the mathematical representation of biological processes to study complex system behaviour and was expected to be less affected by the reproducibility crisis. However, models often fail to reproduce and the reasons for the failure and prevalence were not fully understood. We analyzed 455 kinetic models published in 152 peer-reviewed journals, a collective work of about 1400 scientists from 49 countries. Most of these models were manually encoded from scratch to assess the reproducibility. Our investigation revealed that 49% of the models could not be reproduced using the information provided in the manuscripts. With further effort, an additional 12% could be reproduced either by empirical correction or support from authors. The other 37% remained non-reproducible due to missing parameter values, missing initial concentration, or inconsistent model structure. Among the corresponding authors, we contacted <30% responded. Models from many life science journals failed to reproduce, revealing a common problem in the peer-review process. Hence, we propose an 8-point reproducibility scorecard (Box 1) to assess each model and address the reproducibility crisis.
Read more:
Tiwari et al. 2020 Reproducibility in systems biology modelling. Mol Syst Biol. (2021) 17: e9982; doi: https://doi.org/10.15252/msb.20209982
Click here to download the Reproducibility Scorecard document
|
Box1 |
|
|
Reproducibility scorecard |
|
|
1) Are the mathematical expressions provided and explained in the manuscript/supplementary material? |
|
|
2) Are the parameters and entity initial levels in the manuscript/supplementary material (e.g., listed as a table)? |
|
|
3) Are simulation conditions including software/programming environment, algorithm, changes in parameters/concentration/states, and any data normalization described under each simulation figure or attached as supplementary material? |
|
|
4) Are the model code(s) for the mathematical expression and simulation shared publicly? |
|
|
5) Are the model codes shared in standard formats such as SBML, CellML, and COMBINE archive? Are instructions for simulation experiments and analysis available in standard format such as SED-ML? Are they syntactically validated? |
|
|
6) Are the model codes deposited in a relevant open model database (e.g., BioModels, Physiome, JWSOnline)? |
|
|
7) Are the model codes well documented to unambiguously identify model entities/variables? (with additional annotations characterising reactions, mathematical expressions, events, conditions, etc. where relevant.) |
|
|
8) Are the numerical results shared publicly along with the model codes? |
|
|
Total Score (out of 8) |
|
List of 455 models analysed in this work and their reproducibility status
R = Reproducible
RMC = Reproduced with model correction
RAH = Reproduced with author help
NRP = Non reproducible
|
Model ID |
BioModels link |
Reproducibility status |
|
MODEL1812040007 |
NRP |
|
|
MODEL1708310000 |
R |
|
|
MODEL1006230057 |
R |
|
|
MODEL0911270008 |
R |
|
|
MODEL8687732743 |
R |
|
|
MODEL1006230001 |
R |
|
|
MODEL1604270000 |
RMC |
|
|
MODEL1604270001 |
RMC |
|
|
MODEL1708250001 |
R |
|
|
MODEL1006230019 |
R |
|
|
MODEL1006230004 |
R |
|
|
MODEL1006230005 |
R |
|
|
MODEL1006230007 |
R |
|
|
MODEL0912160005 |
R |
|
|
MODEL1006230002 |
RMC |
|
|
MODEL1006230011 |
RMC |
|
|
MODEL1006230098 |
RMC |
|
|
MODEL1006230006 |
R |
|
|
MODEL1006230062 |
RMC |
|
|
MODEL1006230061 |
RMC |
|
|
MODEL1006230094 |
R |
|
|
MODEL1006230086 |
R |
|
|
MODEL1006230052 |
R |
|
|
MODEL1006230097 |
R |
|
|
MODEL1602280002 |
RMC |
|
|
MODEL1602280003 |
RMC |
|
|
MODEL1006230022 |
R |
|
|
MODEL4968912141 |
R |
|
|
MODEL7817907010 |
RMC |
|
|
MODEL1701090001 |
RMC |
|
|
MODEL1511290000 |
R |
|
|
MODEL1006230091 |
R |
|
|
MODEL1803300000 |
R |
|
|
MODEL0848279215 |
R |
|
|
MODEL1006230054 |
R |
|
|
MODEL6960055446 |
R |
|
|
MODEL1202090002 |
R |
|
|
MODEL1202090003 |
R |
|
|
MODEL0913242281 |
R |
|
|
MODEL2504064544 |
R |
|
|
MODEL1703030000 |
R |
|
|
MODEL1112260000 |
RMC |
|
|
MODEL1112260001 |
RMC |
|
|
MODEL1006230047 |
R |
|
|
MODEL1808280004 |
R |
|
|
MODEL1808280002 |
R |
|
|
MODEL1808280003 |
R |
|
|
MODEL1808280005 |
R |
|
|
MODEL1808280001 |
R |
|
|
MODEL1808280012 |
RMC |
|
|
MODEL1808240002 |
RAH |
|
|
MODEL1808240003 |
RAH |
|
|
MODEL1808280008 |
R |
|
|
MODEL1808280009 |
R |
|
|
MODEL1808280010 |
R |
|
|
MODEL2004060002 |
R |
|
|
MODEL2004060001 |
R |
|
|
MODEL1511020000 |
R |
|
|
MODEL1811150001 |
RMC |
|
|
MODEL1811210001 |
R |
|
|
MODEL1812040008 |
R |
|
|
MODEL1809060006 |
R |
|
|
MODEL1811070001 |
RMC |
|
|
MODEL1811080001 |
R |
|
|
MODEL1811220001 |
RAH |
|
|
MODEL1812110002 |
RMC |
|
|
MODEL1812120004 |
R |
|
|
MODEL1812120005 |
R |
|
|
MODEL1812210001 |
R |
|
|
MODEL1901250001 |
R |
|
|
MODEL1006230038 |
R |
|
|
MODEL1605030002 |
R |
|
|
MODEL1605030003 |
R |
|
|
MODEL1605030004 |
R |
|
|
MODEL1605030005 |
R |
|
|
MODEL1903040001 |
R |
|
|
MODEL1807020001 |
R |
|
|
MODEL1808160002 |
R |
|
|
MODEL1907050001 |
R |
|
|
MODEL1907050002 |
R |
|
|
MODEL1907050003 |
R |
|
|
MODEL1907050004 |
R |
|
|
MODEL1907240003 |
R |
|
|
MODEL1907100001 |
R |
|
|
MODEL1907050005 |
R |
|
|
MODEL1907230001 |
R |
|
|
MODEL1807230003 |
R |
|
|
MODEL1907110001 |
R |
|
|
MODEL1907110002 |
RMC |
|
|
MODEL1907230002 |
R |
|
|
MODEL1907160001 |
R |
|
|
MODEL1907160002 |
R |
|
|
MODEL1907170001 |
R |
|
|
MODEL1907180001 |
R |
|
|
MODEL1907230003 |
R |
|
|
MODEL1907180002 |
R |
|
|
MODEL1907240002 |
R |
|
|
MODEL1907240001 |
R |
|
|
MODEL1907220001 |
R |
|
|
MODEL1907180004 |
R |
|
|
MODEL1907240004 |
R |
|
|
MODEL2005110001 |
R |
|
|
MODEL1908130003 |
R |
|
|
MODEL1910250001 |
R |
|
|
MODEL1907250001 |
R |
|
|
MODEL1907260017 |
RMC |
|
|
MODEL1907290001 |
R |
|
|
MODEL1907290002 |
R |
|
|
MODEL1907290003 |
R |
|
|
MODEL1907300001 |
R |
|
|
MODEL1907310004 |
R |
|
|
MODEL1907310003 |
R |
|
|
MODEL1907310005 |
R |
|
|
MODEL1908010001 |
R |
|
|
MODEL1908010003 |
R |
|
|
MODEL1908010002 |
R |
|
|
MODEL1908020001 |
R |
|
|
MODEL1908040001 |
R |
|
|
MODEL1908020002 |
R |
|
|
MODEL1908050002 |
R |
|
|
MODEL1908050001 |
R |
|
|
MODEL1908120002 |
R |
|
|
MODEL1908060001 |
R |
|
|
MODEL1908120004 |
RMC |
|
|
MODEL1908060002 |
R |
|
|
MODEL1908060003 |
R |
|
|
MODEL1908080001 |
R |
|
|
MODEL1908090001 |
R |
|
|
MODEL1908090002 |
R |
|
|
MODEL1908120001 |
R |
|
|
MODEL1908120003 |
R |
|
|
MODEL1908090003 |
R |
|
|
MODEL1908130001 |
R |
|
|
MODEL1908130002 |
R |
|
|
MODEL1908130004 |
R |
|
|
MODEL1908150001 |
R |
|
|
MODEL1908140001 |
R |
|
|
MODEL1908190001 |
R |
|
|
MODEL1908190002 |
R |
|
|
MODEL1908200002 |
RMC |
|
|
MODEL1908200001 |
RMC |
|
|
MODEL1909170002 |
RAH |
|
|
MODEL1908190003 |
R |
|
|
MODEL1908210001 |
R |
|
|
MODEL1909050001 |
RMC |
|
|
MODEL1909050003 |
RAH |
|
|
MODEL1908290002 |
RMC |
|
|
MODEL1909060001 |
RAH |
|
|
MODEL1909030001 |
RMC |
|
|
MODEL1909180001 |
RAH |
|
|
MODEL1909090001 |
R |
|
|
MODEL1909120001 |
R |
|
|
MODEL1909110001 |
R |
|
|
MODEL1909160001 |
R |
|
|
MODEL1909170001 |
RMC |
|
|
MODEL1909170003 |
R |
|
|
MODEL1909250001 |
R |
|
|
MODEL1909180002 |
RAH |
|
|
MODEL1909230001 |
R |
|
|
MODEL1909170004 |
R |
|
|
MODEL1909240001 |
R |
|
|
MODEL1909240003 |
R |
|
|
MODEL1909250002 |
RAH |
|
|
MODEL1909250003 |
R |
|
|
MODEL1909270001 |
R |
|
|
MODEL1909260001 |
R |
|
|
MODEL1909260002 |
R |
|
|
MODEL1909300001 |
R |
|
|
MODEL1909300005 |
RMC |
|
|
MODEL1603110003 |
RMC |
|
|
MODEL1910020001 |
RMC |
|
|
MODEL1609100001 |
R |
|
|
MODEL1604100005 |
RMC |
|
|
MODEL7898438988 |
RMC |
|
|
MODEL1403250000 |
R |
|
|
MODEL1203070000 |
RMC |
|
|
MODEL1910250002 |
R |
|
|
MODEL1910290001 |
R |
|
|
MODEL1910290002 |
R |
|
|
MODEL1911010001 |
R |
|
|
MODEL1012080000 |
R |
|
|
MODEL1807160001 |
R |
|
|
MODEL1911100001 |
R |
|
|
MODEL1710230000 |
R |
|
|
MODEL1911100002 |
R |
|
|
MODEL1911100003 |
R |
|
|
MODEL1911110001 |
R |
|
|
MODEL1911100004 |
R |
|
|
MODEL1911120002 |
R |
|
|
MODEL1911120003 |
R |
|
|
MODEL1911120005 |
R |
|
|
MODEL1911120006 |
R |
|
|
MODEL1911130001 |
R |
|
|
MODEL1911130002 |
R |
|
|
MODEL1902140001 |
R |
|
|
MODEL1803220002 |
R |
|
|
MODEL1911130004 |
R |
|
|
MODEL1911140001 |
R |
|
|
MODEL1911130005 |
R |
|
|
MODEL1911130006 |
R |
|
|
MODEL1610100004 |
R |
|
|
MODEL1911140003 |
R |
|
|
MODEL1610100000 |
R |
|
|
MODEL1911180001 |
R |
|
|
MODEL1610100001 |
R |
|
|
MODEL1911180002 |
R |
|
|
MODEL1911270003 |
R |
|
|
MODEL1911270004 |
R |
|
|
MODEL1911280001 |
R |
|
|
MODEL1911280003 |
R |
|
|
MODEL1903280001 |
R |
|
|
MODEL1903280002 |
R |
|
|
MODEL1904020002 |
R |
|
|
MODEL1904020003 |
R |
|
|
MODEL1904030001 |
R |
|
|
MODEL1605030001 |
R |
|
|
MODEL1812070001 |
R |
|
|
MODEL1006230079 |
R |
|
|
MODEL8268650277 |
R |
|
|
MODEL1201140002 |
R |
|
|
MODEL1912020001 |
R |
|
|
MODEL1008060001 |
R |
|
|
MODEL1912040001 |
RAH |
|
|
MODEL1912100003 |
R |
|
|
MODEL1912120001 |
R |
|
|
MODEL1912120004 |
R |
|
|
MODEL1912160002 |
R |
|
|
MODEL1912160004 |
R |
|
|
MODEL1912160005 |
RMC |
|
|
MODEL1912170002 |
R |
|
|
MODEL1912170003 |
R |
|
|
MODEL1912090004 |
R |
|
|
MODEL1912130002 |
RAH |
|
|
MODEL1912100002 |
R |
|
|
MODEL1912100006 |
R |
|
|
MODEL1912130001 |
R |
|
|
MODEL1912160003 |
RMC |
|
|
MODEL2001060001 |
R |
|
|
MODEL1912170001 |
R |
|
|
MODEL1912180002 |
R |
|
|
MODEL1912180005 |
R |
|
|
MODEL1912180006 |
R |
|
|
MODEL1912190002 |
R |
|
|
MODEL2001030001 |
R |
|
|
MODEL1912190001 |
R |
|
|
MODEL2001080003 |
R |
|
|
MODEL2001200003 |
R |
|
|
MODEL2001140001 |
R |
|
|
MODEL2001150001 |
R |
|
|
MODEL2001200001 |
R |
|
|
MODEL2003060003 |
R |
|
|
MODEL2003110001 |
RMC |
|
|
MODEL2001220002 |
RMC |
|
|
MODEL2003060002 |
RMC |
|
|
MODEL1909300002 |
RMC |
|
|
MODEL2002280001 |
R |
|
|
MODEL1006230025 |
RMC |
|
|
MODEL2003030001 |
RAH |
|
|
MODEL1803200002 |
R |
|
|
MODEL2003130001 |
R |
|
|
MODEL2003170001 |
R |
|
|
MODEL1808280007 |
R |
|
|
MODEL1808140001 |
R |
|
|
MODEL1910020003 |
R |
|
|
MODEL1704120000 |
RMC |
|
|
MODEL2003190007 |
RAH |
|
|
MODEL2004140001 |
R |
|
|
MODEL2004240001 |
R |
|
|
MODEL1809050002 |
R |
|
|
MODEL1809040003 |
R |
|
|
MODEL1809040002 |
R |
|
|
MODEL1809040004 |
R |
|
|
MODEL1809050001 |
R |
|
|
MODEL1809060003 |
R |
|
|
MODEL2004230001 |
R |
|
|
MODEL1809060001 |
R |
|
|
MODEL1809060005 |
R |
|
|
MODEL1809060002 |
R |
|
|
MODEL1809060004 |
R |
|
|
MODEL1806270001 |
R |
|
|
MODEL1805230002 |
R |
|
|
MODEL1805220002 |
R |
|
|
MODEL1805210001 |
R |
|
|
MODEL1805170002 |
R |
|
|
MODEL1803050002 |
R |
|
|
MODEL1803050001 |
R |
|
|
MODEL9200487367 |
NRP |
|
|
MODEL9147975215 |
NRP |
|
|
MODEL8938094216 |
NRP |
|
|
MODEL8459127548 |
NRP |
|
|
MODEL7896869925 |
NRP |
|
|
MODEL3897771820 |
NRP |
|
|
MODEL2005200001 |
NRP |
|
|
MODEL2005040001 |
NRP |
|
|
MODEL2004300001 |
NRP |
|
|
MODEL2004170001 |
NRP |
|
|
MODEL2004140002 |
NRP |
|
|
MODEL2003200002 |
NRP |
|
|
MODEL2003200001 |
NRP |
|
|
MODEL2003190008 |
NRP |
|
|
MODEL2003190006 |
NRP |
|
|
MODEL2003190005 |
NRP |
|
|
MODEL2003190004 |
NRP |
|
|
MODEL2003190003 |
NRP |
|
|
MODEL2003190002 |
NRP |
|
|
MODEL2003190001 |
NRP |
|
|
MODEL2003180003 |
NRP |
|
|
MODEL2003180002 |
NRP |
|
|
MODEL2003180001 |
NRP |
|
|
MODEL2003160002 |
NRP |
|
|
MODEL2003140001 |
NRP |
|
|
MODEL2003060001 |
NRP |
|
|
MODEL2003050002 |
NRP |
|
|
MODEL2003050001 |
NRP |
|
|
MODEL2003040001 |
NRP |
|
|
MODEL2003030003 |
NRP |
|
|
MODEL2003030002 |
NRP |
|
|
MODEL2002250001 |
NRP |
|
|
MODEL2002030001 |
NRP |
|
|
MODEL2001310001 |
NRP |
|
|
MODEL2001220001 |
NRP |
|
|
MODEL2001200002 |
NRP |
|
|
MODEL2001160002 |
NRP |
|
|
MODEL2001160001 |
NRP |
|
|
MODEL2001140002 |
NRP |
|
|
MODEL2001130003 |
NRP |
|
|
MODEL2001090002 |
NRP |
|
|
MODEL2001090001 |
NRP |
|
|
MODEL2001080002 |
NRP |
|
|
MODEL2001030002 |
NRP |
|
|
MODEL1912170005 |
NRP |
|
|
MODEL1912170004 |
NRP |
|
|
MODEL1912160006 |
NRP |
|
|
MODEL1912120005 |
NRP |
|
|
MODEL1912120003 |
NRP |
|
|
MODEL1912110001 |
NRP |
|
|
MODEL1912100005 |
NRP |
|
|
MODEL1912100004 |
NRP |
|
|
MODEL1912090003 |
NRP |
|
|
MODEL1912090002 |
NRP |
|
|
MODEL1912090001 |
NRP |
|
|
MODEL1911280002 |
NRP |
|
|
MODEL1911270002 |
NRP |
|
|
MODEL1911270001 |
NRP |
|
|
MODEL1911190002 |
NRP |
|
|
MODEL1911190001 |
NRP |
|
|
MODEL1911150001 |
NRP |
|
|
MODEL1911130007 |
NRP |
|
|
MODEL1911130003 |
NRP |
|
|
MODEL1911120004 |
NRP |
|
|
MODEL1911120001 |
NRP |
|
|
MODEL1911100005 |
NRP |
|
|
MODEL1911070001 |
NRP |
|
|
MODEL1910290003 |
NRP |
|
|
MODEL1910240001 |
NRP |
|
|
MODEL1910030001 |
NRP |
|
|
MODEL1910020002 |
NRP |
|
|
MODEL1910010001 |
NRP |
|
|
MODEL1909300004 |
NRP |
|
|
MODEL1909300003 |
NRP |
|
|
MODEL1909240002 |
NRP |
|
|
MODEL1909230002 |
NRP |
|
|
MODEL1909100002 |
NRP |
|
|
MODEL1909090003 |
NRP |
|
|
MODEL1909090002 |
NRP |
|
|
MODEL1909050002 |
NRP |
|
|
MODEL1908270002 |
NRP |
|
|
MODEL1908270001 |
NRP |
|
|
MODEL1907310002 |
NRP |
|
|
MODEL1907310001 |
NRP |
|
|
MODEL1907260002 |
NRP |
|
|
MODEL1907260001 |
NRP |
|
|
MODEL1907180003 |
NRP |
|
|
MODEL1907140001 |
NRP |
|
|
MODEL2005130001 |
NRP |
|
|
MODEL2005070001 |
NRP |
|
|
MODEL1903010001 |
NRP |
|
|
MODEL1812210002 |
NRP |
|
|
MODEL1812130001 |
NRP |
|
|
MODEL1812060002 |
NRP |
|
|
MODEL1812040005 |
NRP |
|
|
MODEL1812040003 |
NRP |
|
|
MODEL1812040001 |
NRP |
|
|
MODEL1811050001 |
NRP |
|
|
MODEL2005150001 |
NRP |
|
|
MODEL1808280011 |
NRP |
|
|
MODEL1808280006 |
NRP |
|
|
MODEL1808210003 |
NRP |
|
|
MODEL1808210002 |
NRP |
|
|
MODEL1808210001 |
NRP |
|
|
MODEL1808160001 |
NRP |
|
|
MODEL1808150002 |
NRP |
|
|
MODEL1808150001 |
NRP |
|
|
MODEL1808130001 |
NRP |
|
|
MODEL1808080001 |
NRP |
|
|
MODEL1807240002 |
NRP |
|
|
MODEL1807240001 |
NRP |
|
|
MODEL1807230001 |
NRP |
|
|
MODEL1807190001 |
NRP |
|
|
MODEL1807180005 |
NRP |
|
|
MODEL1807180004 |
NRP |
|
|
MODEL1807180003 |
NRP |
|
|
MODEL1807180002 |
NRP |
|
|
MODEL1807180001 |
NRP |
|
|
MODEL1806280001 |
NRP |
|
|
MODEL1806260001 |
NRP |
|
|
MODEL1806250002 |
NRP |
|
|
MODEL1806250001 |
NRP |
|
|
MODEL1806150001 |
NRP |
|
|
MODEL1806130002 |
NRP |
|
|
MODEL1806130001 |
NRP |
|
|
MODEL1806120001 |
NRP |
|
|
MODEL1806070001 |
NRP |
|
|
MODEL1806060001 |
NRP |
|
|
MODEL1806050001 |
NRP |
|
|
MODEL1806040001 |
NRP |
|
|
MODEL1806010001 |
NRP |
|
|
MODEL1805230001 |
NRP |
|
|
MODEL1805220001 |
NRP |
|
|
MODEL1805150001 |
NRP |
|
|
MODEL1805140001 |
NRP |
|
|
MODEL1805090001 |
NRP |
|
|
MODEL1710260004 |
NRP |
|
|
MODEL1710260003 |
NRP |
|
|
MODEL1710260002 |
NRP |
|
|
MODEL1710260001 |
NRP |
|
|
MODEL1710260000 |
NRP |
|
|
MODEL1708310001 |
NRP |
|
|
MODEL1604100000 |
NRP |
|
|
MODEL1601250000 |
NRP |
|
|
MODEL1509050002 |
NRP |
|
|
MODEL1303140000 |
NRP |
|
|
MODEL1210260004 |
NRP |
|
|
MODEL1204060000 |
NRP |
|
|
MODEL1204040000 |
NRP |
|
|
MODEL1202170000 |
NRP |
|
|
MODEL1112110004 |
NRP |
|
|
MODEL1006230111 |
NRP |
|
|
MODEL1006230055 |
NRP |
|
|
MODEL1006230044 |
NRP |
|
|
MODEL1006230040 |
NRP |
|
|
MODEL1006230013 |
NRP |
|
|
MODEL1004070001 |
NRP |
|
|
MODEL0911270007 |
NRP |
|
|
MODEL0478895291 |
NRP |
|
|
MODEL1812060001 |
NRP |
|
|
MODEL2003170002 |
NRP |
|
|
MODEL1812050001 |
NRP |
|
|
MODEL1812040006 |
NRP |
|
|
MODEL1812040004 |
NRP |
|
|
MODEL1812040002 |
NRP |
|
|
MODEL1808280013 |
NRP |
|
|
MODEL2006080001 |
NRP |
|
|
MODEL2007090001 |
NRP |
|
|
MODEL1202090001 |
R |