The program in metascience at Phenoscience Laboratories targets the development of advanced scientific methodologies and the testing of the scientific method itself. In addition, the metascience program researches selected issues in the philosophy and the foundations of science.
Phenoscience Laboratories is one of five members of a multi-laboratory research consortium testing the reliability of the scientific method in the context of psychological studies. The other consortium members are Stanford University, UC Berkeley, UC Santa Barbara, and the University of Virginia. This program is sponsored by the Fetzer Franklin Fund (FFF) of the John E. Fetzer Memorial Trust until 2020. The goal of this large-scale metascientific study, which involves 10s of thousands of study participants, is to quantify precisely the reproducibility, or the lack thereof, of experimental findings between different laboratories which implement, as far as is possible, the same experimental routines.
Presently, this is the only extensive multi-laboratory study for evaluating the replicability of scientific findings in a prospective manner. The adoption of a blinded, prospective approach allows the demonstration, should it exist, of the so-called “experimenter effect”, i.e., the potential biasing of experimental results by uncontrolled behaviors of the involved experimenter agents, such as in performing the experiment or in analyzing the collected data. That is, using the scientific method the reliability of the scientific method is quantified; hence, the term metascience.
Phenoscience Laboratories contributes to the overall research design, including the specific protocol implementation criteria, from the start to the finish of this multi-year research consortium initiative. Phenoscience Laboratories was co-organizer of the Metascience symposium in 2012, titled “The Decline Effect: Evidence, Explanations, and Future Directions” (see FFF events), together with Dr. Jonathan Schooler at UC Santa Barbara (META Lab). The 2012 symposium led to the formation of the present multi-laboratory consortium. The symposium was sponsored by the Fetzer Franklin Fund (FFF) of the John E. Fetzer Memorial Trust.
Advanced Meta-experimental Protocol (AMP)
Phenoscience Laboratories has developed an advanced meta-experimental protocol (AMP) that allows the tracking of different kinds of artefacts that may exist in association with any type of experimental design (Walleczek, manuscript forthcoming). Specifically, such artefacts are investigated and discussed as part of the AMP under the term systematic methodological error (SME). The source of any SME can be one of two types of error, or a combination thereof: (1) Systematic experimental error (SEE), and (2) systematic statistical error (SSE). Presently, critical tests for potential artefacts based upon the insertion of an AMP into existing experimental designs are underway (Walleczek and von Stillfried, manuscript forthcoming). The AMP is based on a higher-order extension of the concept of the systematic negative control (SNC) which was originally introduced in 1999 (Walleczek et al., 1999).
Technology Development and Testing for Artefacts
Walleczek, J. (1995) Method and Apparatus for Performing Dual-beam, Dual-wavelength Fluorescence Spectrophotometric Evaluation of a Biological Specimen. US–Patent #5,383,023. Read more
Jones, R.A., Walleczek, J. and Adey, W.R. (1996) Mechanical Vibration in “Double-Wound” Magnetic-Field Exposure Coils. Bioelectromagnetics 17, 516-518. Read more
Walleczek, J., Shiu, E. and Hahn, G. (1999) Increase in Radiation-induced HPRT Gene Mutation Frequency after Nonthermal Exposure to Nonionizing 60 Hz Electromagnetic Fields. Radiat. Res. 151, 489-497. Read more
Synopsis: The article introduces the systematic negative control (SNC) protocol. Its purpose is to detect potential artifacts related to the application of the standard control condition in association with a particular experimental design. The use of the SNC-protocol is summarized in the paper this way: “To monitor and verify the reliability of the methodology employed on an ongoing basis, we implemented a new protocol, which we termed the systematic negative control protocol. The performance of systematic negative controls, i.e., of sham-treatments compared to simultaneous controls alternated with, and equal in number to, real treatments compared to simultaneous controls, may represent a significant improvement over conventional approaches.”
Wolsko, P.M., Eisenberg, D.M., Simon, L.S., Davis, R.B., Walleczek, J., Mayo-Smith, M., Kaptchuk, T.J., and Phillips, R.S. (2004) Double-blind Placebo-controlled Trial of Static Magnets for the Treatment of Osteoarthritis of the Knee: Results of a Pilot Study. Altern. Ther. Health Med.. 10, 36-43. Read more
Jan Walleczek was a co-organizer of a workshop on the The Conceptual Foundations of Science (Parmenides Foundation, Germany). The 3-day workshop, held on September 18-21, 2018, addressed the foundations of science in light of the challenge to conceptualize the relationship between matter, life, and mind. The workshop program was sponsored by the Fetzer Franklin Fund (FFF) of the John E. Fetzer Memorial Trust.
Walleczek, J. and von Stillfried, N. (2018) Testing the Application of the Advanced Meta-experimental Protocol (AMP) in a Weak-effects Study Paradigm. Manuscript forthcoming.
Walleczek, J. (2018) Advanced Meta-experimental Protocol (AMP) with Sham-experiments and Systematic Negative Controls: Detecting Weak Effects with High Specificity and Reproducibility. Manuscript in preparation.