Changing the Channel, Overcoming Crisis : Reproducibility Initiatives

--Strategy--

CHANGING THE CHANNEL

By Mark Zipkin, Staff Writer

Having helped expose the reproducibility crisis about five years ago, Amgen Inc. is staying at the forefront of the issue with its support of F1000Research, an open-science platform to let researchers check each other’s data. BUt as the field wrestles with causes and solutions, some academics are starting to bristle at the intrusion.

The issue was sparked by a pair of papers in 2011 and 2012 from Bayer AG and Amgen, respectively, that showed each company could only reproduce a small percentage of published data when the experiments were tried in-house.

Although many companies routinely rerun published experiments to verify them, they rarely go the distance of publishing their findings.

Alexander Kamb, SVP of discovery research at Amgen, told BioCentury there has long been a “culture of publication” at the company that has included publishing negative data, but the environment in the scientific community is changing to make that rarer. “The nature of the process is more challenging and tends to favor new findings as opposed to reconsideration of something,” said Kamb. He added that in current climate, “more prosaic, disconfirming results aren’t so easily published.”

Now. Amgen has put its weight behind the efforts of life science publisher Faculty of 1000(F1000) to address the problem with the launch of the Preclinical Reproducibility and Robustness channel on the publisher’s F1000Research platform.

On Feb. 4, to mark the channel’s launch, Kamb published a joint editorial with Bruce Alberts, former editor in chief of Science and an F1000Research advisory board member, outlining the goal of the channel as an effort “aimed at strengthening the self-correcting nature of science through the widespread, rapid publication of the failures (as well as the success) of attempts to reproduce published scientific findings.”

The same day, Amgen posted the channel’s first three papers in which the biotech documented experimental details of internal studies that refute published findings from four different groups, including one paper from its own scientists.

"It was kind of known in the genomics world that a certain study was quite difficult to believe. It was like the whispers of the conference hall."

-Michael Markie, F1000Research

Naming Names

The channel was the brainchild of Kamb and Alberts. According to Michael Markie, associate publisher at F1000Research, the increasing unease in the scientific community about the reproducibility crisis prompted the duo to create an alternative publishing outlet to traditional journals, which routinely prioritize the impact of a study over its rigor. According to Markie, publishing in the F1000Research channel differs from publishing in a journal in several key ways.

First is speed to publication: submissions are posed after editorial checks - covering ethical standards, methodology and open access to the data - are performed, but before any peer review. In addition, publication does not hinge on the potential impact on the field. Reviewers are solicited once a prepublished study is posted to the channel’s site, where readers can also post comments. F1000Research authors choose their referees, although the publisher screens the choices to prevent conflicts of interest. Once three referees approve a paper, F1000Research indexes it widely on search engines such as PubMed. Markie said that although there’s still a certain stigma attached to openly contesting published findings, the platform is designed for transparency and dialogue.

The idea that the preclinical research community could go beyond the stigma came from a dispute around results from the Mouse ENCODE Consortium.

In 2014, two papers by the consortium’s scientists published in Nature and the Proceedings of the National Academy of Sciences presented controversial findings on comparative gene regulation data, claiming that the patterns of gene expression data tended to cluster more by species than tissue. The studies contradicted common beliefs in the field. “It was kind of known in the genomics world that a certain study was quite difficult to believe. It was like the whispers of the conference hall,” said Markie. In 2015, two researchers from the University of Chicago refuted the findings in the Genomics, Computational & Systems Biology subject area of F1000Research. After the negative data were posted, four referees — with names attached — approved it, although one approved it with reservations. According to Markie, the widespread attention prompted a healthy discussion.

“What happened there was a big lively debate with a lot of the key stakeholders in that area of science who all chipped in, in a constructive way,” he said, adding that even the original authors contributed to the conversation. “Everything was quite civil.”

NEGATIVE FINDINGS

The studies Amgen posted on the new channel address the roles of specific pathways in obesity, neurodegeneration and Alzheimer’s disease (AD). All three are still awaiting peer review. The first challenged a pair of 2012 publications concerning GPR21 in obesity. A paper in Biochemical and Biophysical Research Communications by Amgen researchers reported that GPR21-knockout mice were resistant to diet-induced obesity, and one in The Journal of Clinical Investigation from the University of California San Diego and Pfizer Inc. showed GPR21 knockout improved insulin sensitivity. But in Amgen’s F1000Research findings, new GPR21-knockout mice were generated that did not replicate either of the earlier studies. Instead, the paper suggested the metabolic phenotypes of the original GPR21 knockout mice were due to unintentional changes in expression of a nearby gene, RABGAP1, caused during generation of the knockouts.

“There is work that might be defined ‘reproducible’ in a very narrow sense, but what we're trying to do here is find mechanisms and hypotheses that are robust enough to really make it in the clinic.”

- Alexander Kamb, Amgen

Because mouse GPR21 is encoded within a RABGAP1 intron, the F1000Research paper’s authors thought the neomycin cassette insertion used to generate the original knockout mice might have altered RABGAP1 expression. In the new knockout mice — which were generated via a 29-base pair deletion in GPR21 using transcription activator-like effector nucleases (TALENs) — RABGAP1 expression was unaffected, and the mice showed no improvements in glucose and insulin metabolism compared with wild-type littermates.

The second paper contested a 2010 Nature report from a group at Harvard Medical School that suggested USP14 slows the degradation of proteasome substrates such as tau and TDP-43, which play an important role in neurodegenerative diseases. Whereas the Harvard group’s evidence showed a catalytically inactive form of USP14 decreased tau and TDP-43 levels in HEK293 cells compared with functional USP14 — supporting the idea that functional USP14 prevents their degradation — the Amgen team found no difference between the two. In addition, siRNA knockdown of USP14 by the Amgen team did not affect endogenous tau expression in a different cell line.

The Amgen team indicated in its paper that differences in the method, such as the expression vector used, could underlie the discrepancy, and noted that follow-up studies by the Harvard group had failed to show the USP14 effect occurred in vivo.

At least one company has been targeting USP14. In 2013, Proteostasis Therapeutics Inc. received a grant from The Michael J. Fox Foundation for Parkinson’s Research to develop a USP14 inhibitor to promote clearance of α-synuclein to treat Parkinson’s disease (PD), aiming for the clinic in 2015. In December 2013, Proteostasis announced a partnership with Biogen Inc. to continue development, and in 2014 it received milestone payments from Biogen. Biogen did not respond to requests for comment. The third paper was the only one to receive a response so far — which highlighted how some in the community are viewing the initiative.

The study centered on results from a 2012 Science paper from an academic group headed by Gary Landreth at Case Western Reserve University School of Medicine. Landreth’s study used a mouse model of AD, and showed the RXR agonist Targretin bexarotene produced more than 50% reduction of β-amyloid plaque area within 72 hours, reversed cognitive and social behavior deficits, and improved neural circuit function. Landreth is a professor of neurosciences and neurology, and director of the Alzheimer Research Laboratory at Case Western. The Amgen group treated wild-type Sprague-Dawley rats with Targretin, but detected no significant change in β-amyloid levels after three days or seven days. It did not perform behavioral assays or examine neural circuit function.

In a response posted on F1000Research, Landreth argued that the reason for the difference is that the Amgen researchers used “the wrong formulation.” In the 2012 study, his group used the clinically approved formulation of Targretin, which is a microcrystalline form of the drug. Amgen’s group used a soluble form of the molecule, which Landreth stated would have a different PK profile that would affect its activity. He added in his response that the importance of the formulation had been well documented in the literature and the FDA filing, and was detailed in a response to four comments on his study published in 2013 in Science. He also noted the use of different species in the Amgen study

“We don’t want it to turn into this ‘pharma can’t do it’ channel.”

- Michael Markie, F1000Research

“The Amgen scientists (and others) clearly did not make an effort to understand and replicate the original study design,” he wrote in his rebuttal on F1000Research. Landreth concluded by stating a “logical flaw” in the Amgen paper undermines its conclusion. “I think this study is emblematic of the problems associated with reporting ‘failure to replicate’ findings in studies that do not genuinely reproduce the published work,” he wrote. According to Landreth, ReXceptor Inc. licensed options from Case Western on the use of bexarotene in the treatment of AD. ReXceptor did not respond to requests for comment. At least one other company is developing an RXR agonist: Io Therapeutics Inc. has IRX4204 in Phase I testing for AD and PD.

“The pharma/biotech industry doesn't do its own basic research anymore. It relies on research that's paid for by NIH, and yet it screams ‘bloody murder’ when things go wrong.”

- Judith Kimble, University of Wisconsin-Madison

WHOSE CRISIS?

Since the problem was brought to light by Amgen and Bayer, the academic community has responded with several initiatives to address reproducibility (see “Reproducibility Initiatives”). According to Kamb, to define what constitutes “reproducible,” it’s important to think about what the data need to support.

“The key thing is that the clinical hypothesis applies robustly in the maelstrom of the clinic,” he said. “So there is work that might be defined ‘reproducible’ in a very narrow sense, but what we’re trying to do here is find mechanisms and hypotheses that are robust enough to really make it in the clinic.”

At the Global Biological Standards Institute (GBSI)’s annual summit on Feb. 9, keynote speaker Judith Kimble said the two papers kicking off the reproducibility crisis were “a bomb” for biomedical researchers. However, she added, “The first question is, is it true? And I think we don’t really know whether or not it’s true yet.”

Kimble is a professor of biochemistry at the University of Wisconsin-Madison and an investigator at Howard Hughes Medical Institute (HHMI). Kimble questioned the accuracy of one of the leading points of the summit: a GBSI-backed study published in a June 2015 Perspective in PLoS Biology which calculated the costs of irreproducible clinical research at $28 billion. “Trying to define what is reproducible and what is not — what is a reproducible paper, what is a reproducible panel — is a science in and of itself,” said Kimble. But regardless of the exact number, she added, “There are clearly problems.”

During her talk, Kimble pointed to a number of well-known causes of irreproducibility, covering inadequate training, problematic stocks, lack of transparency and occasional fraud or misconduct. But Kimble characterized these as symptoms, adding: “The elephant in the room is hypercompetition.” Hypercompetition, she said, has resulted from ratcheting up the healthy competitive pressures of the field to “the point where something starts to break.” “I would say that the system is at that point,” she added.

One culprit is the push from scientific publishers, the job market and industry to see that results and publications are clinically relevant, she said. Another driver is the increase in the number of researchers while overall financial funding has decreased, with NIH funding down 30% in constant dollars since 2003. Although NIH received a budget increase of 3%, bringing its total for FY16 to $31 billion, Kimble said it would require 5% annual increases for the next five years just to get back to 2003 levels, meaning it is unrealistic to expect the public sector alone to solve the hypercompetition issue.

“This bomb that came in 2012 came from the pharma/biotech industry,” said Kimble. She added: “The pharma/biotech industry doesn’t do its own basic research anymore. It relies on research that’s paid for by NIH, and yet it screams ‘bloody murder’ when things go wrong.” F1000Research expects more companies besides Amgen will get involved in the channel as it has actively solicited research from industry, and Markie expects several pharmaceutical companies will jointly publish a paper on the channel in the coming weeks. Still, he said, “We don’t want it to turn into this ‘pharma can’t do it’ channel.” “Obviously Amgen have taken the leadership position here like they did with the Bayer article a few years ago,” said Markie. But he hopes the channel will grow to see 50-100 publications annually, from both industry scientists and academics, “when someone else apart from Amgen has done it.”

April 4, 2016

COMPANIES AND INSTITUTIONS MENTIONED

Amgen Inc. (NASDAQ:AMGN), Thousand Oaks, Calif.

Bayer AG (Xetra:BAYN), Leverkusen, Germany

Biogen Inc. (NASDAQ:BIIB), Cambridge, Mass.

Case Western Reserve University, Cleveland, Ohio

Faculty of 1000 (F1000), London, U.K.

Global Biological Standards Institute (GBSI), Washington, D.C.

Harvard Medical School, Boston, Mass.

Howard Hughes Medical Institute, Chevy Chase, Md.

Io Therapeutics Inc., Santa Ana, Calif.

The Michael J. Fox Foundation for Parkinson’s Research, New York, N.Y.

Mouse ENCODE Consortium, Stanford, Calif.

National Institutes of Health (NIH), Bethesda, Md.

Pfizer Inc. (NYSE:PFE), New York, N.Y.

Proteostasis Therapeutics Inc. (NASDAQ:PTI), Cambridge, Mass.

ReXceptor Inc., Cleveland, Ohio

University of California San Diego, La Jolla, Calif.

University of Chicago, Chicago, Ill.

University of Wisconsin-Madison, Madison, Wis.

TARGETS AND COMPOUNDS

GPR21 - G protein-coupled receptor 21

RABGAP1 - RAB GTPase activating protein 1

RXR - Retinoid X receptor

SNCA - α-synuclein

tau (MAPT; FTDP-17) - Microtubule-associated protein τ

TDP-43 (TARDBP) - TAR DNA binding protein 43

USP14 (TGT) - Ubiquitin specific peptidase 14 tRNA-guanine transglycosylase

REFERENCES

Alberts, B. and Kamb, A. “Publishing confirming and non-confirming data.” F1000Research (2016)

Begley, C. and Ellis, L. “Drug development: Raise standards for preclinical cancer research.” Nature (2012)

Cramer, P., et al. “ApoE-directed therapeutics rapidly clear β-amyloid and reverse deficits in AD mouse models.” Science (2012)

Freedman, L., et al. “The economics of reproducibility in preclinical research.” PLoS Biology (2015)

Gardner, J., et al. “G-protein-coupled receptor GPR21 knockout mice display improved glucose tolerance and increased insulin response.” Biochemical and Biophysical Research Communications (2012)

Gilad, Y. and Mizrahi-Man, O. “A reanalysis of mouse ENCODE comparative gene expression data.” F1000Research (2015)

Lee, B.-H., et al. “Enhancement of proteasome activity by a small-molecule inhibitor of USP14.” Nature (2010)

Ortuno, D., et al. “Does inactivation of USP14 enhance degradation of proteasomal substrates that are associated with neurodegenerative diseases?” F1000Research (2016)

Osborn, O., et al. “G protein–coupled receptor 21 deletion improves insulin sensitivity in diet-induced obese mice.” The Journal of Clinical Investigation (2012)

Osherovich, L. “STK33 strikes out.” SciBX: Science-Business eXchange (2011)

Prinz, F., et al. “Believe it or not: How much can we rely on published data on potential drug targets?” Nature Reviews Drug Discovery (2011)

Wang, J., et al. “GPR21 KO mice demonstrate no resistance to high fat diet induced obesity or improved glucose tolerance.” F1000Research (2016)

Wang, S., et al. “Effect of LXR/RXR agonism on brain and CSF Aβ40 levels in rats.” F1000Research (2016)

EDITORIAL & RESEARCH

NEWSROOM:

pressreleases@biocentury.com

SAN CARLOS, CA:

+1 650-595-5333; Fax: +1 650-595-5589

CHICAGO:

+1 312-755-0798; Fax: +1 650-595-5589

WASHINGTON, DC:

+1 202-462-9582; Fax: +1 202-667-2922

UNITED KINGDOM:

+44 (0)1865-512184; Fax: +1 650-595-5589

Editor: C. Simone Fishburn, Ph.D.

Associate Editor: Michael J. Haas

Senior Writer: Lauren Martz

Staff Writers: Selina Koch, Ph.D.; Mary Romeo; Karen Tkach, Ph.D.; Mark Zipkin

Director of Research: Walter Yang

Copy Editor: Claire Quang

BioCentury®; Because Real Intelligence is Hard to Find™; BCIQ™; The BioCentury 100™; and The Clear Route to ROI™ are trademarks of BIOCENTURY PUBLICATIONS, INC. All contents Copyright © 2016, BIOCENTURY PUBLICATIONS, INC. ALL RIGHTS RESERVED. No part of BioCentury’s Publications or Website may be copied, reproduced, retransmitted, disseminated, sold, distributed, published, broadcast, circulated, commercially exploited or used to create derivative works without the written consent of BioCentury. Information provided by BioCentury’s Publications and Website is gathered from sources that BioCentury believes are reliable; however, BioCentury does not guarantee the accuracy, completeness, or timeliness of the information, nor does BioCentury make any warranties of any kind regarding the information. The contents of BioCentury’s Publications and Website are not intended as investment, business, tax or legal advice, and BioCentury is not responsible for any investment, business, tax or legal opinions cited therein.

CORPORATE, SUBSCRIPTIONS & PRIVACY

BioCentury’s mission is to provide value-added business information & analysis for life science companies, investors, academia and government on the strategic issues essential to the formation, development and sustainability of life science ventures.

BioCentury Publications, Inc.

BioCentury International Inc

MAIN OFFICES

PO Box 1246

San Carlos CA 94070-1246

+1 650-595-5333; Fax: +1 650-595-5589

CORPORATE

Chairman: Karen Bernstein, Ph.D.

President & CEO: David Flores

Vice President/Commercial Operations: Thomas Carey

Vice President/Administration & CFO: Bennet Weintraub

Publisher: Eric Pierce

Executive Editor and Director, New Ventures: Joshua L. Berlin

Senior Director/Commercial Operations: Tim Tulloch

Director/Business Intelligence: Chris Dokomajilar

Director/Multimedia Operations: Jeffrey Fitzgerald

Director/Multimedia Business Development: Jamie Gould

Director/Digital Product Manager: Ravid Lazinsky,

Director/Marketing & Promotional Services: Greg Monteforte

Director/Administration & Human Resources: Susan Morgan

Director/Publishing: Jenny Nichols

SUBSCRIBER SERVICES

Subscriber Services: subscribe@biocentury.com

Account Managers: Orlando Abello; Matt Krebs; Michelle Ortega; Ron Rabinowitz

BUSINESS SERVICES Accounting & Billing: finance@biocentury.com

Conferences: conferences@biocentury.com

Data Solutions Support: support@biocentury.com

Privacy Policy: privacy@biocentury.com

Reprints/Permissions: businessservices@biocentury.com

PRIVACY & ADVERTISING

In accordance with its Privacy Policy, BioCentury does NOT sell its customer information or usage data to third parties. BioCentury does NOT sell advertising in the BioCentury, BioCentury Innovations or BioCentury Week in Review. BioCentury is pleased to acknowledge its conference partners and sponsors through unpaid promotional announcements in its publications. BioCentury MAY accept paid promotional messages from sponsors, which are displayed only on BioCentury’s websites.