The need for core outcome sets in renal cancer clinical effectiveness research

Other contributions in this Special Issue discuss current management strategies of small renal masses, whereas the aim of our contribution is to create awareness about how outcome reporting heterogeneity is a major problem when evaluating the effectiveness of renal cancer treatment strategies, how this impacts renal cancer care and how this problem can be overcome. To do this, it is necessary to summarise how knowledge is produced and applied in medical research and the synthesis of study results.

In research, an outcome is broadly defined as a measurement or observation used to capture and assess the effect of a treatment or intervention, such as the assessment of side effects (risks) or effectiveness (benefits)^[1]. When designing studies of treatment risks and benefits, such as randomised trials, observational studies, or a platform for big data analysis, research teams need to know and be able to succinctly communicate what is known so far about these outcomes. Likewise, when clinical practice guideline panels provide treatment/care management recommendations, they ought to do so based on a balanced consideration of the risks and benefits of treatments/interventions, with patient preferences^[2].

For both of these circumstances, designing new research and making treatment recommendations, to know the totality of the existing evidence base requires effort from various experts and stakeholders, such as clinicians, researchers, and patient advocates. Systematic reviews and meta-analyses are key research methods to address this. Typically, systematic reviews utilise deductive reasoning and set out to answer an a priori research question with strict inclusion and exclusion criteria; then, data are extracted from the included studies on baseline characteristics and outcomes, aiming to minimise bias and random error^[3,4]. If there is sufficient similarity in the populations, measurements, and definition of the outcome across studies, then the outcome may be amenable to meta-analysis, which is a statistical technique whereby estimates from more than one study are combined and can often give more power and precision than individual estimates from any one study alone^[4].

However, outcome reporting heterogeneity is a frequent problem when systematically reviewing and meta-analysing an evidence base. Outcome reporting heterogeneity refers to the interrelated problems of inconsistency (different outcomes reported in different studies) and variability (same outcomes reported across studies but defined and/or measured differently)^[1,5]. This heterogeneity may be further exacerbated by selective outcome reporting, whereby the choice of outcomes to report is based on their statistical significance or some other post-hoc decision^[6].

Outcome reporting heterogeneity exists within the renal cancer treatment effectiveness evidence, as exemplified in comprehensive systematic reviews of outcomes comparing various treatments for localised renal cancer and reporting on oncological outcomes^[7], perioperative outcomes^[8], and quality of life^[8,9]. In the limitation sections of each of these reviews, the authors noted that they found it difficult to compare results across studies due to outcome reporting inconsistency and/or variability, and meta-analyses were either not possible or limited in scope. Instead, they mostly used narrative synthesis^[10] to describe the data. This resulted in unwieldy data tables and inefficient textual summaries, which were burdensome to prepare and difficult to communicate. This, in turn, hampers the guideline-making process when expert panels try to make sense of the evidence and offer actionable recommendations.

A recent systematic review focusing on outcome reporting heterogeneity in renal cancer describes this phenomenon for overall survival, adverse events, and quality of life^[11]. The use of different measurement start and end times for calculating both overall survival and cancer-specific survival makes it difficult to combine the data and provide a critical and concise summary of them. Adverse event reporting used three different approaches: the standardised Clavien-Dindo system^[12] (focusing on the consequence of the event, e.g., requiring further medical treatment), simple lists of events, and “trifecta” or “pentafecta” outcomes (each representing a composite outcome that is also prone to heterogeneity in terms of meeting the criteria for it). This variability means it is not possible to directly compare adverse events across studies, nor is it possible to meta-analyse the data. Even available reverse coding lists for Clavien-Dindo do not solve this problem since they represent an unreasonable additional workload, and the heterogeneity of the source data still exists. Quality of life represents a particularly critical field: only three of 143 studies reported quality of life at all, and all three used different measurement instruments. In particular, the amount of available and used quality of life instruments (which are often not specific to renal cancer) makes it extremely difficult to draw conclusions about these data across studies^[9].

Since the problem is clear - how can we overcome outcome reporting heterogeneity? A methodologically sound and sustainable solution is a core outcome set (COS). A COS is an agreed minimum set of outcomes to be reported in all trials in a clinical area^[1]. There exists extensive methodological guidance on how to develop COS^[1] and how to report the protocols^[12] and results papers^[13]. A methodological hub called the Core Outcome Measures in Effectiveness Trials (COMET) Initiative maintains a register of a completed and ongoing COS. The stages of creating a COS are outlined in [Figure 1].

Figure 1. Overview of the COS development process (reproduced from Williamson et al. 2017)^[1]. COMET: Core outcome measures in effectiveness trials; COS: core outcome set.

COS development in the rheumatoid arthritis field is an example that gives reason for optimism and hopes that this strategy can improve the situation in the field of renal cancer as well. Their COS development programme started in the mid-1990s^[14,15], and between 2002 and 2016, 81% of completed registered trials reported the full rheumatoid arthritis COS with a decreasing rate of selective reporting^[16,17].

The good news for the field of renal cancer effectiveness research is the current development of a COS each for localised, locally advanced, and metastatic renal cancer, broadly following the COMET COS development guidance (registration available here: https://www.comet-initiative.org/studies/details/1406, step 2; all steps are referring to Figure 1). After the publication of the study protocol (step 3)^[18] and the literature review for localised renal cancer^[11], qualitative methods will ensure that the views of the patients are adequately captured (step 4ii) before a Delphi study (step 4iii) and consensus meeting (step 4iv) are conducted. In this way, the outcomes that are considered core by stakeholders, including urology healthcare professionals, researchers, and patients, will be determined, including their definitions and recommendations for their formal assessment (step 5). Regarding the latter, a current project of the European Organisation for Research and Treatment of Cancer (EORTC) Quality of Life Group, in accordance with its methodological requirements, is developing a patient-reported outcome measure specific for renal cell cancer patients^[19,20]. The preliminary module is about to be tested and will likely result in an acceptable and feasible tool.

Awareness-raising efforts are already ongoing to promote COS implementation in general. For instance, the National Institute for Health and Care Research (NIHR), a major UK research funder, requests applicants to utilise applicable COSs in their funding applications. In another example, one of the Core Outcomes in Women’s and Newborn Health (CROWN) initiative COS implementation strategies was to create a consortium of over 80 gynaecology-obstetrics journals to endorse the use of COSs in studies submitted to their journals^[21]. Similarly, the group responsible for co-ordinating Cochrane reviews in the dermatology setting has embedded a COS initiative within their review group, which aims to facilitate implementation^[22]. These three examples are useful to bear in mind to address barriers to uptake.

In conclusion, outcome reporting heterogeneity is problematic for evidence synthesis in renal cancer treatment effectiveness research. It creates difficulties in succinctly summarising the evidence base, which, in turn, makes it difficult for guideline panels to recommend best practices and for clinicians to advise their patients. A solution to this problem exists, and a renal cancer COS is in development. Examples of COS uptake in other disciplines offer optimism. We can learn from other successful instances of COS implementation, such as awareness raising, encouraging researchers to use COS through editorial endorsement and policy, and via research funders requiring a specific statement on the use of or justification to not use the existing COS.