Guide to Beta Site Changes

Description of improvements implemented on 5.14.2018

For a brief overview of the changes discussed below, click here.

Background

The Organ Procurement and Transplantation Network (OPTN) Final Rule requires that the Scientific Registry of Transplant Recipients (SRTR) “Make available to the public timely and accurate program-specific information on the performance of transplant programs. This shall include free dissemination over the Internet, and shall be presented, explained, and organized as necessary to understand, interpret, and use the information accurately and efficiently” (OPTN Final Rule 121.11(b)(iv)). In an effort to improve the program-specific reports (PSRs), SRTR hosted a Consensus Conference on Transplant Program Quality and Surveillance in December 2012 (read a meeting summary). The meeting yielded a number of recommendations, including recommendations to improve the presentation of the data for public consumption. Two key recommendations targeted improved public reporting:

  • Recommendation I.1: PSRs should be better suited to the needs of all users, particularly patients.
  • Recommendation III.1: Enhance reporting of access to transplant and pretransplant outcomes.

These recommendations motivated SRTR to begin a process to enhance and improve the data presented publicly on the SRTR website.

In June 2010, the Agency for Healthcare Research and Quality (AHRQ) published Best Practices in Public Reporting (Hibbard J, Sofaer S, AHRQ Publication No. 10-0082-EF, June 2010). This three-part article makes many recommendations about effectively reporting complex healthcare quality data. SRTR, along with its Visiting Committee, carefully considered these recommendations. Key recommendations that influenced development of the 5-tiered system included:

  • Make it easy for consumers to understand and use the comparative information. Reduce the cognitive burden by summarizing, interpreting, highlighting meaning, and narrowing options.
  • Rank ordering by performance as opposed to alphabetical ordering.
  • Using symbols instead of numbers.
  • Providing an overall summary measure.
  • Including fewer reporting categories (5 vs. 9).

SRTR and the SRTR Visiting Committee considered these recommendations in developing an improved SRTR website aimed at conveying difficult transplant program metrics to the public. Development of new ways to display these metrics and a new website took place over the 5-year period from 2012 to 2016. In December 2016, SRTR launched a new website that included a new 5-tier outcome assessment based on first-year graft failure rates. This system replaced a previous 3-tier system. The 5-tier assessment is described in detail in this article.

HRSA requested that SRTR move the new system to a Beta website while HRSA and SRTR considered the feedback received. Over the past year, SRTR has worked on an improved version of the site. The following improvements have been made in response to feedback received.


Details of the New SRTR Beta Website

Below we detail nine changes made to the search results presentation. Click on each image for a larger view.

Change #1: Renamed headings for program metrics.

Description of the change: The measure “Transplant Volume” was renamed “Transplants in a year” for all organs except kidney and liver. Kidney and liver now include new separate living and deceased donor measures described in change #6 (e.g., “Deceased Donor Transplants in a Year”). “Transplant Rate” was renamed “Getting a Transplant Faster” for organs other than kidney and liver. For kidney and liver, only deceased donor transplants are now included in the calculated transplant rate, and this is specified in the heading. “Outcome Assessment” was renamed “1-Year [organ] Survival,” and [organ] is replaced by the specific organ in each organ-specific search.

Response to feedback: Patient feedback included the suggestion to clarify that “volume” refers to numbers of transplants performed in a year, and the criticism that “transplant rate” and “outcome assessment” are not descriptive and do not help interpret the information. 

Change #2: Added 5-tier assessments for pretransplant metrics.

Description of the change: The new website (launched in December 2016) presented a 5-tier assessment for first-year survival with a functioning transplanted organ, and provided the observed (unadjusted) transplant rate as a number, e.g., 17.1 transplants per 100 people per year. The updated site provides a simple 5-tier summary for three metrics: survival on the waiting list, getting a deceased-donor transplant faster, and 1-year organ survival.

Response to feedback: SRTR made this change in response to feedback that the original version of the website focused too much on first-year outcomes when getting a transplant faster is arguably more important to overall patient survival. The Consensus Conference also recommended enhancing reporting of “access to transplant and pretransplant outcomes.” SRTR agreed with this feedback and developed a methodology to provide 5-tier assessments for both the waitlist mortality rate (survival on the waiting list) and the transplant rate (getting a deceased-donor transplant faster). These assessments are adjusted for patient characteristics in a similar manner to how posttransplant outcomes are adjusted for patient and donor characteristics. Researchers at the Minneapolis Medical Research Foundation (Principle Investigators: Ajay Israni, MD, MS, and Cory Schaffhausen, PhD), through a grant from the AHRQ, have been conducting patient focus groups and randomized trials of website versions. Through a randomized controlled trial, they found that users more often selected programs with a higher transplant rate when the transplant rate was presented as a graphical icon rather than numerically. When a program did not have the highest 1-year organ survival, but did have the highest transplant rate, 45% of patients chose the program with the highest transplant rate when it was presented as a tier icon; only 26% did so when the numeric transplant rate was shown alongside the 5-tier icon for 1-year organ survival.

Change #3: Condensed the 5-tier icons and removed interpretive text below the icons.

Description of the change: Because we added two new 5-tier assessments for pretransplant metrics, we changed the icons slightly to be more compact. In addition, the original version of the website included interpretive text below the 5-tier assessment icons to indicate whether higher numbers where better or worse than lower numbers. The interpretive text consisted of the following statements for tiers 1 through 5, respectively: “Worse than Expected,” “Somewhat Worse than Expected,” “Good (as Expected),” “Somewhat Better than Expected,” and “Better than Expected.”

Response to feedback: SRTR made this change in response to feedback that the interpretive text was inflammatory and did not add clarity; e.g., what does ”as expected” mean? Further feedback noted that these statements did not convey the actual differences between the tiers with regard to actual survival percentages (in the case of the first-year outcomes assessments). The updated version of the website removed these statements in favor of a key placed at the top of the search results page that provides the actual numbers expected within each tier for each of the three outcomes presented in the search results, as detailed in change #4 next.

Change #4: Added a key describing the meaning of the icons and providing actual numbers expected within each tier for each outcome assessed.

Description of the change: Because we removed the interpretive text below the icons as detailed in change #3 above, we added a key at the top of the search results page. The key begins by simply showing that tier 1 is “worse” and tier 5 is “better.” Without this, readers may not correctly interpret that a tier-5 is better than a tier-1 evaluation on each of the outcomes assessed. Below this simple key is a button labeled “Show National Rates >>”. Clicking on this button brings up a table that shows expected numbers within each of the five tiers for each of the three outcomes. These provide context for what each tier means for each outcome and clarifies that tiers reflect a comparison to national outcomes. This also demonstrates that variation in getting a deceased donor transplant faster is generally larger than variation in first-year rates of organ survival. The numbers in this table reflect outcomes for an average-risk patient at an average program within the tier.

Response to feedback: This key was added in response to feedback that the tiers hide the actual numbers behind the evaluations and do not provide context. Additional feedback suggested that getting a transplant faster, i.e., higher transplant rates, were often more important to overall patient survival after listing than posttransplant outcomes. The key allows readers to put each metric in context with the other metrics.

Change #5: Changed the transplant rate to a deceased-donor-only transplant rate.

Description of the change: The original version of the website presented the unadjusted transplant rate as a number, e.g., 17.1 transplants per 100 persons per year. This rate included both living and deceased donor transplants in the numerator for kidney and liver programs. The new 5-tier assessment of getting a deceased donor transplant faster includes only deceased donor transplants in the numerator of the rate, censoring candidate follow-up at the time of living donor transplant.

Response to feedback: This change was made in response to feedback that the “all-donor” transplant rate was misleading, especially for candidates without a living donor. Furthermore, OPTN policy mandates that living donor recipients be added to the deceased donor waiting list prior to undergoing living donor transplant. However, policy does not specify when candidates are added to the waiting list. Therefore, living donor candidates may be added to the waiting list early or late in the process leading up to the living donor transplant. The timing of adding candidates to the deceased donor waiting list when a living donor transplant is intended can bias the transplant rate high or low depending on the program’s practice. The new metric responds to this feedback by constructing the transplant rate evaluation based only on deceased donor transplants, censoring patient follow-up at the time of undergoing living donor transplant.

Change #6: Divided the transplant volume into deceased and living donor transplant volumes.

Description of the change: Because the transplant rate evaluation described in #5 above now describes only the program’s deceased donor transplant rate, we split the program’s transplant volume descriptors into deceased donor and living donor volumes, separately. This allows readers to see which programs perform more or fewer living donor transplants.

Response to feedback: This was done in response to feedback as described in #5 above, and to feedback from patient focus groups indicating that patients want to know which programs perform more or fewer living donor transplants.

Change #7: Indicated which evaluation is most important to overall patient survival after listing.

Description of the change: An icon () has been added to specific columns indicating which component of care has the greatest impact on overall survival after listing. In searches for heart, kidney, and liver programs, this icon appears on the “getting a transplant faster” column. In searches for a lung program, this icon appears on the “1-year lung survival” column. For more information regarding how these columns were determined, click here.

Response to feedback: This was done in response to feedback that the original version of the site focused too much attention on first-year posttransplant organ survival when pretransplant outcomes may be more important for overall survival when choosing a transplant program, given larger variation in transplant rates across programs. A first step to remedy this was adding 5-tier evaluations for pretransplant outcomes, i.e., waitlist mortality and transplant rate. With three evaluations, patients see more information but lack guidance regarding how to use the information. In testing various solutions, this presentation was the only one that succeeded in guiding users toward the columns of most importance to their survival. Without this icon, users often focused on 1-year organ survival as being most important.

Change #8: The sort order was changed to default to the column of most importance to patient survival after listing.

Description of the change: The original version of the website sorted search results by the outcome evaluation tier, bringing tier-5 programs to the top of the search results. The updated version sorts initially by the column with the largest impact on patient survival after listing as described in change #7 above. For heart, kidney, and liver programs, the default sort is by the “getting a transplant faster” column. For lung programs, the default sort is by the “1-year lung survival” column. The default sort order for each organ is determined by the following algorithm:

  • Heart results:
    • Primary sort: getting a transplant faster tier
    • Secondary sort order: first-year organ survival tier, survival on the waiting list tier, deceased donor volume.
  • Kidney results:
    • Primary sort: getting a transplant faster tier
    • Secondary sort order: first-year organ survival tier, survival on the waiting list tier, deceased donor volume, living donor volume.
  • Liver results:
    • Primary sort: getting a transplant faster tier
    • Secondary sort order: first-year organ survival tier, survival on the waiting list tier, deceased donor volume, living donor volume.
  • Lung results:
    • Primary sort: first-year organ survival tier
    • Secondary sort order: getting a transplant faster tier, survival on the waiting list tier, deceased donor volume.
  • Other organs:
    • Primary sort: deceased donor volume
    • Secondary sort order: getting a transplant faster tier, first-year organ survival tier, survival on the waiting list tier.

Users can re-sort based on their preference by clicking on the column headings. The following algorithm is followed for sorts by a particular column:

  • Distance:
    • Secondary sort order: getting a transplant faster tier, first-year organ survival tier, survival on the waiting list tier, deceased donor volume, living donor volume (if applicable).
  • Deceased donor volume:
    • Secondary sort order: getting a transplant faster tier, first-year organ survival tier, survival on the waiting list tier, living donor volume (if applicable).
  • Living donor volume (if applicable):
    • Secondary sort order: getting a transplant faster tier, first-year organ survival tier, survival on the waiting list tier, deceased donor volume.
  • Getting a transplant faster tier:
    • Secondary sort order: first-year graft survival tier, survival on the waiting list tier, deceased donor volume, living donor volume.
  • Survival on the waiting list tier:
    • Secondary sort order: getting a transplant faster tier, first-year organ survival tier, deceased donor volume, living donor volume.
  • First-year organ survival tier:
    • Secondary sort order: getting a transplant faster tier, survival on the waiting list tier, deceased donor volume, living donor volume.

Response to feedback: This was done in response to feedback that the original version of the site focused too much on posttransplant outcomes when pretransplant outcomes may be most important to overall patient survival after listing. The AHRQ’s Best Practices in Public Reporting recommended rank ordering programs by performance, and the updated version of the site rank orders programs by the measure most important to overall patient survival after listing. For more information on how these columns were determined, click here.

Change #9: Disclaimer statement was added below the search results table.

Description of the change: The following statement was added to the site below the table of search results: “Note: Future outcomes may be different from past outcomes. Groups (numbers of bars) do not represent the outcome for each individual.”

Response to feedback: This change was made in response to feedback from transplant professionals requesting a statement that these evaluations are derived based on past experience and may not reflect current or future performance at the program. The AHRQ Best Practices in Public Reporting recommend against adding statements of uncertainty when reporting healthcare evaluations publicly, noting “research shows that consumers tend to discount information when a report communicates that there is uncertainty about the data” (Hibbard J, Sofaer S. Best Practices in Public Reporting No. 1: How to Effectively Present Health Care Performance Data to Consumers. AHRQ Publication No. 10-0082-EF, June 2010). However, Drs. Israni and Schaffhausen’s work with patients did not find a substantial downside to including this statement. SRTR also presented the concept to OPTN’s Patient Affairs Committee, which had no objections and unanimously approved the statement.  

Contact Us

Please submit your comments about the beta site changes by contacting us. The 60-day comment period ends 7.13.2018.