Exploring the role of ChatGPT in decision making for gender-affirming surgery

INTRODUCTION

Large language models (LLMs), such as ChatGPT, have transformed information access by providing immediate and comprehensive responses to a wide range of prompts. For transgender and gender-diverse individuals considering gender-affirming surgery (GAS), having access to accurate, reliable, and easily understandable information is crucial for making informed decisions about such complex and sensitive procedures. Patients require a thorough knowledge of their surgical options, potential risks, recovery processes, and expected outcomes.

LLMs have been shown to be effective in many areas of healthcare delivery. For example, one study found that 45% of LLM-generated clinical summaries were equivalent to, and 36% were superior to, those of medical experts, underscoring their potential to perform at or above expert levels^[1]. Additionally, ChatGPT demonstrated its competency by passing the United States Medical Licensing Examination with a score of 64.4%, further showcasing its understanding of medical knowledge^[2].

Despite these successes, concerns about misinformation, readability, and the oversimplification of complex medical concepts persist. A recent qualitative study among health informatics researchers found that while LLMs significantly benefit patient education, clinical tasks, personalized care, and patient-healthcare interactions, they also pose risks, such as spreading misinformation, biased decision making, and inaccuracies in communication^[3]. These limitations are echoed by other researchers, who acknowledge the potential of LLMs but emphasize the ease with which misinformation can spread due to a lack of transparency and accountability^[4-6]. Furthermore, the readability of LLM-generated content often exceeds the average literacy level of U.S. patients, which could impair their ability to understand essential details, particularly in a sensitive and personal subject like GAS^[7].

While LLMs have risen to prominence for their broader healthcare applications, their role in specialized areas, like GAS, remains largely unexplored. GAS is a rapidly evolving field, with the number of procedures tripling in recent years^[8]. This increase in operations has heightened the need for transgender and gender-diverse individuals to be able to access accurate and reliable information as they navigate the process. Prior research has shown that online forums, social media, and medical websites often serve as primary sources of GAS information for many patients, who frequently seek firsthand experiences and community support^[9-11]. However, there is limited evidence on how this population uses LLMs like ChatGPT and whether these tools meet the unique needs of transgender and gender-diverse patients considering GAS.

Expanding our understanding of this area is crucial for improving patient education and ensuring that individuals considering GAS have access to reliable, accurate, and comprehensive information. As the number of GAS procedures continues to rise, equipping patients with trustworthy resources is key for enhancing their health outcomes and overall satisfaction with the surgical process. Therefore, we aimed to quantitatively assess the extent of ChatGPT use among individuals considering GAS and evaluate how it influences their decision-making process.

METHODS

RESULTS

Among the 207 participants, 100% reported that their current gender differs from their assigned gender at birth. The majority identified as non-binary (40.6%), followed by transgender men (29.5%) and transgender women (13%), with an average age of 30.2 years [Table 1]. A high level of interest in GAS was observed, with 89% expressing interest [Table 2]. When seeking information about GAS, participants primarily relied on online forums or communities such as Reddit and Tumblr (24.6%) and medical websites such as WebMD and Mayo Clinic (21.3%). Social media platforms were also frequently used (17.4%), alongside healthcare professionals (15%) and personal stories (13.5%) [Figure 1].

Figure 1. Primary source of information for GAS. GAS: Gender-affirming surgery.

Regarding the use of ChatGPT, 73% of participants had used the platform, though only a small subset (6.7%) used it specifically as a resource for GAS information. Participants who used ChatGPT for GAS information generally had positive perceptions, with 70% rating the information as moderately to slightly useful. Additionally, 40% of users felt ChatGPT had a positive influence on their decision-making process, and 80% expressed moderate to high levels of trust in the information provided.

When comparing ChatGPT’s information to other sources, 25% of users found it comparable to advice from healthcare providers, while 60% viewed it as similar to information found on medical websites. However, 20% considered ChatGPT’s information somewhat worse than that on medical websites, and 30% found it somewhat worse than scientific journals. Despite these varied opinions, a majority (60%) of ChatGPT users indicated they would recommend the tool to peers considering GAS.

Participants used ChatGPT to search for various GAS-related topics, with the most common being facial feminization, facial masculinization, and mastectomy (18.2% each). Other frequently queried topics included metoidioplasty, phalloplasty, and breast augmentation (13.6% each). 30% of participants used ChatGPT to learn about surgeon information. Many also sought information about surgical risks, techniques, and recovery processes, although some felt the platform provided insufficient detail on financial considerations (23.1%) and recovery information (15.4%). Despite these limitations, ChatGPT was seen by many as a useful supplementary resource.

Several participants shared their personal thoughts and experiences using ChatGPT for GAS information. One participant noted that “ChatGPT gave a detailed explanation of what gender-affirming surgery is”, while another commented that it “can be efficient to an extent”. Others appreciated its straightforwardness, with one remarking that it was “pretty accurate to what I wanted to know... and straight to the point”.

DISCUSSION

This study aimed to characterize the use of ChatGPT as a resource for obtaining information about GAS among transgender and gender-diverse individuals in the United States. While most participants had used ChatGPT, its application specifically for GAS information was notably limited compared to more traditional resources such as online forums and medical websites. These findings align with prior research suggesting that patients navigating complex health decisions, such as GAS, prefer personalized, human-centered resources over automated tools^[9,10]. There are several reasons why participants may not have wanted to use ChatGPT as a leading information source. Participants specifically commented on the efficiency and straightforward nature of ChatGPT compared to other sources. Future studies should seek to evaluate platforms that offer a variety of perspectives beyond efficiency alone - such as those that include high-fidelity images, opportunities for human interaction, or a sense of community - elements that are currently missing from ChatGPT’s functionality^[6,12].

Prior studies have emphasized the importance of personal experience and relatability in health decision making, particularly for marginalized groups such as transgender and gender-diverse individuals^[13,14]. Social media platforms and online forums have become essential spaces for peer support and sharing firsthand experiences, offering a level of emotional validation and specificity that LLMs like ChatGPT are unable to replicate^[15-17]. Although ChatGPT can deliver broad, data-driven responses, its utility in addressing the personal, psychological, and emotional aspects of GAS decision making appears limited^[4,18]. This may explain why it is not yet seen as a primary resource for making such important medical decisions.

The underutilization of ChatGPT for GAS information could also be attributed to broader concerns about the accuracy and trustworthiness of LLMs in healthcare settings. Several studies have raised alarms about the potential for LLMs to disseminate misinformation or oversimplify complex medical concepts, which can lead to patient confusion or misinformed decisions^[4,19]. Furthermore, the ability of LLMs to account for patient-specific factors, such as individual medical histories or co-occurring health conditions, remains limited. For transgender patients, whose healthcare needs are often highly specialized and require tailored interventions, this lack of personalization could further diminish trust in ChatGPT as a reliable resource.

Interestingly, despite the limited use of ChatGPT for information on GAS, some participants did report that it positively influenced their decision making. This indicates that while ChatGPT may not currently serve as a primary resource, it may have greater utility as a supplementary tool, especially as LLMs evolve to integrate more specialized medical data and provide real-time, accurate patient feedback. Furthermore, research in fields such as ophthalmology and urology has shown that, while LLMs can provide reasonably accurate and comprehensive information, they often fall short of addressing the full range of patient needs^[20,21], underscoring ChatGPT’s potential to complement, rather than replace, traditional sources of medical information for GAS^[22].

This study additionally identified several areas where ChatGPT’s content could be improved to better meet the needs of individuals seeking information on GAS. Participants specifically noted that ChatGPT provided insufficient details on financial considerations, surgical techniques, and recovery processes - key elements required in the decision-making process for GAS. This mirrors findings from other research where LLMs have been criticized for their inability to provide comprehensive, context-specific medical information, particularly in areas that require detailed, patient-centered guidance^[18]. These limitations are particularly concerning in fields like transgender healthcare, where access to accurate, personalized, and affirming medical information is often limited^[23-25].

From a broader perspective, the findings of this study emphasize the ongoing need to guide patients toward trusted, reputable sources of medical information, especially for GAS, where misinformation can have serious and life-long consequences. Healthcare providers should guide patients toward high-quality resources, including peer-reviewed medical websites such as GAS websites produced by academic institutions, such as our institution’s transgender care website https://genderaffirmingsurgicalcare.ucsf.edu/, and consultations with trained professionals. It is also equally important to approach the integration of LLMs into healthcare with caution, emphasizing that these technologies should complement, rather than replace, human expertise and empathy.

Our study is not without limitations. The small number of participants who used ChatGPT specifically for information on GAS limits the generalizability of our results. Additionally, the reliance on self-reported data introduces potential biases, such as over-reporting or under-reporting the use of ChatGPT or other information sources. Self-reported demographics such as gender identity and sexual orientation may also introduce bias into the results given differences in participant understanding of specific terms, which could differ from widely accepted definitions. There are limitations to the use of Prolific, including reliance on self-reported data, the potential for participant bias due to financial incentives, and the fact that the participant pool may not fully represent the general population. While Prolific participants may not fully represent the broader population, prior research has shown that data quality from Prolific is comparable to or better than other commonly used platforms such as MTurk. Efforts were also made to reduce the risk of inaccurate data through attention-check questions and Prolific’s required demographic verification. Our survey was not exhaustive and did not offer the opportunity for detailed responses such as reasons for ranking ChatGPT lower than other sources. However, the diversity of our sample, in terms of gender identity and racial background, provides a valuable snapshot of the current landscape of information-seeking behaviors among transgender and gender-diverse individuals considering GAS.

The implications of this study extend beyond the immediate context of GAS. As LLMs become more integrated into the larger healthcare landscape, it is essential for researchers and clinicians to continually assess their utility in various fields, particularly those that require high levels of personalization and patient-centered care. At the time of the study, ChatGPT was not a primary resource for participants seeking information about GAS. However, among those who utilized it, the platform was generally well-recommended. As AI technology continues to gain popularity, it is likely that its use for sourcing information about GAS will increase. Future research should investigate how ChatGPT and other LLMs can be adapted to better address the needs of patients considering complex medical procedures. This may involve incorporating multimedia features, such as video explanations or interactive Q&A formats, to improve user engagement and build trust. Additionally, future studies should aim to increase sample sizes and include more diverse populations to provide a more comprehensive understanding of patient education within complex surgical domains. Including definitions of key terms for participants in future surveys could also help minimize variability in understanding and improve consistency in responses.

In conclusion, this study found that ChatGPT was underutilized as a resource for information on GAS, though it had a positive impact on patient education and provided useful information for those who did use it. Participants rated ChatGPT as moderately helpful but identified gaps in areas such as financial considerations, surgical techniques, and recovery details. These findings highlight that while ChatGPT shows promise as a supplementary resource for patients considering GAS, it is not yet perceived as a primary source of reliable information. Given the critical need for trustworthy and comprehensive guidance during the decision-making process for GAS, this emphasizes the importance of steering patients toward more trusted, human-centered resources, while continuing to refine language models to better meet the needs of diverse patient populations.